Synchronization method of multi-source sensing device based on multi-source fusion and related device

Abstract

The embodiments of the present application provide a method for synchronizing a multi-source sensor device based on multi-source fusion and related devices, which relate to the field of vehicle-road collaborative technology, and may also relate to the field of transportation, the field of autonomous driving, artificial intelligence, and cloud technology, etc. The method includes: synchronizing the multi-source sensor device, and the synchronization processing includes at least one of spatial synchronization processing or temporal synchronization processing. For spatial synchronization processing, the target external parameters between the multi-source sensor devices are determined by acquiring a digitized map. For temporal synchronization processing, the clocks of the sensor devices to be corrected in the multi-source sensor devices are corrected by acquiring the predicted network parameters corresponding to each sensor device to be corrected in at least part of the sensor devices to be corrected in the multi-source sensor devices. The technical problem of poor multi-source fusion effect in the related art can be solved, thereby improving the effect of multi-source fusion.

Description

Technical Field

[0001] The present application relates to the field of vehicle-road collaboration technology, and may also relate to the fields of transportation, autonomous driving, artificial intelligence, cloud technology, etc. Specifically, the present application relates to a multi-source sensing device synchronization method based on multi-source fusion and related devices. Background Art

[0002] In the field of vehicle-road cooperative technology, sensor devices are generally used to sense the environment around the vehicle, and then determine the vehicle's driving guidance decision. In order to improve the accuracy of environmental perception, multi-source sensor devices are generally used to collect the environment. For example, each sensor device in the multi-source sensor device collects sensor data, and then the sensor data collected by the multi-source sensor devices are multi-source fused to improve the accuracy of environmental perception.

[0003] Therefore, how to improve the effect of multi-source fusion is one of the important research directions in this field. Summary of the invention

[0004] The embodiments of the present application provide a multi-source sensing device synchronization method based on multi-source fusion and related devices, which are used to solve the technical problem of poor multi-source fusion effect in related technologies, thereby improving the effect of multi-source fusion.

[0005] On the one hand, an embodiment of the present application provides a multi-source sensor device synchronization method based on multi-source fusion, comprising:

[0006] Performing synchronization processing on the multi-source sensing devices, the synchronization processing including at least one of spatial synchronization processing or temporal synchronization processing;

[0007] Spatial synchronization processing includes:

[0008] Acquire a digitized map, and calibrate each sensor device in the multi-source sensor device based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map;

[0009] Determine a target extrinsic parameter between the multi-source sensing devices based on a first candidate extrinsic parameter corresponding to each sensing device;

[0010] Time synchronization processing includes:

[0011] Acquire predicted network parameters corresponding to each sensor device to be calibrated in at least part of the sensor devices to be calibrated in the multi-source sensor devices, where the predicted network parameters are network parameters in a future time period between the sensor device to be calibrated and the time synchronization source;

[0012] For each sensor device to be calibrated, determining a clock calibration frequency corresponding to the sensor device to be calibrated based on the predicted network parameters corresponding to the sensor device to be calibrated;

[0013] For each sensor device to be calibrated, the clock of the sensor device to be calibrated is calibrated based on the clock correction frequency corresponding to the sensor device to be calibrated, so that the time difference between the corrected clock of the sensor device to be calibrated and the clock of the time synchronization source is smaller than the time difference between the clock before correction and the clock of the time synchronization source.

[0014] On the other hand, an embodiment of the present application provides a multi-source sensor device synchronization device based on multi-source fusion, including:

[0015] A synchronization module, used to perform synchronization processing on the multi-source sensor devices, wherein the synchronization processing includes at least one of spatial synchronization processing or temporal synchronization processing;

[0016] When performing spatial synchronization processing, the synchronization module is used to:

[0017] Acquire a digitized map, and calibrate each sensor device in the multi-source sensor device based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map;

[0018] Determine a target extrinsic parameter between the multi-source sensing devices based on a first candidate extrinsic parameter corresponding to each sensing device;

[0019] When performing time synchronization processing, the synchronization module is used to:

[0020] Acquire predicted network parameters corresponding to each sensor device to be calibrated in at least part of the sensor devices to be calibrated in the multi-source sensor devices, where the predicted network parameters are network parameters in a future time period between the sensor device to be calibrated and the time synchronization source;

[0021] For each sensor device to be calibrated, determining a clock calibration frequency corresponding to the sensor device to be calibrated based on the predicted network parameters corresponding to the sensor device to be calibrated;

[0022] For each sensor device to be calibrated, the clock of the sensor device to be calibrated is calibrated based on the clock correction frequency corresponding to the sensor device to be calibrated, so that the time difference between the corrected clock of the sensor device to be calibrated and the clock of the time synchronization source is smaller than the time difference between the clock before correction and the clock of the time synchronization source.

[0023] Optionally, when the synchronization module calibrates each sensor device in the multi-source sensor device based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map, it can be used to:

[0024] Calibrate each sensor device in the multi-source sensor device based on the digitized map to obtain a first current external parameter between each sensor device and the digitized map;

[0025] For each sensor device in the multi-source sensor device, if the sensor device is calibrated for the first time, the corresponding first current external parameter is determined as the first candidate external parameter; if the sensor device is calibrated for at least the second time, the first external parameter deviation between the first current external parameter and the first historical external parameter is determined, and the first candidate external parameter corresponding to the sensor device is determined based on the first external parameter deviation, and the first historical external parameter is the external parameter determined by the calibration before this calibration.

[0026] Optionally, when the synchronization module calibrates each sensor device in the multi-source sensor device based on the digitized map to obtain a first current external parameter between each sensor device and the digitized map, it can be used to:

[0027] Acquiring sensor data collected by each sensor device;

[0028] For each sensing device, first road information within a preset range of the sensing device is determined based on the sensing data collected by the sensing device, and a matching map range in the digitized map that matches the preset range is determined, and second road information matching the first road information is determined from the matching map range, and a first current external parameter between the sensing device and the digitized map is determined based on the relative relationship between the first road information and the second road information.

[0029] Optionally, the multi-source sensing device includes at least one of a point cloud sensing device or an image sensing device, the sensing data collected by the point cloud sensing device includes point cloud data, the first road information determined based on the point cloud data includes at least one of vehicle flow information or guardrail information, and the second road information matching the first road information determined based on the point cloud data includes at least one of vehicle flow information or guardrail information, the sensing data collected by the image sensing device includes image data, the first road information determined based on the image data includes at least one of vehicle flow information, guardrail information, or lane line information, and the second road information matching the first road information determined based on the image data includes at least one of vehicle flow information, guardrail information, or lane line information;

[0030] When the synchronization module determines the first current external parameter between the sensor device and the digitized map based on the relative relationship between the first road information and the second road information, it can be used to:

[0031] For the point cloud sensing device, determine a first relative relationship corresponding to the point cloud sensing device, and determine a first current external parameter between the point cloud sensing device and the digitized map based on each relative sub-relationship in the first relative relationship, wherein the first relative relationship includes at least one of a first relative sub-relationship or a second relative sub-relationship, the first relative sub-relationship is a relationship between vehicle flow information determined based on the point cloud data and vehicle flow information in the digitized map, and the second relative sub-relationship is a relationship between guardrail information determined based on the point cloud data and guardrail information in the digitized map;

[0032] For the image sensing device, a second relative relationship corresponding to the image sensing device is determined, and based on each relative sub-relationship in the second relative relationship, a first current external parameter between the image sensing device and the digitized map is determined, wherein the second relative relationship includes at least one of a third relative sub-relationship, a fourth relative sub-relationship, or a fifth relative sub-relationship, the third relative sub-relationship is a relationship between traffic information determined based on image data and traffic information in the digitized map, the fourth relative sub-relationship is a relationship between guardrail information determined based on image data and guardrail information in the digitized map, and the fifth relative sub-relationship is a relationship between lane line information determined based on image data and lane line information in the digitized map.

[0033] Optionally, when the synchronization module determines the first candidate external parameter corresponding to the sensor device based on the first external parameter deviation, it can be used to perform at least one of the following:

[0034] If the first external parameter deviation is greater than the first external parameter deviation threshold, determining the first current external parameter as the first candidate external parameter;

[0035] If the first external parameter deviation is less than or equal to the first external parameter deviation threshold, the first historical external parameter of the most recent calibration is determined as the first candidate external parameter corresponding to the sensor device;

[0036] Obtain a first weight corresponding to the first current external parameter, and obtain a second weight corresponding to the first historical external parameter of the most recent calibration of this calibration, and perform weighted calculation based on the first current external parameter, the first weight, the first historical external parameter, and the second weight to obtain a first candidate external parameter corresponding to the sensing device, wherein when the first external parameter deviation is greater than the first external parameter deviation threshold, the first weight is greater than the second weight; when the first external parameter deviation is less than or equal to the first external parameter deviation threshold, the first weight is less than the second weight.

[0037] Optionally, when the synchronization module determines the target external parameter between the multi-source sensor devices based on the first candidate external parameter corresponding to each sensor device, it can be used to:

[0038] Determine at least K-1 pairs of sensor devices, wherein the sensor device in each sensor device pair is one of the multi-source sensor devices, and K is the total number of the multi-source sensor devices;

[0039] For each sensor device pair, determining a second candidate extrinsic parameter between the sensor devices in the sensor device pair based on the first candidate extrinsic parameter corresponding to each sensor device in the sensor device pair;

[0040] For each sensing device pair, a target extrinsic parameter between the sensing devices in the sensing device pair is determined based on a second candidate extrinsic parameter between the sensing devices in the sensing device pair.

[0041] Optionally, when the synchronization module determines the target extrinsic parameter between the sensor devices in the sensor device pair based on the second candidate extrinsic parameter between the sensor devices in the sensor device pair, it can be used to perform any of the following:

[0042] Determining a second candidate extrinsic parameter between the sensor devices in the sensor device pair as a target extrinsic parameter between the sensor devices in the sensor device pair;

[0043] The first road information corresponding to each sensor device in the sensor device pair is obtained, and based on the first road information corresponding to each sensor device in the sensor device pair, a third candidate extrinsic parameter between the sensor devices in the sensor device pair is determined, and the second candidate extrinsic parameter and the third candidate extrinsic parameter between the sensor devices in the sensor device pair are fused to obtain a target extrinsic parameter between the sensor devices in the sensor device pair, wherein the first road information corresponding to the sensor device is determined based on the sensor data collected by the sensor device.

[0044] Optionally, the multi-source sensing device includes a roadside sensing device and a vehicle-mounted sensing device within a preset range of the roadside sensing device, and the synchronization module, when performing spatial synchronization processing, can be used to:

[0045] Acquire sensor data collected by roadside sensor devices;

[0046] Determine the traffic flow information within a preset range based on the sensing data collected by the roadside sensing device, and determine the traffic volume within the preset range based on the traffic flow information within the preset range;

[0047] A spatial synchronization frequency for performing spatial synchronization processing is determined based on a vehicle flow rate within a preset range, wherein the vehicle flow rate is positively correlated with the spatial synchronization frequency.

[0048] Optionally, the network parameter includes at least one of network delay, network jitter or network available bandwidth. When the synchronization module determines the clock correction frequency corresponding to the sensor device to be corrected based on the predicted network parameter corresponding to the sensor device to be corrected, it can be used for any of the following:

[0049] If the predicted network parameter corresponding to the sensor device to be corrected meets the first correction condition, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the first clock correction frequency, and the first correction condition includes at least one of the predicted network delay being less than the network delay threshold, the predicted network jitter being less than the network jitter threshold, or the predicted network available bandwidth being greater than the network available bandwidth threshold;

[0050] If the predicted network parameters corresponding to the sensor device to be corrected meet the second correction condition, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the second clock correction frequency, the second correction condition includes at least one of the predicted network delay being greater than or equal to the network delay threshold, the predicted network jitter being greater than or equal to the network jitter threshold, or the predicted network available bandwidth being less than or equal to the network available bandwidth threshold, and the first clock correction frequency is higher than the second clock correction frequency;

[0051] If the predicted network parameters corresponding to the sensor device to be corrected do not meet the first correction condition and the second correction condition, the predicted network parameters are fused to obtain a fused network state prediction value, and when the fused network state prediction value is greater than the network state value threshold, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the first clock correction frequency, and when the fused network state prediction value is less than or equal to the network state value threshold, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the second clock correction frequency.

[0052] Optionally, for each sensor device to be calibrated, the synchronization module is further used to:

[0053] Acquire the time difference between the sensor device to be calibrated and the time synchronization source during at least one clock calibration;

[0054] If the time difference corresponding to each clock correction in at least one clock correction is greater than the time difference threshold, determining that the amplitude of subsequent clock correction of the sensing device to be corrected is the first correction amplitude;

[0055] If the time difference corresponding to one of the at least one clock corrections is less than or equal to the time difference threshold, it is determined that the amplitude of the subsequent clock correction of the sensor device to be calibrated is a second correction amplitude, which is smaller than the first correction amplitude.

[0056] Optionally, the synchronization module is also used to:

[0057] For each sensor device to be calibrated, obtaining the real network parameters in the past time period between the sensor device to be calibrated and the time synchronization source;

[0058] For each sensor device to be calibrated, prediction is performed based on the real network parameters corresponding to the sensor device to be calibrated by using the trained network prediction model to obtain the predicted network parameters corresponding to the sensor device to be calibrated;

[0059] Among them, the trained network parameter prediction model is trained in the following way:

[0060] The actual network parameters within a first preset time period are used as the input of the network prediction model to be trained, and the actual network parameters within a second preset time period are used as the output of the network prediction model to be trained, and the network prediction model to be trained is trained, wherein the first preset time period is the time period before the second preset time period.

[0061] On the other hand, an embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method of any embodiment of the present application.

[0062] On the other hand, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method of any embodiment of the present application is implemented.

[0063] On the other hand, an embodiment of the present application further provides a computer program product, including a computer program, which implements the method of any embodiment of the present application when the computer program is executed by a processor.

[0064] The technical solution of the embodiment of the present application, for spatial synchronization processing, obtains a digitized map, and calibrates each sensor device in the multi-source sensor device based on the digitized map, obtains the first candidate external parameter between each sensor device and the digitized map, and determines the target external parameter between the multi-source sensor devices based on the first candidate external parameter corresponding to each sensor device. In various complex situations, the external parameters between the multi-source sensor devices can be accurately determined, which improves the versatility of multi-source fusion, thereby improving the effect of multi-source fusion of data. In addition, since each sensor device in the multi-source sensor device is calibrated based on the digitized map, no matter what the scene is, it can be calibrated by the method of the embodiment of the present application, and it can be calibrated without manual work, thereby improving the efficiency of calibration and the ability to generalize scenes.

[0065] For time synchronization processing, the predicted network parameters corresponding to each sensor device to be corrected in at least part of the sensor devices to be corrected in the multi-source sensor devices are obtained. The predicted network parameters are the network parameters in the future time period between the sensor device to be corrected and the time synchronization source. For each sensor device to be corrected, the clock correction frequency corresponding to the sensor device to be corrected is determined based on the predicted network parameters corresponding to the sensor device to be corrected. For each sensor device to be corrected, the clock of the sensor device to be corrected is corrected based on the clock correction frequency corresponding to the sensor device to be corrected, so that the time difference between the corrected clock of the sensor device to be corrected and the clock of the time synchronization source is less than the time difference between the clock before correction and the clock of the time synchronization source, that is, the clock correction frequency can be combined with the future network status. By adaptively adjusting the clock correction frequency, the accuracy of time synchronization can be improved, thereby improving the effect of multi-source data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0067] Figure 1 A schematic diagram of an application environment of a multi-source sensing device synchronization method based on multi-source fusion provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of an application environment of another multi-source sensing device synchronization method based on multi-source fusion provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of an implementation environment of a multi-source sensing device synchronization method based on multi-source fusion provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of a flow chart of a multi-source sensing device synchronization method based on multi-source fusion provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of the principle of a centralized mode of TSN network configuration provided in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of 5G and TSN integration provided in an embodiment of the present application;

[0073] Figure 7 A schematic diagram of a process based on multi-source fusion provided in an embodiment of the present application;

[0074] Figure 8 A schematic diagram of a calibration process between multi-source sensor devices provided in an embodiment of the present application;

[0075] Fig. 9 A schematic diagram of a time synchronization process provided by an embodiment of the present application;

[0076] Fig.10 A schematic diagram of the structure of a multi-source sensor device synchronization device based on multi-source fusion provided in an embodiment of the present application;

[0077] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0079] Those skilled in the art will appreciate that, unless expressly stated, the singular forms "one", "an", and "the" used herein may also include plural forms. It should be further understood that the terms "include" and "comprise" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to the one element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here may include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" indicates that it is implemented as "A", or is implemented as "A", or is implemented as "A and B". "Multiple" refers to "two or more"

[0080] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0081] See also Figure 1 , Figure 1 A schematic diagram of an application environment of a multi-source sensing device synchronization method based on multi-source fusion provided in an embodiment of the present application.

[0082] like Figure 1 As shown, when a vehicle is driving on a road, the vehicle's environmental information can be sensed by a roadside sensor unit (RSU) installed on the road, thereby providing the vehicle with driving guidance decisions, such as guiding the vehicle to go straight or change lanes. At this time, the vehicle's driving guidance decision can be determined by integrating the vehicle's environmental information sensed by the roadside sensor unit.

[0083] Optionally, the roadside sensing device may be at least one of multiple types such as a magnetic sensing device, a piezoelectric sensing device, or a microwave radar sensing device. In this embodiment, the roadside sensing device may include a point cloud sensing device and an image sensing device. The point cloud sensing device refers to a sensing device for collecting point cloud data, such as at least one of a laser radar or a millimeter wave radar. The image sensing device refers to a sensing device for collecting image data, such as a camera. The camera may be a monocular camera or a multi-camera camera, which is not limited here.

[0084] Optionally, the vehicle may also be equipped with an onboard sensor device, and the vehicle's environmental information may be sensed by the roadside sensor device and the onboard sensor device. In this case, the vehicle's posture guidance decision may be determined by integrating the vehicle's environmental information sensed by the roadside sensor device and the onboard sensor device. Optionally, the onboard sensor device may also include a point cloud sensor device and an image sensor device.

[0085] See also Figure 2 , Figure 2 A schematic diagram of an application environment of another multi-source sensing device synchronization method based on multi-source fusion provided in an embodiment of the present application.

[0086] like Figure 2 As shown, the vehicle may be equipped with one or more sensing devices. Specifically, each vehicle may sense the vehicle's environmental information through the multiple sensing devices installed thereon. In addition, each vehicle to be driven and guided may sense the environmental information of the vehicle to be driven and guided through the one or more sensing devices installed thereon and the sensing devices installed by other vehicles.

[0087] For example, the environmental information of vehicle A can be sensed by a sensor device installed in vehicle A and a sensor device installed in vehicle B.

[0088] It is understandable that for each vehicle on the road, the sensor devices installed on other vehicles within a certain range of the vehicle can be used to sense the environmental information of the vehicle, and one or more sensor devices installed on other vehicles can be selected for fusion as needed, without limitation here.

[0089] See also Figure 3 , Figure 3 A schematic diagram of an implementation environment of a synchronization method of a multi-source sensing device 310 based on multi-source fusion provided in an embodiment of the present application.

[0090] like Figure 3 The implementation environment shown may include a multi-source sensor device 310, a synchronization server 320, and a prediction server 330. Each of the multi-source sensor devices 310 is used to collect sensor data, and the sensor data may reflect the environment around the sensor device 310. The synchronization server 320 may be used to perform synchronization processing on the multi-source sensor devices 310, for example, performing at least one of spatial synchronization processing or time synchronization processing on the multi-source sensor devices 310. In the case of performing time synchronization processing, the role of the synchronization server 320 may be to serve as a synchronization source or as a time synchronization service to provide time synchronization to the sensor device 310.

[0091] When the synchronization server 320 needs to perform time synchronization processing, the prediction server 330 can be used to determine the predicted network parameters corresponding to each sensor device 310 to be corrected in at least some of the sensor devices 310 to be corrected in the multi-source sensor devices 310, and then send the predicted network parameters corresponding to the sensor devices 310 to be corrected to the synchronization server 320, so that the synchronization server 320 can determine the clock correction frequency corresponding to each sensor device 310 to be corrected based on the predicted network parameters corresponding to each sensor device 310 to be corrected, and then correct the clock of each sensor device 310 to be corrected based on the clock correction frequency corresponding to each sensor device 310 to be corrected.

[0092] The prediction server 330 may be a QoS (Quality of Service) prediction server 330. The QoS prediction server 330 may be used to predict the network status within a future time scale.

[0093] Optionally, a perception server may also be included. The perception server may be used to perceive and fuse three-dimensional objects on the road, such as traffic, vehicles, etc., by fusing the multi-source sensor device 310 .

[0094] In this embodiment, the perception algorithm server, synchronization server 320, and QoS prediction server 330 all have signal processing and data processing units, which can be CPU (Central Processing Unit) / GPU (Graphics Processing Unit) machines with physical entities, or they can be cloud servers.

[0095] It should be noted that, in this embodiment, a terminal within the periphery of the multi-source sensing device 310 may also be used as a server, such as a terminal in a vehicle.

[0096] Specifically, the cloud server can be implemented as an independent server or a server cluster composed of multiple servers. The terminal can be at least one of various personal computers, laptops, smart phones, tablet computers, Internet of Things devices or portable wearable devices, etc. The Internet of Things device can be at least one of smart speakers, smart TVs, smart air conditioners or smart car-mounted devices, etc. The portable wearable device can be at least one of smart watches, smart bracelets or head-mounted devices, etc.

[0097] It is understandable that, in a possible implementation, the synchronization server 320 and the prediction server 330 may be the same server, which is not limited here.

[0098] In the related art, during spatial synchronization processing, generally, each of the multiple sensor devices 310 collects sensor data from the same static object in the real scene, and then uses the difference between the sensor data collected by the multiple sensor devices 310 to determine the external parameters between the multi-source sensor devices 310. However, the actual situation may be more complicated. For example, the multiple sensor devices 310 are far apart, resulting in one of the multiple sensor devices 310 being unable to completely collect the same static object in the real scene to obtain complete sensor data. At this time, the external parameters between the multi-source sensor devices may not be accurately determined, and the versatility of multi-source fusion is low, resulting in poor fusion effect.

[0099] In addition, in the related art, in the time synchronization process, time synchronization is generally performed by sending a synchronization data packet, and the frequency of sending the synchronization data packet is fixed in the related art, and the frequency of clock correction is fixed. However, the road environment of vehicle-road collaboration is relatively complex, and the clock of the sensor device to be corrected 310 in the multi-source sensor device 310 is corrected with a fixed clock correction frequency, and the result of time synchronization may be inaccurate, resulting in poor fusion effect.

[0100] Therefore, in view of at least one of the above-mentioned technical problems or areas that need improvement in the related art, the present application proposes a multi-source sensor device synchronization method based on multi-source fusion and related devices. The scheme performs synchronization processing on the multi-source sensor devices, and the synchronization processing includes at least one of spatial synchronization processing or temporal synchronization processing; the spatial synchronization processing includes: obtaining a digitized map, and calibrating each sensor device in the multi-source sensor device based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map; determining a target external parameter between the multi-source sensor devices based on the first candidate external parameter corresponding to each sensor device; the temporal synchronization processing includes: obtaining at least part of the multi-source sensor devices The predicted network parameters corresponding to each sensor device to be corrected are divided into the sensor devices to be corrected, and the predicted network parameters are the network parameters in a future time period between the sensor device to be corrected and the time synchronization source; for each sensor device to be corrected, the clock correction frequency corresponding to the sensor device to be corrected is determined based on the predicted network parameters corresponding to the sensor device to be corrected; for each sensor device to be corrected, the clock of the sensor device to be corrected is corrected based on the clock correction frequency corresponding to the sensor device to be corrected, so that the time difference between the corrected clock of the sensor device to be corrected and the clock of the time synchronization source is smaller than the time difference between the clock before correction and the clock of the time synchronization source, so as to achieve the effect of improving multi-source fusion.

[0101] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0102] Optionally, the present application relates to the field of vehicle-road collaboration technology. For example, a roadside sensor device is used as one of the multi-source sensor devices.

[0103] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS) are a development direction of Intelligent Traffic Systems (ITS). The IVICS uses advanced wireless communications and new generation Internet technologies to implement all-round dynamic real-time information interaction between vehicles and roads, and conducts active vehicle safety control and road cooperative management based on the collection and integration of dynamic traffic information in all time and space, fully realizing the effective coordination of people, vehicles and roads, ensuring traffic safety, and improving traffic efficiency, thus forming a safe, efficient and environmentally friendly road traffic system.

[0104] Optionally, the embodiments of the present application may also relate to the field of transportation, for example, after a multi-source sensor device synchronization method is used for multi-source fusion, the sensor data collected by the multi-source sensor devices are fused to guide the driving of vehicles on the traffic road.

[0105] Intelligent transportation system, also known as intelligent transport system (Intelligent Transportation System), effectively integrates advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control and vehicle manufacturing, strengthens the connection between vehicles, roads and users, and thus forms a comprehensive transportation system that ensures safety, improves efficiency, improves the environment and saves energy.

[0106] Optionally, the embodiments of the present application may also relate to the field of autonomous driving. For example, in the process of autonomous driving, the solution of the embodiments of the present application is used to perform at least one of spatial synchronization or temporal synchronization between sensor devices.

[0107] Autonomous driving is a comprehensive technology that brings together many high-tech technologies. Environmental information acquisition and intelligent decision-making control, as key links, rely on the innovation and breakthrough of a series of high-tech technologies such as sensor device technology, image recognition technology, electronic and computer technology and control technology. Autonomous driving depends on breakthroughs and innovations in many aspects of technology. Key technologies related to autonomous driving systems include environmental perception, logical reasoning and decision-making, motion control, processor performance, etc. With the advancement of machine vision (such as 3D camera technology), pattern recognition software (such as optical character recognition programs) and lidar systems (which have been combined with global positioning technology and spatial data), on-board computers can control the driving of cars by combining machine vision, sensor data and spatial data. On the other hand, there are still some key technical issues that need to be solved for popularization, including the specification of communication protocols between vehicles, the problem of shared lanes for manned and unmanned vehicles, the establishment of a general software development platform, the fusion of information between multiple sensor devices, and the adaptability of visual algorithms to the environment.

[0108] Optionally, the present application may also relate to the field of artificial intelligence (AI) technology, for example, predicting network parameters in future time periods through artificial intelligence.

[0109] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0110] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensor devices, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. Pre-trained models, an important technology for model training in the field of artificial intelligence, are developed from large language models in the field of natural language processing (NLP). Large language models are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. LLMs are characterized by their large scale and billions of parameters, which help them learn complex patterns in language data. These models are usually based on deep learning architectures such as transformers, which helps them achieve impressive performance on various NLP tasks. Among them, the initial generative language model in the embodiment of the present application can be a large language model.

[0111] Optionally, the present application may also involve the field of cloud technology. For example, a server is deployed in the cloud, and cloud computing is used to achieve fusion of multi-source sensor data and perception of environmental information.

[0112] Among them, cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. Cloud storage is a new concept extended and developed from the concept of cloud computing. Distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide data storage and target business access functions to the outside world.

[0113] See also Figure 4 , Figure 4 The flowchart of a multi-source sensor device synchronization method based on multi-source fusion provided in an embodiment of the present application is as follows. Figure 4 The method shown can be performed by an electronic device, which can be a server. Figure 4 The method shown includes.

[0114] S400: Synchronize the multi-source sensor devices.

[0115] Wherein, the synchronization processing includes at least one of S410, spatial synchronization processing or S420, time synchronization processing. In the present embodiment, the spatial synchronization processing may be a process of determining the target external parameters between the multi-source sensing devices. Wherein, the external parameter may be understood as the relative relationship between one reference object and another reference object. Optionally, the external parameter is generally represented by a rotation matrix and a translation vector. In the present embodiment, the external parameter may refer to the relative relationship between the multi-source sensing devices. The time synchronization processing may be a process of correcting the clock of the sensing device to be corrected in the multi-source sensing device so that the time difference between the corrected clock and the time synchronization source is smaller.

[0116] First, the spatial synchronization process will be described.

[0117] Wherein, S410, spatial synchronization processing includes:

[0118] S411, obtaining a digitized map, and calibrating each sensor device in the multi-source sensor device based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map.

[0119] Among them, the multi-source sensing device may include multiple sensing devices. A digitized map may refer to a high-precision, high-resolution map of details such as ground roads, buildings, and terrain based on precise surveying and mapping technology and high-precision positioning technology, including satellite navigation and sensors and other technical means. This type of map contains not only basic geographic information, but also detailed information such as lanes, traffic signs, traffic lights, building heights, traffic flow, etc., and can be widely used in autonomous driving, intelligent transportation, urban planning and other fields. In this embodiment, the digitized map may be a high-precision map. The first candidate external parameter can be used to represent the relative relationship between the sensing device and the digitized map.

[0120] In this embodiment, the calibration of each sensor device in the multi-source sensor device based on the digitized map may not be limited by the installation position of the multi-source sensor device. Specifically, for the actual sensor data collected by each sensor device, matching elements can be found from the digitized map, and then the first candidate external parameter between the sensor device and the digitized map is determined.

[0121] S412: Determine a target extrinsic parameter among the multi-source sensor devices based on the first candidate extrinsic parameter corresponding to each sensor device.

[0122] The target extrinsic parameter can be used to represent the relative relationship between two sensor devices in the multi-source sensor device. In this embodiment, since the first candidate extrinsic parameter between the sensor device and the digitized map is determined, the target extrinsic parameter between the multi-source sensor devices can be determined based on the same reference (digitized map).

[0123] It can be understood that this embodiment obtains a digitized map and calibrates each sensor device in the multi-source sensor device based on the digitized map to obtain the first candidate external parameter between each sensor device and the digitized map, and determines the target external parameter between the multi-source sensor devices based on the first candidate external parameter corresponding to each sensor device. In response to various complex situations, the external parameters between the multi-source sensor devices can be accurately determined, thereby improving the versatility of multi-source fusion and thus improving the effect of multi-source fusion of data. In addition, since each sensor device in the multi-source sensor device is calibrated based on the digitized map, no matter what the scene is, it can be calibrated by the method of the embodiment of the present application, and it can be calibrated without manual work, thereby improving the efficiency of calibration and the ability to generalize scenes.

[0124] Next, the time synchronization process will be described.

[0125] Wherein, S420, time synchronization processing includes:

[0126] S421. Acquire predicted network parameters corresponding to each sensor device to be calibrated in at least part of the sensor devices to be calibrated in the multi-source sensor devices, where the predicted network parameters are network parameters in a future time period between the sensor device to be calibrated and the time synchronization source.

[0127] In this embodiment, if the time synchronization source is one of the sensor devices in the multi-source sensor devices, the sensor devices in the multi-source sensor devices except the time synchronization source are determined as the sensor devices to be corrected; if the time synchronization source is a time synchronization server other than the multi-source sensor devices, any sensor device in the multi-source sensor devices is determined as the sensor device to be corrected. The network parameter may refer to a parameter characterizing the network state between the sensor device to be corrected and the time synchronization source. Optionally, the network parameter may include but is not limited to at least one of network delay, network jitter or network available bandwidth. Optionally, the network parameter in the future time period may be a network parameter in the future time period with the current time as the time starting point.

[0128] S422: For each sensor device to be calibrated, determine a clock calibration frequency corresponding to the sensor device to be calibrated based on the predicted network parameters corresponding to the sensor device to be calibrated.

[0129] The clock correction frequency may refer to the number of times the clock correction is performed per unit time. In this embodiment, the network state between the sensor device to be corrected and the time synchronization source may be determined based on the network parameters between the sensor device to be corrected and the time synchronization source in the future time period, and then the clock correction frequency corresponding to the sensor device to be corrected may be determined based on the network state between the sensor device to be corrected and the time synchronization source.

[0130] S423: For each sensor device to be calibrated, calibrate the clock of the sensor device to be calibrated based on the clock calibration frequency corresponding to the sensor device to be calibrated.

[0131] In this embodiment, the time synchronization source can be used as a synchronization source or as a time synchronization service to provide time synchronization to the sensor device. Specifically, since the network status between different correction sensor devices and the time synchronization source may be different, the clock correction frequencies corresponding to different correction sensor devices may also be different. The clock of the sensor device to be corrected is corrected based on the clock correction frequency corresponding to the sensor device to be corrected, so that the time difference between the corrected clock of the sensor device to be corrected and the clock of the time synchronization source is less than the time difference between the clock before correction and the clock of the time synchronization source.

[0132] Optionally, for each sensing device to be calibrated, the clock correction method may be that the sensing device to be calibrated and the time synchronization source send time synchronization data packets to each other. The time difference between the sensing device to be calibrated and the time synchronization source can be determined based on the time carried in the time synchronization data packet. The clock of the sensing device to be calibrated can be adjusted based on the time difference, so that the time difference between the corrected clock of the sensing device to be calibrated and the clock of the time synchronization source is smaller than the time difference before correction.

[0133] Specifically, since the clock of the time synchronization source is used as the reference clock, the smaller the time difference between each device in the multi-source sensor device and the time synchronization source is, the smaller the time difference between the multi-source sensor devices is.

[0134] It can be understood that, in this embodiment, the predicted network parameters corresponding to each sensor device to be corrected in at least part of the sensor devices to be corrected in the multi-source sensor devices are obtained, and the predicted network parameters are the network parameters in the future time period between the sensor device to be corrected and the time synchronization source. For each sensor device to be corrected, the clock correction frequency corresponding to the sensor device to be corrected is determined based on the predicted network parameters corresponding to the sensor device to be corrected. For each sensor device to be corrected, the clock of the sensor device to be corrected is corrected based on the clock correction frequency corresponding to the sensor device to be corrected, so that the time difference between the corrected clock of the sensor device to be corrected and the clock of the time synchronization source is less than the time difference between the clock before correction and the clock of the time synchronization source, that is, the clock correction frequency can be combined with the future network status, and the accuracy of time synchronization can be improved by adaptively adjusting the clock correction frequency, thereby improving the effect of multi-source data fusion.

[0135] Optionally, in this embodiment, time synchronization can be performed based on TSN (Time-Sensitive Networking) technology. TSN is a network technology defined by IEEE to achieve time synchronization and deterministic communication between nodes, mainly used for real-time communication and network control, including TSN centralized user configuration (CUC), centralized network configuration (CNC), TSN bridge, data stream transmitter (TSN talker station) and data stream receiver (TSN listenerStation), etc. Optionally, the time synchronization source can be used as a TSN talker station, and the sensor device to be corrected can be used as a TSN listener Station. At present, IEEE has defined a subset of the IEEE 802.1 standard. These specifications jointly define various aspects of TSN, including clock synchronization, frame reservation, configuration and management, frame shortening, bridges and gateways, interoperability testing, process identification and time slot management, etc. The relevant protocols defined by TSN include the following:

[0136] IEEE 802.1AS-2011: TSN universal clock synchronization protocol;

[0137] IEEE 802.1Qbv-2015: TSN frame reservation protocol;

[0138] IEEE 802.1Qca-2018: TSN configuration and management protocol;

[0139] IEEE 802.1Qcc-2017: TSN switch and end system configuration and management;

[0140] IEEE 802.1Qbu-2016: TSN frame shortening protocol;

[0141] IEEE 802.1Qbg-2012: TSN bridge and gateway standard;

[0142] IEEE 802.1Qci-2019: TSN interoperability test specification;

[0143] IEEE 802.1CB-2017: TSN process identification protocol;

[0144] IEEE 802.1Qch-2020: TSN timeslot management protocol.

[0145] For TSN, after clock synchronization and scheduling strategy, network configuration must be considered. In AVB (Ethernet Audio Video Bridging), SRP (stream reservation protocol) is a distributed network configuration mechanism. In more stringent industrial applications, a more efficient and easy-to-use configuration method is required. IEEE 802.1Qcc is currently a generally accepted configuration standard. The principle of the centralized mode of TSN network configuration is as follows: Figure 5 shown. Figure 5 A schematic diagram of the principle of a centralized mode of TSN network configuration provided in an embodiment of the present application.

[0146] For the SRP mechanism provided by IEEE802.1Qat, this is a distributed network demand and resource allocation mechanism. New registration or deregistration, any changes and requests will cause network delays and overloads, reducing the transmission efficiency of the network. Therefore, the TSN working group also provides IEEE 802.1Qcc to support centralized registration and stream reservation services, called SRP enhanced mode. In this mode, the system reduces the size and frequency of reservation messages (relaxes the timer) to trigger updates when link status and reservations change.

[0147] In addition, IEEE802.1Qcc provides a set of tools for global management and control of the network, enhancing SRP through UNI and supplemented by a centralized network configuration (CNC) node. UNI provides a common L2 layer service method. CNC interacts with UNI to provide reservation, scheduling and other types of remote management protocols such as NETCONF or RESTCONF for running resources; at the same time, IEEE 802.1Qcc is compatible with IETF YANG / NETCONF data modeling language. For a fully centralized network, the optional CUC node communicates with CNC through a standard API for discovering terminal nodes, retrieving terminal node capabilities and user requirements, and configuring optimized TSN terminal node capabilities. Its interaction with more advanced stream reservation protocols (such as RSVP) is seamless, similar to AVB's use of existing SRP mechanisms. IEEE 802.1Qcc still supports the original SRP's fully distributed configuration mode, allowing coexistence between centrally managed systems and distributed systems. In addition, IEEE 802.1Qcc supports a so-called hybrid configuration mode to provide migration services for legacy devices. This configuration management mechanism, combined with IEEE 802.1Qca path control and reservation, and TSN shaper, can achieve zero congestion loss for end-to-end transmission.

[0148] For the entire network, there must be an efficient and easy-to-use network configuration to obtain the resources of terminal nodes and bridge nodes, the bandwidth of each node, data load, target address, clock and other information, and gather them to the central node for unified scheduling to obtain the best transmission efficiency.

[0149] Optionally, 5G (fifth generation mobile communication technology) can also be used for time synchronization. The 5G system constitutes a TSN switch, in which DS-TT and NW-TT are TSN translators on the terminal side and the network side, respectively.

[0150] See also Figure 6 , Figure 6 A schematic diagram of 5G and TSN integration provided in an embodiment of the present application. In this network architecture, the 5G network is virtualized into a logical switching node (also referred to as a virtual switching node) in the TSN network. Devices / network elements in the TSN network, such as Figure 6 The TSN switching node / data terminal and TSN system in the system can transmit data through the logical switching node. Figure 6As shown in the figure, in order to integrate the 5G network well into the TSN network, adaptation functions on the network side and the terminal side are added between the TSN network and the 5G network. Figure 6 In the example, the control plane (CP) with network-side TSN adaptation function is added to the application function (AF) network element, the user plane (UP) with network-side TSN adaptation function is added to the user plane function (UPF) network element, and the terminal (UE) with terminal-side TSN adaptation function is added to the terminal. These three together with the 5G network form a logical switching node as a switching node in TSN. Figure 6 In the figure, the UPF and the UP of the TSN adaptation function on the network side are drawn separately, and the UE and the TSN adaptation function on the terminal side are drawn separately. However, in practice, the UP of the TSN adaptation function on the network side can be deployed on the UPF network element, or the UP of the TSN adaptation function on the network side can be an internal functional module of the UPF network element. Correspondingly, the TSN adaptation function on the terminal side can be deployed on the UE, or the TSN adaptation function on the terminal side can be an internal functional module of the UE. Among them, the CP of the TSN adaptation function on the network side, the UP of the TSN adaptation function on the network side, and the TSN adaptation function on the terminal side are used to adapt the characteristics and information of the 5G network to the information required by TSN, and communicate with the network elements in TSN through the interface defined by TSN. Based on Figure 6 In the network architecture shown, when the TSN network transmits data through the 5G network, the downlink transmission path of the data can be: TSN system->TSN adaptation function (UP) on the network side->UPF->RAN->UE->TSN adaptation function on the terminal side->TSN switching node / data terminal, and the uplink transmission path of the data can be: TSN switching node / data terminal->TSN adaptation function on the terminal side->UE->RAN->UPF->TSN adaptation function (UP) on the network side->TSN system.

[0151] Understandably, Figure 6 In the network architecture that integrates the 5G network with the TSN network, the TSN switching nodes / data terminals and the TSN system belong to the TSN network, and all nodes except the TSN switching nodes / data terminals and the TSN system belong to the 5G network.

[0152] It should be noted that Figure 6 The network architecture in the example of integrating 5G network and TSN network is illustrated. Figure 6The 5G network in the present application can also be replaced by other third-generation partnership project networks, for example, it can be replaced by a 4.5G network, a 4G network, a 3G network, a 2G network or a future 6G network, etc. The method provided in the embodiment of the present application can be applied to a network architecture that integrates any 3GPP network with a TSN network. The method provided in the embodiment of the present application is not limited to being used for Figure 6 The network architecture shown.

[0153] It should be noted that the spatial synchronization processing and the temporal synchronization processing may be performed simultaneously, or the temporal synchronization processing may be performed after the spatial synchronization processing is completed, or the spatial synchronization processing may be performed after the temporal synchronization processing is completed, and no limitation is made here.

[0154] For ease of understanding, the following embodiments further illustrate the spatial synchronization processing and the temporal synchronization processing respectively on the basis of the above embodiments.

[0155] In a possible implementation, each sensor device in the multi-source sensor device is calibrated based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map, including:

[0156] Calibrate each sensor device in the multi-source sensor device based on the digitized map to obtain a first current external parameter between each sensor device and the digitized map;

[0157] For each sensor device in the multi-source sensor device, if the sensor device is calibrated for the first time, the corresponding first current external parameter is determined as the first candidate external parameter; if the sensor device is calibrated for at least the second time, the first external parameter deviation between the first current external parameter and the first historical external parameter is determined, and the first candidate external parameter corresponding to the sensor device is determined based on the first external parameter deviation, and the first historical external parameter is the external parameter determined by the calibration before this calibration.

[0158] The first current external parameter may refer to an external parameter determined by the current calibration, that is, an external parameter determined by the latest calibration.

[0159] In this embodiment, if the sensor device is calibrated for the first time, the corresponding first current external parameter can be determined as the first candidate external parameter. If the sensor device is calibrated for the second or more times, the first external parameter deviation between the first current external parameter and the first historical external parameter is determined, and the first candidate external parameter corresponding to the sensor device is determined based on the first external parameter deviation, that is, the first candidate external parameter is determined in combination with the historical calibration situation, which can improve the calibration accuracy of the determined first candidate external parameter, and further improve the accuracy of multi-source fusion.

[0160] Specifically, the first current extrinsic parameter may be subtracted from the first historical extrinsic parameter to obtain a subtracted matrix, and then a value corresponding to the subtracted matrix may be determined, and the value corresponding to the subtracted matrix may be determined as the first extrinsic parameter deviation.

[0161] It should be noted that the starting point of the calibration times can be set as needed. For example, when the calibration times reach a threshold, the calibration times are recalculated, or the calibration times are recalculated within a certain period of time after the first calibration.

[0162] Optionally, the certain time may be related to the current traffic volume on the road. The greater the current traffic volume, the shorter the interval time for recalculating the calibration times may be. There is no limitation here.

[0163] It is understandable that when the multi-source sensor device that needs to fuse sensor data includes a vehicle-mounted sensor device, the accuracy of the calibration can be further improved by determining the interval time for recalculating the calibration times according to the current traffic flow on the road.

[0164] In a possible implementation, each sensor device in the multi-source sensor device is calibrated based on the digitized map to obtain a first current external parameter between each sensor device and the digitized map, including:

[0165] Acquiring sensor data collected by each sensor device;

[0166] For each sensing device, first road information within a preset range of the sensing device is determined based on the sensing data collected by the sensing device, and a matching map range in the digitized map that matches the preset range is determined, and second road information matching the first road information is determined from the matching map range, and a first current external parameter between the sensing device and the digitized map is determined based on the relative relationship between the first road information and the second road information.

[0167] Among them, the preset range may be related to the detection capability of the sensor device. Optionally, the preset range is not greater than the maximum detection range supported by the sensor device. The first road information may refer to the road information determined based on the sensor data. The second road information may refer to the road information in the digitized map that matches the first road information. Optionally, the road elements referred to by the matched first road information and the second road information may be consistent. For example, both refer to traffic or guardrails, etc., which are not limited here.

[0168] Exemplarily, the first road information may include at least one of traffic flow information, guardrail information or lane line information. Traffic flow information may be used to characterize the size of traffic flow. Traffic flow may be the number of vehicles passing through the sensing device per unit time. Optionally, the traffic flow information may be current traffic flow information characterizing the current size of traffic flow, or may be accumulated traffic flow information characterizing the accumulated size of traffic flow. Specifically, the current traffic flow information may be determined as the traffic flow information. It may also be the sum of the traffic flow information determined by each calibration from the first calibration to the current calibration, so that the sum of the traffic flow information determined by each calibration is determined as the accumulated traffic flow information. Guardrail information may be used to characterize the features of the guardrail, such as at least one of the features of the length of the guardrail, the height of the guardrail or the width of the guardrail. Lane line information may be used to characterize the features of the lane line, such as at least one of the features of the lane line, such as the lane line of the lane number or the line width of the lane line, which is not limited here.

[0169] Specifically, if the first road information includes traffic flow information, the second road information matching the first road information includes traffic flow information; if the first road information includes guardrail information, the second road information matching the first road information includes guardrail information; if the first road information includes lane line information, the second road information matching the first road information includes lane line information.

[0170] In this embodiment, optionally, a matching map range in the digitized map that matches the preset range can be searched based on the location information of the sensor device. Optionally, the center of the matching map range can be the location information of the sensor device, and the matching map range can be no larger than the preset range.

[0171] Specifically, the multi-source sensor device may include at least one of a point cloud sensor device and an image sensor device, and the following embodiments describe the point cloud sensor device and the image sensor device respectively. The image sensor device may identify road information through a machine vision algorithm.

[0172] First, a point cloud sensing device is used for explanation.

[0173] The sensing data collected by the point cloud sensing device includes point cloud data, the first road information determined based on the point cloud data includes at least one of vehicle flow information or guardrail information, the second road information matched with the first road information determined based on the point cloud data includes at least one of vehicle flow information or guardrail information, and the first current external parameter between the sensing device and the digitized map is determined based on the relative relationship between the first road information and the second road information, including:

[0174] Determine a first relative relationship corresponding to the point cloud sensing device, and determine a first current external parameter between the point cloud sensing device and the digitized map based on each relative sub-relationship in the first relative relationship, wherein the first relative relationship includes at least one of a first relative sub-relationship or a second relative sub-relationship, the first relative sub-relationship being the relationship between the traffic information determined based on the point cloud data and the traffic information in the digitized map, and the second relative sub-relationship being the relationship between the guardrail information determined based on the point cloud data and the guardrail information in the digitized map.

[0175] In this embodiment, the traffic flow information and the guardrail information can be understood as the information of a three-dimensional target. The traffic flow information can be understood as the information of a dynamic target, and the guardrail information can be understood as the information of a static target.

[0176] It should be noted that, when there are at least two relative sub-relationships in the first relative relationship, the first current external parameter can be determined based on one of the relative sub-relationships, or at least two relative sub-relationships in the first relative relationship can be fused to further determine the first current external parameter between the point cloud sensing device and the digitized map.

[0177] It can be understood that by fusing at least two relative sub-relations in the first relative relationship to further determine the first current external parameter between the point cloud sensing device and the digitized map, the accuracy of the determined first current external parameter can be improved.

[0178] Next, an image sensor device will be used for explanation.

[0179] The sensing data collected by the image sensing device includes image data, the first road information determined based on the image data includes at least one of vehicle flow information, guardrail information, or lane line information, and the second road information matched with the first road information determined based on the image data includes at least one of vehicle flow information, guardrail information, or lane line information;

[0180] Determining a first current external parameter between the sensor device and the digitized map based on a relative relationship between the first road information and the second road information includes:

[0181] Determine a second relative relationship corresponding to the image sensing device, and determine a first current external parameter between the image sensing device and the digitized map based on each relative sub-relationship in the second relative relationship, wherein the second relative relationship includes at least one of a third relative sub-relationship, a fourth relative sub-relationship, or a fifth relative sub-relationship, the third relative sub-relationship being a relationship between traffic flow information determined based on the image data and traffic flow information in the digitized map, the fourth relative sub-relationship being a relationship between guardrail information determined based on the image data and guardrail information in the digitized map, and the fifth relative sub-relationship being a relationship between lane line information determined based on the image data and lane line information in the digitized map.

[0182] In this embodiment, the lane line information can be understood as the information of a two-dimensional target. In addition, the lane line information can also be understood as the information of a static target.

[0183] It should be noted that, when there are at least two relative sub-relationships in the second relative relationship, the first current external parameter can be determined based on one of the relative sub-relationships, or at least two relative sub-relationships in the second relative relationship can be fused to further determine the first current external parameter between the image sensing device and the digitized map.

[0184] It can be understood that by fusing at least two relative sub-relations in the second relative relationship to determine the first current external parameter between the image sensing device and the digitized map, the accuracy of the determined first current external parameter can be improved.

[0185] In the above embodiments, it is mentioned that the first road information may include at least one of traffic flow information, hurdle information or lane line information. The following embodiments further illustrate how to select the first road information for calibration based on any of the above embodiments.

[0186] Optionally, if the multi-source sensor device includes a mobile sensor device, the target extrinsic parameters between the multi-source sensor devices can be calibrated based on dynamic vehicle flow information.

[0187] Exemplarily, the multi-source sensing device is described as including a vehicle-mounted sensing device and a roadside sensing device.

[0188] Specifically, the vehicle-mounted sensor device is installed on the vehicle and will move with the vehicle, while the roadside sensor device is generally fixed beside the road, so the motion relationship between the vehicle-mounted sensor device and the roadside sensor device is generally relative motion. Since the vehicle-mounted sensor device and the roadside sensor device correspond to the same reference object (digital map), the absolute traffic information in the digital map is consistent, but the vehicle-mounted sensor device is mobile, while the roadside sensor device is fixed, that is, the first relative traffic information between the first traffic information determined by the vehicle-mounted sensor device and the second traffic information on the digital map is different from the second relative traffic information between the first traffic information determined by the roadside sensor device and the second traffic information. Therefore, based on the first relative traffic information and the second relative traffic information, the relative change relationship between the vehicle-mounted sensor device and the roadside sensor device in unit time can be determined.

[0189] As another example, the multi-source sensor device includes at least two vehicle-mounted sensor devices for illustration. For ease of understanding, the at least two vehicle-mounted sensor devices are respectively named as one vehicle-mounted sensor device and another vehicle-mounted sensor device for illustration.

[0190] Specifically, if one of the vehicle-mounted sensor devices and the other vehicle-mounted sensor device are installed on different vehicles, then one of the vehicle-mounted sensor devices and the other vehicle-mounted sensor device will respectively move with the vehicles on which they are installed. However, the actual road conditions are complicated, so the vehicle on which one of the vehicle-mounted sensor devices is installed and the vehicle on which the other vehicle-mounted sensor device is installed may be relatively stationary or relatively moving, depending on the actual situation. Similarly, the two vehicle-mounted sensor devices correspond to a unified reference object (digital map), and the absolute traffic information in the digital map is consistent. Therefore, the third relative traffic information between the first traffic information and the second traffic information determined by one of the vehicle-mounted sensor devices and the fourth relative traffic information between the first traffic information and the second traffic information determined by the other vehicle-mounted sensor device may be different or the same. Therefore, based on the third relative traffic information and the fourth relative traffic information, the relative change relationship between one of the vehicle-mounted sensor devices and the other vehicle-mounted sensor device within a unit time can be determined.

[0191] It should be noted that the dynamic information in the digitized map of this embodiment may be updated in real time, such as the dynamic traffic information in the digitized map. Optionally, the dynamic traffic information in the digitized map may be updated based on the traffic information detected by the roadside sensor device, such as determining the traffic information detected by the roadside sensor device as the latest traffic information in the digitized map.

[0192] In a possible implementation manner, determining the first candidate extrinsic parameter corresponding to the sensing device based on the first extrinsic parameter deviation includes at least one of the following:

[0193] If the first external parameter deviation is greater than the first external parameter deviation threshold, determining the first current external parameter as the first candidate external parameter;

[0194] If the first external parameter deviation is less than or equal to the first external parameter deviation threshold, the first historical external parameter of the most recent calibration is determined as the first candidate external parameter corresponding to the sensor device;

[0195] Obtain a first weight corresponding to the first current external parameter, and obtain a second weight corresponding to the first historical external parameter of the most recent calibration of this calibration, and perform weighted calculation based on the first current external parameter, the first weight, the first historical external parameter, and the second weight to obtain a first candidate external parameter corresponding to the sensing device, wherein when the first external parameter deviation is greater than the first external parameter deviation threshold, the first weight is greater than the second weight; when the first external parameter deviation is less than or equal to the first external parameter deviation threshold, the first weight is less than the second weight.

[0196] In the present embodiment, the first external parameter deviation threshold value can be set as needed and is not limited here. In the present embodiment, when the first external parameter deviation is greater than the first external parameter deviation threshold value, it can be indicated that the deviation is large, that is, the change of the external parameter is large, and the first current external parameter is determined as the first candidate external parameter. Alternatively, when the first external parameter deviation is less than or equal to the first external parameter deviation threshold value, it can be indicated that the deviation is small, that is, the change of the external parameter is small, and the first historical external parameter calibrated most recently in this calibration is determined as the first candidate external parameter corresponding to the sensing device. In addition, the first current external parameter can be directly combined with the first historical external parameter for weighted calculation, thereby improving the calculation accuracy of the first candidate external parameter.

[0197] In a possible implementation, determining a target extrinsic parameter between multiple source sensor devices based on a first candidate extrinsic parameter corresponding to each sensor device includes:

[0198] Determine at least K-1 pairs of sensor devices, wherein the sensor device in each sensor device pair is one of the multi-source sensor devices, and K is the total number of the multi-source sensor devices;

[0199] For each sensor device pair, determining a second candidate extrinsic parameter between the sensor devices in the sensor device pair based on the first candidate extrinsic parameter corresponding to each sensor device in the sensor device pair;

[0200] For each sensing device pair, a target extrinsic parameter between the sensing devices in the sensing device pair is determined based on a second candidate extrinsic parameter between the sensing devices in the sensing device pair.

[0201] Exemplarily, assuming that the multi-source sensor device includes sensor device 1, sensor device 2 and sensor device 3, at least a second candidate extrinsic parameter between sensor device 1 and sensor device 2, and a second candidate extrinsic parameter between sensor device 1 and sensor device 3 need to be determined.

[0202] As another example, assuming that the multi-source sensing device includes sensing device 1, sensing device 2, sensing device 3 and sensing device 4, at least the second candidate external parameter between sensing device 1 and sensing device 2, the second candidate external parameter between sensing device 1 and sensing device 3, and the second candidate external parameter between sensing device 1 and sensing device 4 must be determined.

[0203] It can be understood that the second candidate extrinsic parameter of each of the K-1 pairs of sensor devices can be determined, which can reduce the computing resources required for spatial synchronization processing.

[0204] In a possible implementation, determining a target extrinsic parameter between the sensor devices in the sensor device pair based on the second candidate extrinsic parameter between the sensor devices in the sensor device pair includes any one of the following:

[0205] Determining a second candidate extrinsic parameter between the sensor devices in the sensor device pair as a target extrinsic parameter between the sensor devices in the sensor device pair;

[0206] The first road information corresponding to each sensor device in the sensor device pair is obtained, and based on the first road information corresponding to each sensor device in the sensor device pair, a third candidate extrinsic parameter between the sensor devices in the sensor device pair is determined, and the second candidate extrinsic parameter and the third candidate extrinsic parameter between the sensor devices in the sensor device pair are fused to obtain a target extrinsic parameter between the sensor devices in the sensor device pair, wherein the first road information corresponding to the sensor device is determined based on the sensor data collected by the sensor device.

[0207] Specifically, the second candidate external parameter between the sensor devices in the sensor pair may be determined as the target external parameter between the sensor devices in the sensor pair, thereby reducing the computing resources required for calibration. In addition, the third candidate external parameter between the sensor devices in the sensor pair may be determined by the first road information corresponding to each sensor device in the sensor pair, and then the second candidate external parameter and the third candidate external parameter between the sensor devices in the sensor pair may be fused to obtain the target external parameter between the sensor devices in the sensor pair, thereby improving the accuracy of the determined target external parameter.

[0208] In a possible implementation, the multi-source sensing device includes a roadside sensing device and a vehicle-mounted sensing device within a preset range of the roadside sensing device, and the spatial synchronization processing includes:

[0209] Acquire sensor data collected by roadside sensor devices;

[0210] Determine the traffic flow information within a preset range based on the sensing data collected by the roadside sensing device, and determine the traffic volume within the preset range based on the traffic flow information within the preset range;

[0211] A spatial synchronization frequency for performing spatial synchronization processing is determined based on a vehicle flow rate within a preset range, wherein the vehicle flow rate is positively correlated with the spatial synchronization frequency.

[0212] In this embodiment, the traffic volume may be the current traffic volume. Specifically, the greater the traffic volume, the more complex the road conditions, and the relative relationship between the vehicle-mounted sensor device and the roadside sensor device will change frequently. Therefore, the spatial synchronization frequency for performing spatial synchronization processing may be determined based on the traffic volume within a preset range, and the greater the traffic volume, the higher the spatial synchronization frequency.

[0213] The technical solution of this embodiment is to obtain sensor data collected by a roadside sensor device; determine the traffic information within a preset range based on the sensor data collected by the roadside sensor device, and determine the traffic volume within a preset range based on the traffic information within the preset range; and determine the spatial synchronization frequency for spatial synchronization processing based on the traffic volume within the preset range, which can further improve the accuracy of spatial synchronization processing.

[0214] The following embodiments further illustrate the solution for time synchronization based on any of the above embodiments.

[0215] In a possible implementation, the network parameter includes at least one of network delay, network jitter, or network available bandwidth, and determining the clock correction frequency corresponding to the sensor device to be corrected based on the predicted network parameter corresponding to the sensor device to be corrected includes at least one of the following:

[0216] If the predicted network parameter corresponding to the sensor device to be corrected meets the first correction condition, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the first clock correction frequency, and the first correction condition includes at least one of the predicted network delay being less than the network delay threshold, the predicted network jitter being less than the network jitter threshold, or the predicted network available bandwidth being greater than the network available bandwidth threshold;

[0217] If the predicted network parameters corresponding to the sensor device to be corrected meet the second correction condition, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the second clock correction frequency, the second correction condition includes at least one of the predicted network delay being greater than or equal to the network delay threshold, the predicted network jitter being greater than or equal to the network jitter threshold, or the predicted network available bandwidth being less than or equal to the network available bandwidth threshold, and the first clock correction frequency is higher than the second clock correction frequency;

[0218] If the predicted network parameters corresponding to the sensor device to be corrected do not meet the first correction condition and the second correction condition, the predicted network parameters are fused to obtain a fused network state prediction value, and when the fused network state prediction value is greater than the network state value threshold, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the first clock correction frequency, and when the fused network state prediction value is less than or equal to the network state value threshold, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the second clock correction frequency.

[0219] In this embodiment, the predicted network parameters are fused to obtain a fused network state prediction value, which can be a fused network state estimation value obtained by normalizing the network parameters and performing weighted calculation according to the weights. The network state estimation value can reflect the network state between the sensor device to be corrected and the time synchronization source. The network state value threshold is used to evaluate whether the network state is suitable for clock correction. If the fused network state prediction value is greater than the network state value threshold, it means that the network state is good and the network fluctuation may be small, then correction can be performed at a higher clock correction frequency. If the fused network state prediction value is less than or equal to the network state value threshold, it means that the network state is poor and the network fluctuation may be large. At this time, correction can be performed at a lower clock correction frequency, thereby avoiding invalid correction caused by packet loss, thereby taking into account both the accuracy and efficiency of clock correction.

[0220] In a possible implementation, for each sensor device to be calibrated, the method further includes:

[0221] Acquire the time difference between the sensor device to be calibrated and the time synchronization source during at least one clock calibration;

[0222] If the time difference corresponding to each clock correction in at least one clock correction is greater than the time difference threshold, determining that the amplitude of subsequent clock correction of the sensing device to be corrected is the first correction amplitude;

[0223] If the time difference corresponding to one of the at least one clock corrections is less than or equal to the time difference threshold, it is determined that the amplitude of the subsequent clock correction of the sensor device to be calibrated is a second correction amplitude, which is smaller than the first correction amplitude.

[0224] In this embodiment, specifically, the stage of correcting the clock of the sensor device to be corrected with the first correction amplitude can be considered as the adjustment stage, in which a large-scale clock correction is performed to quickly achieve the optimal synchronization progress. The stage of correcting the clock of the sensor device to be corrected with the second correction amplitude can be considered as the punctual stage. The punctual stage is a small-scale clock adjustment to minimize the interference caused by fluctuations.

[0225] In some example situations, the network status may be unstable and fluctuate, which may easily lead to abnormal situations such as loss of time synchronization messages and delayed arrival, which in turn leads to a large time difference between the sensor device to be corrected and the time synchronization source. Therefore, at least one clock correction can be performed in a round of time synchronization processing, and a time difference can be obtained for each clock correction. According to the time difference obtained in a round of time synchronization processing, it is determined whether it is a short-term fluctuation or a continuous fluctuation. A counter method is used to determine whether the fluctuation is a short-term fluctuation or a continuous fluctuation: in each time synchronization, if the network is in a fluctuating state for 50 consecutive times (that is, the time offset is greater than 8ms), the program will consider the current mode to be a continuous fluctuation, otherwise, it is considered to be a short-term fluctuation.

[0226] In a possible implementation, the predicted network parameters corresponding to each of the sensor devices to be calibrated in at least part of the sensor devices to be calibrated in the multi-source sensor devices are determined in the following manner:

[0227] For each sensor device to be calibrated, obtaining the real network parameters in the past time period between the sensor device to be calibrated and the time synchronization source;

[0228] For each sensor device to be calibrated, prediction is performed based on the real network parameters corresponding to the sensor device to be calibrated by using the trained network prediction model to obtain the predicted network parameters corresponding to the sensor device to be calibrated;

[0229] Among them, the trained network parameter prediction model is trained in the following way:

[0230] The actual network parameters within a first preset time period are used as the input of the network prediction model to be trained, and the actual network parameters within a second preset time period are used as the output of the network prediction model to be trained, and the network prediction model to be trained is trained, wherein the first preset time period is the time period before the second preset time period.

[0231] Optionally, the past time period may be a time period determined with the current time as the time end point. Optionally, the past time period may be consistent with the time period length of the first preset time period. The future time period may be consistent with the time period length of the second preset time period. The time end point of the first preset time period may be used as the time start point of the second preset time period.

[0232] The technical solution of this embodiment can improve the accuracy of network status prediction by training the network prediction model through machine learning of time series prediction. Optionally, the network prediction model of this embodiment may include but is not limited to at least one of RNN (Recurrent Neutral Network), LGBM (Light Gradient Boosting Machine) or RF (Random Forest), which is not limited here. In addition, by setting the past time period to the time period determined with the current time as the time end, the past time period is consistent with the time period length of the first preset time period, the future time period is consistent with the time period length of the second preset time period, and the time end of the first preset time period is used as the time starting point of the second preset time period, the accuracy of network status prediction can be further improved.

[0233] See also Figure 7 , Figure 7 A schematic diagram of a multi-source fusion process provided in an embodiment of the present application. This embodiment is described by taking a synchronization server as a time synchronization source as an example.

[0234] S710: The synchronization server calibrates each sensor device in the multi-source sensor device based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map.

[0235] S720. The synchronization server determines a target external parameter between the multi-source sensor devices based on the first candidate external parameter corresponding to each sensor device, and sends the target external parameter between the multi-source sensor devices to the perception server.

[0236] S730. The prediction server determines the predicted network parameters corresponding to each sensor device, and sends the predicted network to the synchronization server.

[0237] S740: The synchronization server determines, for each sensor device, a clock correction frequency corresponding to the sensor device based on the predicted network parameters corresponding to the sensor device.

[0238] S750: The synchronization server calibrates the clock of each sensor device based on a clock calibration frequency corresponding to the sensor device.

[0239] In this embodiment, the synchronization server corrects the clock of the sensor device based on the clock correction frequency corresponding to the sensor device. The server may transmit time synchronization data packets to and from the sensor device according to the clock correction frequency. The time synchronization data packets carry time information. Based on the time information, the clock time difference between the sensor device and the time synchronization source may be determined. Then, the sensor device may correct its own clock based on the clock time difference.

[0240] S760: The sensor device calibrates its own clock.

[0241] S770. The sensor device sends the collected sensor data to the perception server.

[0242] S780. The perception server performs multi-source fusion perception based on the target external parameters between the multi-source sensor devices and the sensor data sent by each sensor device.

[0243] It can be understood that the relevant description of S710-S780 can refer to the description of any of the above embodiments and will not be repeated here.

[0244] The technical solution of this embodiment can improve the accuracy of multi-source fusion by improving the precision of spatiotemporal synchronization and solving the problems of fusion misalignment caused by multi-source sensor fusion.

[0245] For ease of understanding, the above embodiments are summarized and described below. This embodiment is described by taking the digitized map as a high-precision map and the multi-source sensing device including a millimeter wave radar and a camera.

[0246] Specifically, for spatial synchronization processing, it is mainly divided into automatic calibration between multi-source sensor devices and high-precision maps, and automatic calibration between multi-source sensor devices. Figure 8 , Figure 8 A schematic diagram of a calibration process between multi-source sensor devices provided in an embodiment of the present application.

[0247] (1) Multi-sensor (multi-source sensing device) and high-precision map automatic calibration algorithm:

[0248] The information statistically identified by the sensor (including at least one of the dynamically accumulated traffic flow, static guardrails or lane markings) is matched with the corresponding elements in the high-precision map (at least one of the traffic flow, guardrails or lane markings), and then the external parameter changes between the millimeter-wave radar and the high-precision map are confirmed through optimization methods. After the changes occur, the update is detected in real time. The update method is: first calculate the external parameters, detect the current external parameter status, determine whether the offset exceeds the threshold, and update the external parameters in real time. In addition, in addition to the calibration between multiple sensors and high-precision maps, there is also a unified joint calibration between multiple sensors of the same type on the road section. The dynamic targets and map elements identified by different sensors need to be matched with each other, and finally a joint global optimization is performed.

[0249] For example: Assume that sensor 1 is a millimeter-wave radar and sensor 2 is a camera. The steps of multi-sensor and high-precision map calibration are as follows: For the millimeter-wave radar, first obtain the dynamic traffic data and static guardrail data of the millimeter-wave radar sensor, project them into the high-precision map, match the associated elements with the relevant elements of the high-precision map, calculate the external parameter deviation, and judge in real time whether the external parameter offset exceeds the specified threshold through external parameter status monitoring. If it exceeds the threshold, the automatic calibration program is triggered to update the external parameters of the millimeter-wave radar and high-precision map; for the camera, first obtain the lane line information of the camera sensor through the lane line detection algorithm, match the associated elements with the lane line data of the high-precision map, calculate the external parameters of the camera relative to the high-precision map, and judge whether the offset exceeds the threshold through external parameter status monitoring. If it exceeds the specified threshold, the automatic calibration program is triggered to update the camera sensor and the external parameters of the high-precision map.

[0250] (2) Automatic calibration algorithm between multiple sensors:

[0251] Optionally, the external parameters between multiple sensors can be determined based on the results of automatic calibration of multiple sensors and high-precision maps. In addition, the dynamic vehicle objects identified by sensor 1 can be matched with the dynamic vehicle objects identified by the visual perception algorithm in sensor 2, and then the changes in external parameters between sensor 1 and sensor 2 and the high-precision map can be confirmed through optimization methods, and the updates can be detected in real time after the changes occur. Update method: First, the external parameters are calculated, and the current external parameter status is detected, and it is determined whether the offset exceeds the threshold, and the external parameters are updated in real time.

[0252] For example: Assume that sensor 1 is a millimeter-wave radar and sensor 2 is a camera. First, obtain the dynamic traffic data collected by the millimeter-wave radar, and extract the vehicle image features through the visual target detection algorithm; then perform feature correlation matching between the radar and visual observation target elements, calculate the sensor external parameters, and use the external parameter status monitoring to determine in real time whether the external parameter offset exceeds the specified threshold. If it exceeds the threshold, the automatic calibration program is triggered to update the external parameters of the millimeter-wave radar and the high-precision map.

[0253] Specifically, the time synchronization process mainly involves an adjustment phase and a timekeeping phase.

[0254] See also Fig. 9 , Fig. 9 A schematic diagram of a time synchronization process provided in an embodiment of the present application.

[0255] (3) Time synchronization scheme among multiple sensors.

[0256] A dynamic adjustment based on network status is proposed. The core of the algorithm is to reduce the frequency of clock correction and select the synchronization period with good network status for clock correction. First, the synchronization source uses the machine learning method to predict the network status in the future time scale through the collected data (the network status here refers to the network status between multiple sensors and the synchronization source), and then the synchronization strategy is divided into the adjustment stage and the punctuality stage according to the network situation.

[0257] Note: Machine learning methods predict the network status in the future time scale by performing a machine learning analysis on the collected network data. Machine learning methods based on time series prediction can be used, such as RNN (Recurrent Neutral Network), LGBM (Light gradient boosting machine), RF (Random Forest), etc.

[0258] In the adjustment phase, the network will make large clock corrections based on the prediction, striving to quickly achieve the optimal synchronization accuracy; the punctual phase is a small clock adjustment to minimize the interference caused by fluctuations. In the initial stage of the synchronization process, if a good network is predicted (low network latency, low jitter, and large available bandwidth), we will set the program to the adjustment phase. When it is detected that the synchronization accuracy reaches the ideal range, the program will be set to the punctual phase; if a poor network is predicted (high network latency, high jitter, and small available bandwidth), then the packet sending frequency in the adjustment phase will be reduced. How to set the adjustment phase of the program according to the synchronization accuracy? According to our experience, in a wide area network environment, if the following inequality holds, we will judge that the program is in the punctual phase, otherwise, we will set the program to the adjustment phase:

[0259] t 2 -t 1 -t 4 +t 3 ≤8ms.

[0260] Among them, t 2 ,t 1 ,t 4 and t 3 They respectively represent the time information carried in the time synchronization data packet.

[0261] However, due to the instability of the network in a wireless environment, it is easy to cause the loss or abnormality of synchronization messages, so it is necessary to handle abnormal situations. When it is predicted that the network will have obvious fluctuations in the future, the network status is continuously monitored to determine whether the network is a short-term fluctuation or a continuous fluctuation. A counter method is used to determine whether the fluctuation is a short-term fluctuation or a continuous fluctuation: in each time synchronization, if the program is in a network fluctuation state for 50 consecutive times (that is, the time offset is greater than 8ms), the program will consider the current mode to be a continuous fluctuation, otherwise, it will be considered a short-term fluctuation. Short-term fluctuations do not have a significant impact on synchronization accuracy. If it is a continuous fluctuation, the program will be changed from the timekeeping stage to the adjustment stage and synchronization will be restarted. Finally, after each round of timing, the parameters of the current network are re-reported and updated based on the current clock adjustment results.

[0262] Specifically, this embodiment optimizes the spatiotemporal synchronization algorithm and the QoS prediction algorithm, which can, on the one hand, realize the automation of the calibration algorithm and, on the other hand, improve the scene generalization capability; improve the spatiotemporal synchronization accuracy and solve problems such as fusion misalignment caused by multi-source sensor fusion.

[0263] See also Fig.10 , Fig.10 The structure diagram of a multi-source sensor device synchronization device based on multi-source fusion provided in an embodiment of the present application is shown in FIG. Fig.10 The illustrated apparatus 1000 may include a synchronization module 1010, wherein:

[0264] A synchronization module 1010, configured to perform synchronization processing on the multi-source sensor devices, wherein the synchronization processing includes at least one of spatial synchronization processing or temporal synchronization processing;

[0265] When performing spatial synchronization processing, the synchronization module 1010 is used to:

[0266] Acquire a digitized map, and calibrate each sensor device in the multi-source sensor device based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map;

[0267] Determine a target extrinsic parameter between the multi-source sensing devices based on a first candidate extrinsic parameter corresponding to each sensing device;

[0268] When performing time synchronization processing, the synchronization module 1010 is used to:

[0269] Acquire predicted network parameters corresponding to each sensor device to be calibrated in at least part of the sensor devices to be calibrated in the multi-source sensor devices, where the predicted network parameters are network parameters in a future time period between the sensor device to be calibrated and the time synchronization source;

[0270] For each sensor device to be calibrated, determining a clock calibration frequency corresponding to the sensor device to be calibrated based on the predicted network parameters corresponding to the sensor device to be calibrated;

[0271] For each sensor device to be calibrated, the clock of the sensor device to be calibrated is calibrated based on the clock correction frequency corresponding to the sensor device to be calibrated, so that the time difference between the corrected clock of the sensor device to be calibrated and the clock of the time synchronization source is smaller than the time difference between the clock before correction and the clock of the time synchronization source.

[0272] Optionally, when the synchronization module 1010 calibrates each sensor device in the multi-source sensor device based on the digitized map to obtain the first candidate external parameter between each sensor device and the digitized map, it can be used to:

[0273] Calibrate each sensor device in the multi-source sensor device based on the digitized map to obtain a first current external parameter between each sensor device and the digitized map;

[0274] For each sensor device in the multi-source sensor device, if the sensor device is calibrated for the first time, the corresponding first current external parameter is determined as the first candidate external parameter; if the sensor device is calibrated for at least the second time, the first external parameter deviation between the first current external parameter and the first historical external parameter is determined, and the first candidate external parameter corresponding to the sensor device is determined based on the first external parameter deviation, and the first historical external parameter is the external parameter determined by the calibration before this calibration.

[0275] Optionally, when the synchronization module 1010 calibrates each sensor device in the multi-source sensor device based on the digitized map to obtain a first current external parameter between each sensor device and the digitized map, it can be used to:

[0276] Acquiring sensor data collected by each sensor device;

[0277] For each sensing device, first road information within a preset range of the sensing device is determined based on the sensing data collected by the sensing device, and a matching map range in the digitized map that matches the preset range is determined, and second road information matching the first road information is determined from the matching map range, and a first current external parameter between the sensing device and the digitized map is determined based on the relative relationship between the first road information and the second road information.

[0278] Optionally, the multi-source sensing device includes at least one of a point cloud sensing device or an image sensing device, the sensing data collected by the point cloud sensing device includes point cloud data, the first road information determined based on the point cloud data includes at least one of vehicle flow information or guardrail information, and the second road information matching the first road information determined based on the point cloud data includes at least one of vehicle flow information or guardrail information, the sensing data collected by the image sensing device includes image data, the first road information determined based on the image data includes at least one of vehicle flow information, guardrail information, or lane line information, and the second road information matching the first road information determined based on the image data includes at least one of vehicle flow information, guardrail information, or lane line information;

[0279] When determining the first current external parameter between the sensor device and the digitized map based on the relative relationship between the first road information and the second road information, the synchronization module 1010 can be used to:

[0280] For the point cloud sensing device, determine a first relative relationship corresponding to the point cloud sensing device, and determine a first current external parameter between the point cloud sensing device and the digitized map based on each relative sub-relationship in the first relative relationship, wherein the first relative relationship includes at least one of a first relative sub-relationship or a second relative sub-relationship, the first relative sub-relationship is a relationship between vehicle flow information determined based on the point cloud data and vehicle flow information in the digitized map, and the second relative sub-relationship is a relationship between guardrail information determined based on the point cloud data and guardrail information in the digitized map;

[0281] For the image sensing device, a second relative relationship corresponding to the image sensing device is determined, and based on each relative sub-relationship in the second relative relationship, a first current external parameter between the image sensing device and the digitized map is determined, wherein the second relative relationship includes at least one of a third relative sub-relationship, a fourth relative sub-relationship, or a fifth relative sub-relationship, the third relative sub-relationship is a relationship between traffic information determined based on image data and traffic information in the digitized map, the fourth relative sub-relationship is a relationship between guardrail information determined based on image data and guardrail information in the digitized map, and the fifth relative sub-relationship is a relationship between lane line information determined based on image data and lane line information in the digitized map.

[0282] Optionally, when determining the first candidate extrinsic parameter corresponding to the sensor device based on the first extrinsic parameter deviation, the synchronization module 1010 may be configured to perform at least one of the following:

[0283] If the first external parameter deviation is greater than the first external parameter deviation threshold, determining the first current external parameter as the first candidate external parameter;

[0284] If the first external parameter deviation is less than or equal to the first external parameter deviation threshold, the first historical external parameter of the most recent calibration is determined as the first candidate external parameter corresponding to the sensor device;

[0285] Obtain a first weight corresponding to the first current external parameter, and obtain a second weight corresponding to the first historical external parameter of the most recent calibration of this calibration, and perform weighted calculation based on the first current external parameter, the first weight, the first historical external parameter, and the second weight to obtain a first candidate external parameter corresponding to the sensing device, wherein when the first external parameter deviation is greater than the first external parameter deviation threshold, the first weight is greater than the second weight; when the first external parameter deviation is less than or equal to the first external parameter deviation threshold, the first weight is less than the second weight.

[0286] Optionally, when determining the target extrinsic parameter between the multi-source sensor devices based on the first candidate extrinsic parameter corresponding to each sensor device, the synchronization module 1010 may be used to:

[0287] Determine at least K-1 pairs of sensor devices, wherein the sensor device in each sensor device pair is one of the multi-source sensor devices, and K is the total number of the multi-source sensor devices;

[0288] For each sensor device pair, determining a second candidate extrinsic parameter between the sensor devices in the sensor device pair based on the first candidate extrinsic parameter corresponding to each sensor device in the sensor device pair;

[0289] For each sensing device pair, a target extrinsic parameter between the sensing devices in the sensing device pair is determined based on a second candidate extrinsic parameter between the sensing devices in the sensing device pair.

[0290] Optionally, when the synchronization module 1010 determines the target extrinsic parameter between the sensor devices in the sensor device pair based on the second candidate extrinsic parameter between the sensor devices in the sensor device pair, it can be used to perform any of the following:

[0291] Determining a second candidate extrinsic parameter between the sensor devices in the sensor device pair as a target extrinsic parameter between the sensor devices in the sensor device pair;

[0292] The first road information corresponding to each sensor device in the sensor device pair is obtained, and based on the first road information corresponding to each sensor device in the sensor device pair, a third candidate extrinsic parameter between the sensor devices in the sensor device pair is determined, and the second candidate extrinsic parameter and the third candidate extrinsic parameter between the sensor devices in the sensor device pair are fused to obtain a target extrinsic parameter between the sensor devices in the sensor device pair, wherein the first road information corresponding to the sensor device is determined based on the sensor data collected by the sensor device.

[0293] Optionally, the multi-source sensing device includes a roadside sensing device and a vehicle-mounted sensing device within a preset range of the roadside sensing device. When performing the spatial synchronization process, the synchronization module 1010 may be used to:

[0294] Acquire sensor data collected by roadside sensor devices;

[0295] Determine the traffic flow information within a preset range based on the sensing data collected by the roadside sensing device, and determine the traffic volume within the preset range based on the traffic flow information within the preset range;

[0296] A spatial synchronization frequency for performing spatial synchronization processing is determined based on a vehicle flow rate within a preset range, wherein the vehicle flow rate is positively correlated with the spatial synchronization frequency.

[0297] Optionally, the network parameter includes at least one of network delay, network jitter, or network available bandwidth. When the synchronization module 1010 determines the clock correction frequency corresponding to the sensor device to be corrected based on the predicted network parameter corresponding to the sensor device to be corrected, it can be used for any of the following:

[0298] If the predicted network parameter corresponding to the sensor device to be corrected meets the first correction condition, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the first clock correction frequency, and the first correction condition includes at least one of the predicted network delay being less than the network delay threshold, the predicted network jitter being less than the network jitter threshold, or the predicted network available bandwidth being greater than the network available bandwidth threshold;

[0299] If the predicted network parameters corresponding to the sensor device to be corrected meet the second correction condition, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the second clock correction frequency, the second correction condition includes at least one of the predicted network delay being greater than or equal to the network delay threshold, the predicted network jitter being greater than or equal to the network jitter threshold, or the predicted network available bandwidth being less than or equal to the network available bandwidth threshold, and the first clock correction frequency is higher than the second clock correction frequency;

[0300] If the predicted network parameters corresponding to the sensor device to be corrected do not meet the first correction condition and the second correction condition, the predicted network parameters are fused to obtain a fused network state prediction value, and when the fused network state prediction value is greater than the network state value threshold, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the first clock correction frequency, and when the fused network state prediction value is less than or equal to the network state value threshold, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the second clock correction frequency.

[0301] Optionally, for each sensor device to be calibrated, the synchronization module 1010 is further used to:

[0302] Acquire the time difference between the sensor device to be calibrated and the time synchronization source during at least one clock calibration;

[0303] If the time difference corresponding to each clock correction in at least one clock correction is greater than the time difference threshold, determining that the amplitude of subsequent clock correction of the sensing device to be corrected is the first correction amplitude;

[0304] If the time difference corresponding to one of the at least one clock corrections is less than or equal to the time difference threshold, it is determined that the amplitude of the subsequent clock correction of the sensor device to be calibrated is a second correction amplitude, which is smaller than the first correction amplitude.

[0305] Optionally, the synchronization module 1010 is further used for:

[0306] For each sensor device to be calibrated, obtaining the real network parameters in the past time period between the sensor device to be calibrated and the time synchronization source;

[0307] For each sensor device to be calibrated, prediction is performed based on the real network parameters corresponding to the sensor device to be calibrated by using the trained network prediction model to obtain the predicted network parameters corresponding to the sensor device to be calibrated;

[0308] Among them, the trained network parameter prediction model is trained in the following way:

[0309] The actual network parameters within a first preset time period are used as the input of the network prediction model to be trained, and the actual network parameters within a second preset time period are used as the output of the network prediction model to be trained, and the network prediction model to be trained is trained, wherein the first preset time period is the time period before the second preset time period.

[0310] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0311] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, and the processor executes the above computer program to implement the steps of the method of any embodiment of the present application.

[0312] In an alternative embodiment, an electronic device is provided, such as Fig.11 As shown, Fig.11The electronic device 1100 shown includes: a processor 1101 and a memory 1103. The processor 1101 and the memory 1103 are connected, such as through a bus 1102. Optionally, the electronic device 1100 may also include a transceiver 1104, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 1104 is not limited to one, and the structure of the electronic device 1100 does not constitute a limitation on the embodiments of the present application.

[0313] Processor 1101 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 1101 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0314] The bus 1102 may include a path to transmit information between the above components. The bus 1102 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 1102 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0315] The memory 1103 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0316] The memory 1103 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 1101. The processor 1101 is used to execute the computer program stored in the memory 1103 to implement the steps shown in the above method embodiment.

[0317] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0318] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0319] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0320] The above are only optional implementation methods for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A method for synchronizing a multi-source sensing device based on multi-source fusion, characterized in that: include: Performing synchronization processing on the multi-source sensing devices, wherein the synchronization processing includes at least one of spatial synchronization processing or temporal synchronization processing; The spatial synchronization process includes: Acquire a digitized map, and calibrate each sensor device in the multi-source sensor device based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map; Determine a target extrinsic parameter between the multi-source sensing devices based on a first candidate extrinsic parameter corresponding to each sensing device; The time synchronization process includes: Acquire predicted network parameters corresponding to each sensor device to be calibrated in at least part of the sensor devices to be calibrated in the multi-source sensor devices, wherein the predicted network parameters are network parameters in a future time period between the sensor device to be calibrated and a time synchronization source; For each sensor device to be calibrated, determining a clock calibration frequency corresponding to the sensor device to be calibrated based on the predicted network parameters corresponding to the sensor device to be calibrated; For each sensing device to be calibrated, the clock of the sensing device to be calibrated is calibrated based on the clock correction frequency corresponding to the sensing device to be calibrated, so that the time difference between the corrected clock of the sensing device to be calibrated and the clock of the time synchronization source is smaller than the time difference between the clock before correction and the clock of the time synchronization source.

2. The method according to claim 1, characterized in that The step of calibrating each sensor device in the multi-source sensor device based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map includes: Calibrate each sensor device in the multi-source sensor device based on the digitized map to obtain a first current external parameter between each sensor device and the digitized map; For each sensor device in the multi-source sensor device, if the sensor device is calibrated for the first time, the corresponding first current external parameter is determined as the first candidate external parameter; if the sensor device is calibrated for at least the second time, a first external parameter deviation between the first current external parameter and a first historical external parameter is determined, and a first candidate external parameter corresponding to the sensor device is determined based on the first external parameter deviation, where the first historical external parameter is an external parameter determined by the calibration before this calibration.

3. The method according to claim 2, characterized in that The step of calibrating each sensor device in the multi-source sensor device based on the digitized map to obtain a first current external parameter between each sensor device and the digitized map includes: Acquiring sensor data collected by each sensor device; For each sensing device, first road information within a preset range of the sensing device is determined based on the sensing data collected by the sensing device, and a matching map range in the digitized map that matches the preset range is determined, and second road information matching the first road information is determined from the matching map range, and a first current external parameter between the sensing device and the digitized map is determined based on a relative relationship between the first road information and the second road information.

4. The method according to claim 3, characterized in that The multi-source sensing device includes at least one of a point cloud sensing device or an image sensing device, the sensing data collected by the point cloud sensing device includes point cloud data, the first road information determined based on the point cloud data includes at least one of vehicle flow information or guardrail information, and the second road information matched with the first road information determined based on the point cloud data includes at least one of vehicle flow information or guardrail information, the sensing data collected by the image sensing device includes image data, the first road information determined based on the image data includes at least one of vehicle flow information, guardrail information or lane line information, and the second road information matched with the first road information determined based on the image data includes at least one of vehicle flow information, guardrail information or lane line information; The determining of a first current external parameter between the sensing device and the digitized map based on the relative relationship between the first road information and the second road information includes: For the point cloud sensing device, determine a first relative relationship corresponding to the point cloud sensing device, and determine a first current external parameter between the point cloud sensing device and the digitized map based on each relative sub-relationship in the first relative relationship, wherein the first relative relationship includes at least one of a first relative sub-relationship or a second relative sub-relationship, the first relative sub-relationship is a relationship between vehicle flow information determined based on the point cloud data and vehicle flow information in the digitized map, and the second relative sub-relationship is a relationship between guardrail information determined based on the point cloud data and guardrail information in the digitized map; For the image sensing device, a second relative relationship corresponding to the image sensing device is determined, and based on each relative sub-relationship in the second relative relationship, a first current external parameter between the image sensing device and the digitized map is determined, wherein the second relative relationship includes at least one of a third relative sub-relationship, a fourth relative sub-relationship, or a fifth relative sub-relationship, the third relative sub-relationship is a relationship between traffic flow information determined based on the image data and traffic flow information in the digitized map, the fourth relative sub-relationship is a relationship between guardrail information determined based on the image data and guardrail information in the digitized map, and the fifth relative sub-relationship is a relationship between lane line information determined based on the image data and lane line information in the digitized map.

5. The method according to claim 2, characterized in that: The determining of the first candidate extrinsic parameter corresponding to the sensor device based on the first extrinsic parameter deviation includes at least one of the following: If the first external parameter deviation is greater than a first external parameter deviation threshold, determining the first current external parameter as the first candidate external parameter; If the first external parameter deviation is less than or equal to a first external parameter deviation threshold, determining the first historical external parameter of the most recent calibration as the first candidate external parameter corresponding to the sensor device; Obtain a first weight corresponding to a first current external parameter, and obtain a second weight corresponding to a first historical external parameter of a most recent calibration of this calibration, and perform weighted calculation based on the first current external parameter, the first weight, the first historical external parameter, and the second weight to obtain a first candidate external parameter corresponding to the sensing device, wherein, when the first external parameter deviation is greater than a first external parameter deviation threshold, the first weight is greater than the second weight; when the first external parameter deviation is less than or equal to the first external parameter deviation threshold, the first weight is less than the second weight.

6. The method according to claim 1, characterized in that Determining a target extrinsic parameter between the multi-source sensing devices based on a first candidate extrinsic parameter corresponding to each sensing device includes: Determine at least K-1 pairs of sensor devices, wherein the sensor device in each sensor device pair is one of the multi-source sensor devices, and K is the total number of the multi-source sensor devices; For each sensor device pair, determining a second candidate extrinsic parameter between the sensor devices in the sensor device pair based on the first candidate extrinsic parameter corresponding to each sensor device in the sensor device pair; For each sensing device pair, a target extrinsic parameter between the sensing devices in the sensing device pair is determined based on a second candidate extrinsic parameter between the sensing devices in the sensing device pair.

7. The method according to claim 6, characterized in that The determining the target extrinsic parameter between the sensor devices in the sensor device pair based on the second candidate extrinsic parameter between the sensor devices in the sensor device pair includes any one of the following: Determining a second candidate extrinsic parameter between the sensor devices in the sensor device pair as a target extrinsic parameter between the sensor devices in the sensor device pair; The first road information corresponding to each sensor device in the sensor device pair is obtained, and based on the first road information corresponding to each sensor device in the sensor device pair, a third candidate extrinsic parameter between the sensor devices in the sensor device pair is determined, and the second candidate extrinsic parameter and the third candidate extrinsic parameter between the sensor devices in the sensor device pair are fused to obtain a target extrinsic parameter between the sensor devices in the sensor device pair, wherein the first road information corresponding to the sensor device is determined based on the sensor data collected by the sensor device.

8. The method according to any one of claims 1 to 7, characterized in that: The multi-source sensing device includes a roadside sensing device and a vehicle-mounted sensing device within a preset range of the roadside sensing device, and the spatial synchronization processing includes: Acquiring sensor data collected by the roadside sensor device; Determine the traffic flow information within the preset range based on the sensing data collected by the roadside sensing device, and determine the traffic volume within the preset range based on the traffic flow information within the preset range; A spatial synchronization frequency for performing spatial synchronization processing is determined based on the vehicle flow within the preset range, wherein the vehicle flow is positively correlated with the spatial synchronization frequency.

9. The method according to claim 1, characterized in that: The network parameter includes at least one of network delay, network jitter or network available bandwidth, and the step of determining the clock correction frequency corresponding to the sensor device to be corrected based on the predicted network parameter corresponding to the sensor device to be corrected includes at least one of the following: If the predicted network parameter corresponding to the sensor device to be corrected meets the first correction condition, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the first clock correction frequency, and the first correction condition includes at least one of the predicted network delay being less than the network delay threshold, the predicted network jitter being less than the network jitter threshold, or the predicted network available bandwidth being greater than the network available bandwidth threshold; If the predicted network parameter corresponding to the sensor device to be corrected meets the second correction condition, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the second clock correction frequency, the second correction condition includes at least one of the predicted network delay being greater than or equal to the network delay threshold, the predicted network jitter being greater than or equal to the network jitter threshold, or the predicted network available bandwidth being less than or equal to the network available bandwidth threshold, and the first clock correction frequency is higher than the second clock correction frequency; If the predicted network parameters corresponding to the sensor device to be corrected do not meet the first correction condition and the second correction condition, the predicted network parameters are fused to obtain a fused network status prediction value, and when the fused network status prediction value is greater than the network status value threshold, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the first clock correction frequency, and when the fused network status prediction value is less than or equal to the network status value threshold, the clock correction frequency corresponding to the sensor device to be corrected is determined to be the second clock correction frequency.

10. The method according to claim 1, characterized in that For each sensor device to be calibrated, the method further comprises: Acquire the time difference between the sensor device to be calibrated and the time synchronization source during at least one clock calibration; If the time difference corresponding to each clock correction in the at least one clock correction is greater than the time difference threshold, determining that the amplitude of the subsequent clock correction of the sensor device to be corrected is the first correction amplitude; If the time difference corresponding to one of the at least one clock corrections is less than or equal to the time difference threshold, it is determined that the amplitude of the subsequent clock correction of the sensing device to be calibrated is a second correction amplitude, and the second correction amplitude is smaller than the first correction amplitude.

11. The method according to claim 9 or 10, characterized in that: The predicted network parameters corresponding to each of the sensor devices to be calibrated in at least part of the sensor devices to be calibrated in the multi-source sensor devices are determined in the following manner: For each sensor device to be calibrated, obtaining the real network parameters between the sensor device to be calibrated and the time synchronization source in the past time period; For each sensor device to be calibrated, prediction is performed based on the real network parameters corresponding to the sensor device to be calibrated by using the trained network prediction model to obtain the predicted network parameters corresponding to the sensor device to be calibrated; The trained network parameter prediction model is obtained by training in the following way: The network prediction model to be trained is trained by using the actual network parameters within a first preset time period as the input of the network prediction model to be trained and using the actual network parameters within a second preset time period as the output of the network prediction model to be trained, wherein the first preset time period is the time period before the second preset time period.

12. A multi-source sensor device synchronization device based on multi-source fusion, characterized in that: include: A synchronization module, used for performing synchronization processing on the multi-source sensor devices, wherein the synchronization processing includes at least one of spatial synchronization processing or temporal synchronization processing; When performing spatial synchronization processing, the synchronization module is used to: Acquire a digitized map, and calibrate each sensor device in the multi-source sensor device based on the digitized map to obtain a first candidate external parameter between each sensor device and the digitized map; Determine a target extrinsic parameter between the multi-source sensing devices based on a first candidate extrinsic parameter corresponding to each sensing device; When performing time synchronization processing, the synchronization module is used to: Acquire predicted network parameters corresponding to each sensor device to be calibrated in at least part of the sensor devices to be calibrated in the multi-source sensor devices, wherein the predicted network parameters are network parameters in a future time period between the sensor device to be calibrated and a time synchronization source; For each sensor device to be calibrated, determining a clock calibration frequency corresponding to the sensor device to be calibrated based on the predicted network parameters corresponding to the sensor device to be calibrated; For each sensing device to be calibrated, the clock of the sensing device to be calibrated is calibrated based on the clock correction frequency corresponding to the sensing device to be calibrated, so that the time difference between the corrected clock of the sensing device to be calibrated and the clock of the time synchronization source is smaller than the time difference between the clock before correction and the clock of the time synchronization source.

13. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images