A method and device for enhancing vision based on formation systems

Abstract

This invention relates to the field of vehicle perception technology and discloses a method and apparatus for enhancing the field of vision based on a platooning system. The method is applied to a first vehicle in the platooning system, which includes a first vehicle and a second vehicle. The method includes: acquiring perceived object data fed back by the second vehicle; converting the perceived object data from the second vehicle's coordinate system to the first vehicle's coordinate system to obtain enhanced data for the first vehicle; and displaying the perceived object corresponding to the enhanced data for the first vehicle. Applying the technical solution of this invention can effectively expand the field of vision of vehicles in a platooning system and improve the driver's perception of the surrounding environment.

Description

Technical Field

[0001] This invention relates to the field of vehicle perception technology, specifically to a method and apparatus for enhancing the field of vision based on a platooning system. Background Technology

[0002] Currently, with the development of autonomous driving and intelligent connected vehicles, vehicle platooning has become an important technical means to improve traffic efficiency, reduce energy consumption, and enhance driving safety. In platooning systems, vehicles share information through V2X (Vehicle-to-Everything) communication technology to achieve collaborative perception and control.

[0003] In a platooning system, the limited sensing range of the vehicle's sensors creates a bottleneck in the vehicle's ability to perceive its surroundings. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a vision enhancement method based on a formation system to solve the problem of limited perception range of vehicles in the prior art.

[0005] According to one aspect of the present invention, a vision enhancement method based on a formation system is provided, the method being applied to a first vehicle in the formation system, the formation system including the first vehicle and a second vehicle, the method comprising:

[0006] Obtain the perceived object data fed back by the second vehicle;

[0007] The perceived object data is transformed from the second vehicle coordinate system to the first vehicle coordinate system to obtain the first vehicle augmentation data.

[0008] Displays the perceived object corresponding to the augmented data of the first vehicle.

[0009] According to another aspect of the present invention, a vision enhancement device based on a formation system is provided, comprising:

[0010] The target acquisition module is used to acquire the perceived object data fed back by the second vehicle;

[0011] The pose alignment module is used to convert the perceived object data from the second vehicle coordinate system to the first vehicle coordinate system to obtain the first vehicle augmentation data.

[0012] The field-of-view enhancement module is used to display the perceived objects corresponding to the enhanced data of the first vehicle.

[0013] According to another aspect of the present invention, a vision enhancement device based on a formation system is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0014] The memory is used to store at least one executable instruction that causes the processor to perform the operation of the vision enhancement method based on the formation system described above.

[0015] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction that causes a formation-based vision enhancement device / apparatus to perform the operation of the formation-based vision enhancement method described above.

[0016] This invention, through obtaining object perception data fed back by a second vehicle, transforms the object perception data from the second vehicle coordinate system to the first vehicle coordinate system to obtain first vehicle augmentation data. This ensures that the first vehicle can accurately understand the spatial location of the object perception data. After obtaining the first vehicle augmentation data, the perceived object corresponding to the first vehicle augmentation data is displayed. This allows the driver to clearly obtain information about various objects in the surrounding environment, including objects in blind spots, effectively expanding the field of vision of vehicles in the platooning system, improving the driver's perception of the surrounding environment, and thus enhancing driving safety.

[0017] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0018] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0019] Figure 1 A schematic diagram of a typical urban occlusion scenario is shown, illustrating a first embodiment of the vision enhancement method based on a formation system provided by the present invention.

[0020] Figure 2 A flowchart of a first embodiment of the vision enhancement method based on a formation system of the present invention is shown;

[0021] Figure 3 A schematic diagram of a formation system is shown, illustrating a first embodiment of the vision enhancement method based on a formation system provided by the present invention.

[0022] Figure 4A flowchart of a second embodiment of the vision enhancement method based on a formation system of the present invention is shown;

[0023] Figure 5 This diagram illustrates the coordinate relationships of a second embodiment of the vision enhancement method based on a formation system provided by the present invention.

[0024] Figure 6 This diagram illustrates a fine alignment schematic of a second embodiment of the vision enhancement method based on a formation system provided by the present invention.

[0025] Figure 7 A schematic diagram of transmission delay is shown for a second embodiment of the vision enhancement method based on a formation system provided by the present invention.

[0026] Figure 8 This diagram illustrates a second embodiment of the vision enhancement method based on a formation system provided by the present invention, showing a data association and deduplication process.

[0027] Figure 9 A flowchart of a third embodiment of the vision enhancement method based on a formation system of the present invention is shown;

[0028] Figure 10 A schematic diagram of the time synchronization architecture of the third embodiment of the vision enhancement method based on the formation system provided by the present invention is shown.

[0029] Figure 11 A simplified flowchart of a third embodiment of the vision enhancement method based on a formation system provided by the present invention is shown.

[0030] Figure 12 A flowchart of a fourth embodiment of the vision enhancement method based on a formation system of the present invention is shown;

[0031] Figure 13 A schematic diagram of the structure of a first embodiment of the vision enhancement device based on a formation system provided by the present invention is shown.

[0032] Figure 14 A schematic diagram of an embodiment of the vision enhancement device based on a formation system provided by the present invention is shown. Detailed Implementation

[0033] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0034] In practical applications of platooning systems, the first vehicle's perception of its surroundings is limited due to issues such as limited sensing range and susceptibility to obstruction by its sensors (e.g., cameras, lidar, millimeter-wave radar). This is particularly problematic in complex traffic scenarios (such as curves, slopes, and urban road sections with numerous obstructions), where the first vehicle struggles to acquire timely and accurate complete information about vehicles ahead or to the side, thus limiting its field of vision and consequently restricting the information available to the user or driver. Therefore, effectively integrating information from other vehicles to enhance the first vehicle's field of vision when its view is obstructed is a key challenge in platooning systems.

[0035] like Figure 1 As shown, Figure 1 The diagram illustrates a typical urban occlusion scenario of the first embodiment of the vision enhancement method based on a platooning system provided by the present invention. In the typical urban occlusion scenario shown in the diagram, the driver of vehicle 1 can only obtain very limited information, such as traffic lights at intersections and road topology information, which will hinder the driver from making relevant driving decisions. Similarly, for autonomous driving, the sensors equipped on a single vehicle, such as cameras and lasers, will be blocked by larger vehicles, resulting in the inability to obtain effective traffic light and road topology information.

[0036] Figure 2 A flowchart illustrating a first embodiment of the vision enhancement method based on a platooning system according to the present invention is shown. This method is performed by a first vehicle in the platooning system, which includes a first vehicle and a second vehicle. Figure 2 As shown, the method includes the following steps:

[0037] Step 210: Obtain the perceived object data fed back by the second vehicle.

[0038] The formation system consists of a lead vehicle and multiple formation members. The first vehicle is the one whose field of vision needs to be enhanced, such as a member of the formation whose perception may be limited due to obstruction or other factors. The second vehicle is the one in the formation that can provide sensory data to enhance the field of vision of the first vehicle, such as the lead vehicle, whose sensors can acquire environmental information that the first vehicle may have difficulty perceiving due to its advantageous position.

[0039] In one implementation, the second vehicle may be the vehicle located in front of the first vehicle in the platooning system during the platooning process, i.e. the lead vehicle of the first vehicle, which can perceive the surrounding environment information through its own sensors.

[0040] In one implementation, in vehicle platooning applications, vehicles in a platoon can share sensing information with each other. That is, in this scenario, the sensing information such as traffic lights at intersections, road topology, and traffic participant information obtained by the lead vehicle can be simultaneously shared with its member vehicles.

[0041] like Figure 3 As shown, Figure 3 The diagram illustrates a formation system based on a first embodiment of the vision enhancement method for formation systems provided by this invention. In the system shown, the lead vehicle, as the core of the formation, acquires GNSS time through a Global Navigation Satellite System (GNSS) antenna, and calculates the precise Physical Layer (PHY) hardware time, which serves as the main time source for the entire formation.

[0042] Furthermore, after calculating and synchronizing to the hardware time, the intelligent driving terminal device obtains this time and uses it for the timing of sensor data and algorithm modules within the intelligent driving domain. In the diagram, the intelligent driving terminal device represents the software and hardware modules related to intelligent driving of platooned vehicles.

[0043] Furthermore, the sensor module represents the scene sensors equipped in the vehicle, including cameras, LiDAR, and inertial measurement units (IMUs), which acquire data such as images, point clouds, and motion data. The mapping and perception module mainly consists of modules related to intelligent driving algorithms. It acquires target-level data, feature-level data, and high-precision positioning data of the first vehicle. Target-level data includes traffic light positions, traffic light signal states, and the positions and motion states of traffic participants. Feature-level data includes extracted feature data from images or point clouds, used to describe the environment. The high-precision positioning data of the first vehicle includes high-precision pose obtained through GNSS or Real-Time Kinematic (RTK) positioning technology, used to describe the vehicle's position and attitude in the global coordinate system. The platoon management and control module is used for decision-making and execution related to path planning, speed control, and interaction between the lead vehicle and platoon members. Information sharing can be achieved through the switches / gateways of the lead vehicle and platoon members.

[0044] In one implementation, the perceived object data is obtained by the second vehicle through its sensor module and mapping and perception module. It is used to describe the relevant data of various targets in the surrounding environment, including the type of target, such as pedestrians, two-wheeled vehicles, other vehicles, cones, curbs, traffic lights, etc., as well as the position and motion state of the target in the vehicle coordinate system. The motion state includes speed, acceleration, heading angle, etc.

[0045] For example, the sensor module of the second vehicle collects images, point clouds, and motion data of the surrounding environment in real time. This raw data is transmitted to the mapping and perception module. The mapping and perception module processes the raw data, such as identifying the location and signal status of traffic lights, determining the location and motion parameters of traffic participants (e.g., two-wheeled vehicles, other vehicles), and extracting image or point cloud features of curbs or buildings, ultimately forming perceived object data. Subsequently, the second vehicle uses V2X communication technology to feed this perceived object data back to the first vehicle at a preset transmission frequency (e.g., 10Hz), ensuring that the first vehicle can obtain the latest environmental information in a timely manner.

[0046] Step 220: Transform the perceived object data from the second vehicle coordinate system to the first vehicle coordinate system to obtain the first vehicle augmentation data.

[0047] The second vehicle coordinate system refers to a three-dimensional coordinate system established with the second vehicle as its origin. It can be established with the center of the rear axle of the second vehicle as the origin, defining the forward direction of the second vehicle as the positive X-axis, the leftward direction perpendicular to the forward direction as the positive Y-axis, and the upward direction perpendicular to the ground as the positive Z-axis. The first vehicle coordinate system refers to a three-dimensional coordinate system established with the first vehicle as its origin. It can be established with the center of the rear axle of the first vehicle as the origin, and its coordinate axis directions are defined in the same way as the second vehicle coordinate system to facilitate coordinate transformation calculations.

[0048] In one implementation, the enhanced data of the first vehicle is information that has been adapted to the coordinate system of the first vehicle after the perceived object data has been transformed. This data not only contains the accurate position of the perceived object in the coordinate system of the first vehicle, but also retains key information such as the object's motion state, type, and feature-level data. This data can make up for the deficiencies of the first vehicle's own perception and expand the perception range of the first vehicle.

[0049] For example, during coordinate transformation, the position information of the perceived object in the second vehicle coordinate system is transformed to the first vehicle coordinate system based on the relative pose information of the two vehicles, ensuring that the transformed position information accurately reflects the actual position of the object relative to the first vehicle.

[0050] Step 230: Display the perceived object corresponding to the augmented data of the first vehicle.

[0051] The perceived object corresponding to the first vehicle augmentation data refers to the actual object described by the first vehicle augmentation data. These objects may be objects that the first vehicle's own sensors cannot perceive (such as objects in blind spots caused by occlusion), or objects that are the same as the targets perceived by the first vehicle itself.

[0052] In one implementation, after obtaining the first vehicle augmentation data, the perceived objects corresponding to the first vehicle augmentation data are presented to the user, such as the driver, in a visual manner. Specifically, this can be achieved through in-vehicle display devices (such as instrument panel displays, central control displays, head-up displays, etc.).

[0053] By acquiring the perceived object data fed back by the second vehicle, the perceived object data is transformed from the second vehicle coordinate system to the first vehicle coordinate system to obtain the first vehicle augmentation data. This ensures that the first vehicle can accurately understand the spatial position of the perceived object. After obtaining the first vehicle augmentation data, the perceived object corresponding to the first vehicle augmentation data is displayed. This allows the driver to clearly obtain information about various objects in the surrounding environment, including objects in blind spots, effectively expanding the field of vision of vehicles in the formation system, improving the driver's perception of the surrounding environment, and thus enhancing driving safety.

[0054] Figure 4 A flowchart of another embodiment of the vision enhancement method based on a formation system of the present invention is shown, the method being performed by the first vehicle of the formation system. Figure 4 As shown, Figure 4 A flowchart illustrating the step of converting perceived object data from a second vehicle coordinate system to a first vehicle coordinate system to obtain first vehicle augmentation data, provided in an embodiment of the present invention, includes the following steps:

[0055] Step 410: Match the environmental features at the same location in the perception data of the second vehicle and the first vehicle to obtain the relative pose.

[0056] Among them, perception data refers to the data collected by the vehicle through sensors and processed to describe the surrounding environment, which includes image data, point cloud data and environmental features extracted from them.

[0057] In one implementation, the same location refers to a place where the perception ranges of the second vehicle and the first vehicle overlap or are associated in an actual traffic environment, such as the curb of the same road, the wall of the same roadside building, or the installation location of the same traffic sign.

[0058] In one implementation, environmental features refer to information extracted from the perceived data that can represent the unique attributes of the location. For image perceived data, environmental features can be corners, edges, textures, etc. in the image; for point cloud perceived data, environmental features can be geometric structures in the point cloud, such as the broken line shape of a curb, the right-angle structure of a building corner, the outline point cloud of a traffic sign, etc. Environmental features at the same location are unique and identifiable and can be used to identify a specific location.

[0059] In one implementation, relative pose refers to the positional and attitude relationships between the second vehicle and the first vehicle obtained through environmental feature matching. The positional relationship includes the distance between the two vehicles in space, relative coordinates, etc., and the attitude relationship includes the differences in heading angle, pitch angle, etc. between the two vehicles.

[0060] In one implementation, the second vehicle continuously collects perception data of its surrounding environment through its sensors, including the features of the roadside curb, the outlines of roadside buildings, and information on other vehicles, pedestrians, and other objects on the road. Simultaneously, the first vehicle also collects perception data of its own surrounding environment through its sensors, such as traffic sign features to its side and information on nearby traffic cones. Through inter-vehicle communication, the second vehicle sends its collected perception data to the first vehicle. The first vehicle receives this data and stores it along with its own collected perception data, preparing for subsequent environmental feature matching and object information processing.

[0061] For example, in a platooning scenario on urban roads, the second vehicle captures traffic light images at the intersection ahead using a camera and scans for pedestrian locations at the intersection using a LiDAR, thus obtaining the second vehicle's perception data. The first vehicle scans for curb point cloud data in its right lane using a LiDAR and captures images of traffic cones not far ahead using a camera, thus obtaining the first vehicle's perception data. The first vehicle will acquire the traffic light and pedestrian-related perception data sent by the second vehicle, as well as the curb and traffic cone-related perception data collected by itself.

[0062] In one implementation, environmental feature matching can be used to compare and correlate environmental features at the same location in the perception data of the second vehicle and the first vehicle to determine whether they belong to the same environmental feature, and to calculate the position and pose differences between them. In specific implementations, the feature matching method needs to be matched with the feature extraction method. If features are extracted based on 3D point cloud data, a corresponding matching method combined with a random sampling consistency method can be used to obtain the relative relationship.

[0063] For example, during the movement of the second vehicle, point cloud data of a fixed building can be collected using LiDAR, and the geometric features of the building can be extracted as environmental features. When the first vehicle moves near the fixed building, point cloud data of the building is also collected using LiDAR, and its geometric features are extracted. The geometric features of the fixed building extracted by the second vehicle and the first vehicle are matched, and the positional and orientation differences between the two vehicles when collecting the building features are calculated using a matching algorithm, thereby obtaining the relative pose between the second vehicle and the first vehicle.

[0064] Step 420: Based on the relative pose, transform the perceived object data from the second vehicle coordinate system to the first vehicle coordinate system to obtain the first vehicle transformed data.

[0065] The first vehicle transformation data refers to the relevant data describing the target object perceived by the second vehicle in the first vehicle coordinate system after coordinate transformation, including position data, attitude data, and motion state data.

[0066] In one implementation, after obtaining the position coordinates of a target object in the second vehicle coordinate system, the position offset and angular offset of the second vehicle relative to the first vehicle contained in the relative pose can be converted into the pose of the target object in the second vehicle coordinate system using a coordinate transformation formula, while retaining the motion state data. After such a conversion process, the target data of the target object in the first vehicle coordinate system, i.e., the first vehicle conversion data, is obtained.

[0067] Step 430: Perform data association based on the first vehicle perception target data and the first vehicle conversion data to obtain the first vehicle augmentation data.

[0068] Among them, the target perception data of the first vehicle refers to the target information obtained by the first vehicle through its own sensors after collection and processing. This information describes the type, position, attitude, motion state and other data of the target directly perceived by the first vehicle in the first vehicle coordinate system.

[0069] For example, by comparing and analyzing each target object in the first vehicle's perceived target data with each target object in the first vehicle's converted data, it can be determined whether they belong to the same actual target object. The two sets of information belonging to the same actual target object are deduplicated. A data association algorithm can be used to associate the first vehicle's perceived target data and the first vehicle's converted data to obtain first vehicle augmented data that includes both the target object information directly perceived by the first vehicle and the target object information perceived and processed by the second vehicle.

[0070] By matching environmental features at the same location with the perception data of the second and first vehicles to obtain relative poses, and then transforming the target information of the second vehicle to the coordinate system of the first vehicle based on the relative poses, the relative relationship between the second and first vehicles in space can be accurately determined, ensuring the accuracy of coordinate transformation. By integrating the target data perceived by the first vehicle and the transformed data through data association, duplicate information can be removed and target information not directly perceived by the first vehicle can be supplemented, resulting in target data for enhanced field of view. This method solves the problem of increased relative pose estimation errors leading to large positional deviations in the enhanced field of view results, ensuring the accuracy of expanding the perception field of view of the first vehicle based on the formation system.

[0071] Based on the foregoing embodiments, the method provided in this embodiment of the invention for matching environmental features at the same location in the perception data of the second vehicle and the first vehicle to obtain a relative pose can be achieved through steps A1 to A4:

[0072] Step A1: Obtain the second vehicle location data and the first vehicle location data.

[0073] The first vehicle positioning data includes the three-dimensional position coordinates and three-dimensional attitude angles of the first vehicle in the global coordinate system. The second vehicle positioning data includes the three-dimensional position coordinates and three-dimensional attitude angles of the second vehicle in the global coordinate system.

[0074] In one implementation, this positioning data can be acquired via GNSS / RTK and includes the vehicle's three-dimensional position coordinates and three-dimensional attitude angles in a global coordinate system. The first vehicle can directly collect its own positioning data and acquire the positioning data of the second vehicle collected by the second vehicle through the vehicle-to-vehicle communication of the formation system.

[0075] Step A2: Determine the initial relative pose based on the second vehicle positioning data and the first vehicle positioning data.

[0076] The initial relative pose is the preliminary spatial relationship between the second vehicle and the first vehicle, which includes position offset and attitude offset. Position offset refers to the difference in three-dimensional coordinates of the second vehicle in the coordinate system of the first vehicle, and attitude offset refers to the difference in three-dimensional attitude angles of the second vehicle relative to the first vehicle.

[0077] In one implementation, based on the high-precision positioning information contained in the positioning data of the second vehicle and the positioning data of the first vehicle, the initial positional and attitude relationships of the second vehicle relative to the first vehicle can be derived, thus obtaining the initial relative pose. The accuracy of the initial relative pose depends on the accuracy of the poses of the second vehicle and the following vehicle in the global coordinate system. Under normal circumstances, high-precision positioning can meet the requirements for enhanced field of view. However, high-precision positioning calculations rely on satellite signals, which are easily blocked, leading to a decrease in the accuracy of high-precision positioning.

[0078] like Figure 5 As shown, Figure 5 This diagram illustrates the coordinate relationships of a second embodiment of the vision enhancement method based on a formation system provided by the present invention. The scenario shown is illustrated using an XY plane coordinate system; however, in practical applications, a three-dimensional coordinate system is more commonly used, where XYO... Word It is a global coordinate system with its origin at O. 全局 (O) WordThe global coordinate system can be the EastNorth Up (ENU) navigation coordinate system. The ENU navigation coordinate system uses a fixed reference point as its origin, with the positive X-axis pointing east, the positive Y-axis pointing north, and the positive Z-axis pointing vertically upwards. This provides a unified position and attitude reference for all vehicles. Front The coordinate system for the current position of the preceding vehicle (i.e., the second vehicle) is O. 前1 (O) Front The X-axis points in the direction the vehicle in front is moving, and the Y-axis points to the left of the vehicle in front. XYO Back For the following vehicle, i.e., the current position of the first vehicle, the coordinate system has its origin at O. 后 (O) Back The X-axis points in the direction the following vehicle is moving, and the Y-axis points to the left side of the following vehicle. Cone 1 (P) Cone-F The pose of the cone is the pose of the cone in the vehicle coordinate system as identified by the second vehicle.

[0079] Step A3: Match the environmental features at the same location in the perception data of the second vehicle and the first vehicle to obtain feature matching information.

[0080] Among them, the environmental features at the same location refer to the features with the same attributes collected by the second vehicle and the first vehicle when they pass through the same location in the same static environmental area. The static environmental area includes roadside curbs, fixed buildings, trees, traffic sign poles, etc. The shape, position, texture and other attributes of these environmental features are relatively stable and can be used as the basis for matching.

[0081] In one implementation, after the first vehicle receives feature descriptions of the same location from both vehicles, it can transform the features from the second vehicle's coordinate system to the rear vehicle's coordinate system using an initial relative pose. Theoretically, the feature descriptions of the same location from both vehicles should overlap. However, due to errors in high-precision positioning, the two features may not overlap. The relative pose relationship between the two features can be obtained through feature matching, thereby calibrating the relative pose.

[0082] In one implementation, feature matching information refers to various types of data generated during the matching process, including successfully matched feature pairs, positional deviations, angular deviations, similarity values, etc. This information can reflect the correspondence and differences between the environmental features of the two vehicles and is a key basis for subsequent calibration of the initial relative pose.

[0083] In one implementation, after extracting environmental features at the same location from the perception data of the two vehicles, a feature matching algorithm can be used to compare the environmental features at the same location in the perception data of the second vehicle and the first vehicle one by one, calculate the distance and angle difference between each feature point pair, select feature point pairs with high similarity as successfully matched feature pairs, and record the positional deviation and angle deviation between these feature pairs to obtain feature matching information.

[0084] For example, environmental features can be extracted from image or point cloud data. Feature extraction methods can employ traditional geometric feature extraction, such as PFH / FPFH, or methods based on deep learning models, such as SupperPoint / ORB. PFH (Point Feature Histogram) is a local feature descriptor for 3D point clouds, while FPFH (Fast Point Feature Histogram) is an optimized version of PFH with higher computational efficiency. ORB (Oriented Fast and Rotated BRIEF) is a fast and rotation-invariant image feature detection and description algorithm.

[0085] Furthermore, the feature matching method needs to be compatible with the feature extraction method. For example, for 3D point cloud data, SuperPoint can be used to extract features, and SuperPoint can be used for feature matching, while the Random Sample Consensus (RANSAC) method can be used to obtain the relative pose. In addition, cross-modal methods can also be used to utilize image and point cloud data simultaneously; this embodiment does not impose specific limitations on this.

[0086] like Figure 6 As shown, Figure 6 This diagram illustrates a fine alignment schematic of a second embodiment of the field-of-view enhancement method based on a formation system provided by the present invention. In the scene shown, feature 1 (FEA) Back This represents the static environmental features acquired by the following vehicle at its current location. Static environmental features include curbs, buildings, or trees, etc. Feature 2 ( This indicates a feature extracted from the same location by the second vehicle. This feature can be obtained from the second vehicle's current location or when the second vehicle passes the location of the following vehicle. This feature is stored in the map created by the second vehicle and transmitted to the following vehicle via inter-vehicle communication. XYO Front The coordinate system for the current position of the preceding vehicle (i.e., the second vehicle) is O. 前1 (O) Front ); XYOBack For the following vehicle, i.e., the current position of the first vehicle, the coordinate system has its origin at O. 后 (O) Back ); This is the vehicle coordinate system of the preceding vehicle (i.e., the second vehicle) at the current position of the following vehicle, with its origin at O. 前2 ( ).

[0087] Step A4: Based on feature matching information, calibrate the initial relative pose to obtain the relative pose.

[0088] The initial relative pose, which depends on positioning data, may be biased due to factors such as satellite signal obstruction and positioning equipment errors. Feature matching information directly reflects the differences in how the two vehicles perceive the same environmental features, and can provide a precise basis for calibration.

[0089] In one implementation, the positional and angular deviations between successfully matched feature pairs can be determined based on feature matching information. Statistical values ​​of these deviations, such as average deviation and maximum deviation, are calculated to determine the error direction and magnitude of the initial relative pose, thereby obtaining the relative pose of the feature match. The positional and attitude offsets in the initial relative pose are adjusted based on the relative pose of the feature match to reduce the deviation between features, thus obtaining the final relative pose.

[0090] By determining the initial relative pose based on the positioning data of the two vehicles, the preliminary relative relationship between the first and second vehicles in the formation system can be established. After obtaining the initial relative pose, matching environmental features at the same location yields feature matching information, which accurately reflects the differences in the perception features of the two vehicles, providing a reliable basis for calibration. Calibrating the initial relative pose based on the feature matching information eliminates errors in the initial relative pose, resulting in a high-precision relative pose. In this way, the final relative pose accurately reflects the spatial relationship between the second and first vehicles, ensuring the accuracy of coordinate transformation.

[0091] Based on the foregoing embodiments, the vision enhancement method based on a formation system provided in this invention, which determines the initial relative pose based on the second vehicle positioning data and the first vehicle positioning data, can be achieved through steps A21 to A23:

[0092] Step A21: Obtain the pose of the second vehicle in the global coordinate system based on the second vehicle's localization data to obtain the global pose of the second vehicle.

[0093] The global coordinate system refers to the unified coordinate system used in the formation system to describe the spatial position of the entire traffic environment, which can be the ENU navigation coordinate system. The pose of the second vehicle refers to the position and attitude of the second vehicle in space. The position is represented by three-dimensional coordinates in the global coordinate system, namely x, y, and z, and the attitude is represented by rotation angles around three coordinate axes, namely yaw, pitch, and roll.

[0094] In one implementation, the second vehicle positioning data includes satellite observation data and IMU motion data. Based on this positioning data, the satellite observation data is processed by a positioning algorithm to obtain the three-dimensional position coordinates of the second vehicle in a global coordinate system. Simultaneously, by using an attitude calculation algorithm and combining the acceleration and angular velocity data from the IMU, the heading angle, pitch angle, and roll angle of the second vehicle can be calculated. Integrating the three-dimensional position coordinates and attitude angles of the second vehicle yields its global pose.

[0095] Step A22: Obtain the pose of the first vehicle in the global coordinate system based on the first vehicle's positioning data to obtain the global pose of the first vehicle.

[0096] The pose of the first vehicle refers to its position and attitude in space. The position is represented by three-dimensional coordinates in the global coordinate system, and the attitude is represented by the heading angle, pitch angle, and roll angle.

[0097] In one implementation, the first vehicle positioning data includes satellite observation data and IMU motion data. Based on this first vehicle positioning data, the satellite observation data is processed by a positioning calculation algorithm to obtain the three-dimensional position coordinates of the first vehicle in a global coordinate system. Simultaneously, by using an attitude calculation algorithm and combining the acceleration and angular velocity data from the IMU, the heading angle, pitch angle, and roll angle of the first vehicle can be calculated. Integrating the three-dimensional position coordinates and attitude angles of the first vehicle yields its global pose.

[0098] Step A23: Determine the initial relative pose based on the global pose of the second vehicle and the global pose of the first vehicle.

[0099] In one implementation, by utilizing the position and attitude data contained in the global poses of the second vehicle and the first vehicle in the same global coordinate system, and through coordinate transformation, the initial position and attitude relationship between the second vehicle and the first vehicle can be obtained, thus obtaining the initial relative pose.

[0100] For example, suppose P Front P represents the global pose of the second vehicle. Back P represents the global pose of the first vehicle. Cone-FGiven the pose of the target identified by the second vehicle in its vehicle coordinate system, the formula for transforming the target to the first vehicle coordinate system using the initial relative pose is as follows:

[0101]

[0102] In the formula, Indicates the initial relative pose. The pose is obtained by transforming the target into the first vehicle coordinate system through the initial relative pose.

[0103] Furthermore, after obtaining the feature matching information, the first vehicle can determine the relative pose of the feature match based on the feature matching information. Then, based on the relative pose obtained from feature matching, the initial relative pose is calibrated to obtain the relative pose. The formula for transforming the target to the first vehicle coordinate system using the relative pose is as follows:

[0104]

[0105] In the formula, This indicates the calibrated relative pose. The pose is obtained by transforming the target into the first vehicle coordinate system through the calibrated relative pose.

[0106] By extracting the poses of the first and second vehicles in the global coordinate system from the positioning data, the global poses of the second vehicle and the first vehicle can be obtained, which can provide basic data for the calculation of the initial relative pose.

[0107] Based on the foregoing embodiments, the method provided in this embodiment of the invention for obtaining first vehicle augmentation data by associating first vehicle perception target data and first vehicle conversion data can be implemented through steps B1 to B3:

[0108] Step B1: Determine the moving target from the first vehicle conversion data, perform motion compensation on the moving target based on the formation transmission delay, and obtain motion compensation data;

[0109] Among them, a moving target object refers to a target object that is in motion in the converted target data. The determination is based on the target object's motion state data. If the target object's velocity is not zero or there is a significant trend of positional change, then the target object is a moving target object. Formation transmission delay refers to the time interval from the acquisition, processing, and transmission of the second vehicle's sensing data to the reception and processing of the first vehicle.

[0110] In one implementation, motion-compensated data refers to target data that, after motion compensation, eliminates position and angle deviations caused by transmission delays, and can accurately reflect the current actual position and attitude of the moving target.

[0111] In one implementation, each target in the first vehicle conversion data can be judged by detecting the velocity parameter in its motion state information. If the velocity is greater than zero, it is determined to be a moving target. The formation transmission delay is obtained. Based on the motion parameters of the moving target, such as vehicle speed, yaw rate, acceleration, etc., and combined with the formation transmission delay, the position change and angle change of the moving target during the transmission delay period are calculated. The position information and attitude information of the moving target in the first vehicle conversion data are then added to the position change and angle change, respectively, to obtain motion compensation data.

[0112] Step B2: Update the first vehicle conversion data based on the motion compensation data to obtain the updated first vehicle conversion data;

[0113] Among them, the updated first vehicle conversion data is a set of updated object information, which includes both the corrected and accurately positioned moving target information and the stable positional information of stationary target information.

[0114] In one implementation, when updating the first vehicle conversion data based on motion compensation data, the moving target in the first vehicle conversion data corresponding to the motion compensation data can be found through the object's unique identifier or features such as object type and initial position. Then, the original position information of the moving target in the first vehicle conversion data is replaced with the corrected position information from the motion compensation data. For stationary targets in the first vehicle conversion data (such as cones and curbs), since their positions do not change over time, no position update is needed, and the original information is directly retained. After updating the positions of all moving targets, the updated moving targets and the unchanged stationary targets are integrated to form the updated first vehicle conversion data.

[0115] Step B3: Perform data association based on the first vehicle's perceived target data and the updated first vehicle's transformed data to obtain the first vehicle's enhanced data.

[0116] In one implementation, each target object in the first vehicle's perceived target data is compared and analyzed with each target object in the updated first vehicle's converted data to determine whether they belong to the same actual target object. The two sets of information belonging to the same actual target object are deduplicated. A data association algorithm can be used to associate the first vehicle's perceived target data and the first vehicle's converted data to obtain first vehicle augmented data that includes both the target object information directly perceived by the first vehicle and the target object information perceived and processed by the second vehicle.

[0117] By performing motion compensation on moving targets in the first vehicle conversion data, positional deviations caused by formation transmission delays can be eliminated, ensuring that the moving target information acquired by the first vehicle in the first vehicle conversion data is real-time and accurate, thereby improving the field of vision accuracy of the first vehicle augmentation data.

[0118] Based on the foregoing embodiments, the motion compensation of a moving target object based on formation transmission delay, as provided in this embodiment of the invention, to obtain motion compensation data, can be achieved through steps B11 to B13:

[0119] Step B11: Obtain the queuing transmission delay.

[0120] Due to factors such as communication link transmission speed and data processing time, platoon transmission delays objectively exist, which will cause the information of the moving target received by the first vehicle to correspond to the state of the target at a certain point in the past, rather than the current real-time state.

[0121] In one implementation, the platooning transmission delay can be obtained through timestamp recording. When the second vehicle generates information about a moving target, it adds a generation timestamp to the information, recording the specific time the information was generated. After receiving the information and completing preliminary processing, the first vehicle records the time when the reception and processing is completed. Subtracting the information generation time recorded by the second vehicle from the time when the reception and processing is completed recorded by the first vehicle gives the platooning transmission delay.

[0122] Step B12: Based on the pose of the moving target in the first vehicle coordinate system, obtain the x-coordinate, y-coordinate and heading angle to be compensated.

[0123] The pose of the moving target in the first vehicle coordinate system refers to the position and attitude information of the moving target in that coordinate system.

[0124] In one implementation, the x-coordinate to be compensated refers to the position coordinate of the moving target object in the X-axis direction under the first vehicle coordinate system, and the y-coordinate to be compensated refers to the position coordinate of the moving target object in the Y-axis direction under the first vehicle coordinate system. The x-coordinate to be compensated and the y-coordinate to be compensated together determine the position of the moving target object on the horizontal plane of the first vehicle coordinate system. The heading angle to be compensated refers to the current heading angle of the moving target object. The changes in the position and direction of the moving target object during the transmission delay are mainly reflected by the changes in these three parameters.

[0125] Step B13: Perform motion compensation based on the formation transmission delay for the x-coordinate, y-coordinate, and heading angle to be compensated, and obtain motion compensation data.

[0126] In one implementation, based on the motion state data of the moving target, the changes in the target's x-coordinate, y-coordinate, and heading angle during the transmission delay can be calculated. These changes are then added to their corresponding compensation parameters to obtain the target's current actual x-coordinate, y-coordinate, and heading angle, thus completing motion compensation. The motion state data of the moving target includes the speed, heading angle, and angular velocity received by the first vehicle from the second vehicle. After motion compensation, a complete data set containing the target's current actual x-coordinate, y-coordinate, heading angle, and other relevant states (such as speed and angular velocity) can be obtained.

[0127] like Figure 7 As shown, Figure 7 A schematic diagram illustrating the transmission delay of a second embodiment of the vision enhancement method based on a formation system provided by the present invention is shown. In the scenario depicted, a moving two-wheeled vehicle exists within the field of vision of the second vehicle, and this two-wheeled vehicle is in the blind spot of the vehicle behind it. It is assumed that the communication delay between the two vehicles is... Seconds. Due to communication delay, the second vehicle obtains the location of the two-wheeled vehicle as two-wheeled vehicle 1 ( The following vehicle obtains the position of the two-wheeled vehicle as shown in the image. (Two-wheeled vehicle 1) When ), the position of the two-wheeled vehicle has changed to two-wheeled vehicle 2 ( The location shown is shown in the image.

[0128] In one implementation, a Constant Turn Rate and Velocity (CTRV) model can be used for state updates to achieve motion compensation. The CTRV model is a target motion model that assumes the target moves at a constant speed and turn rate, used to describe the object's motion state (including position, velocity, heading angle, and angular velocity) in a two-dimensional plane. This embodiment does not impose specific limitations on the motion compensation model.

[0129] For example, assuming time k, the motion state of the moving object is defined by the following formula:

[0130]

[0131] In the formula, , This represents the position of the moving target on the horizontal plane at time k (unit: m). This represents the velocity of the moving target at time k (unit: m / s). The heading angle of the moving target at time k is expressed in rad. This represents the angular velocity of the moving target at time k (unit: rad / s).

[0132] Furthermore, for time k+m, the state is updated using the CTRV model, and the formula for calculating the motion state of the moving target is as follows:

[0133]

[0134]

[0135]

[0136]

[0137]

[0138] In the formula, , This represents the position of the moving target on the horizontal plane at time k+m (unit: m). This represents the velocity of the moving target at time k+m (unit: m / s). The heading angle of the moving target at time k+m is expressed in rad. This represents the angular velocity of the moving target at time k+m (unit: rad / s). Indicates the formation transmission delay (unit: seconds).

[0139] By acquiring the formation transmission delay, the time difference between the generation and reception of moving target information is determined. The abscissa, ordinate, and heading angle to be compensated are extracted from the target's pose, identifying the core parameters requiring compensation. Based on the formation transmission delay and the target's motion characteristics, motion compensation is applied to these parameters, allowing the calculation of the target's current actual state and obtaining the first vehicle's conversion data. This method solves the problem of inaccurate moving target information caused by formation transmission delay.

[0140] Based on the foregoing embodiments, the method provided in this embodiment of the invention for obtaining first vehicle augmentation data by associating first vehicle perception target data and updated first vehicle transformation data can be implemented through steps B31 to B32:

[0141] Step B31: Perform data association based on the first vehicle perceived target data and the updated first vehicle conversion data to obtain the duplicate target matching result.

[0142] The duplicate target matching result is a set of record matching relationships generated after data association, including targets that were successfully matched in the first vehicle conversion data and targets that were not successfully matched in the updated first vehicle conversion data. A successfully matched target pair is a target in the updated first vehicle conversion data that is marked as the same actual target as a target in the first vehicle's perceived target data. Targets that were not successfully matched in the target data are newly added targets that only exist in the updated first vehicle conversion data (targets perceived by the second vehicle but not observed by the first vehicle).

[0143] In one implementation, targets with similar locations can be associated with the same target ID through data association to obtain duplicate target matching results. Data association algorithms can include Nearest Neighbor (NN), Joint Probabilistic Data Association (JPDA), Multiple Hypothesis Tracking (MHT), etc., and this embodiment does not impose specific limitations on them.

[0144] Step B32: Based on the duplicate target matching results, remove duplicate target data from the updated first vehicle transformation data to obtain the first vehicle augmentation data.

[0145] Among them, duplicate target data refers to the target information in the updated first vehicle conversion data that has a matching relationship with the first vehicle perceived target data, that is, the targets that are successfully matched in the duplicate target matching results. The actual environmental targets corresponding to these targets have been directly perceived by the first vehicle. If they are retained, it will cause target duplication in subsequent display or decision-making, affecting the accuracy of information.

[0146] like Figure 8 As shown, Figure 8 This diagram illustrates a second embodiment of the vision enhancement method based on a formation system provided by the present invention, showing data association and deduplication. In the scenario shown, object A (Obj_A) is in the blind spot of the following vehicle, and its information is shared by the second vehicle with the following vehicle and obtained through motion compensation. Object B (Obj_B) is within the shared perception range of both the second and following vehicles. Assuming that at time k, the following vehicle receives information about object B (Obj_B) shared by the second vehicle from object 2, due to communication delay, its position may be different from that of object B1 (Obj_B). The position shown is determined by the relative pose of the front and rear vehicles. This information is then transformed into the rear vehicle coordinate system and motion compensated for, resulting in the position of object B2 in the figure. As shown in the figure. Object B3 ( ) is the location of the object detected by the rear vehicle, object B2 ( The position of the object is obtained from the vehicle in front.

[0147] In one implementation, based on the duplicate target matching results, successfully matched targets are filtered out from the updated first vehicle conversion data and deleted to remove duplicate target data in the updated first vehicle conversion data. Based on the deduplicated target data and the first vehicle perception target data, visual enhancement target data without redundant information can be obtained and displayed to the user.

[0148] Data association accurately identifies duplicate targets in the first vehicle's perceived target data and the updated first vehicle transformed data, preventing misjudgments due to overlapping perception ranges. Based on the matching results, duplicate targets in the updated data are removed, and then the first vehicle's perceived targets are integrated with the remaining newly added targets to form enhanced vision target data. This method eliminates information redundancy, expanding the environmental perception field of view while preventing duplicate targets from interfering with driving decisions, thus providing a more comprehensive road condition reference for driving decisions.

[0149] Figure 9 A flowchart of another embodiment of the vision enhancement method based on a formation system of the present invention is shown, the method being performed by the first vehicle of the formation system. Figure 9 As shown, Figure 9 The flowchart illustrating the steps preceding the acquisition of perception data from the second vehicle and the first vehicle, as provided in this embodiment of the invention, includes the following steps:

[0150] Step 910: Obtain the time synchronization message broadcast by the lead vehicle.

[0151] In traditional formation systems, while the times of multiple vehicles within a formation can be aligned using GNSS time, millisecond-level errors can occur in extreme scenarios due to factors such as GNSS signal obstruction or differences in GNSS processing between different terminal devices. This makes it difficult to achieve good spatiotemporal alignment when multiple vehicles are performing target-level data fusion.

[0152] In one implementation, the lead vehicle refers to the vehicle in the platoon system responsible for providing a time synchronization reference during platooning. It is typically located at the front of the platoon and is responsible for generating and sending unified time reference information to provide a reference for clock synchronization of all other vehicles in the platoon. A time synchronization message is a data frame generated by the lead vehicle according to a specific protocol format. This data frame contains the lead vehicle's precise system time information (such as year, month, day, hour, minute, second, millisecond, or even microsecond-level timestamps), lead vehicle identification information, and verification information. The lead vehicle identification information is used to distinguish other vehicles in the platoon, and the verification information is used to ensure that the message has not been tampered with or damaged during transmission.

[0153] In one implementation, the first and second vehicles can receive time synchronization messages broadcast by the lead vehicle via a switch / gateway. Since the lead vehicle broadcasts the message, all vehicles in the platoon can receive it simultaneously, ensuring the uniformity and timeliness of time synchronization.

[0154] Step 920: Synchronize the clocks of the first vehicle and the second vehicle based on the time synchronization message;

[0155] The clock refers to the timing device inside the first and second vehicles used to record time, including hardware clocks and software clocks. Its initial time may deviate from the clock of the lead vehicle. If it is not synchronized, the perception data and positioning data collected by the first vehicle will not match the data of other vehicles in the formation, especially the second vehicle, in the time dimension, which will affect the subsequent data fusion and vision enhancement effect.

[0156] In one implementation, the first vehicle and the second vehicle calculate the time difference between their own clocks and the clock of the lead vehicle based on the precise timestamp of the lead vehicle in the time synchronization message, and adjust the hardware clocks and software clocks of the first vehicle and the second vehicle to keep their time values ​​consistent with the clock of the lead vehicle, thereby achieving synchronization of the clocks of the first vehicle and the second vehicle.

[0157] Step 930: With the clocks of the first vehicle and the second vehicle synchronized, perform the operation of acquiring the perceived object data fed back by the second vehicle.

[0158] In this context, "synchronization of the clocks of the first and second vehicles" refers to the precise alignment of their clocks with the clock of the lead vehicle, ensuring that the vehicle position data and communication data of the first and second vehicles remain consistent in the time dimension. The platooning system stores key information about all vehicles in the entire platoon, including each vehicle's identifier (such as VIN code and platoon number), real-time position (i.e., coordinates in the global coordinate system), vehicle type, and relative position within the platoon. This data is updated in real time and shared within the platoon.

[0159] In one implementation, other member vehicles in the formation system also receive time synchronization messages broadcast by the lead vehicle, calculate the deviation between their own clock and the lead vehicle's clock, and then adjust their own hardware and software clocks to ensure that the clocks of all vehicles in the formation system are unified, providing a unified time reference for operations such as sharing perception data, coordinate transformation, and motion compensation between vehicles.

[0160] In one implementation, the road topology, traffic light locations, and road topology information acquired by the lead vehicle are all in its vehicle coordinate system. For member vehicles to use this information, they need to use the relative poses of the two vehicles to convert the target-level information and feature-level information they perceive into the member vehicle's coordinate system.

[0161] like Figure 10 As shown, Figure 10 This diagram illustrates the time synchronization architecture of a third embodiment of the vision enhancement method based on a platoon system provided by the present invention. The lead vehicle, acting as the master / grandmaster, receives GNSS time and calculates precise time information. Subsequently, the lead vehicle's switch / gateway broadcasts this precise time information as a time synchronization message to all other vehicles in the platoon system. Each vehicle in the platoon, from platoon vehicle 1 to platoon vehicle n, receives these synchronization messages through its internal switch / gateway and synchronizes its own time accordingly. The synchronization message format can use protocols such as PTP / 1588v2 or gPTP / 802.1AS. Other platoon vehicles, acting as slave / ordinary clock-clients, receive the synchronization messages and synchronize their internal hardware time and system time.

[0162] like Figure 11 As shown, Figure 11 A simplified flowchart of the third embodiment of the vision enhancement method based on a formation system provided by the present invention is shown. Assuming the first vehicle is the rear vehicle and the second vehicle is the front vehicle, the content transmitted by the front and rear vehicles is distinguished as sensing data and high-precision positioning data, transmitted at frequencies of 10Hz and 100Hz respectively. Since the positioning data is small in volume, the transmission delay can be ignored; only the impact of the sensing data transmission delay is considered. At time t0, the rear vehicle updates its relative pose through its own high-precision positioning data update (100Hz) and shares its high-precision positioning data with the front vehicle; simultaneously, the front vehicle senses and acquires information such as the target's motion state, position, velocity, and acceleration through image / point cloud feature data (frequency 10Hz) and shares its sensing data with the rear vehicle. At time t0+m, data sharing continues. The preceding vehicle continuously updates and shares its high-precision positioning and perception data. After receiving the data shared by the preceding vehicle, the following vehicle performs a series of processes: first, it refines the relative pose through relative pose calibration (feature matching); then, it converts the preceding vehicle's perception data to the following vehicle's coordinate system; subsequently, it performs motion compensation and associates the data with the first vehicle's perception data; it removes duplicates of the same targets and finally obtains the first vehicle's augmented data.

[0163] By acquiring the time synchronization message broadcast by the lead vehicle and synchronizing the clock of the first vehicle based on this message, time discrepancies between the first vehicle and the lead vehicle, as well as other vehicles in the platoon system, can be eliminated, preventing timing chaos in the perception data caused by inconsistencies. The second vehicle can be accurately identified from the platoon system, while non-lead vehicles also synchronize their clocks using the same time synchronization message, ensuring clock uniformity across all vehicles in the platoon. This method provides a unified time basis and a clear interaction object for subsequent acquisition of perception data from the first and second vehicles, environmental feature matching, coordinate transformation, and motion compensation for vision enhancement, ensuring the accuracy and reliability of the vision enhancement process.

[0164] Figure 12 A flowchart of another embodiment of the vision enhancement method based on a formation system of the present invention is shown, the method being performed by the first vehicle of the formation system. Figure 12 As shown, Figure 12 The flowchart illustrating the steps preceding the acquisition of perception data from the second vehicle and the first vehicle, as provided in this embodiment of the invention, includes the following steps:

[0165] Step 1210: If the field of view of the first vehicle is obstructed, perform the operation of obtaining the perceived object data fed back by the second vehicle.

[0166] The field of view of the first vehicle refers to the range of its surrounding environment that it can perceive through its own sensor modules (such as cameras, lidar, and millimeter-wave radar). This range is affected by sensor performance (such as sensing distance and detection angle) and external environmental factors (such as obstructions). Obstruction of the field of view means that the first vehicle's sensors are blocked by external objects (such as large vehicles in front, roadside buildings, construction barriers, etc.), resulting in the inability to obtain environmental information normally within part or all of the sensing range.

[0167] In one implementation, the perception and diagnostic module of the first vehicle receives image data and point cloud data transmitted by the sensor module in real time. For the image data, the module analyzes the grayscale distribution and edge features of the image. If a large, uniform area without edge features appears in multiple consecutive frames of the image in a certain direction, such as directly in front, and the area of ​​this area accounts for a preset proportion of the total image area, such as more than 60%, it is preliminarily determined that there may be occlusion in the field of view in that direction. At the same time, the module analyzes the point cloud data in the corresponding direction. If there are no valid point clouds in that direction in multiple consecutive frames of point cloud data, it is further confirmed that there is occlusion in the field of view in that direction.

[0168] Furthermore, once the field-of-view occlusion determination is established, the communication module of the first vehicle automatically sends a request for perceived object data to the second vehicle (such as the vehicle in front of the first vehicle, whose field of view is not obstructed). The request includes the position of the first vehicle and the direction of occlusion. After receiving the request, the second vehicle feeds back the perceived object data in the direction of occlusion of the first vehicle to the first vehicle via V2X communication. The first vehicle receives and stores this data for subsequent field-of-view enhancement processing.

[0169] By detecting whether the field of view of the first vehicle is obstructed, it can be ensured that the first vehicle can obtain comprehensive environmental information even if the field of view is obstructed in complex traffic scenarios.

[0170] Figure 13 A schematic diagram of an embodiment of the vision enhancement device based on a formation system of the present invention is shown. Figure 13 As shown, the device 1300 includes: a target acquisition module 1310, a pose alignment module 1320, and a field of view enhancement module 1330.

[0171] The target acquisition module 1310 is used to acquire the perceived object data fed back by the second vehicle;

[0172] The pose alignment module 1320 is used to convert the perceived object data from the second vehicle coordinate system to the first vehicle coordinate system to obtain the first vehicle augmentation data.

[0173] The field-of-view enhancement module 1330 is used to display the perceived object corresponding to the enhanced data of the first vehicle.

[0174] In an alternative approach, the pose alignment module 1320 is also used to match environmental features at the same location in the perception data of the second vehicle and the first vehicle to obtain a relative pose.

[0175] Based on the relative pose, the perceived object data is transformed from the second vehicle coordinate system to the first vehicle coordinate system to obtain the first vehicle transformed data.

[0176] Data association is performed based on the first vehicle's perceived target data and the first vehicle's conversion data to obtain the first vehicle's enhanced data.

[0177] In one alternative approach, the pose alignment module 1320 is also used to acquire second vehicle positioning data and first vehicle positioning data;

[0178] The initial relative pose is determined based on the second vehicle positioning data and the first vehicle positioning data;

[0179] The environmental features at the same location in the perception data of the second vehicle and the first vehicle are matched to obtain feature matching information;

[0180] The initial relative pose is calibrated based on feature matching information to obtain the relative pose.

[0181] In an alternative embodiment, the pose alignment module 1320 is further configured to obtain the pose of the second vehicle in the global coordinate system based on the second vehicle positioning data, thereby obtaining the global pose of the second vehicle.

[0182] The pose of the first vehicle in the global coordinate system is obtained based on the positioning data of the first vehicle, and the global pose of the first vehicle is obtained.

[0183] The initial relative pose is determined based on the global pose of the second vehicle and the global pose of the first vehicle.

[0184] In an alternative approach, the pose alignment module 1320 is further configured to determine a moving target from the first vehicle conversion data, perform motion compensation on the moving target based on the formation transmission delay, and obtain motion compensation data.

[0185] The first vehicle conversion data is updated based on the motion compensation data to obtain the updated first vehicle conversion data.

[0186] Data association is performed based on the first vehicle's perceived target data and the updated first vehicle's transformed data to obtain the first vehicle's enhanced data.

[0187] In an alternative approach, the pose alignment module 1320 is also used to obtain the formation transmission delay;

[0188] Based on the pose of the moving target in the first vehicle coordinate system, obtain the x-coordinate, y-coordinate and heading angle to be compensated;

[0189] Motion compensation data is obtained by performing motion compensation on the x-coordinate, y-coordinate, and heading angle to be compensated based on the formation transmission delay.

[0190] In an alternative approach, the pose alignment module 1320 is also used to perform data association based on the first vehicle perceived target data and the updated first vehicle transformation data to obtain a duplicate target matching result.

[0191] The duplicate target data in the updated first vehicle transformation data is obtained by removing duplicate target data based on the duplicate target matching results.

[0192] In an alternative embodiment, the target acquisition module 1310 is also used to acquire the time synchronization message broadcast by the lead vehicle;

[0193] The clocks of the first and second vehicles are synchronized based on time synchronization messages;

[0194] With the clocks of the first and second vehicles synchronized, the operation of acquiring the perceived object data fed back by the second vehicle is performed.

[0195] In an alternative embodiment, the target acquisition module 1310 is further configured to perform an operation of acquiring perceived object data fed back by the second vehicle when an obstruction is detected in the field of view of the first vehicle.

[0196] The vision enhancement device based on the formation system provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0197] Figure 14 The diagram shows a structural schematic of an embodiment of the vision enhancement device based on a formation system according to the present invention. The specific embodiments of the present invention do not limit the specific implementation of the vision enhancement device based on a formation system.

[0198] like Figure 14 As shown, the vision enhancement device based on the formation system may include: a processor 1402, a communications interface 1404, a memory 1406, and a communications bus 1408.

[0199] The processor 1402, communication interface 1404, and memory 1406 communicate with each other via communication bus 1408. Communication interface 1404 is used to communicate with other network elements such as clients or other servers. The processor 1402 executes program 1410, specifically performing the relevant steps described in the embodiment of the vision enhancement method for a formation-based system.

[0200] Specifically, program 1410 may include program code, which includes computer-executable instructions.

[0201] Processor 1402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The vision enhancement device based on the formation system includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0202] Memory 1406 is used to store program 1410. Memory 1406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0203] Specifically, program 1410 can be called by processor 1402 to cause the vision enhancement device based on the formation system to execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0204] This invention provides a computer-readable storage medium storing at least one executable instruction that, when executed on a formation-based vision enhancement device / apparatus, causes the formation-based vision enhancement device / apparatus to perform the formation-based vision enhancement method in any of the above method embodiments.

[0205] The executable instructions can be used to cause the vision enhancement device / apparatus based on the formation system to perform the method provided in the above method embodiment. The implementation principle and technical effect are similar, and will not be described in detail here.

[0206] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Furthermore, the embodiments of this invention are not directed to any particular programming language.

[0207] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. Similarly, for the sake of brevity and to aid in understanding one or more aspects of the invention, in the description of exemplary embodiments of the invention above, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0208] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

[0209] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A vision enhancement method based on a formation system, characterized in that, The method is applied to a first vehicle in a platooning system, the platooning system including the first vehicle and a second vehicle, the method comprising: Obtain the perceived object data fed back by the second vehicle; Obtain the location data of the second vehicle and the location data of the first vehicle; Based on the high-precision positioning information contained in the second vehicle positioning data and the first vehicle positioning data, the initial positional relationship and attitude relationship of the second vehicle relative to the first vehicle are derived to obtain the initial relative pose. The environmental features at the same location in the perception data of the second vehicle and the first vehicle are matched to obtain feature matching information. The environmental features at the same location are features with the same attributes collected by the second vehicle and the first vehicle when they pass through the same location and in the same static environment area. The initial relative pose is calibrated based on the feature matching information to obtain the relative pose; Based on the relative pose, the perceived object data is transformed from the second vehicle coordinate system to the first vehicle coordinate system to obtain the first vehicle augmentation data; Displays the perceived object corresponding to the augmented data of the first vehicle.

2. The method according to claim 1, characterized in that, The step of transforming the perceived object data from the second vehicle coordinate system to the first vehicle coordinate system based on the relative pose to obtain the first vehicle augmentation data includes: Based on the relative pose, the perceived object data is converted from the second vehicle coordinate system to the first vehicle coordinate system to obtain the first vehicle converted data. First vehicle augmentation data is obtained by associating the first vehicle perception target data and the first vehicle conversion data.

3. The method according to claim 1, characterized in that, The step of determining the initial relative pose based on the second vehicle positioning data and the first vehicle positioning data includes: Based on the second vehicle's positioning data, the pose of the second vehicle in the global coordinate system is obtained, thus obtaining the global pose of the second vehicle. Based on the first vehicle positioning data, the pose of the first vehicle in the global coordinate system is obtained to obtain the global pose of the first vehicle. The initial relative pose is determined based on the global pose of the second vehicle and the global pose of the first vehicle.

4. The method according to claim 2, characterized in that, The process of associating the first vehicle perception target data and the first vehicle conversion data to obtain the first vehicle augmentation data includes: The moving target is determined from the first vehicle conversion data, and motion compensation is performed on the moving target based on the formation transmission delay to obtain motion compensation data; The first vehicle conversion data is updated based on the motion compensation data to obtain the updated first vehicle conversion data. First vehicle augmentation data is obtained by associating the first vehicle perception target data and the updated first vehicle conversion data.

5. The method according to claim 4, characterized in that, The motion compensation of the moving target based on the formation transmission delay to obtain motion compensation data includes: Get the formation transmission delay; Based on the pose of the moving target in the first vehicle coordinate system, obtain the horizontal coordinate to be compensated, the vertical coordinate to be compensated, and the heading angle to be compensated. Motion compensation is performed on the x-coordinate, y-coordinate, and heading angle to be compensated based on the formation transmission delay to obtain motion compensation data.

6. The method according to claim 4, characterized in that, The process of associating the first vehicle perception target data and the updated first vehicle conversion data to obtain the first vehicle augmentation data includes: Data association is performed based on the first vehicle perceived target data and the updated first vehicle conversion data to obtain duplicate target matching results; Based on the duplicate target matching results, duplicate target data is removed from the updated first vehicle transformation data to obtain the first vehicle augmentation data.

7. The method according to any one of claims 1-6, characterized in that, Before acquiring the perceived object data fed back by the second vehicle, the method further includes: Obtain the time synchronization message broadcast by the lead vehicle; The clocks of the first vehicle and the second vehicle are synchronized based on the time synchronization message; When the clocks of the first vehicle and the second vehicle are synchronized, the operation of acquiring the perceived object data fed back by the second vehicle is performed.

8. The method according to any one of claims 1-6, characterized in that, Before acquiring the perceived object data fed back by the second vehicle, the method further includes: If the field of view of the first vehicle is obstructed, the operation of acquiring the perceived object data fed back by the second vehicle is performed.

9. A target field-of-view enhancement device, characterized in that, The device includes: The target acquisition module is used to acquire the perceived object data fed back by the second vehicle; The pose alignment module is used to acquire second vehicle positioning data and first vehicle positioning data; derive the initial position and attitude relationship of the second vehicle relative to the first vehicle based on the high-precision positioning information contained in the second vehicle positioning data and the first vehicle positioning data, and obtain the initial relative pose; match the environmental features at the same position in the perception data of the second vehicle and the first vehicle to obtain feature matching information, wherein the environmental features at the same position are features with the same attributes collected by the second vehicle and the first vehicle from the same static environmental area when passing through the same position; calibrate the initial relative pose based on the feature matching information to obtain the relative pose. The pose alignment module is also used to convert the perceived object data from the second vehicle coordinate system to the first vehicle coordinate system according to the relative pose to obtain the first vehicle augmentation data. The field-of-view enhancement module is used to display the perceived objects corresponding to the enhanced data of the first vehicle.

Images