A bus-mounted multimodal target detection AI intelligent agent system
By using a multimodal data access and target evidence management unit, combined with credibility assessment and conflict review mechanisms, the problem of insufficient consistency of multi-source data in bus target detection is solved, thereby improving the accuracy and consistency of detection results.
Patent Information
- Application Number
- CN202611577056.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-13
- Publication Date
- 2026-11-10
AI Technical Summary
In existing bus-mounted target detection systems, the multi-source detection data are inconsistent in position and category due to occlusion and vehicle movement, and lack processing for specific targets, resulting in insufficient consistency of detection results.
Data from cameras, lidar, and millimeter-wave radar is acquired through a multimodal data access unit. Combined with target evidence management, credibility assessment, and conflict review mechanisms, time and space correspondence processing is performed to verify and correct target detection results.
It realizes consistency verification and correction of target detection results under multimodal data, reduces redundant processing, and improves the accuracy and consistency of target detection and feature recognition.
Smart Images

Figure CN123068928A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and intelligent transportation technology, and in particular to a bus-mounted multimodal target detection AI intelligent agent system. Background Technology
[0002] In the fields of artificial intelligence and intelligent transportation, data collection, transmission, calculation, judgment, and storage are typically carried out around road environment perception, vehicle operation status analysis, and traffic target recognition. Vehicle-mounted cameras, lidar, millimeter-wave radar, positioning equipment, and vehicle interfaces can respectively acquire road images, point clouds, ranging information, location trajectories, and operating parameters. These data are then organized and associated by vehicle-mounted computing equipment according to time, spatial location, and sensor correspondence. The bus-mounted multimodal target detection AI agent refers to a system designed for bus driving scenarios that detects and identifies features of vehicles, pedestrians, and other road targets. It typically performs temporal correspondence and spatial matching on data obtained from different vehicle-mounted sensors. Based on the target contour, position, category, motion state, and adjacent time records, it completes target localization, category determination, feature extraction, and target association, and stores the processing results on the vehicle-mounted equipment or transmits them to a server.
[0003] In existing bus-mounted target detection, data collected by cameras, lidar, and millimeter-wave radar are correlated according to time and spatial location. Target association is completed based on target location, category, characteristics, and motion state. When affected by occlusion and vehicle movement, the location, category, or association results of the same target may differ under different data sources. Traditional processes rely on current observations and records from adjacent times for judgment, lacking specific processing basis for differences in data sources. Conflicting targets and consistent targets enter similar processing processes, and the data processing load of onboard computing equipment increases with the number of targets. Summary of the Invention
[0004] This invention aims to provide a multimodal target detection AI intelligent agent system for buses. By performing time-space correspondence processing on multimodal data collected by onboard cameras, lidar, and millimeter-wave radar, and combining target evidence recording, credibility state assessment, conflict back-checking, and task scheduling mechanisms, the system correlates, verifies, and corrects target detection results from different data sources. This solves the problem of insufficient consistency in target detection and feature recognition results caused by positional deviations, inconsistent categories, and uncertain target associations in multi-source detection data in the bus environment.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: a bus-mounted multimodal target detection AI intelligent agent system, comprising a multimodal data access unit, a target evidence management unit, a credibility assessment unit, a conflict review unit, and an AI intelligent agent task scheduling unit; The multimodal data access unit is used to acquire multimodal data collected by vehicle-mounted camera devices, lidar and millimeter-wave radar, and record the acquisition time and data source of each multimodal data, so that the detection data from different data sources have corresponding data time information and source information; The target evidence management unit is used to process multimodal data according to the collection time and spatial location, convert data acquired by different acquisition devices into corresponding relationships, perform target detection on the corresponding multimodal data, obtain the position, category, features and motion state of each target, and determine the identity association status by combining the position, features and motion state under adjacent acquisition time, forming a target evidence record corresponding to each target.
[0006] Furthermore, the target evidence management unit identifies multimodal data whose collection time falls within the same corresponding time interval as data to be matched, and converts the data to be matched to a unified spatial location based on the spatial calibration relationship between the vehicle-mounted camera device, lidar, and millimeter-wave radar. It then detects candidate targets from different data sources to obtain the position, category, characteristics, and motion state of the candidate targets. Based on the spatial position correspondence and category consistency relationship between the candidate targets, it establishes the same target correspondence relationship and writes the position, category, characteristics, motion state, and data source of the corresponding candidate target into the target evidence record.
[0007] Furthermore, the target evidence management unit determines the predicted position based on the position changes and movement status of the target in adjacent acquisition times, compares the target position in the next acquisition time with the predicted position, and performs a consistency comparison of the target features in adjacent acquisition times. When the position deviation meets the association distance condition and the target features meet the consistency condition, an association established state is generated. When either condition is not met, an association pending verification state is generated, and the association established state or the association pending verification state is written into the target evidence record as the identity association state.
[0008] Furthermore, when the identity association status in the target evidence record is in the association pending verification state, the target position, target features and motion state of the corresponding target under continuous collection time are extracted to form a historical observation sequence. The predicted position of the corresponding collection time is determined based on the motion state in the historical observation sequence. The positional relationship between the target position and the predicted position and the consistency relationship of the target features are reviewed. When the positional deviation within the continuous review time window continuously meets the association distance condition and the target features continuously meet the consistency condition, the association pending verification state is updated to the association established state. When either condition is not met, the association pending verification state is maintained and the data review of the subsequent collection time continues. The credibility assessment unit is used to determine the credibility status of each data source based on the location deviation, category consistency, and identity association status of the same target under different data sources, and writes the credibility status into the target evidence record.
[0009] Furthermore, the credibility assessment unit extracts the location, category, and identity association status of the same target at the same collection time from the target evidence record across different data sources. It compares the location of the target corresponding to any data source with the locations of the target corresponding to other data sources, forming a location consistency result based on the location consistency relationship and a category consistency result based on the category consistency relationship. When both the location consistency result and the category consistency result meet the consistency conditions and the identity association status is established, the corresponding data source is determined to be in a credible participation state. When either condition is not met, the corresponding data source is determined to be in a pending review state, and the credible participation state or pending review state is written as a credible state into the target evidence record.
[0010] Furthermore, for data sources in the pending review state, the location consistency result, category consistency result, and identity association status are reacquired at the next collection time. When both the location consistency result and the category consistency result meet the consistency condition and the identity association status is in the association established state, the corresponding data source is updated from the pending review state to the trusted participation state. When any condition is not met, the pending review state is maintained, and the review continues to be performed based on the data from the next collection time.
[0011] The conflict lookup unit is used to compare the location, category, and identity association status of the same target under different data sources. When the comparison results do not meet the consistency conditions, a conflict status is generated, and the lookup scope is determined according to the location and movement status in the target evidence record to form the corresponding lookup task. When the comparison results meet the consistency conditions, the corresponding target is marked as consistent.
[0012] Furthermore, the conflict re-investigation unit extracts the location, category, and identity association status of the same target from the target evidence record under different data sources, and compares the deviation between locations, the consistency between categories, and the consistency between identity association status. When the location deviation exceeds the location consistency range, the category is inconsistent, or the identity association status is inconsistent, a conflict status is generated. Based on the target location and movement status that generated the conflict status, the predicted location corresponding to the next acquisition time is determined, and the re-investigation range is determined by the acquisition area of the vehicle-mounted camera device, the acquisition area of the lidar, and the acquisition area of the millimeter-wave radar corresponding to the predicted location.
[0013] Furthermore, the conflict re-investigation unit determines the data source to be re-investigated from the data sources involved in the conflict state determination according to the trust status, extracts the multimodal data corresponding to the target within the re-investigation scope, and matches the extracted multimodal data according to the collection time. It then re-matches the target based on the location, category, and identity association status. When the re-matching result satisfies the consistency of location, category, and identity association status, the re-matching result is written into the target evidence record and the re-investigation task ends. If any consistency condition is not met, the conflict state is retained and the re-investigation task continues to be executed based on the multimodal data corresponding to the next collection time.
[0014] The AI intelligent agent task scheduling unit is used to sort the back-check tasks according to the conflict status and the load of the vehicle processor, control the conflict back-check unit to select the corresponding data source according to the trust status, and re-match the multimodal data within the back-check range to the target, and write the matching result into the target evidence record.
[0015] Furthermore, the AI intelligent agent task scheduling unit reads the back-check tasks corresponding to the conflict state and the load of the vehicle processor. It determines the conflict level based on the number of location conflicts, category conflicts, and identity association state conflicts in the back-check tasks. The conflict level is used as the first sorting criterion, and the back-check task generation time is used as the second sorting criterion to form a back-check task queue. When the load of the vehicle processor is lower than the preset load limit, the back-check tasks are sent to the conflict back-check unit in sequence according to the order in the back-check task queue. When the load of the vehicle processor reaches the preset load limit, the order of the unsent back-check tasks in the back-check task queue is maintained, and the back-check tasks continue to be sent after the load of the vehicle processor is lower than the preset load limit.
[0016] Furthermore, after the conflict lookup unit completes the first lookup task in the ranking, the AI intelligent agent task scheduling unit removes the completed lookup task from the lookup task queue and reacquires the vehicle processor load. When the lookup task queue is not empty and the vehicle processor load is lower than the preset load limit, the AI intelligent agent task scheduling unit sends the re-ranked first lookup task to the conflict lookup unit. When the lookup task queue is not empty and the vehicle processor load reaches the preset load limit, the lookup task queue is maintained and the sending is paused. The sending is resumed after the vehicle processor load is lower than the preset load limit. The current lookup scheduling ends when the lookup task queue is empty.
[0017] For targets corresponding to consistent states and targets that have completed the backtracking task, the target location, target category, and target features are determined based on the corresponding target evidence records, forming target detection and feature recognition results. Through the above technical solution, the corresponding management, reliability status assessment, conflict backtracking, and task scheduling processing of target detection results in bus multimodal data are realized, enabling target information from different data sources to be associated and corrected based on target evidence records.
[0018] The working principle and advantages of this invention are as follows: In this invention, by collecting multimodal data corresponding to time and location, target location, category, features, motion state, and identity association state are used to form a target evidence record. A credible state is determined based on location deviation, category consistency, and identity association state, so that the backtracking selects data sources according to the credible state. When the comparison results do not meet the consistency conditions, the backtracking range is determined based on the target location and motion state and a backtracking task is formed. When the comparison results meet the consistency conditions, a consistent state is formed. The backtracking task is sorted by combining the conflict state and the load of the vehicle processor. The multimodal data within the backtracking range is re-matched to the target and written into the target evidence record, reducing the repeated processing of targets in the consistent state. The target detection and feature recognition results are formed based on the target evidence record. Attached Figure Description
[0019] Figure 1 This is a flowchart of the multimodal target detection and feature recognition processing for buses according to the present invention; Figure 2 This is a data flow diagram formed from the objective evidence records of the present invention; Figure 3 This is a diagram illustrating the identity association status determination and verification status transition of this invention. Figure 4 This is the trusted state assessment and updated state transition diagram of the present invention; Figure 5 This invention provides a data flow diagram for determining the scope of conflict back-checking and re-matching. Figure 6 This is a collaborative scheduling diagram of task sorting and processor load in the present invention. Detailed Implementation
[0020] The following detailed explanation illustrates the specific implementation methods: The multimodal data in this invention includes data collected by vehicle-mounted cameras, LiDAR, and millimeter-wave radar. The acquisition time is used to establish a temporal correspondence between different data sources, and the data source is used to distinguish whether the corresponding data comes from a vehicle-mounted camera, LiDAR, or millimeter-wave radar.
[0021] Target evidence records are established according to the target and stored separately according to the data source, including target location, target category, target characteristics, movement status, identity association status, and credibility status. Identity association status includes established association and pending verification status; credibility status includes credible participation status and pending review status. When different data sources do not meet the consistency conditions for the location, category, or identity association status of the same target, a conflicting state is formed; when the consistency conditions are met, a consistent state is formed.
[0022] Data from vehicle-mounted cameras, lidar, and millimeter-wave radar within the same target evidence record are stored separately. When comparing location, category, and identity-related states, data from different sources is retrieved from the same target evidence record, without pre-merging data from different sources into a single dataset.
[0023] In this embodiment, the AI intelligent agent task scheduling unit receives the conflict status, the time of the backtracking task generation, and the load of the on-board processor. It saves the backtracking tasks to be executed in the backtracking task queue and performs backtracking task issuance, suspension issuance, resumption issuance, and task removal according to the conflict status and the load of the on-board processor.
[0024] Vehicle-mounted cameras, lidar, and millimeter-wave radar acquire road environment data using their respective acquisition methods, and record the acquisition time using a unified clock reference or a timestamp that can be converted to a unified clock reference. During actual installation, hardware synchronization triggering, precise time synchronization protocols, or sensor driver layer time synchronization methods can be used to establish the time relationship between the various acquisition devices. The recorded acquisition time is the time corresponding to the actual acquisition of the image frame, point cloud frame, or radar scan data; the target detection processing completion time is not used as the acquisition time.
[0025] The vehicle-mounted camera device acquires image data using a vehicle-mounted digital camera method. Light from the road scene enters the image sensor through the lens, where it performs photoelectric conversion to form a digital image. The image signal processing unit performs exposure, noise reduction, and image format processing on the digital image to obtain an image frame with the acquisition time. The image frame is input to an image target detector to obtain target detection boxes and target categories. The image region corresponding to the target detection box is then input into target feature encoding processing to obtain the corresponding target features.
[0026] After installation, the vehicle-mounted camera device undergoes calibration to obtain the calibration relationship between the camera and the unified coordinate system of the bus. For vehicles, pedestrians, and non-motorized vehicles, the road contact reference position corresponding to the target detection box is used as the target spatial reference position. The target reference position in the image is transformed to the target position in the unified coordinate system of the bus through the road plane projection relationship. The same target in consecutive image frames establishes a trajectory relationship through multi-target tracking processing, and the motion state of the target is obtained based on the position change and time interval between adjacent effective acquisition times.
[0027] LiDAR uses laser ranging to acquire point cloud data. The laser emitting unit emits laser signals into the road environment, and the laser receiving unit receives the target's return signal. The target distance is determined based on the laser signal propagation time, and combined with the azimuth and elevation angles of the corresponding laser channel, a three-dimensional measurement point is obtained. The three-dimensional measurement points formed within one scanning cycle constitute a point cloud frame, and the corresponding acquisition time is recorded.
[0028] Point cloud frames are input into 3D target detection processing to obtain the 3D position, target category, and target point set of candidate targets. The target point set is processed through point cloud feature encoding to form corresponding target features. The same target in consecutive point cloud frames establishes trajectory relationships through 3D multi-target tracking processing, and the motion state is obtained based on the changes in the position of consecutive targets.
[0029] Millimeter-wave radar employs the FMCW radar acquisition method. The millimeter-wave radar transmits frequency-modulated continuous wave signals and receives reflected signals from road targets. The received signals are processed by an analog front-end and analog-to-digital conversion to form digital sampled data. The target distance is obtained based on the beat frequency between the transmitted and received signals. The target radial velocity is obtained based on the phase change between the continuous frequency modulation periods or through Doppler processing. The target azimuth is obtained based on the phase difference between multiple receiving channels through angle estimation or beamforming processing.
[0030] The target distance and azimuth obtained by millimeter-wave radar are converted into a unified coordinate relationship for buses to form the target position. After target detection, clustering and continuous scanning target tracking, the corresponding radar target and motion status are formed.
[0031] When millimeter-wave radar participates in category consistency judgment, it classifies the radar detection results into targets. Target range, radial velocity, azimuth angle, echo intensity, and continuous target trajectory characteristics are used as classification inputs, and the classification results are mapped to a unified category set adopted by this system.
[0032] Target categories formed from different data sources are all converted to a unified category set before being written into the target evidence record. For example, categories such as "pedestrians," "small passenger vehicles," and "non-motorized vehicles" can be uniformly set. Subsequent category consistency judgments compare the categories in the unified category set, and do not directly compare the category numbers used internally by different devices.
[0033] Target features are stored separately according to their data sources. Target features from vehicle-mounted cameras, lidar, and millimeter-wave radar are not directly used to calculate cosine similarity. Consistency assessment of target features from adjacent acquisition times is performed under the conditions that they originate from the same data source, use the same feature encoding process, and have consistent target feature dimensions.
[0034] After the vehicle-mounted camera, lidar, and millimeter-wave radar are installed, spatial calibration is performed to determine the conversion relationship between the coordinates of each acquisition device and the unified coordinates of the bus. The vehicle-mounted camera determines the correspondence between the image position and the vehicle coordinates through camera calibration and road plane calibration; the lidar and millimeter-wave radar determine the spatial conversion relationship relative to the vehicle coordinates through calibration targets that can be observed by the corresponding acquisition devices.
[0035] Target locations from different data sources are compared using the same road reference location. The vehicle-mounted camera uses the road contact reference location corresponding to the target detection frame; the lidar converts the 3D target detection results to the corresponding road reference location; and the millimeter-wave radar uses the road planar location corresponding to the target distance and azimuth. Spatial distance comparisons are performed only after all data sources have achieved this location unification.
[0036] If an image frame lacks effective acquisition time or the image data is incomplete, the image frame will not be used as the current valid data to be matched; if a lidar point cloud frame lacks effective acquisition time or cannot form a valid target detection result, the missing data will not be used to construct the target position; if a millimeter-wave radar does not form a valid target or lacks effective acquisition time in the current scanning cycle, the candidate target corresponding to the data source will not be constructed.
[0037] When a data source is missing in the current processing cycle, the other validly acquired data retains the corresponding collection time and data source. Location consistency or category consistency results require comparison with at least two data sources. If a valid comparison object is missing, a single data source is not directly determined as a trusted participant; verification continues once new valid data is acquired.
[0038] The onboard processor load is obtained from the operational status monitoring data of the onboard computing platform. For computing platforms that can provide processor runtime statistics, the total processor runtime and idle runtime at adjacent sampling times are read, and the processor utilization rate is obtained based on the proportion of non-idle runtime in the total runtime increment; alternatively, the processor utilization rate status field provided by the onboard computing platform can be read. All processor load percentages mentioned below use the same processor utilization metric.
[0039] Under a specific set of implementation parameters, the bus travels at approximately 30 km / h through an urban road area containing motor vehicles, non-motor vehicles, and pedestrians. The onboard camera acquires road images at 25 Hz, the lidar acquires point cloud data at 10 Hz, and the millimeter-wave radar acquires radar data at 20 Hz, with corresponding acquisition periods of approximately 40 ms, 100 ms, and 50 ms, respectively.
[0040] The system uses 0.1s as the continuous target state processing cycle and takes 50ms before and after the reference acquisition time as the time range for the original observation to directly enter the current time correspondence. That is, the absolute value of the difference between the acquisition time of the original observation and the reference acquisition time is no greater than 50ms.
[0041] During a single road operation, 12 road targets were identified within the vehicle's sensing range for processing, including 6 vehicle targets, 4 pedestrian targets, and 2 non-motorized vehicle targets. The longitudinal distances of each target relative to the bus ranged from approximately 8m to 46m, while their lateral positions were distributed within a road area approximately 4m to the left and right of the bus.
[0042] Please see Figures 1 to 6 This embodiment provides a bus-mounted multimodal target detection AI intelligent agent system, including a multimodal data access unit, a target evidence management unit, a credibility assessment unit, a conflict review unit, and an AI intelligent agent task scheduling unit.
[0043] The multimodal data access unit receives data collected by vehicle-mounted cameras, LiDAR, and millimeter-wave radar, and records the collection time and data source; the target evidence management unit performs time correspondence, spatial transformation, target detection, and identity association, and establishes target evidence records; the credibility assessment unit determines the credibility status based on the location, category, and identity association status of different data sources; the conflict review unit reviews inconsistencies between different data sources; and the AI intelligent agent task scheduling unit controls the execution order of review tasks based on the conflict status and the load of the vehicle-mounted processor.
[0044] Multimodal data access.
[0045] The multimodal data access unit acquires multimodal data collected by the vehicle-mounted camera, LiDAR, and millimeter-wave radar. Each data source records its corresponding acquisition time and data source. The acquisition time uses a timestamp generated by the corresponding acquisition device or converted using a unified clock relationship; the data source uses a source identifier that can distinguish between the vehicle-mounted camera, LiDAR, and millimeter-wave radar.
[0046] The data generated by vehicle-mounted cameras, lidar, and millimeter-wave radar have different formats, therefore the data source is retained when each data enters subsequent processing. The target evidence management unit determines which acquisition device the target's location, category, characteristics, and motion state originated from based on the data source.
[0047] When time interpolation is required between adjacent acquisition times from the same acquisition device, the target tracking processing of the data source itself is first used to confirm that two adjacent valid observations belong to the same target trajectory. Time interpolation is only performed using the two observation positions when the preceding and following valid observations belong to the same target trajectory.
[0048] For example, around 15.240s, the acquisition time for a single frame of image by the vehicle-mounted camera is 15.240s; the acquisition time for the corresponding target observation by the millimeter-wave radar is 15.250s; and the two effective observation times for the same target trajectory by the lidar are 15.200s and 15.300s, respectively.
[0049] The acquisition time difference between the vehicle-mounted camera and the millimeter-wave radar is 10ms, which is within a time range of 50ms before and after the reference acquisition time.
[0050] The lidar observation at 15.300s differs from the reference acquisition time at 15.240s by 60ms. Therefore, the observation at 15.300s is not considered as the original observation within the current direct time range, but is used together with the previous valid observation of the same target trajectory at 15.200s to determine the position corresponding to 15.240s.
[0051] Linear interpolation is used to obtain the reference acquisition time position for effective observations before and after the same target trajectory: ; in, Indicates the target at the reference acquisition time Location; This indicates that the target trajectory was acquired in the previous valid acquisition time. Location; This indicates that the target trajectory was acquired in the next valid acquisition time. Location; and These represent the effective acquisition times before and after the reference acquisition time, respectively. Indicates the current reference acquisition time and satisfies . , and All are position vectors, and the unit of each coordinate component is m; , and The unit is s; This is a dimensionless scale. This interpolation is performed only between two already acquired observations of the same trajectory and is not used for position prediction at a later acquisition time.
[0052] For example, the lidar obtains the position of a pedestrian target in 15.200 seconds. The location of the pedestrian target was obtained at 15.300s. m. When the reference acquisition time is 15.240s, the time scale is 0.4, and the corresponding location is... m.
[0053] The longitudinal position of the other vehicle target at 15.200s and 15.300s is 27.00m and 27.90m respectively, and the longitudinal position at the reference acquisition time of 15.240s is 27.36m; the lateral position is -2.12m and -2.07m respectively, corresponding to a lateral position of -2.10m.
[0054] If the first processing cycle after system startup only has data from the current collection time and no target record from the previous collection time, then the current cycle first performs time mapping, target detection, and spatial mapping between different data sources to form the current target evidence record. After obtaining data from the next processing cycle, the target evidence record from the previous cycle is used as the previous collection time data for identity association processing.
[0055] Target evidence management.
[0056] The target evidence management unit receives the aforementioned multimodal data, establishes a correspondence between different data sources according to the collection time and spatial location, and performs target detection on the corresponding data to obtain the position, category, target features and motion state of the candidate targets.
[0057] Identify the data to be matched.
[0058] The target evidence management unit reads the acquisition time of each data point, identifies the original observations whose acquisition time falls within the same time interval, and the position data that has been aligned to the reference acquisition time based on the valid observations before and after the same target trajectory as the data to be matched.
[0059] Taking a reference acquisition time of 15.240s as an example, the direct time range corresponding to the 50ms before and after is 15.190s to 15.290s. The 15.240s image from the vehicle-mounted camera and the effective observations from the millimeter-wave radar around 15.200s and 15.250s can fall within this time range; after the lidar obtains the 15.240s position based on the same target trajectory at 15.200s and 15.300s, this position is included in the target correspondence at the current reference acquisition time.
[0060] When multiple valid observations of the same millimeter-wave radar target trajectory exist within the corresponding time interval, the valid observation closest to the reference acquisition time is selected for target correspondence between different data sources. Other observations of the same trajectory are used to update the radar target motion state and are not repeatedly written into the target evidence record as new independent road targets.
[0061] If the reference acquisition time for the next processing cycle is, for example, 15.340s, the corresponding direct time range is 15.290s to 15.390s.
[0062] If an observation exceeds the current time interval and a reference acquisition time position cannot be obtained from valid observations of the same target trajectory before and after the same data source, the observation will not participate in the target correspondence of the current reference acquisition time, but will continue to be processed in its corresponding subsequent processing cycle.
[0063] When only data from a portion of the data sources is obtained in a given processing cycle, the already obtained data is still saved according to the collection time and data source. If other data sources cannot be obtained for location and category comparison at present, the trusted participation status is not directly determined; the determination will continue after other valid data is obtained subsequently.
[0064] Unify spatial location and obtain candidate targets.
[0065] The target evidence management unit converts the target locations from different data sources to a unified coordinate system for buses based on the spatial calibration relationships of onboard cameras, lidar, and millimeter-wave radar.
[0066] In this embodiment, the direction of travel of the bus is Direction, the lateral direction of the vehicle is Direction, vertical direction is Direction, and the unit of each spatial coordinate is meters.
[0067] The candidate target position of the vehicle-mounted camera device is obtained from the target detection result and the camera calibration relationship; the candidate target position of the lidar is obtained from the target point cloud or the three-dimensional target detection result; the candidate target position of the millimeter-wave radar is obtained by converting the target distance and azimuth angle.
[0068] The target location, target category, target features, and motion state are written into the candidate target data corresponding to the respective data sources.
[0069] The spatial distance between different candidate targets is calculated using Euclidean distance: ; in, Indicate candidate target With candidate targets Spatial distance between them; , and Representing candidate targets The three spatial coordinate components; , and Representing candidate targets The three spatial coordinate components. The unit of each spatial coordinate is meters (m), therefore... The unit is m.
[0070] For example, with a reference acquisition time of 15.240s, the vehicle-mounted camera device obtains the location of a candidate pedestrian target as follows: m; The corresponding candidate target position of the lidar is m; The candidate target location corresponding to the millimeter-wave radar is m.
[0071] The spatial distance between the vehicle-mounted camera and the lidar is approximately 0.16m, the distance between the vehicle-mounted camera and the millimeter-wave radar is approximately 0.25m, and the distance between the lidar and the millimeter-wave radar is approximately 0.15m.
[0072] In this embodiment, the spatial candidate threshold for establishing the correspondence between candidate targets from different data sources is set to 1.20m. The distance between the three candidate targets is less than 1.20m, and they all belong to the same target category, "pedestrian," so they can be processed under the same target correspondence.
[0073] For the other vehicle target, the location of the onboard camera is... m, the candidate target position of the lidar is The distance between the two vehicles is approximately 0.52m. The location of the other vehicle is... When m, it is the same as The distance between m is approximately 5.90m, therefore they are not considered as candidate combinations of the same target.
[0074] If the spatial distance between candidate targets exceeds 1.20m or the target categories are inconsistent, the same target evidence record will not be written into the current initial matching process. Candidate targets for which no corresponding target has been obtained will remain in an unmatched state under this data source and will continue to be processed in subsequent data collection times.
[0075] Establish a correspondence between the same target and form a record of target evidence.
[0076] The target evidence management unit establishes a correspondence for the same target based on the spatial location correspondence and initial category consistency between candidate targets.
[0077] When the vehicle-mounted camera can acquire valid candidate targets, these candidate targets can be used as a reference target set to establish correspondences between lidar candidate targets, millimeter-wave radar candidate targets, and the reference targets. If the vehicle-mounted camera currently does not have a corresponding candidate target, it will not force the writing of candidate targets from other data sources into a specific vehicle-mounted camera's target record; instead, it will retain the candidate trajectory from the corresponding data source, awaiting subsequent target matching.
[0078] When multiple candidate targets of the same category exist in the same area, they are assigned one by one after category consistency screening. The spatial distance between candidate targets is used as the allocation cost. ; And satisfy: ; as well as: ; in, This indicates the number of candidate targets corresponding to the current category from the reference data source; This indicates the number of candidate targets participating from another data source; Indicates the first The first reference candidate target and the first Spatial distance between candidate targets; Indicates whether a correspondence exists between two candidate targets; This represents the maximum number of valid matches that can be established among candidate combinations that are of the same category and whose spatial distance does not exceed the spatial candidate threshold.
[0079] First, based on the unified target category and the 1.20m spatial candidate threshold, candidate combinations that do not meet the conditions are eliminated. The combinations that do not meet the conditions are then... Determine the number of one-to-one correspondences that can be established for the remaining candidate combinations. When the number of matches is Under the condition that, the total spatial distance of effective corresponding combinations is minimized through linear allocation.
[0080] For example, a vehicle-mounted camera acquires three candidate targets for matching within the same category during a single acquisition time, with their locations being... m、 m and m; the positions of the three candidate targets of the lidar are respectively m、 m and m.
[0081] The distances between the three candidate targets in the same order are approximately 0.16m, 0.52m, and 0.23m, respectively. The distance between the first vehicle-mounted camera candidate target and the second lidar candidate target is approximately 9.93m, and the distance between the first and third lidar candidate targets is approximately 8.32m; the distance between the second vehicle-mounted camera candidate target and the first lidar candidate target is approximately 9.37m, and the distance between the second and third lidar candidate targets is approximately 17.80m.
[0082] After eliminating the above-mentioned cross-candidate combinations based on the unified target category and spatial candidate threshold, the three in-order candidate targets form a one-to-one correspondence.
[0083] After establishing the correspondence, the location, category, target characteristics, motion state, and data source of each data source corresponding to the candidate target are written into the same target evidence record.
[0084] For example, the location of the vehicle-mounted camera is preserved in the target evidence record of the aforementioned pedestrian target at 15.240 seconds. m and category "pedestrian"; LiDAR location m and category "pedestrian"; millimeter-wave radar location The system defines m and the category "pedestrian", and saves the target features and motion states corresponding to each data source.
[0085] The established target evidence record is continuously updated in subsequent collection times. The data source trajectory corresponding to the target evidence record has been established in the previous collection time, and in the subsequent collection time, it is updated to the target evidence record based on the trajectory relationship of this data source, the predicted location, and the target characteristics.
[0086] When a data source experiences a temporary location shift, category change, or pending verification status at a later acquisition time, the target correspondence established at the previous acquisition time is not immediately deleted. Instead, the current actual observation is written into the corresponding field of the data source before a confidence status judgment and conflict lookup are performed.
[0087] The category consistency condition used when establishing the correspondence between targets from different data sources for the first time is aimed at the selection of candidate targets when establishing target evidence records. After the target evidence records have been established, if the category of a certain data source changes during subsequent collection time, the corresponding trajectory relationship is retained, and the current category is compared with the categories of other data sources, thus forming a category conflict.
[0088] Determine the identity association status.
[0089] The target evidence management unit performs identity association for each data source based on adjacent collection times.
[0090] For the same data source, read the target's position, motion state, and target features at the previous acquisition time. Based on the target's position and motion state at the previous acquisition time, determine the predicted position at the next acquisition time. Then, compare the actual target position at the next acquisition time with the predicted position, and compare the target features at adjacent acquisition times.
[0091] In this embodiment and This represents the relative velocity component of the target within the unified coordinate system of the bus. The onboard camera and lidar obtain the relative motion state based on the positional changes of the corresponding target trajectory within adjacent effective acquisition times; the millimeter-wave radar obtains the corresponding motion state based on the Doppler velocity and the positional changes of the continuous trajectory.
[0092] Target position prediction is performed using a constant velocity relationship between adjacent short intervals. ; in, and Indicates the target's longitudinal and lateral positions at the previous acquisition time; and This represents the longitudinal and lateral velocity components of the target relative to the bus. and Indicates the elapsed time interval The predicted location was obtained subsequently; and This indicates the predicted velocity component used in the next acquisition time. This represents the time interval between two adjacent identity association processing times. Location is measured in meters (m), and velocity is measured in meters per second (m / s). The unit is seconds (s).
[0093] For example, a pedestrian's position at 15.240s is... m, with a relative longitudinal velocity of -0.20 m / s and a relative lateral velocity of 1.10 m / s. When At time s, the predicted location is m. If the actual detection position is 15.340s. If m, then the distance between the actual position and the predicted position is approximately 0.085m, rounded to two decimal places as 0.09m.
[0094] The current location of the other vehicle target is m, relative longitudinal velocity is 8.40 m / s, relative lateral velocity is 0.20 m / s, and the predicted position after 0.10 s is m. If the actual location at the next acquisition time is... If m, then the positional deviation is approximately 0.171m, which is rounded to two decimal places as 0.17m.
[0095] Among them, 8.40 m / s and 0.20 m / s are the relative motion speeds of the target with respect to the coordinates of the bus, and are not directly algebraically calculated with the bus's own running speed of approximately 30 km / h.
[0096] Target features from adjacent acquisition times are compared using cosine similarity: ; in, This represents the target feature vector obtained by target feature encoding processing from the same data source at the previous acquisition time; This indicates the target feature vector obtained by using the same target feature encoding process in the subsequent acquisition time. This represents the inner product of two target feature vectors; and These represent the L2 norms of the corresponding target feature vectors; This represents the cosine similarity of the target features.
[0097] The two target features being compared use the same data source, the same target feature encoding method, and have the same dimension. Both the target feature vectors and cosine similarity are treated as dimensionless data.
[0098] When target feature extraction fails, target features are missing, or target features are zero vectors, cosine similarity judgment is not performed, the current adjacent collection time cannot form an established association state, and the corresponding identity association state is set to the association pending verification state.
[0099] Under a set of implementation parameters, the identity association distance condition is set to 1.20m, and the target feature consistency condition is set to: ; The predicted position of the pedestrian deviates from the actual position by approximately 0.09m. Furthermore, when the cosine similarity of the target features at adjacent acquisition times is 0.91, both the position deviation and the target features meet the conditions, thus establishing a valid association.
[0100] Another pedestrian in a partially occluded state has a positional deviation of 0.34m, but the cosine similarity of the target features is 0.72. The target features do not meet the consistency condition, forming a state of association pending verification.
[0101] The cosine similarity of a certain non-motorized vehicle target feature is 0.89, but the deviation between the actual position and the predicted position is 1.46m. The position deviation does not meet the association distance condition, and thus the association is in a pending state.
[0102] The "Association Pending Verification" status indicates that at least one of the following conditions—location continuity or target feature continuity—is not met in the current adjacent acquisition timeframes: the association criteria are not met. This status retains the current target trajectory and proceeds to subsequent continuous verification.
[0103] The vehicle-mounted camera, lidar, and millimeter-wave radar each establish a status of association established or a status of association pending verification corresponding to their respective data sources, and write this status into the data source field corresponding to the target evidence record.
[0104] The status of the associated verification is pending.
[0105] When the identity association status of a certain data source in the target evidence record is in the association pending verification state, the target evidence management unit extracts the target location, target characteristics and motion status of the target corresponding to the data source during the continuous collection time to form a historical observation sequence.
[0106] Under a set of implementation parameters, the continuous review time window uses three consecutive processing cycles, each with a duration of 0.1s. The total length of the three processing cycles is approximately 0.3s. If calculated using three reference acquisition times, the time interval between the first and third reference acquisition times is approximately 0.2s.
[0107] For example, in the first verification cycle, a certain pedestrian to be verified has a positional deviation of 0.48m and a target feature cosine similarity of 0.82; in the second verification cycle, the positional deviation is 0.32m and the target feature cosine similarity is 0.87; and in the third verification cycle, the positional deviation is 0.27m and the target feature cosine similarity is 0.90.
[0108] Within the continuous review time window, each processing cycle with valid target location and valid target features must simultaneously satisfy the correlation distance condition and the target feature consistency condition.
[0109] The cosine similarity of the target features in the first review period is 0.82, which is lower than 0.85. Therefore, the continuous review time window does not meet the condition of continuous consistency, and the associated pending review status is not updated.
[0110] In subsequent consecutive review time windows, if the positional deviations of the three consecutive periods are 0.31m, 0.28m and 0.25m respectively, and the cosine similarity of the target features are 0.87, 0.90 and 0.92 respectively, then all three periods meet the association distance condition and the target feature consistency condition, and the association pending review state is updated to the association established state.
[0111] If the cosine similarity of the target features of another non-motorized vehicle target in the three verification cycles is 0.90, 0.91 and 0.88 respectively, but the position deviation in the second cycle is 1.34m, then the cycle does not meet the association distance condition and continues to maintain the association pending verification state.
[0112] If a valid target feature cannot be obtained in a certain review cycle due to complete target occlusion, the cycle will not be considered as meeting the target feature consistency condition, and artificially generated target features will not be used to supplement the cycle.
[0113] If the continuous consistency condition is not met after a continuous review window ends, the associated pending review status will continue. After obtaining subsequent valid data, review will continue based on the actual target location, target characteristics, and motion status.
[0114] Credibility assessment.
[0115] The credibility assessment unit reads the location, category, and identity association status of the same target in different data sources at the same reference collection time from the target evidence record, and determines the credibility status corresponding to each data source.
[0116] This results in both location-consistent and category-consistent outcomes.
[0117] For any data source, its target location is compared with the target locations of other currently valid data sources in the same target evidence record, and its uniform target category is compared with the uniform target category of other valid data sources.
[0118] Under a set of implementation parameters, the location consistency range for multiple data sources is set to 1.50m. The 1.50m location consistency range is used to determine the consistency between target locations of different data sources under the same reference acquisition time; the aforementioned 1.20m association distance condition is used to determine the identity association between the predicted location and the actual location of the same data source across adjacent acquisition times.
[0119] For example, at the current reference data collection time, the location of the vehicle-mounted camera is... m, the location of the lidar is m, the location of the millimeter-wave radar is The location distance between all data sources is less than 1.50m, and the target category is uniformly "pedestrian".
[0120] For a given data source, the location consistency result is satisfied when the location distance between the data source and other data sources that can be used for valid comparison does not exceed 1.50m; the category consistency result is satisfied when the data source shares the same unified target category with other valid data sources.
[0121] If there is only one valid data source in the current target evidence record, and it is not possible to make a comparison of the position and category between different data sources, then the data source is temporarily uncertain as a credible participant and remains in a pending review state, waiting for the formation of valid comparison conditions in the future.
[0122] The data source trajectory for the same target evidence record has been established in the previous acquisition time, and it continues to be updated to the same target evidence record in subsequent acquisition times. Therefore, if the location of a data source shifts or the category changes later, the current observation is still retained in the corresponding target evidence record before the location consistency and category consistency judgment are performed.
[0123] For example, for a vehicle that has already established a correspondence between different data sources, the current location of the onboard camera is... m, the location of the lidar is m, the distance between the two is approximately 0.52m; the current target trajectory position corresponding to the millimeter-wave radar is The distance between the location of the vehicle-mounted camera and the location of the vehicle-mounted camera is approximately 2.82m.
[0124] The target trajectory of the millimeter-wave radar has already been associated with the corresponding target evidence record in the previous acquisition time. Therefore, the current observation is written into the original target evidence record, resulting in a positional inconsistency.
[0125] If the vehicle-mounted camera and lidar both target "small passenger vehicles", while the millimeter-wave radar currently targets "non-motorized vehicles", then the millimeter-wave radar will result in a category inconsistency with the other two data sources.
[0126] Determine whether the participant status is trusted or pending verification.
[0127] When the location consistency result of a certain data source meets the condition, the category consistency result meets the condition, and the identity association status of the data source is in the association established state, the corresponding data source will be determined as a trusted participant.
[0128] If any one of the three conditions is not met, the corresponding data source will be designated as pending review.
[0129] The "Pending Review" status indicates that the data source currently has at least one of the following issues: inconsistency in location, inconsistency in category, or lack of established identity association. The actual observations from this data source are not deleted if the data is in the "Pending Review" status.
[0130] For example, in the aforementioned vehicle, the distance between the vehicle-mounted camera and the lidar is 0.52m, the unified target category is "small passenger vehicle", and the corresponding identity association status is an established association status. When other comparison relationships also meet the conditions, they can be determined as trusted participation statuses respectively.
[0131] The distance between the millimeter-wave radar and the vehicle-mounted camera is approximately 2.82m. Since there are differences in the current unified target category, the data source corresponding to the millimeter-wave radar is pending verification.
[0132] If the location difference of a pedestrian across three data sources is less than 0.40m, and the target category is "pedestrian" in all three data sources, but the identity association status of one of the data sources is still in the "association pending verification" state, then that data source will continue to remain in the "pending verification" state.
[0133] Update the status to be pending review.
[0134] For data sources that are pending verification, the corresponding target location, unified target category, and identity association status are re-acquired at the next data collection time, and the location and category are re-consistent with other valid data sources.
[0135] For example, in the next 0.1s processing cycle, the onboard camera acquires the vehicle's position. m, location obtained by lidar m, position obtained by millimeter-wave radar m.
[0136] The distance between the vehicle-mounted camera and the lidar is approximately 0.19m, the distance between the vehicle-mounted camera and the millimeter-wave radar is approximately 0.28m, and the distance between the lidar and the millimeter-wave radar is approximately 0.16m, all less than 1.50m.
[0137] If the current unified target category of the millimeter-wave radar is restored to "small passenger vehicle", and the identity association status of the millimeter-wave radar is established, then the millimeter-wave radar will be updated from the pending verification status to the trusted participation status.
[0138] If the location is restored to consistency but the category is still inconsistent, the status of pending review will remain unchanged; if both the location and category are restored to consistency but the identity association status is still pending review, the status of pending review will also remain unchanged.
[0139] Data sources that still do not meet the trusted participation criteria after the continuous review period ends will remain in the pending review status, and the corresponding data will be re-acquired and verified during the conflict re-investigation process.
[0140] Conflict review.
[0141] The conflict re-investigation unit extracts the location, category, and identity association status of the same target from the target evidence record under different data sources, and performs location comparison, category comparison, and identity association status comparison respectively.
[0142] A state of conflict or a state of agreement can be formed.
[0143] When the distance between target locations from different data sources exceeds 1.50m, a location conflict occurs; when the target categories are different, a category conflict occurs; when the identity association statuses corresponding to different data sources are different, an identity association status conflict occurs.
[0144] For example, the distance between the vehicle's onboard camera and the millimeter-wave radar is approximately 2.82m, which exceeds the 1.50m range of location consistency, thus creating a location conflict.
[0145] If the target category unified by the millimeter-wave radar differs from the "small passenger vehicle" category from the other two data sources, a category conflict will occur.
[0146] If the vehicle-mounted camera and lidar are in an established association state, while the millimeter-wave radar is in an association pending verification state, then an identity association state conflict occurs.
[0147] A conflict state indicates that at least one inconsistency regarding the same target exists between different data sources and has not yet been resolved. After a conflict occurs, the observations from the corresponding data source are retained and a retrospective review is initiated.
[0148] When the location, unified target category, and identity association status of different valid data sources all meet the consistency conditions, the corresponding target will be marked as consistent. No retrospective task will be generated for this target corresponding to the current collection time.
[0149] Define the scope of the review and create a review task.
[0150] For targets that are in conflict, the conflict retrieval unit reads the target location and motion state corresponding to the trusted participating data source from the target evidence record, and determines the predicted location for the next acquisition time based on the target motion state.
[0151] For example, the current trusted participation data source location of the vehicle that caused the conflict is... m, with a relative longitudinal velocity of 8.40 m / s and a relative lateral velocity of 0.20 m / s, the predicted position after 0.10 s is m.
[0152] In this embodiment, the predicted location is taken as the center, and an area of 4.0m longitudinally in front and behind the predicted location and 3.0m laterally in the left and right directions is taken in the road plane as the unified spatial reference range corresponding to the vehicle. The total longitudinal span is about 8.0m and the total lateral span is about 6.0m.
[0153] For vehicle-mounted camera devices, the unified spatial back-lookup range is projected onto the next acquisition time image frame through the camera calibration relationship to obtain the corresponding image ROI. Target detection and target feature extraction are then performed again within this ROI.
[0154] For LiDAR, based on the unified coordinate relationship of the vehicle, point clouds whose spatial location is within the unified spatial back-lookup range are selected from the point cloud frames of the next acquisition time, and target detection and target matching are re-performed in the selected local point clouds.
[0155] For millimeter-wave radar, based on the target range and azimuth interval corresponding to the unified spatial lookup range, the corresponding observations are selected from the subsequent radar scan data or radar target list, and the target matching is performed again.
[0156] The local retrospective data used by vehicle-mounted cameras, lidar, and millimeter-wave radar are all obtained by converting the same predicted location and a unified spatial retrospective range.
[0157] For example, if 9 out of the 12 objectives have reached a consistent state and 3 objectives still conflict, then a backtracking task will be generated only for the 3 conflicting objectives.
[0158] Rematch according to trusted status.
[0159] After the AI agent task scheduling unit issues the review task, the conflict review unit reads the trusted status corresponding to different data sources from the target evidence record.
[0160] Data sources that are in a trusted participation state and have not formed a current conflict are used as reference sources for retrospective review; data sources that are in a pending review state or have participated in forming a current conflict are used as data sources to be retrospectively reviewed.
[0161] When multiple data sources are in a state of pending verification, the actual observations of the corresponding data sources within the scope of the review should be obtained again, without pre-designating a particular data source as the correct result.
[0162] The re-extracted multimodal data are first grouped into data with the same reference collection time according to the aforementioned time correspondence, and then the target is re-matched according to the unified spatial location, unified target category, and identity association status of each data source.
[0163] For example, after performing a retrospective search on the aforementioned vehicle targets, the location of the vehicle-mounted camera device can be obtained in the next data collection time. m, LiDAR location m and millimeter-wave radar positions m.
[0164] The spatial distance between the vehicle-mounted camera and the lidar is approximately 0.19m, the distance between the vehicle-mounted camera and the millimeter-wave radar is approximately 0.28m, and the distance between the lidar and the millimeter-wave radar is approximately 0.16m, all of which are within the 1.50m range of positional alignment.
[0165] If the unified target category of the three data sources is "small passenger vehicle", and the identity association status of each data source is in the established association state, then the rematch result satisfies the consistency of location, category and identity association status. The rematch result is written to the target evidence record and the back-search task is ended.
[0166] If the positional deviation of another target being checked decreases from 2.10m to 0.60m, but the target category of a certain data source is still different from that of other data sources, then the positional conflict has been eliminated, but the category conflict still exists, and the conflict status remains.
[0167] If the unified target category of a pedestrian from three data sources is "pedestrian", the maximum position deviation is 0.44m, but the identity association status of one data source is still in the association pending verification state, then the position and category meet the consistency condition, but the identity association status does not meet the consistency condition, and the subsequent backtracking will continue.
[0168] Location inconsistency refers to the fact that the location distance between at least one set of data sources participating in the comparison exceeds the range of location consistency; category inconsistency refers to the difference in the unified target category of different data sources; and identity association inconsistency refers to the fact that different data sources do not jointly form an established association. Once all three conditions are met, the corresponding backtracking task ends.
[0169] AI agent task scheduling and result generation.
[0170] The AI intelligent agent task scheduling unit reads the current back-check task, conflict status, back-check task generation time, and vehicle processor load, determines the execution order of back-check tasks based on the conflict situation, and controls task distribution based on the vehicle processor load.
[0171] Determine the conflict level and create a queue of backtracking tasks.
[0172] The AI agent task scheduling unit reads the location conflict, category conflict, and identity association state conflict corresponding to each back-check task.
[0173] A conflict corresponds to a specific data source comparison relationship resulting in inconsistency under a comparison dimension.
[0174] For example, if the distance between the vehicle-mounted camera and the millimeter-wave radar exceeds the range for positional consistency, a positional conflict occurs; similarly, if the distance between the lidar and the millimeter-wave radar also exceeds the range for positional consistency, another positional conflict occurs. Comparisons based on the same data source are not counted repeatedly within the same comparison dimension.
[0175] Category conflicts and identity-related state conflicts are counted in the same way.
[0176] The current conflict count for a regression task is calculated by adding the number of location conflicts, category conflicts, and identity-related status conflicts. The conflict level is then determined based on the current conflict count. The higher the current conflict count, the higher the conflict level.
[0177] For example, Task A has 2 location conflicts, 2 category conflicts, and 2 identity-related state conflicts, for a total of 6 conflicts; Task C has 1 location conflict, 1 category conflict, and 1 identity-related state conflict, for a total of 3 conflicts; Task B has 1 location conflict and 1 category conflict, for a total of 2 conflicts; and Task D has only 1 location conflict.
[0178] The conflict levels are ranked from highest to lowest as A, C, B, and D.
[0179] When two tasks have the same number of current conflicts, their initial generation times are compared. For example, if task B was first generated at 15.342s and task E was first generated at 15.366s, and both currently have 2 conflicts, then task B will be ranked before task E.
[0180] When old back-review tasks are re-matched at a later collection time, and some conflicts are resolved, the remaining conflict items are re-counted and the conflict level is updated based on the target evidence records. Incomplete old tasks and newly generated tasks are re-formed into a back-review task queue according to the updated conflict level and the task's initial generation time.
[0181] The task is assigned based on the load control of the onboard processor.
[0182] The AI intelligent agent task scheduling unit reads the current processor utilization from the operating status monitoring data of the vehicle computing platform and compares the processor utilization with the preset load limit.
[0183] Under a set of implementation parameters, the preset load limit is 80%.
[0184] When the current processor utilization is 63%, which is lower than 80%, the AI agent task scheduling unit sends the first-ranked back-up task in the back-up task queue to the conflict back-up unit.
[0185] After the top-ranked backtracking task is completed, the processor utilization is reread. If the current processor utilization is 82%, reaching the preset load limit, the current order of unassigned tasks in the backtracking task queue is maintained, and the assignment of new backtracking tasks is paused.
[0186] When the processor utilization drops to, for example, 74%, below 80%, the current top-ranked callback task continues to be sent to the conflict callback unit.
[0187] The onboard processor load is reread at least when preparing to issue a new back-up task and after the currently prioritized task has been executed.
[0188] When the onboard processor load reaches the preset load limit, only the issuance of new back-check tasks is paused. Back-check tasks are not deleted, nor are the target location, target category, target characteristics, identity association status, and trust status in the target evidence record modified.
[0189] Queue update after task completion.
[0190] After the conflict lookup unit completes the first lookup task in the current sorting, the AI agent task scheduling unit removes the completed lookup tasks from the lookup task queue and rereads the conflict status and vehicle processor load corresponding to the remaining lookup tasks.
[0191] Taking the aforementioned queues A, C, B, and D as an example, after task A is completed, A is removed from the task queue and the current conflict level of C, B, and D is re-determined.
[0192] If the conflict state remains unchanged, the remaining queues are C, B, and D. If the current processor utilization is 74%, task C is assigned. After task C completes, task C is removed from the queue, and the remaining task status and processor utilization are read again.
[0193] If only task D remains and the current processor utilization is 81%, then task D will be kept in the backlog task queue; when the processor utilization drops to, for example, 72%, task D will be reassigned.
[0194] When the task queue is empty, the current task scheduling ends. The vehicle-mounted camera, lidar, and millimeter-wave radar continue to collect data. If a new conflict occurs during the next data collection session, a new task is generated based on the corresponding conflict.
[0195] The results of target detection and feature recognition are generated.
[0196] For targets that have reached a consistent state and targets that have completed the back-check task, the target location, target category, and target characteristics are determined based on the corresponding target evidence records.
[0197] When a target has location data from two or more trusted participating data sources, first confirm that the location consistency condition is met among the trusted participating data sources, and then take the arithmetic mean of the location coordinates of the trusted participating data sources in the unified vehicle coordinate relationship to obtain the output target location of the target.
[0198] For example, the locations of the three trusted data sources are respectively m、 m and m, then the corresponding target position is: ; Get about m.
[0199] The above position averaging only uses data sources that have met the position consistency condition and are in a trusted participation state, and does not include positions corresponding to the pending verification state in the averaging.
[0200] The target category is the common unified target category after the category consistency judgment is completed. Targets that have already entered this step meet the category consistency condition, and no further majority voting is required for conflicting categories.
[0201] Target features from different data sources are retained separately, and target features from vehicle-mounted cameras, lidar, and millimeter-wave radar are not directly vector-added or arithmetic-averaged. Target evidence records retain valid target features that have passed identity verification according to their data source, and the correspondence between target features and data sources is preserved when outputting target features.
[0202] For targets that are still in a conflict state and have not completed the backtracking task, the data that has not yet passed the consistency verification will not be used to form the target detection and feature recognition results after the backtracking is completed. Instead, the corresponding conflict state will be retained and subsequent backtracking will be performed.
[0203] For example, among the current 12 targets, 9 targets reach a consistent state after the initial comparison of different data sources, while the other 3 targets enter a conflict review process. The first conflicting target completes consistency recovery within the next 0.1s processing cycle; the second conflicting target completes consistency recovery after two consecutive processing cycles; the identity association state of the third target remains inconsistent, and it continues to remain in a conflicting state.
[0204] At this point, the 11 targets that have met the conditions for forming the results form the target detection and feature recognition results based on the corresponding target evidence records; the third target continues to perform the back-check.
[0205] When the positional deviation of the third target at the next acquisition time is 0.36m, the cosine similarity of the target features at adjacent acquisition times from the same data source is 0.89, the unified target category from different data sources is restored to consistency, and the identity association status of each data source is in the established association state, the rematch result is written into the target evidence record, and the back-search task corresponding to that target ends.
[0206] In the above processing, the collection time and data source are obtained by the multimodal data access unit; the target location, target category, target features, and motion state are written into the target evidence record after being processed by the target detection and target tracking corresponding to each data source; each data source forms an identity association state based on the location, motion state, and target features of adjacent collection times; the credibility assessment unit determines the credible participation state or the pending verification state based on the location, unified target category, and identity association state of different data sources at the same reference collection time; the conflict review unit forms a conflict state based on the differences in location, category, and identity association state of different data sources, and determines the review scope based on the predicted location; the AI intelligent agent task scheduling unit issues the review task based on the conflict level, review task generation time, and vehicle processor load; the rematching result is written back to the target evidence record, and the target evidence record that has completed consistency verification forms the target detection and feature recognition result.
[0207] In this embodiment, the 1.20m identity association distance condition is used to determine the predicted and actual locations between adjacent data collection times from the same data source; the 1.50m location consistency range is used to determine the consistency between target locations from different data sources at the same reference collection time; the 0.85 target feature consistency condition is used to determine the cosine similarity between target features from adjacent data collection times from the same data source; three consecutive processing cycles are used to verify the associated pending status; and the 80% vehicle processor load limit is used to control the issuance of new back-check tasks.
[0208] 1.20m and 1.50m are both length values; the target feature cosine similarity is a dimensionless value; the number of processing cycles is the number of discrete cycles; and the on-board processor load is the processor utilization rate. Each parameter is used for its corresponding processing condition, and direct addition or subtraction operations between data of different dimensions are not performed.
[0209] Linear interpolation is used for the time correspondence between two valid observations of the same target trajectory; Euclidean distance is used for comparing the spatial distance between candidate targets in a unified spatial location and the positional deviation of different data sources; linear assignment is used for the one-to-one correspondence between multiple candidate targets of the same category; constant velocity state prediction is used to determine the predicted position of the next acquisition time based on the current target position and motion state; cosine similarity is used for comparing the consistency of target features at adjacent acquisition times of the same data source.
[0210] The data acquisition time is obtained from the sensor acquisition timestamp and the unified time synchronization relationship; the images from the vehicle-mounted camera are obtained from the image sensor, the target category and target features are obtained from image target detection and target feature encoding processing, and the target position is obtained from camera calibration and road space mapping; the lidar point cloud is obtained from laser ranging and scanning angle, and the target position, target category and target features are obtained from point cloud target detection processing; the millimeter-wave radar target range, radial velocity and azimuth angle are obtained from FMCW radar signal processing, and the target category is obtained from radar target classification processing; the motion state is obtained from continuous target tracking from various data sources; the vehicle-mounted processor load is obtained from the vehicle computing platform operation status monitoring; the retrospective task generation time is written into the corresponding retrospective task according to the unified system time when the retrospective task is formed.
[0211] The target evidence record is used to save the target data and status corresponding to different data sources; the association status and the association pending verification status are used to indicate the identity association of the target under the same data source at adjacent collection times; the trusted participation status and the pending verification status are used to indicate whether the corresponding data source meets the current trusted processing conditions; the conflict status is used to indicate that there are still differences in the location, category or identity association status between different data sources of the same target; the consistency status is used to indicate that the location, category and identity association status of different valid data sources of the same target all meet the consistency conditions.
[0212] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A bus-mounted multimodal target detection AI intelligent agent system, characterized in that, The system includes: a multimodal data access unit, a target evidence management unit, a credibility assessment unit, a conflict review unit, and an AI agent task scheduling unit; The multimodal data access unit acquires multimodal data collected by the vehicle-mounted camera device, lidar, and millimeter-wave radar, and records the acquisition time and data source corresponding to each multimodal data. The target evidence management unit matches the multimodal data according to the collection time and spatial location, performs target detection on the matched multimodal data, obtains the position, category, characteristics and motion state of each target, and determines the identity association state by combining the position, characteristics and motion state under adjacent collection time, forming a target evidence record corresponding to each target; The credibility assessment unit determines the credibility status of each data source based on the location deviation, category consistency, and identity association status of the same target under different data sources, and writes the credibility status into the target evidence record; The conflict review unit compares the location, category, and identity association status of the same target under different data sources. When the comparison results do not meet the consistency conditions, a conflict status is generated, and the review scope is determined according to the location and movement status in the target evidence record to form a corresponding review task. When the comparison results meet the consistency condition, the corresponding target is marked as consistent. The AI intelligent agent task scheduling unit sorts the back-check tasks according to the conflict state and the load of the vehicle processor, controls the conflict back-check unit to select the corresponding data source according to the trust state, and re-matches the multimodal data within the back-check range, and writes the matching result into the target evidence record. For the target corresponding to the consistent state and the target that has completed the backtracking task, the target location, target category and target features are determined based on the corresponding target evidence records to form target detection and feature recognition results.
2. The bus-mounted multimodal target detection AI intelligent agent system according to claim 1, characterized in that, The processing procedure of the target evidence management unit includes: The multimodal data whose collection time falls within the same corresponding time interval are identified as the data to be matched; Based on the spatial calibration relationship between the vehicle-mounted camera device, lidar and millimeter-wave radar, the data to be matched is converted to a unified spatial location, and target detection is performed on the converted data to be matched to obtain the position, category, features and motion state of candidate targets under different data sources; Establish a correspondence between candidate targets based on their spatial location and category consistency, and write the location, category, characteristics, motion state, and data source of the corresponding candidate target into the target evidence record.
3. The bus-mounted multimodal target detection AI intelligent agent system according to claim 2, characterized in that, The process of determining the identity association status includes: According to the order of collection time, targets of the same category at adjacent collection times are selected from the target evidence records, and the predicted position corresponding to the next collection time is determined based on the target position and movement state at the previous collection time. The target location at the next acquisition time is compared with the predicted location, and the consistency of the target features at adjacent acquisition times is compared. When the position deviation meets the correlation distance condition and the target features meet the consistency condition, the targets at adjacent acquisition times are determined to be the same target and a correlation established state is generated. When either condition is not met, a correlation pending state is generated. The established association status or the pending association status is written into the target evidence record as the identity association status.
4. The bus-mounted multimodal target detection AI intelligent agent system according to claim 3, characterized in that, The process of reviewing the associated pending status includes: When the identity association status in the target evidence record is the association pending verification status, the target location, target features and motion status of the corresponding target under continuous acquisition time are extracted to form a historical observation sequence. Based on the motion state in the historical observation sequence, the predicted position for the corresponding acquisition time is determined one by one, and the target position is compared with the predicted position. At the same time, the consistency of target features under adjacent acquisition time is compared. When the positional deviation within the continuous verification time window continuously meets the association distance condition and the target feature continuously meets the consistency condition, the association pending verification state is updated to the association established state. When either condition is not met, the association pending verification state is maintained and the verification continues with the data corresponding to the next collection time until the continuous verification time window ends.
5. The bus-mounted multimodal target detection AI intelligent agent system according to claim 1, characterized in that, The processing procedure of the credibility assessment unit includes: Extract the location, category, and identity association status of the same target at the same collection time from the target evidence records; For any data source, the corresponding target location is compared with the target locations of other data sources. A location consistency result is formed based on whether the spatial distance meets the location consistency condition, and a category consistency result is formed based on whether the corresponding categories are the same. When both the location consistency result and the category consistency result meet the consistency condition, and the identity association status is an established association status, the corresponding data source is determined to be a trusted participation status. When either condition is not met, the corresponding data source is determined to be a pending review status, and the trusted participation status or the pending review status is written as the trusted status into the target evidence record.
6. The bus-mounted multimodal target detection AI intelligent agent system according to claim 5, characterized in that, The process of updating the trusted state includes: For data sources in the pending review state, the location consistency result, the category consistency result, and the identity association status are reacquired at the next collection time. When the re-acquired location consistency result and category consistency result both meet the consistency condition, and the identity association status is the association established status, the corresponding data source is updated from the pending review status to the trusted participation status. If any condition is still not met, the pending review status is maintained, and the data from the next collection time is reviewed again until the status update condition is met or the continuous review time window ends.
7. The bus-mounted multimodal target detection AI intelligent agent system according to claim 1, characterized in that, The processing procedure of the conflict lookup unit includes: Extract the location, category, and identity association status of the same target from different data sources from the target evidence records, and compare the deviations between locations, the consistency between categories, and the consistency between identity association statuses respectively. The conflict state is generated when the position deviation exceeds the range of position consistency, category inconsistency, or identity association status inconsistency. Based on the target location and motion state that caused the conflict, the predicted location of the target at the next acquisition time is determined. The back-search range is determined by the acquisition area of the vehicle-mounted camera device, the acquisition area of the lidar and the acquisition area of the millimeter-wave radar corresponding to the predicted location, and the back-search task corresponding to the back-search range is formed.
8. The bus-mounted multimodal target detection AI intelligent agent system according to claim 7, characterized in that, The execution process of the backtracking task includes: Based on the trusted status, determine the data source to be reviewed from the data sources involved in the conflict status determination, and extract the multimodal data corresponding to the target within the review scope; The extracted multimodal data were matched according to the collection time, and the target was rematched based on the location, category, and identity association status. When the rematch results meet the requirements of consistent location, consistent category, and consistent identity association status, the rematch results are written to the target evidence record and the backtracking task ends. When any consistency condition is not met, the conflict state is retained and the backtracking task continues to be executed after waiting for the multimodal data corresponding to the next collection time.
9. The bus-mounted multimodal target detection AI intelligent agent system according to claim 1, characterized in that, The processing steps of the AI agent task scheduling unit include: Read the back-lookup task and vehicle processor load corresponding to the conflict state, determine the conflict level according to the number of location conflicts, category conflicts and identity association state conflicts in each back-lookup task, and form a back-lookup task queue with the conflict level as the first sorting criterion and the generation time of the back-lookup task as the second sorting criterion. When the load of the vehicle processor is lower than the preset load limit, the back-check tasks are sequentially sent to the conflict back-check unit according to the order in the back-check task queue. When the on-board processor load reaches the preset load limit, the unsent back-up tasks are kept in the back-up task queue and are sent out again after the on-board processor load falls below the preset load limit.
10. The bus-mounted multimodal target detection AI intelligent agent system according to claim 9, characterized in that, After the conflict lookup unit completes the first lookup task in the ranking, the AI intelligent agent task scheduling unit removes the completed lookup task from the lookup task queue and reacquires the vehicle processor load. When the back-check task queue is not empty and the load of the vehicle processor is lower than the preset load limit, the back-check task that is reordered and placed first is sent to the conflict back-check unit. When the back-check task queue is not empty and the vehicle processor load reaches the preset load limit, the back-check task queue is maintained and the delivery is paused. Delivery is resumed after the vehicle processor load is lower than the preset load limit. The current callback scheduling ends when the callback task queue is empty.
