Methods, systems, and computer programs for highly reliable runtime detection, classification, and tracking of object actors.

Abstract

The system is provided with a roadside unit (RSU) transceiver equipped with a sensor suite consisting of one or more electro-optic sensors, camera sensors, thermal imaging sensors, lidar sensors, ultrasonic sensors, and / or radar sensors, and capable of receiving authenticated participant Society of Automotive Engineers (SAE) standard J2735 basic safety messages, RTK-corrected GNSS positioning data, and SAE standard J3224 sensor data sharing messages. The system may include one or more processors programmed or configured to receive data from the system's own sensor suite, reporting transceiver, and other network-connected devices, and to construct high-reliability object actor detection, classification, and tracking perception messages representing object actors within the system's field of view for use in other internal system processes, and to transmit them to other external devices for use.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 453,614, filed on March 21, 2023, the content of which is incorporated herein by reference.

[0002] The disclosed subject matter generally relates to methods, systems, and products for human and computer control of vehicles operating in the vicinity of road areas, training of machine learning and / or perception systems of vehicle control systems, and development, verification, and validation of vehicle control systems before (e.g., simulation), during, and after road operation.

Background Art

[0003] A significant proportion of object actors in the road network environment (both vehicles and vulnerable road users) do not participate in V2X. Therefore, without V2X - based cooperative perception, the perception environment model has to rely entirely on sensor - based perception of these object actors. However, establishing the degree of confidence in this perception depends on ground - truth knowledge.

[0004] The premise of this disclosure is that cooperative perception messages (BSM in North America / CAM in Europe) can be used as a basis for establishing ground - truth for processing the perception system. Thus, the deviation between the position reported by the sensor and the RTK - corrected position reported in the cooperative perception message is an accurate and quantifiable measure of ground - truth and serves as a precursor data set for determining the confidence level of the output of the perception system.

[0005] Ground truth proxies are reliable collaborative perceptions that provide confidence levels for both location and object classification. These confidence level fields form part of the basic message structure (collaborative perception message) as defined in SAE J3224. However, without establishing ground truth and incorporating it into the perception systems of participating contributors, the accuracy of perception messages cannot be made reliable, and the message source becomes unreliable even if the message itself is not unusable by the receiver. It is necessary to establish ground truth, incorporate it into the training and validation of perception systems, and then establish it periodically in the runtime operation of the system so that a confidence level for the perception system can be determined. This is especially true for outputs in perception systems concerning participating object actors other than those specified in SAE J3224 / J2735 (e.g., vulnerable road users).

[0006] Current practices in machine learning / computer vision (ML / CV) based collaborative perception involve developing object actor recognition through consensus. Consensus is established through various processes to verify the independent reports of two or more participants regarding the perceived output of each object actor within the field of view. According to SAE J3224, participants should report their confidence in various data fields shared by them. However, there are no widely adopted industry standards for establishing confidence in machine learning / computer vision inference or predictive perception. Current industry and academic practices for establishing confidence levels may rely on some form of measuring the inherent differences or marginal differences in the contributing object actor reports used to generate consensus. Neither incorporating objective ground truth nor using ground truth to establish reliable perception and measure confidence in perceived output has been done.

[0007] Determining the confidence level of the output of the perceptual system is essential for collaborative cognition. The system described below not only constructs an established ground truth to provide reliable perception, but also enables reliable collaborative cognition by measuring the confidence of the output of the system's perceptual system.

[0008] Uncertainty and reliability: Attitude refers to the position and velocity state of an object or actor at a given time for a given sample. A certain level of uncertainty exists when vehicles equipped with advanced driver-assistance systems (ADAS) and automated driving systems (ADS) are in motion. The nature of ADAS and ADS vehicles in motion inherently introduces uncertainty in the measurement of both the vehicle's own attitude and the attitude of the object actors they sense and perceive. Attitude states can change over time as ADAS / ADS sense and process sensor data to perceive and perform attitude measurements. This uncertainty is combined with the overall uncertainty of ADAS / ADS sensors, which acquire a changing background with each data acquisition cycle, and a moving vehicle presents its sensors with new pixels or equivalent sensor discrete rendering modalities with each cycle. The relationship between moving object actors and static environments also introduces inherent uncertainty. In addition to attitude, uncertainty is present in the presence of object actors in the field of view, the type / instance of object actors, and the detection and classification processes for establishing the follow-on placement or attitude state of object actors. Assuming uncertainty exists, an ADS / ADAS system should have both means of determining its own confidence level in its own system perception and means of receiving the confidence level indicated when the system consumes perception reports from sources outside of its system. Confidence and indicated confidence provide a constructive framework for the system to process commands and decisions for safety and driving tasks. In the case of collaborative perception shared among two or more ADS / ADAS or V2X participants, an objective method for determining "confidence level" and "indicated confidence level" is required to meaningfully utilize reports from others.

[0009] Basic safety message / collaborative recognition message: The SAE J2735 Basic Safety Message (BSM), or the European equivalent Cooperative Awareness Message (CAM), is a standardized data object encapsulated in an abstract syntactic notation-based message format traded to provide situational awareness in road operating environments. The Roadside Unit (RSU) / Onboard Unit (OBU) ecosystem receives real-time kinematic GNSS positioning updates to ensure accurate global coordinate localization for system use in position reporting.

[0010] The situational awareness data shared in BSM / CAM self-reports the vehicle's status 10 times per second (10Hz) and includes data attributes such as latitude, longitude, altitude, coordinate accuracy, speed, direction of travel, steering angle, acceleration, brake system status, and vehicle length and width. BSM / CAM information digested by advanced driver assistance systems (ADAS) and / or automated driving systems (ADS) enables a more reliable object perception process that considers enhanced road situation awareness, out-of-line recognition and predictive behavior detection, and infrastructure that reports third-party vehicles and their respective objective coordinate positions and dimensions. The benefits of this enhanced perception include improved safety and vehicle mobility efficiency.

[0011] BSM / CAM transactions are conducted in one-to-many (broadcast) mode via V2X wireless communication protocols such as IEEE 1609, also known as WAVE (Wireless Access Vehicular Environment), but are not limited to this, and the media access control (MAC) layer is IEEE 802.11p or PC5 (i.e., C-V2X). BSM / CAM can be shared both peer-to-peer or "vehicle-to-vehicle" (V2V) or vehicle-to-infrastructure (V2I), with the extension of V2I being vehicle-to-network (V2N). The communication ecosystem for V2V, V2I, and V2N is commonly referred to as vehicle-to-everything or "V2X".

[0012] To exchange BSM / CAM, participating mobile object actors are equipped with radio frequency transceiver modems called Onboard Units (OBUs). Fixed infrastructure radio frequency transceivers are called Roadside Units (RSUs). The IEEE 1609.2 component of the WAVE stack uses SCMS certificates (i.e., "credentials") to ensure the security and authentication of data traded within the ecosystem, thereby enabling RSU / OBU units to determine the trustworthiness of the BSM / CAM sender and, consequently, the trustworthiness of their messages. Devices compliant with IEEE 1609.2 and provisioned with SCMS credentials, i.e., RSUs and OBUs, are called End Entities (EEs).

[0013] Security credential management system: SCMS is an instantiation of Public Key Infrastructure (PKI), which generates and delivers certificates to the EE to provide trust and assurance for messages broadcast by the EE (SAE J3275 and SAE J3224 messages encapsulating sensor data for in-vehicle ADAS / ADS). SCMS can invalidate credentials if it cannot establish or maintain trust. Each EE participant receives SCMS credentials, which are used to sign issued messages, read by recipients, and establish the reliability of the message's reporting source. The SCMS system enables the EE to incorporate BSM / CAM and Sensor Data Sharing Messages (SDSM) into its operational picture with greater reliability than would be obtained from messaging without credentials.

[0014] Sensor data sharing message: The SAE J3224 defined Sensor Data Sharing Message (SDSM) is a standardized text object message format for sharing object actor detection and classification between participant RSU and OBU-equipped vehicles. Under the J3224 standard, participants form a Sensor Sharing Service System (SSS System). Participant vehicles, equipped with an OBU and corresponding GNSS receivers, report object actor detection and classification in SDSM format, receiving GNSS positioning data and RTK corrections applied to sensors such as LiDAR, Radar, and cameras, along with corresponding calculations for processing the sensor data. Near the SSS system, participant RSU and OBU-equipped vehicles may send and receive SDSMs, which may be used in downstream systems to enhance and confirm their own road action awareness. To utilize an SDSM transmitted by another participant, the sender must indicate a level of confidence to provide a means for the receiver to determine the weighting of the message data when incorporating it into their own worldview.

[0015] Ground Truth: Ground truth is a term that describes information known to be real or true through direct observation and measurement. Ground truth is the confirmed known location of an object actor at the same time that the object actor is detected in a sensor data capture cycle (e.g., a frame or item in a data stream). Ground truth may be established in the actual execution process, where RTK GNSS reports from detected object actors occur actively at the same time as sensor capture on the detected object actor. Ground truth may also be established post-execution, in which case the location is subsequently measured or determined and attached to the corresponding sensor capture data frame or stream. An example of a post-execution ADAS / ADS perception system requires ground truth data for the development of a perception model and operates with high reliability when ground truth can be provided during runtime operation. The perception model is trained and validated by providing ground truth sensor data in which object actors are a priori detected and classified. During actual runtime operation, the perception model compares the runtime sensor data with the runtime known ground truth on object actors in the operating environment.

[0016] High reliability: High reliability is defined as the accuracy of object actor presence, classification, position, velocity, and acceleration being determined to be "high" in post-event analysis that can determine ground truth, even without direct measurement of ground truth. Through statistical analysis, runtime reliability is judged to be high if the system producing the evaluated output meets a statistically significant threshold of agreement with or approaching ground truth, and / or if, based on pre-set parameters, other participants' object actor reports agree with the system's own predictions within a statistically significant threshold. If ground truth data is provided periodically and / or occasionally, the system can self-determine its confidence levels for multiple parameters, making it possible to measure the difference between perceptual inferences and ground truth during system runtime. A system that performs confidence checks in the following manner consistently maintains high reliability on a binary state basis: (A) high confidence in the perceptual inferences the system is generating, or (B) high confidence that the system is not generating perceptual inferences of high reliability. These two high-confidence states are maintained across multiple perceptual and ground truth parameters, such as object actor category type, position, velocity, and acceleration, and measurements are performed based on these. [Overview of the project] [Problems that the invention aims to solve]

[0017] Accordingly, the subject matter of this disclosure is to provide a method, system, and computer program product for establishing highly reliable detection, classification, and tracking perception outputs on object actors using J2735 BSM / CAM and J3224 SDSM messages, RTK-corrected GNSS positioning data, and sensor data. The method, system, and computer program can reside on a host-based system, which functions as a physical platform for sensors and calculations that outputs sensor data of the perceived road environment in the vicinity of the host system. [Means for solving the problem]

[0018] These and other advantages are achieved, for example, by a system including one or more processors programmed or configured to perform the steps of (i) receiving data related to road environment object actor sensor-based detection and classification prediction, IEEE 1609 standard security credential management system (SCMS) cryptographic signature SAE J2735 standard basic safety message (BSM) / cooperative recognition message (CAM), and IEEE 1609 standard SCMS cryptographic signature SAE J3224 standard sensor data sharing message (SDSM); and ii) determining runtime detection, classification, and tracking of object actors in the road operating environment with respect to a system relative coordinate reference map.

[0019] When determining the detection, classification, and tracking of an object actor, one or more processors (iii) extract object actor-related data from the received SCMS cryptographic signature SAE J2735 participant broadcast basic security message / cooperative recognition message and then determine the orientation of the object actor, wherein the received data includes, but is not limited to, the reported real-time kinematic GNSS corrected position of the object actor, spatial dimensions (e.g., length × width × height), and the category type classification of the object actor, and the BSM / CAM extracted data is converted into a format for storing and retrieving the object actor represented as an artifact in a system relative coordinate reference map, and (iv) the received SCMS cryptographic signature SAE (v) Steps of extracting object actor-related data from a J3224 participant broadcast sensor data sharing message (SDSM), wherein the received data may include the location of the sensed object actor, classification of the category type, observation timestamp, and external observer reports of the location in GNSS-based global coordinates, and the SDSM extracted data is converted into a format for storing and retrieving the represented object actor as an artifact in a system relative coordinate reference map; and (v) Steps of constructing a representative object actor artifact in a system relative coordinate reference map, wherein the coordinate system includes a data structure and road environment Persistent and / or transient data representing features and object actors are related to the host base system in a normalized reference unit format associated with the current position of the sensing system and / or the reference frame, and the object actors represented in the reference map are data artifacts generated from sensor-based predictive perception-related output data, SDSM-based message data, or BSM / CAM-based message data, which include position and occupancy attributes normalized to the base system reference coordinate system, and in addition to applicable object actor artifacts, a priori and / or runtime mapped and sensed road environment features are also represented, step,(vi) A step of predicting an object actor present in the field of view of one or more sensors of the system, wherein the output of the prediction includes predicted detection and classification of the presence, location, spatial dimensions, and category type of the object actor associated with received sensor data, the received sensor data being associated with one or more cameras, lidars, thermal, or radar sensors, and the prediction is based on the output of one or more machine learning models that predict the presence, location, category type, and geometric dimensions of the object actor, and the predicted object actor output is sent to the system relative coordinate reference map as a predicted object actor artifact; (vii) A step of determining the presence of an actual object actor in the road environment associated with a representative object actor artifact in the system relative coordinate reference map, wherein one or more procedures and / or machine learning models determine that two or more artifacts correspond to the same object actor; (viii) A step to determine the confidence level for object actor predictive detection, classification, and tracking, wherein the accuracy of aggregated weighted predictions of object actor detection, classification, and tracking over the course of previous and current runtime outputs using available BSM / CAM-based object actor data artifacts is measured to determine the error between the prediction and the actual detection, classification, and tracking, the measured accuracy is retained in system memory for reference in subsequent runtime operations, the measured accuracy established from artifacts in the received BSM / CAM database is determined as an objective comparison for measuring internally generated predictive perception, externally generated SDSM, and aggregated combinations thereof against actual detection and classification, as well as tracking parameter values, the system determines confidence intervals based on priori and runtime-updated accuracy thresholds, the relevant thresholds consist of parameter value ranges determined to satisfy statistically significant accuracy requirements indicating low uncertainty regarding the predicted values ​​and that the underlying predicted values ​​are worth using for perception runtime use in a road environment, and the output is,(ix) A confidence interval for the current reconciled state parameter values ​​of a crossmatched object-actor, which includes a high confidence indicator for predicted object-actor-related state values ​​or a high confidence indicator that the system's perception cannot detect, classify, or track object-actors within a threshold range; and (ix) a step of sustaining the matched object-actor artifact as ground truth or high confidence determination object-actor in the system relative reference coordinate map, which is then logged with crossmatched and reconciled data, including confidence attributes related to basic safety messages / cooperative recognition messages, sensor data sharing messages, and road environment sensor perception data.

[0020] At least one of the two or more artifacts is associated with the received BSM / CAM data, and based on this, one or more algorithms and / or machine learning models establish the existence of a relationship between the two or more object actor representative artifacts. A pair or set of matched artifacts, including a BSM / CAM-based artifact, is reduced to a ground truth object actor in a system reference map, and the match reference is applied to the cross-matched artifacts to construct a ground truth object actor artifact that exists in an applicable state in the system relative coordinate reference map, and if a sensor-related object actor artifact is not identified as a match with the BSM / CAM artifact, the BSM / CAM is established as a ground truth perceptual object actor.

[0021] If no BSM / CAM-related artifacts exist, but representative object artifacts exist based on predicted perceptual data and / or SDSM messages, a module having one or more algorithms and / or machine learning models measures the artifact parameters relative to each other with respect to the system relative coordinate system map, establishes the existence of a match between two or more object actor representative artifacts, determines the confidence level of the crossmatch between the two or more object actor artifacts, and the parameters used to determine the crossmatch include, but are not limited to, the object actor classification category type, position, velocity, and acceleration.

[0022] Using ground truth, or otherwise predicted by a matched perception process, or reported by SDSM, precursor artifacts used to determine matching are removed from the reference map, and the output of matched artifacts with the corresponding map update is an aggregated and matched collection of runtime-determined ground truth and high-confidence detection, classification, and tracking of object actors in the road environment, and the system relative coordinate reference map is updated to reflect the determined state for subsequent system runtime use and / or external transmission of ground truth data and high-confidence base data.

[0023] One or more processors are further programmed or configured to perform the step of determining the current runtime confidence interval based on the current measurable environmental conditions of the current runtime and the frequency, accuracy, and timeliness of BSM / CAM-based accuracy measurements performed prior to this determination, wherein the determination relates to the measured performance of matching predicted object actor outputs and BSM / CAM-related ground truth outputs, the outputs being the difference between actual object actor state attributes including data and predicted object actor state attributes, and the outputs being compared with time and condition weight data to determine the confidence interval for each state attribute.

[0024] One or more processors are further programmed or configured to perform steps of determining a current adjustment factor and associating this with run-time and current environmental conditions and the system's measurement accuracy performance, wherein the adjustment factor is time-stamped, and the time-stamped factor is used to determine a damping factor that adjusts a confidence interval related tolerance based on the statistical significance of historical uncertainty introduced beyond the moment of an immediate ground-truth versus predicted collation, and the output is used to generate internal data that is the basis for determining system reliability in a perception decision.

[0025] One or more processors are further programmed or configured to perform steps of constructing a high-reliability perception report for use in both internal and external systems, wherein the output of the external system is transmitted via one or more network devices or radio frequency transceivers.

[0026] One or more processors are further programmed or configured to perform steps of storing sensor data, collated ground-truth, and high-confidence perception outputs together with a time stamp, wherein the time stamp associates the collated perception output as a labeled feature within the corresponding logged sensor data, and the labeled logged data collection provides machine learning model training material for training a priori machine learning models and constructing subsequent high-confidence perception models.

[0027] These advantages and other advantages are achieved, for example, by the method of steps (i) to (ix) above.

[0028] These advantages and other advantages are further achieved, for example, by at least one non-transitory computer-readable medium storing at least one computer program product including one or more instructions that cause at least one processor to perform steps (i) to (ix) above. [Brief explanation of the drawing]

[0029] Preferred embodiments described herein and illustrated by the following drawings are included not to limit the invention, but to illustrate it, and similar titles indicate similar elements.

[0030] [Figure 1] A non-limiting, exemplary system overview is provided. In another embodiment of the system shown in Figure 1, the RSU transceiver can be replaced with an OBU transceiver. [Modes for carrying out the invention]

[0031] The following detailed description is essentially illustrative and is not intended to limit the embodiments or uses and applications of the embodiments described. All implementations described below are illustrative implementations provided to enable those skilled in the art to create or use embodiments of the disclosure and are not intended to limit the scope of the disclosure as defined by the claims. It should also be understood that the drawings included herein provide only schematic representations of currently preferred structures of the invention, and that structures within the scope of the invention may include structures different from those shown in the drawings.

[0032] The disclosed invention provides a method, system, and computer program product for establishing highly reliable detection, classification, and tracking perception outputs on an object actor using J2735 BSM / CAM and J3224 SDSM messages, RTK-corrected GNSS positioning data, and sensor data. The method, system, and computer program can reside on a host-based system, which functions as a physical platform for sensors and calculations that outputs sensor data of the sensed road environment in the vicinity of the host system.

[0033] Referring to Figure 1, an overview of a non-limiting exemplary system is shown. In another embodiment of the disclosed invention, the RSU transceiver 402 shown in Figure 1 may be replaced by an OBU transceiver. The RSU 402 is equipped with either a direct RTK receiver or an RTK correction update via a network interface that transmits RTK corrections to the RSU (408). The system receives a Real-Time Kinematic Positioning (RTK) error correction 400 for Global Navigation Satellite System (GNSS) positioning. The RTK correction update provides high accuracy to the position data, which the RSU transceiver 402 may convert into an SAE J2735 RTCM (Radio Technical Commission for Maritime Services) message 404, broadcast within the ecosystem, and used by the OBU 405 to apply the correction to the position data in their BSM / CAM. Alternatively, the OBU 405, equipped with an RF receiver 403, may directly acquire the RTK correction 400.

[0034] The IEEE 1609.2 module 403, integrated into the OBU405, signs outgoing BSM / CAM and SDSMs using Security Credential Management System (SCMS) credentials. The equivalent module in the RSU402 verifies the received BSM / CAM and SDSMs. This ensures that only authenticated participants can report BSM / CAM and SDSMs, thereby preventing unauthorized participants from providing false data to the system and providing shared perceptual participants with assurance of the integrity of BSM / CAM data and subsequent system SDSM broadcasts.

[0035] The System RSU radio transceiver module 402 receives both BSM / CAM and SDSM 401 from the reporting participant, and the IEEE1609.2 module 406 verifies the received messages before passing the message data to the J2735 / J3224 message extraction, transformation, and load (ETL) module 410. The System Incoming BSM / CAM ETL module 410 extracts the BSM / CAM and / or SDSM data, transforms the provided position and spatial dimension data into system relative coordinates, and constructs object actor data artifacts. The ETL module 401 loads the BSM / CAM and SDSM-related artifacts, along with the reported object actor category type classification and position observation timestamps, into the system relative coordinate reference map 500, as indicated by the reporting J2735 / J3224 participant. The BSM / CAM-related artifacts, rather than SDSM artifacts, within Reference Map 500 form the basis for the ground truth determination, matching, and correlating processes for object actor artifacts predicted by the system's perception system and SDSM-related object actor artifacts reported by participants.

[0036] In some embodiments, the system may consume a sensor data stream, which may include, but is not limited to, one or more of the camera, lidar, radar, and thermal spectral modalities 101a, 101b, 101c, and 101d. The sensor data passes through perception models for each respective sensor spectral type 200, 201, 202, and 203 and is jointly supplied to the sensor fusion processor module 204. The sensor fusion preprocessor module 204 transmits the sensor stream, which has been formatted and combined for use by the fusion stream perception model set 205. Each respective sensor perception module performs object detection and classification using a priori constructed calibrated algorithm model set.

[0037] Independent of and simultaneously with BSM / CAM and SDSM message processing, the system's perception module processes sensor data to predict the detection and classification of object actors. In some embodiments, the system's sensor suite may be calibrated on a fixed, unfolded road area. Calibration data is input as a base reference layer in the system relative coordinate reference map 500. Known points and coordinates for the sensors remain within the fixed reference field of view of each sensor. All static or other persistent object actors within the field of view of the sensor suite may be assigned pixel and / or voxel spatial system classifications with relative and objective (e.g., GNSS global) map locations.

[0038] In some non-limiting embodiments, the system's perceptual models 200, 201, 202, 203, and 205 may provide a pixel (or respective modality unit) pre-mapping configuration that aligns received sensor data pixels with measured pixel fields (or equivalent sets of sensor data units) of a system relative coordinate reference map. The pre-mapped pixel fields assign known static occupied pixel voxel spaces. Changes between known pixel pre-mapped pixel fields indicate changes to the environment and / or object actors occupying the space in front of the a priori measured pixels. The perceptual model, with a pre-processed set of pixel data fields corresponding to the system relative coordinate reference map, may use the pixel occupancy change input to infer the presence of object elements. Combined with adjacent pixel deltas, the model set provides an augmented dataset beyond the raw sensor-captured data to be used to predict the detection and classification of object actors (along with corresponding position inference for placement on the system relative coordinate reference map 500).

[0039] Independent of the sensor-specific perception modules, the sensor fusion processor module merges sensor data into a multispectral dataset frame stream 304 for use by the fusion stream perception model 205 to perform detection and classification. Detection performed by perception models 200, 201, 202, 203, and 205 is the process of determining the presence of discrete object actors within the sensor's field of view and the corresponding output. Classification performed by perception models 200, 201, 202, 203, and 205 is the process of determining the type, class, and / or instance of detected object actors (e.g., sedan, tractor trailer, SUV, pedestrian, etc.) and the corresponding output. The outputs of the sensor-specific perception models 300, 301, 302, and 303 and the fusion stream perception model set 205 are fed to the aggregate confidence voting module 501 for matching and pruning the predicted detection and classification outputs. The aggregate confidence voting module processes internal predictive perception based on the accuracy of the multiple sensor modality perception models and the agreement or disagreement of the fusion perception model set. In some non-limiting embodiments, this process may occur before the system determines the overall system confidence level based on its own perception.

[0040] The system relative coordinate reference map 500 maintains both object actor artifacts associated from external participants and object actor artifacts predicted by the system's internal perception model. The system relative coordinate reference map 500 includes a data structure relating to the host base system in the format of normalized reference units of measurement related to the current position and / or reference frame of the sensing system, with persistent and / or transient data artifacts representing object actors and other data items representing road environment features. The map 500 hosts runtime generated artifacts representing either predicted object actors or BSM / CAM reported object actors from the system predictive perception processes in perception-related modules 204, 205 and the aggregate confidence voting module 501 and BSM / CAM ETL module 410, respectively. The generated artifacts include position and occupancy attributes normalized to the base system reference coordinate system included in the map. In some non-limiting embodiments, the map may host a priori determined and / or runtime mapped and perceived road environment features.

[0041] The perceptual matching module 502, through an algorithm and / or model set, performs the matching and matching process for object actor artifacts. The module references artifacts, performs measurements, and compares the artifacts to criteria including object actor geometry, category type classification, orientation, and tracked orientation over a process of multiple consecutive sample periods. The module calculates cross-matching between available artifacts from artifact sources in three different databases (BSM / CAM-based artifacts, SDSM-based artifacts, and predicted perceptual object actor-based artifacts). Thus, orientation is defined as the object actor's position, velocity (speed and direction of motion), and spatial occupancy relative to the system relative coordinate system that constitutes the base of the reference map 500. The perceptual matching module 502 process measures the variance between multiple object actor artifacts representing the same object actor. The module uses the measurements to cross-match the represented object actors. The sequence of module measurements, combined with the count of contributing artifacts, provides additional criteria for determining the reliability of the matched artifacts.

[0042] In some non-limiting embodiments, if BSM / CAM artifacts are available, matched matching establishes which sensor-detected object actor artifacts and SDSMs are the same object actor within the BSM / CAM-related object actor artifacts. BSM / CAM-based matched artifact groups are established, having ground truth where corresponding artifacts are replaced by multiple artifacts representing the object actor. If BSM / CAM is not available, the perceptual matching module 502, with perceptual predictions of the system itself, uses one or more J3224 message inputs to identify participant-reported object actor artifacts indicating agreement on the object actor's location, geometric dimensions, and classification.

[0043] The perceptual matching module 502 performs object velocity tracking by processing two or more consecutive established object actor detection and classification-based artifacts populated on a system relative coordinate reference map, determining the velocity by determining the change between map occupancy. Next, the perceptual matching module 502 performs acceleration tracking by processing the change between two or more consecutive velocity tracking data items.

[0044] The perceptual matching module 502 overcomes detection uncertainties introduced by different participants or the system's own perceptual modules through the match and prune process, which can lead to variability in object actor detection decisions. The perceptual matching module 502 overcomes classification uncertainties introduced by different participants or the system's own perceptual modules through the match and prune process, which can lead to variability in object actor detection decisions. Once determined high-confidence detections and classifications are provided, the perceptual matching module 502 tracks sequential updates to determine velocity and acceleration tracks on the object.

[0045] The perceptual matching module 502 determines the runtime confidence level by measuring the difference between predicted detected, classified, and tracked object actor artifacts, SDSM-based artifacts, and BSM / CAM-based artifacts. Module 502 uses BSM / CAM messages as base artifacts, where available, to establish a base ground truth reference for cross-matching predicted and SDSM-based artifact determinations. Module 502 sorts the data in captured timestamp order. It then measures the error between predicted artifact values ​​and BSM / CAM-based actual values ​​for detection, classification, and tracking parameters, including but not limited to object actor category type, location, dimensions, velocity, and acceleration. The measured error differences are summed by the unit of measurement for each parameter. The module processes the parameter value differences through a weighted decay function to compute an overall view of system perceptual output accuracy. Using the determined accuracy values, the system provides an indicator confidence level based on the total number of ground truth validation predictions made during the runtime sample period and the difference between ground truth and system predictions. These confidence levels may be presented as aggregated weights, or as individual components with indicated confidence levels, such as specific measurements and weighted confidence levels for position, velocity, acceleration, etc.

[0046] Furthermore, the perceptual matching module 502 may use accumulated confidence checks to process the indicated confidence levels for additional parameters, including but not limited to road environment behavior design domains, such as usage area, time of day, weather conditions, traffic speed, road network feature type, traffic signal status, and combinations thereof. The module maintains a timer for each indicated confidence level parameter from the last received ground truth check. When no BSM / CAM received messages are received during the recent sample period for the system to perform confidence checks, the module updates the system confidence in its own indicated confidence level determination process through a time or event decay function that adjusts the system confidence level. Through this process, the system will maintain high confidence in system perceptual inference, or under the condition that the system is actually highly confident in not generating highly confident perceptual inferences.

[0047] Using BSM / CAM, which is available for both runtime processing and offline model / algorithm training and refinement, the matching module combines the matching and collated object actor detection, classification, and tracking recognition processes with the runtime confidence level determination process. This allows the matching module to output a statistically significant confidence index for subsequent runtime use in the system's own perceptual data processing, and to transmit the confidence index to other systems outside the system.

[0048] The perceptual matching module formats object actor classification, position, velocity, and acceleration data into a high-reliability perceptual data message 504 and transmits it to the Subscriber internal system process 600 and system network interface 601, which include a system relative coordinate reference map 500, thereby enabling the message to be transmitted to external recipients such as participant ADAS-equipped vehicles and automated driving system vehicles.

[0049] The system sensor and perception logger 503 record runtime operation data for analysis and subsequent model improvement and continuous refinement.

[0050] A priori model sets - algorithm training and calibration: The system's perceptual models 200, 201, 202, 203, 205 and perceptual matching module 502 are trained and tuned on ground truth model training data obtained from J2735 BSM / CAM messages 406 of participant vehicles and object actors operating in the vicinity of the installation system during calibration. RTK corrected position reports 409 are annotated with log data about participant vehicles and other object actors through the matching process. The system uses these BSM / CAM messages as ground truth object actors with known locations (e.g., object type: vehicle-SUV; geometry box dimensions: width 2.2m, height 2.3m, length 4.2m; location: latitude 32.553385°, longitude -96.822104°, height 197.815m). The BSM / CAM message reports are converted into a format that places the known object actors on the system reference coordinate system map. Known object actors are cross-matched to the system perception module outputs that are simultaneously populated on the system reference coordinate system map. The cross-matching feedback loop allows reviewers to evaluate the performance of the system's perception system and support further improvements in perception tasks.

[0051] Since many modifications, variations, and alterations can be made to the preferred embodiments described herein, all matters shown in the foregoing description and accompanying drawings are intended to be interpreted as illustrative rather than restrictive. Accordingly, the scope of the invention should be determined by the accompanying claims and their legal equivalents.

Claims

1. A system comprising one or more processors, wherein the one or more processors are Steps include receiving data related to road environment object actor sensor-based detection and classification prediction, IEEE 1609 standard security credential management system (SCMS) cryptographic signature SAE J2735 standard basic safety message (BSM) / cooperative recognition message (CAM), and IEEE 1609 standard SCMS cryptographic signature SAE J3224 standard sensor data sharing message (SDSM), A step of determining the runtime detection, classification, and tracking of object actors in a road operating environment related to a system relative coordinate reference map, Programmed or configured to do, For the step of determining the detection, classification, and tracking of object actors, one or more processors: A step of extracting object actor-related data from received SCMS cryptographic signature SAE J2735 participant broadcast basic security message / cooperative recognition message, and then determining the orientation of the object actor, wherein the received data includes, but is not limited to, the spatial dimensions of the reported object actor, including real-time kinematic GNSS corrected position, length, width and / or height, and the category type classification of the object actor, and the BSM / CAM extracted data is converted into a format for storing and retrieving the object actor represented as an artifact in a system relative coordinate reference map, Steps include: extracting object actor-related data from a received SCMS cryptographic signature SAE J3224 participant broadcast sensor data sharing message (SDSM), wherein the received data may include the location of the sensed object actor, its category type classification, observation timestamp, and an external observer's report of its location in GNSS-based global coordinates, and the SDSM-extracted data is converted into a format for storing and retrieving the object actor represented as an artifact in a system relative coordinate reference map; A step of constructing representative object actor artifacts in a system relative coordinate reference map, wherein the coordinate system includes a data structure, and persistent and / or transient data representing road environment features and object actors are related to the host base system in a normalized reference unit format related to the current position of the sensing system and / or the reference frame, and the object actors represented in the reference map are data artifacts generated from sensor-based predictive perception-related output data, SDSM-based message data, or BSM / CAM-based message data, which include position and occupation attributes normalized with respect to the base system reference coordinate system, and in addition to applicable object actor artifacts, a priori and / or runtime mapped and sensed road environment features are also represented, A step of predicting object actors present within the field of view of one or more sensors of the system, wherein the output of the prediction includes predicted detection and classification of the presence, location, spatial dimensions, and category type of object actors related to received sensor data, the received sensor data being associated with one or more cameras, lidars, thermal, or radar sensors, the prediction being based on the output of one or more machine learning models predicting the presence, location, category type, and geometric dimensions of object actors, and the predicted object actor output being sent to the system relative coordinate reference map as predicted object actor artifacts, A step of determining the presence of an actual object actor in the road environment associated with a representative object actor artifact in the system relative coordinate reference map, wherein one or more procedures and / or machine learning models determine that two or more artifacts correspond to the same object actor. At least one of the two or more artifacts is associated with the received BSM / CAM data, and based thereon, one or more algorithms and / or machine learning models establish the existence of a relationship between the two or more object actor representative artifacts, and a pair or set of matched artifacts, including BSM / CAM-based artifacts associated with the received BSM / CAM data, is reduced to a ground truth object actor in the system reference map, the match reference is applied to the cross-matched artifacts to construct a ground truth object actor artifact that exists in an applicable state in the system relative coordinate reference map, and if the sensor-related object actor artifact is not identified as a match with the BSM / CAM artifact, the BSM / CAM is established as a ground truth perception object actor. If BSM / CAM-related artifacts are absent, but representative object artifacts based on predicted perceptual data and / or SDSM messages exist, a module having one or more algorithms and / or machine learning models measures the artifact parameters relative to each other with respect to the system relative coordinate system map, establishes the existence of a match between two or more object actor representative artifacts, determines the confidence level of the crossmatch between the two or more object actor artifacts, and the parameters used to determine the crossmatch include, but are not limited to, object actor classification category type, position, velocity, and acceleration. A step to determine the confidence level for object actor predictive detection, classification, and tracking, wherein the accuracy of aggregated weighted predictions of object actor detection, classification, and tracking across the process of previous and current runtime output using available BSM / CAM-based object actor data artifacts is measured, the error between the prediction and the actual detection, classification, and tracking is determined, the measured accuracy is retained in system memory for reference in subsequent runtime operations, and the measured accuracy established from the received BSM / CAM database artifacts is used as an objective ratio to measure internally generated predictive perception, externally generated SDSM, and aggregated combinations thereof against the actual detection and classification, as well as tracking parameter values. The system determines confidence intervals based on priori and runtime-updated accuracy thresholds, the relevant thresholds comprising a range of parameter values ​​determined to satisfy statistically significant accuracy requirements indicating low uncertainty regarding predicted values ​​and worthy use of underlying predicted values ​​for perceptual runtime use in a road environment, the output being a confidence interval for the current matched state parameter values ​​of the cross-matched object actor, which includes a high confidence indicator for predicted object actor-related state values ​​or a high confidence indicator that the system's perception cannot detect, classify, or track object actors within the threshold range, and the steps are as follows: Using ground truth, or otherwise predicted by a matched perception process, or reported by SDSM, precursor artifacts used to determine matching are removed from the reference map, and the output of matched artifacts with the corresponding map update is an aggregated and matched collection of runtime-determined ground truth and high-confidence detection, classification, and tracking of object actors in the road environment, and the system relative coordinate reference map is updated to reflect the determined state for subsequent system runtime use and / or external transmission of ground truth data. The steps include: maintaining the matched object actor artifact as ground truth or high confidence determination object actor in the system relative reference coordinate map, which is then logged with cross-matched and matched data including basic safety messages / collaborative recognition messages, sensor data sharing messages, and confidence attributes related to road environment sensor perception data; A system that is programmed or configured to perform a certain action.

2. The one or more processors described above are: The system according to claim 1, further programmed or configured to perform the step of determining the current runtime confidence interval based on the current measurable environmental conditions at the current runtime and the frequency, accuracy, and timeliness of BSM / CAM-based accuracy measurements performed prior to the determination, wherein the determination relates to the measured performance of matching predicted object actor outputs and BSM / CAM-related ground truth outputs, the outputs being the difference between actual object actor state attributes including data and predicted object actor state attributes, and the outputs being compared with time and condition weight data to determine the confidence interval for each state attribute.

3. The one or more processors described above are: The system according to claim 1, further programmed or configured to perform the steps of determining a current adjustment factor and relating it to runtime and current environmental conditions and the measurement accuracy performance of the system, wherein the adjustment factor is timestamped, the timestamped factor is used to determine a decay factor that adjusts the confidence interval-related tolerance based on the statistical significance of historical uncertainty introduced beyond the instant of immediate ground truth versus predictive matching, and the output is used to generate internal data that forms the basis for determining system reliability in perceptual determination.

4. The one or more processors described above are: The system according to claim 1, further programmed or configured to perform the step of constructing a highly reliable perception report for use in both an internal and an external system, wherein the output of the external system is transmitted via one or more network devices or radio frequency transceivers.

5. The one or more processors described above are: The system according to claim 1, further programmed or configured to perform the steps of storing sensor data and matched ground truth and high confidence perceptual outputs together with timestamps, wherein the timestamps associate the matched perceptual outputs with labeled features in the corresponding logged sensor data, and the labeled logged data collection provides machine learning model training material to train a machine learning model a priori and build a subsequent high confidence perceptual model.

6. Steps include receiving data related to road environment object actor sensor-based detection and classification prediction, IEEE 1609 standard security credential management system (SCMS) cryptographic signature SAE J2735 standard basic safety message (BSM) / cooperative recognition message (CAM), and IEEE 1609 standard SCMS cryptographic signature SAE J3224 standard sensor data sharing message (SDSM), A step of determining the runtime detection, classification, and tracking of object actors in a road operating environment related to a system relative coordinate reference map, A method including, The steps for determining the detection, classification, and tracking of object actors are: A step of extracting object actor-related data from received SCMS cryptographic signature SAE J2735 participant broadcast basic security message / cooperative recognition message, and then determining the orientation of the object actor, wherein the received data includes, but is not limited to, the spatial dimensions of the reported object actor, including real-time kinematic GNSS corrected position, length, width and / or height, and the category type classification of the object actor, and the BSM / CAM extracted data is converted into a format for storing and retrieving the object actor represented as an artifact in a system relative coordinate reference map, Steps include: extracting object actor-related data from a received SCMS cryptographic signature SAE J3224 participant broadcast sensor data sharing message (SDSM), wherein the received data may include the location of the sensed object actor, its category type classification, observation timestamp, and an external observer's report of its location in GNSS-based global coordinates, and the SDSM-extracted data is converted into a format for storing and retrieving the object actor represented as an artifact in a system relative coordinate reference map; A step of constructing representative object actor artifacts in a system relative coordinate reference map, wherein the coordinate system includes a data structure, and persistent and / or transient data representing road environment features and object actors are related to the host base system in a normalized reference unit format related to the current position of the sensing system and / or the reference frame, and the object actors represented in the reference map are data artifacts generated from sensor-based predictive perception-related output data, SDSM-based message data, or BSM / CAM-based message data, which include position and occupation attributes normalized with respect to the base system reference coordinate system, and in addition to applicable object actor artifacts, a priori and / or runtime mapped and sensed road environment features are also represented, A step of predicting object actors present within the field of view of one or more sensors of the system, wherein the output of the prediction includes predicted detection and classification of the presence, location, spatial dimensions, and category type of object actors related to received sensor data, the received sensor data being associated with one or more cameras, lidars, thermal, or radar sensors, the prediction being based on the output of one or more machine learning models predicting the presence, location, category type, and geometric dimensions of object actors, and the predicted object actor output being sent to the system relative coordinate reference map as predicted object actor artifacts, A step of predicting object actors present within the field of view of one or more sensors of the system, wherein the output of the prediction includes predicted detection and classification of the presence, location, spatial dimensions, and category type of object actors related to received sensor data, the received sensor data being associated with one or more cameras, lidars, thermal, or radar sensors, the prediction being based on the output of one or more machine learning models predicting the presence, location, category type, and geometric dimensions of object actors, and the predicted object actor output being sent to the system relative coordinate reference map as predicted object actor artifacts, At least one of the two or more artifacts is associated with the received BSM / CAM data, and based thereon, one or more algorithms and / or machine learning models establish the existence of a relationship between two or more object actor representative artifacts, and a pair or set of matched artifacts including the BSM / CAM-based artifact is reduced to a ground truth object actor in the system reference map, and the match reference is applied to the cross-matched artifacts to construct a ground truth object actor artifact that exists in an applicable state in the system relative coordinate reference map, and if the sensor-related object actor artifact is not identified as a match with the BSM / CAM artifact, the BSM / CAM is established as a ground truth perception object actor. If BSM / CAM-related artifacts are absent, but representative object artifacts based on predicted perceptual data and / or SDSM messages exist, a module having one or more algorithms and / or machine learning models measures the artifact parameters relative to each other with respect to the system relative coordinate system map, establishes the existence of a match between two or more object actor representative artifacts, determines the confidence level of the crossmatch between the two or more object actor artifacts, and the parameters used to determine the crossmatch include, but are not limited to, object actor classification category type, position, velocity, and acceleration. A step to determine the confidence level for object actor predictive detection, classification, and tracking, wherein the accuracy of aggregated weighted predictions of object actor detection, classification, and tracking across the process of previous and current runtime output using available BSM / CAM-based object actor data artifacts is measured, the error between the prediction and the actual detection, classification, and tracking is determined, the measured accuracy is retained in system memory for reference in subsequent runtime operations, and the measured accuracy established from the received BSM / CAM database artifacts is used as an objective ratio to measure internally generated predictive perception, externally generated SDSM, and aggregated combinations thereof against the actual detection and classification, as well as tracking parameter values. The system determines confidence intervals based on priori and runtime-updated accuracy thresholds, the relevant thresholds comprising a range of parameter values ​​determined to satisfy statistically significant accuracy requirements indicating low uncertainty regarding predicted values ​​and worthy use of underlying predicted values ​​for perceptual runtime use in a road environment, the output being a confidence interval for the current matched state parameter values ​​of the cross-matched object actor, which includes a high confidence indicator for predicted object actor-related state values ​​or a high confidence indicator that the system's perception cannot detect, classify, or track object actors within the threshold range, and the steps are as follows: Using ground truth, or otherwise predicted by a matched perception process, or reported by SDSM, precursor artifacts used to determine matching are removed from the reference map, and the output of matched artifacts with the corresponding map update is an aggregated and matched collection of runtime-determined ground truth and high-confidence detection, classification, and tracking of object actors in the road environment, and the system relative coordinate reference map is updated to reflect the determined state for subsequent system runtime use and / or external transmission of ground truth data. The steps include: maintaining the matched object actor artifact as ground truth or high confidence determination object actor in the system relative reference coordinate map, which is then logged with cross-matched and matched data including basic safety messages / collaborative recognition messages, sensor data sharing messages, and confidence attributes related to road environment sensor perception data; Methods that include...

7. The method according to claim 6, further comprising the step of determining the current runtime confidence interval based on the current measurable environmental conditions of the current runtime and the frequency, accuracy, and timeliness of BSM / CAM-based accuracy measurements performed prior to the determination, wherein the determination relates to the measured performance of matching predicted object actor outputs and BSM / CAM-related ground truth outputs, the outputs being the difference between actual object actor state attributes including data and predicted object actor state attributes, the outputs being compared with time and condition weight data to determine the confidence interval for each state attribute.

8. The method according to claim 6, further comprising the steps of determining a current adjustment factor and relating it to runtime and current environmental conditions and the measurement accuracy performance of the system, wherein the adjustment factor is timestamped, the timestamped factor is used to determine a damping factor that adjusts the confidence interval-related tolerance based on the statistical significance of historical uncertainty introduced beyond the instant of immediate ground truth versus predictive matching, and the output is used to generate internal data that forms the basis for determining system reliability in perceptual determination.

9. The method according to claim 6, further comprising the step of constructing a highly reliable perception report for use in both an internal and an external system, wherein the output of the external system is transmitted via one or more network devices or radio frequency transceivers.

10. The method according to claim 6, further comprising the steps of storing sensor data and matched ground truth and high confidence perceptual outputs together with timestamps, wherein the timestamps associate the matched perceptual outputs with labeled features in the corresponding logged sensor data, and the labeled logged data collection provides machine learning model training material for prioritizing the training of a machine learning model and for constructing a subsequent high confidence perceptual model.

11. A non-temporary computer-readable medium storing at least one computer program product comprising one or more instructions causing at least one processor to perform an operation, wherein the operation is Steps include receiving data related to road environment object actor sensor-based detection and classification prediction, IEEE 1609 standard security credential management system (SCMS) cryptographic signature SAE J2735 standard basic safety message (BSM) / cooperative recognition message (CAM), and IEEE 1609 standard SCMS cryptographic signature SAE J3224 standard sensor data sharing message (SDSM), A step of determining the runtime detection, classification, and tracking of object actors in a road operating environment related to a system relative coordinate reference map, This includes doing, The steps for determining the detection, classification, and tracking of object actors are: A step of extracting object actor-related data from received SCMS cryptographic signature SAE J2735 participant broadcast basic security message / cooperative recognition message, and then determining the orientation of the object actor, wherein the received data includes, but is not limited to, the spatial dimensions of the reported object actor, including real-time kinematic GNSS corrected position, length, width and / or height, and the category type classification of the object actor, and the BSM / CAM extracted data is converted into a format for storing and retrieving the object actor represented as an artifact in a system relative coordinate reference map, Steps include: extracting object actor-related data from a received SCMS cryptographic signature SAE J3224 participant broadcast sensor data sharing message (SDSM), wherein the received data may include the location of the sensed object actor, its category type classification, observation timestamp, and an external observer's report of its location in GNSS-based global coordinates, and the SDSM-extracted data is converted into a format for storing and retrieving the object actor represented as an artifact in a system relative coordinate reference map; A step of constructing representative object actor artifacts in a system relative coordinate reference map, wherein the coordinate system includes a data structure, and persistent and / or transient data representing road environment features and object actors are related to the host base system in a normalized reference unit format related to the current position of the sensing system and / or the reference frame, and the object actors represented in the reference map are data artifacts generated from sensor-based predictive perception-related output data, SDSM-based message data, or BSM / CAM-based message data, which include position and occupation attributes normalized with respect to the base system reference coordinate system, and in addition to applicable object actor artifacts, a priori and / or runtime mapped and sensed road environment features are also represented, A step of predicting object actors present within the field of view of one or more sensors of the system, wherein the output of the prediction includes predicted detection and classification of the presence, location, spatial dimensions, and category type of object actors related to received sensor data, the received sensor data being associated with one or more cameras, lidars, thermal, or radar sensors, the prediction being based on the output of one or more machine learning models predicting the presence, location, category type, and geometric dimensions of object actors, and the predicted object actor output being sent to the system relative coordinate reference map as predicted object actor artifacts, A step of determining the presence of an actual object actor in the road environment associated with a representative object actor artifact in the system relative coordinate reference map, wherein one or more procedures and / or machine learning models determine that two or more artifacts correspond to the same object actor. At least one of the two or more artifacts is associated with the received BSM / CAM data, and based thereon, one or more algorithms and / or machine learning models establish the existence of a relationship between two or more object actor representative artifacts, and a pair or set of matched artifacts including the BSM / CAM-based artifact is reduced to a ground truth object actor in the system reference map, and the match reference is applied to the cross-matched artifacts to construct a ground truth object actor artifact that exists in an applicable state in the system relative coordinate reference map, and if the sensor-related object actor artifact is not identified as a match with the BSM / CAM artifact, the BSM / CAM is established as a ground truth perception object actor. If BSM / CAM-related artifacts are absent, but representative object artifacts based on predicted perceptual data and / or SDSM messages exist, a module having one or more algorithms and / or machine learning models measures the artifact parameters relative to each other with respect to the system relative coordinate system map, establishes the existence of a match between two or more object actor representative artifacts, determines the confidence level of the crossmatch between the two or more object actor artifacts, and the parameters used to determine the crossmatch include, but are not limited to, object actor classification category type, position, velocity, and acceleration. A step to determine the confidence level for object actor predictive detection, classification, and tracking, wherein the accuracy of aggregated weighted predictions of object actor detection, classification, and tracking across the process of previous and current runtime output using available BSM / CAM-based object actor data artifacts is measured, the error between the prediction and the actual detection, classification, and tracking is determined, the measured accuracy is retained in system memory for reference in subsequent runtime operations, and the measured accuracy established from the received BSM / CAM database artifacts is used as an objective ratio to measure internally generated predictive perception, externally generated SDSM, and aggregated combinations thereof against the actual detection and classification, as well as tracking parameter values. The system determines confidence intervals based on priori and runtime-updated accuracy thresholds, the relevant thresholds comprising a range of parameter values ​​determined to satisfy statistically significant accuracy requirements indicating low uncertainty regarding predicted values ​​and worthy use of underlying predicted values ​​for perceptual runtime use in a road environment, the output being a confidence interval for the current matched state parameter values ​​of the cross-matched object actor, which includes a high confidence indicator for predicted object actor-related state values ​​or a high confidence indicator that the system's perception cannot detect, classify, or track object actors within the threshold range, and the steps are as follows: Using ground truth, or otherwise predicted by a matched perception process, or reported by SDSM, precursor artifacts used to determine matching are removed from the reference map, and the output of matched artifacts with the corresponding map update is an aggregated and matched collection of runtime-determined ground truth and high-confidence detection, classification, and tracking of object actors in the road environment, and the system relative coordinate reference map is updated to reflect the determined state for subsequent system runtime use and / or external transmission of ground truth data. The steps include: maintaining the matched object actor artifact as ground truth or high confidence determination object actor in the system relative reference coordinate map, which is then logged with cross-matched and matched data including basic safety messages / collaborative recognition messages, sensor data sharing messages, and confidence attributes related to road environment sensor perception data; Computer-readable media, including [specific examples of computer-readable media].

12. The aforementioned operation is, The computer-readable medium according to claim 11, further comprising the step of determining the current runtime confidence interval based on the current measurable environmental conditions of the current runtime and the frequency, accuracy, and timeliness of BSM / CAM-based accuracy measurements performed prior to the determination, wherein the determination relates to the measured performance of matching predicted object actor outputs and BSM / CAM-related ground truth outputs, the outputs being the difference between actual object actor state attributes including data and predicted object actor state attributes, the outputs being compared with time and condition weight data to determine the confidence interval for each state attribute.

13. The aforementioned operation is, A computer-readable medium according to claim 11, further comprising the steps of determining a current adjustment factor and relating it to runtime and current environmental conditions and the measurement accuracy performance of the system, wherein the adjustment factor is timestamped, the timestamped factor is used to determine a decay factor that adjusts the confidence interval-related tolerance based on the statistical significance of historical uncertainty introduced beyond the instantaneous ground truth versus predictive matching moment, and the output is used to generate internal data that forms the basis for determining system reliability in the determination of perception.

14. The aforementioned operation is, The computer-readable medium according to claim 11, further comprising the step of constructing a highly reliable perception report for use in both an internal and an external system, wherein the output of the external system is transmitted via one or more network devices or radio frequency transceivers.

15. The aforementioned operation is, A computer-readable medium according to claim 11, further comprising the steps of storing sensor data and matched ground truth and high confidence perceptual outputs together with timestamps, wherein the timestamps associate the matched perceptual outputs with labeled features in the corresponding logged sensor data, and the labeled logged data collection provides machine learning model training material for prioritizing the training of a machine learning model and for constructing a subsequent high confidence perceptual model.

Images