Methods / systems / computer programs that enable ground truth runtime perception for BSM / RTCM / SCMS

Abstract

A roadside unit (RSU) transceiver or on-board unit (OBU) transceiver is provided, equipped with a sensor suite consisting of one or more electro-optic sensors, camera sensors, thermal imaging sensors, lidar sensors, ultrasonic sensors, GNSS receivers, and / or radar sensors, and capable of receiving authenticated participant SAE J2735 basic safety messages and RTK-corrected GNSS positioning data. The system may include one or more processors programmed or configured to receive data from the system's own sensor suite, reporting transceivers, and other network-connected devices, and to construct ground truth object detection, classification, and tracking messages for object actors within the system's field of view.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application claims priority to U.S. Provisional Patent Application No. 63 / 453,592, filed on March 21, 2023, the content of which is incorporated herein by reference.

[0002] The disclosed subject matter generally relates to methods, systems, and products for human and computer control of vehicles operating in the vicinity of road regions, training of machine learning and / or perception systems of vehicle control systems, and development, verification, and validation of vehicle control systems before (e.g., simulation), during, and after road operation.

Background Art

[0003] The premise of this disclosure is that cooperative awareness messages (BSM in North America / CAM in Europe) can be used as a basis for establishing ground truth for processing perception systems. Thus, the deviation between the position reported by a sensor and the RTK - corrected position reported in a cooperative awareness message is an accurate and quantifiable measure of ground truth and can be used to develop, train, verify, validate and check the perception system at runtime, and perform performance measurements after execution, and can be used as a set of precursor parameters for determining the confidence level of the output of the perception system.

[0004] Those skilled in the art envision V2X-based collaborative recognition providing a means to augment sensor-based perception and object detection processes. This approach treats BSM / CAM as an additional input to the perceived environment model. However, this approach does not incorporate objective ground truth into the runtime perception process for downstream system use. A popular handling or use of the idea of ​​collaborative recognition is that BSM / CAM broadcast by object actors can help either extend the receiver's perception model beyond the field of view or merge them with the BSM / CAM to improve the accuracy and reliability of sensor-based object actor detection reports. However, again, this is not equivalent to the direct establishment and incorporation of objective ground truth.

[0005] In industry and academia, mainstream methods for training and validating ML / CV (machine learning / computer vision) perception model sets require a priori labeled data on large datasets. First, a large data corpus must be captured representing the static environment, the changing environmental factors and conditions, and the changing types and instances of object actors present or moving within the environment. Currently, many state-of-the-art ML / CV recognition systems require offline labeling after capture, a labeling task often performed by humans manually annotating sensor-captured frames, a cumbersome and costly process. The system described below, in addition to establishing ground truth at runtime, mitigates the need for offline, post-execution human labeling inherent in state-of-the-art ML / CV recognition systems.

[0006] Current practices in ML / CV-based collaborative perception involve developing object actor recognition through consensus. Consensus is established through various processes to verify the independent reports of two or more participants on the perceived output of each object actor within the field of view. According to SAE J3224, participants should report their confidence in various data fields shared by them. However, there are no widely adopted industry standards for establishing confidence in machine learning / computer vision inference perception. Current industry and academic practices for establishing confidence levels may rely on some form of measuring the inherent differences or marginal differences in the contributing object actor reports used to generate consensus. Neither incorporating objective ground truth nor using ground truth to establish reliable perception and measure confidence in perceived output has been done.

[0007] Establishing ground truth is a requirement for higher-level capabilities and downstream processes in automotive assistance and automation technologies. The system described below establishes ground truth and makes it available throughout the entire sensor data acquisition process used in the system's perception process.

[0008] Ground Truth: Ground truth is a term that describes information known to be real or true through direct observation and measurement. An example of an ADAS / ADS perception system is that it requires ground truth data for the development of its perception model and operates with high reliability when ground truth is available during runtime. The perception model is trained and validated by providing ground truth sensor data from which object actors are a priori detected and classified. During actual runtime, the perception model establishes ground truth on object actors in the operating environment by matching predictions from the runtime sensor database with runtime known location, classification type, and spatial geometry data based on BSM / CAM messages.

[0009] Basic safety message / collaborative recognition message: The SAE J2735 Basic Safety Message (BSM), or the European equivalent Cooperative Awareness Message (CAM), is a standardized data object encapsulated in an abstract syntactic notation-based message format traded to provide situational awareness in road operating environments. The RSU / OBU ecosystem receives real-time kinematic GNSS positioning updates to ensure accurate global coordinate localization for use in position reporting.

[0010] The situational awareness data shared in BSM / CAM self-reports the vehicle's status 10 times per second (10Hz), including data attributes such as latitude, longitude, altitude, speed, direction of travel, steering angle, acceleration, brake system status, and vehicle length and width. BSM / CAM information digested by advanced driver assistance systems (ADAS) and / or automated driving systems (ADS) enables a more reliable object perception process that considers enhanced road situation awareness, out-of-line recognition and predictive behavior detection, and infrastructure that reports third-party vehicles and their respective objective coordinate positions and dimensions. The benefits of this enhanced perception include improved safety and vehicle mobility efficiency.

[0011] BSM / CAM transactions are conducted in one-to-many (broadcast) mode via V2X wireless communication protocols such as IEEE 1609, also known as WAVE (Wireless Access Vehicular Environment), but are not limited to this, and the media access control (MAC) layer is IEEE 802.11p or PC5 (i.e., C-V2X). BSM / CAM can be shared both peer-to-peer or "vehicle-to-vehicle" (V2V) or vehicle-to-infrastructure (V2I), with the extension of V2I being vehicle-to-network (V2N). The communication ecosystem for V2V, V2I, and V2N is generally referred to as Vehicle-to-Everything or "V2X".

[0012] To exchange BSM / CAM signals, participating vehicles are equipped with radio frequency transceiver modems called Onboard Units (OBUs). Fixed infrastructure radio frequency transceivers are called Roadside Units (RSUs). The IEEE 1609.2 components of the WAVE stack use SCMS certificates (i.e., "credentials") to ensure the security and authentication of data traded within the ecosystem, thereby enabling RSU / OBU units to determine the authenticity of the BSM / CAM sender and, consequently, the authenticity of their messages. Devices compliant with IEEE 1609.2 and provisioned with SCMS credentials, i.e., RSUs and OBUs, are called End Entities (EEs).

[0013] Security Credential Management System (SCMS): SCMS is an instantiation of Public Key Infrastructure (PKI), which generates and delivers certificates to the EE to provide trust and assurance for messages broadcast by the EE (SAE J3275 and SAE J3224 messages encapsulating sensor data for in-vehicle ADAS / ADS). SCMS can invalidate credentials if it cannot establish or maintain trust. Each EE participant receives SCMS credentials, which are used to sign issued messages, read by recipients, and establish the reliability of the message's reporting source. The SCMS system enables the EE to incorporate BSM / CAM and Sensor Data Sharing Messages (SDSM) into its operational picture with greater reliability than would be obtained from messaging without credentials. [Overview of the Initiative] [Problems that the invention aims to solve]

[0014] Accordingly, the subject matter of this disclosure is to provide a method, system, and computer program product for establishing ground truth perception output on an object actor using BSM / CAM, RTK-corrected GNSS positioning data, and sensor data. The method, system, and computer program can reside on a host-based system, which functions as a physical platform for sensors and calculations that outputs sensor data of the sensed road environment in the vicinity of the host system. [Means for solving the problem]

[0015] These and other advantages are achieved, for example, by a system including one or more processors, which are programmed or configured to perform the steps of: (i) receiving data related to road environment object actor sensor-based detection and classification prediction and IEEE 1609 standard security credential management system (SCMS) cryptographic signature SAE J2735 standard basic safety message (BSM) and / or collaborative recognition message (CAM); and (ii) determining runtime ground truth perception based on collated and matched data artifacts constructed in a system relative coordinate reference map associated with BSM / CAM reported object actor data and predicted object actor data artifacts.

[0016] When determining runtime ground truth perception, one or more processors (iii) extract object actor-related data from received SCMS cryptographic signature SAE J2735 participant broadcast basic safety message / cooperative recognition message and then determine the orientation of the object actor, wherein the received data includes, but is not limited to, the reported real-time kinematic GNSS corrected position of the object actor, spatial dimensions (e.g., length × width × height), and category type classification of the object actor, and the BSM / CAM extracted data is converted into a format for storing and retrieving object actors represented as artifacts in a system relative coordinate reference map, and (iv) construct a representative object actor artifact in a system relative coordinate reference map, wherein the coordinate system includes a data structure, and persistent and / or transient data representing road environment features and object actors are related to the host base system in a format of normalized reference units related to the current position and / or reference frame of the sensing system and represented in the reference map. (v) a prediction of an object actor which is a data artifact generated from sensor-based predictive perception-related output data or BSM / CAM-based message data, which includes position and occupancy attributes normalized to a base system reference coordinate system, and in addition to applicable object actor artifacts, also represents a priori and / or runtime mapped and perceived road environment features, (v) a prediction of an object actor which is present in the field of view of one or more sensors of the system, the output of the prediction which includes the predicted presence of the object actor and its respective classification and position state associated with the sensor data, the received sensor data which is associated with one or more cameras, lidars, or radar sensors, and the prediction which is based on the output of one or more machine learning models which predict the presence, position, category type and geometric dimensions of the object actor, and the predicted object actor output which is sent as a predicted object actor artifact to the system relative coordinate reference map,(vi) A step of determining the presence of two or more artifacts loaded into the system relative coordinate reference map corresponding to the same object actor in a road environment, wherein at least one of the two or more artifacts is associated with BSM / CAM data, and thereafter one or more algorithms and / or machine learning models establish the presence of a match between the two or more object actor representative artifacts, and the set of matched artifacts constitutes a ground truth object actor in the system reference map, and if a sensor-related object actor artifact is not identified as a match to a BSM / CAM artifact, the unmatched BSM / CAM is the basis for constituting a ground truth perception object actor (vii) The system is programmed or configured to perform the following steps: (vii) After the determined ground truth object actor artifact is constructed, precursor artifacts used for matching are removed from the reference map, the matched artifact and the result of the map update is the runtime ground truth perception, the system relative coordinate reference map is updated to reflect the resulting state for subsequent system runtime use and / or external transmission of ground truth data, and (vii) the matched object actor artifact is persisted as the ground truth object actor data artifact in the system relative coordinate map, which is then logged with cross-matched and reconciled data associated with basic safety messages / cooperative perception messages and road environment sensor perception data.

[0017] One or more processors are further programmed or configured to perform the step of determining the ground truth velocity of an object actor based on the progression of a set of established runtime ground truth perception data. One or more processors are further programmed or configured to perform the step of determining the ground truth acceleration of an object actor based on the progression of a set of established runtime ground truth perception data. One or more processors are further programmed or configured to perform the step of building labeled data for post-run training of a machine learning model, wherein runtime determined ground truth perception data and raw data from one or more sensors of the system are logged and timestamped, and runtime generated logged and timestamped ground truth perception data includes labeled features associated with the logged sensor data.

[0018] These and other advantages are also achieved by a method that includes, for example, (i) receiving data related to road environment object actor sensor-based detection and classification prediction and IEEE 1609 standard security credential management system (SCMS) cryptographic signature SAE J2735 standard basic safety message (BSM) and / or collaborative recognition message (CAM); and (ii) determining runtime ground truth perception based on collated and matched data artifacts constructed in a system relative coordinate reference map associated with BSM / CAM reported object actor data and predicted object actor data artifacts.

[0019] When determining runtime ground truth perception, the method includes (iii) extracting object actor-related data from received SCMS cryptographic signature SAE J2735 participant broadcast basic safety message / cooperative recognition message and then determining the pose of the object actor, wherein the received data includes, but is not limited to, the reported real-time kinematic GNSS corrected position of the object actor, spatial dimensions (e.g., length × width × height), and category type classification of the object actor, and the BSM / CAM extracted data is converted into a format for storing and retrieving object actors represented as artifacts in a system relative coordinate reference map, and (iv) constructing representative object actor artifacts in a system relative coordinate reference map, wherein the coordinate system includes a data structure, and persistent and / or transient data representing road environment features and object actors are related to the host base system in a normalized reference unit format related to the current position and / or reference frame of the sensing system and represented in the reference map. (v) a prediction of an object actor which is a data artifact generated from sensor-based predictive perception-related output data or BSM / CAM-based message data, which includes position and occupancy attributes normalized to a base system reference coordinate system, and in addition to applicable object actor artifacts, also represents a priori and / or runtime mapped and perceived road environment features, (v) a prediction of an object actor which is present in the field of view of one or more sensors of the system, the output of the prediction which includes the predicted presence of the object actor and its respective classification and position state associated with the sensor data, the received sensor data which is associated with one or more cameras, lidars, or radar sensors, and the prediction which is based on the output of one or more machine learning models which predict the presence, position, category type and geometric dimensions of the object actor, and the predicted object actor output which is sent as a predicted object actor artifact to the system relative coordinate reference map,(vi) A step of determining the presence of two or more artifacts loaded into the system relative coordinate reference map corresponding to the same object actor in a road environment, wherein at least one of the two or more artifacts is associated with BSM / CAM data, and thereafter one or more algorithms and / or machine learning models establish the presence of a match between the two or more object actor representative artifacts, and the set of matched artifacts constitutes a ground truth object actor in the system reference map, and if a sensor-related object actor artifact is not identified as a match to a BSM / CAM artifact, the unmatched BSM / CAM is the basis for constituting a ground truth perception object actor, and the determined ground (vii) After the ground truth object actor artifact is constructed, precursor artifacts used for matching are removed from the reference map, the result of the matched artifact and map update is runtime ground truth perception, the system relative coordinate reference map is updated to reflect the resulting state for subsequent system runtime use and / or external transmission of ground truth data, and (vii) the matched object actor artifact is persisted as ground truth object actor data artifact in the system relative coordinate map, which is then logged with associated data as cross-matched and matched data associated with basic safety messages / cooperative perception messages and road environment sensor perception data.

[0020] These advantages and other advantages are further achieved, for example, by storing at least one non-temporary computer-readable medium containing at least one computer program product that includes one or more instructions causing at least one processor to perform the method steps described above. [Brief explanation of the drawing]

[0021] The preferred embodiments described herein and illustrated by the following drawings are included to illustrate the present invention, not to limit it, and like reference numerals indicate like elements.

[0022] [Figure 1] Shows an overview of a non-limiting exemplary system.

Best Mode for Carrying Out the Invention

[0023] The following detailed description is merely exemplary in nature and is not intended to limit the embodiments described or the application and uses of the embodiments described. All of the implementations described below are exemplary implementations provided to enable those skilled in the art to make or use the embodiments of the present disclosure, and are not intended to limit the scope of the present disclosure as defined by the claims. It should also be understood that the drawings included herein provide only a schematic representation of the presently preferred structure of the present invention, and that structures within the scope of the present invention may include structures different from those shown in the drawings.

[0024] The disclosed invention provides a method, system, and computer program product for establishing ground truth perception output on an object actor using BSM / CAM, RTK-corrected GNSS positioning data, and sensor data. The method, system, and computer program can reside on a host-based system, which functions as a physical platform for sensors and calculations that output sensor data for the sensed road environment in the vicinity of the host system.

[0025] The system uses BSM / CAM as a known object actor with a known position, category type classification, and spatial dimensions (e.g., object type: vehicle - SUV; geometry box dimensions: width 2.2 meters, height 2.3 meters, length 4.2 meters; position: latitude 32.553385°, longitude -96.822104°, height 197.815 meters). The BSM / CAM reports are converted into formatted data artifacts to place the known object actor on the system reference coordinate system map. The known object actor is cross - matched to the system perception module output artifacts that are simultaneously populated on the system reference coordinate system map. The system uses these cross - matches to establish the ground truth of the internal perception system and may also be used for offline ML (machine learning) model / algorithm training refinement. The feedback loop of the cross - match enables the reviewer to evaluate the performance of the system's perception system and assist in further improvements in perception tasks. System users can not only establish runtime and offline training data with ground truth but also use the cross - matched object actor data to train a high - confidence model set / algorithm for detecting, classifying, and tracking object actors that do not report BSM / CAM.

[0026] Referring to Figure 1, an overview of a non-limiting, exemplary system is shown. The RSU includes either a direct RTK receiver or an RTK correction update 107 via a network interface that transmits RTK corrections to the RSU. The system receives Real-Time Kinematic Positioning (RTK) error corrections 100 for Global Navigation Satellite System (GNSS) positioning. The RTK correction update provides high accuracy to the position data, which the RSU transceiver 102 converts into an SAE J2735 RTCM (Radio Technical Commission for Maritime Services) message 104, broadcasts within the ecosystem, and is used by the OBU 105 to apply the corrections to the position data in their BSM / CAMs. Alternatively, the OBU 105, equipped with an RF receiver 108, may directly acquire the RTK corrections 100.

[0027] The IEEE1609.2 module 103, integrated into OBU105, signs the outgoing BSM / CAM using SCMS credentials. The equivalent module 106 within RSU102 verifies the received BSM / CAM. This ensures that only BSM / CAMs reported by authenticated participants are considered reliable, thereby preventing unauthorized participants from providing false data to the system and providing shared perceptual participants with assurance of the integrity of the BSM / CAM data and subsequent system broadcasts or messages.

[0028] The system RSU / OBU radio transceiver module 102 receives the BSM / CAM 101 from the reporting participant, and the IEEE1609.2 module 106 validates the message data before passing it to the BSM / CAM Extract, Transform, and Load (ETL) module 402. The BSM / CAM ETL module 402 extracts the input BSM / CAM data and transforms the provided position data into data artifacts, which include, but are not limited to, system coordinates, classifications, and occupied geometry messages 403 placed within a system relative coordinate reference map 500. The BSM / CAM-related data artifacts can then be used to compare, adjust, and match simultaneously generated predictions of the object actor's state, further enabling, adjusting, confirming, and / or enhancing the perception, prediction, and / or motion planning process.

[0029] The system relative coordinate reference map 500 includes a data structure in which persistent and / or transient data artifacts representing object actor artifacts and other data items representing road environment features are related to the host base system in a normalized reference unit format related to the current position of the sensing system and / or the frame of the reference. The map 500 hosts runtime-generated artifacts representing either object actors predicted from the system sensing process in the sensing module 200, the aggregate fusion sensing module 300, and the BSM / CAM ETL module 402, respectively, or object actors reported by the BSM / CAM. The generated artifacts include position and occupancy attributes normalized to the base system reference coordinate system included in the map. In some non-limiting embodiments, the map may host a priori determined and / or runtime-mapped and sensed road environment features.

[0030] The system's perception module 200 performs runtime detection and classification tasks to determine the presence of discrete object actors within the sensor's field of view, classify the category type of the object actors, and determine the position of the object actors relative to the sensor's data frame.

[0031] The system's perception module 200 may receive sensor inputs from one or more of the following types of sensors (including, but not limited to, cameras 201a, lidar 201b, thermal 201c, and radar 201d). Following any preprocessing and formatting data processing procedures, the system's perception module 200 performs object detection and classification tasks to extract distinct object actor entities from the sensor data. The object actor detection and classification messages 300 output by module 200 describe the classification type of the object actor and the inferred detected location and spatial occupancy. The object actor detection and classification messages become data related to predicted object actor data artifacts and are sent downstream to a system relative coordinate reference map, and in some non-limiting embodiments, the sensor-specific perception module outputs are sent to an intermediate perception aggregation processing module 301. Message 300 is sent to the aggregate fused stream perception module 301, which compares the outputs of multiple sensor models and determines an aggregate classification detection decision based on the fidelity and confidence level of each streamed sensor independent model output, and determines multiple sensor enhanced classification and detection decisions. The aggregate fused stream perception module sends the fused output prediction object actor artifact 302 message to the system relative coordinate reference map 500, within which the fused output message is therefore called a prediction object actor artifact.

[0032] The system's perceptual ground truth matching module 600 processes both the BSM / CAM derived object actor artifact data 601 and the perceptual fusion output report of the predicted object actor artifacts 302, both of which reside within the system relative coordinate reference map.

[0033] Module 600 calculates cross-matching between two different dataset artifacts (BSM / CAM reported vs. predicted object actor perception process) based on geometry, category type classification, and attitude and tracked attitude over multiple consecutive sample periods. Thus, attitude is defined as the object actor's position, velocity (speed and direction of motion), and spatial occupancy relative to the system-relative coordinate system that forms the basis of the reference map 500. The module performs a simultaneous matching process of two or more artifacts with respect to classification, position, attitude, and tracked attitude criteria. The matching establishes which sensor-detected object actor artifact is the same object actor as the associated object actor artifact in a known BSM / CAM message report. The matching of two different dataset artifacts on the object artifact generates ground truth, relating the system's own attitude to the BSM / CAM-reported object actor and the system's determined predicted object actor associated with the sensor data. The resulting output is established as ground truth on specific object actors that the system can include in subsequent tasks. The matched object actor output from the matching between the BSM / CAM and the predicted object actor artifact in the system relative coordinate reference map is the ground truth for the system's perceived system. If no corresponding predicted object actor artifact exists, or if a BSM / CAM-based artifact exists, the BSM / CAM artifact is the basis for the ground truth report output, transformed into system-related coordinates on the system relative coordinate reference map.

[0034] The perceptual ground truth matching module updates the system relative coordinate reference map with matched ground truth data messages 601 to other system downstream processes 701 or network interface 702 for transmission to an external system, and also updates the system relative coordinate reference map with ground truth object actor data artifacts for further runtime object actor tracking and matching. The downstream system process 701 performs object actor tracking with respect to the system reference coordinate reference map and the matched object actors determined by the perceptual ground truth matching module 600. Tracking is a process and corresponding output that determines the current position, velocity, and acceleration of the detected object actors across a series of artifacts and matching updates. The perceptual ground truth matching module determines changes in the position and direction of movement of ground truth object actors by relating sequential ground truth determination outputs, which is the ground truth of the object actor velocity relative to the system relative coordinate reference map. Furthermore, along with the sequential determination of the object actor's position and velocity ground truth output, the matching module determines changes in position and velocity to determine the object actor's ground truth acceleration. The matching module 600 further writes the established ground truth position, geometric occupancy, velocity, and acceleration information to a persistent log as timestamped output. The corresponding timestamped sensor data is persisted, and the ground truth data is annotated as feature labels related to the logged sensor data.

[0035] Since many modifications, variations, and alterations can be made to the preferred embodiments described herein, all matters shown in the foregoing description and accompanying drawings are intended to be interpreted as illustrative rather than restrictive. Accordingly, the scope of the invention should be determined by the accompanying claims and their legal equivalents.

Claims

1. A system comprising one or more processors, wherein the one or more processors are Steps include receiving data related to road environment object actor sensor-based detection and classification prediction, and IEEE 1609 standard security credential management system (SCMS) cryptographic signature SAE J2735 standard basic safety message (BSM) and / or cooperative recognition message (CAM), The steps include determining runtime ground truth perception based on matched and coordinated data artifacts constructed in a system relative coordinate reference map associated with BSM / CAM reported object actor data and predicted object actor data artifacts, Programmed or configured to do, For the step of determining runtime ground truth perception, one or more processors, A step of extracting object actor-related data from received SCMS cryptographic signature SAE J2735 participant broadcast basic security message / cooperative recognition message, and then determining the orientation of the object actor, wherein the received data includes, but is not limited to, the spatial dimensions of the reported object actor, including real-time kinematic GNSS corrected position, length, width and / or height, and the category type classification of the object actor, and the BSM / CAM extracted data is converted into a format for storing and retrieving the object actor represented as an artifact in a system relative coordinate reference map, A step of constructing representative object actor artifacts in a system relative coordinate reference map, wherein the coordinate system includes a data structure, and persistent and / or transient data representing road environment features and object actors are related to the host base system in a normalized reference unit format related to the current position of the sensing system and / or the reference frame, and the object actors represented in the reference map are data artifacts generated from sensor-based predictive perception-related output data or BSM / CAM-based message data, which include position and occupation attributes normalized with respect to the base system reference coordinate system, and in addition to applicable object actor artifacts, a priori and / or runtime mapped and sensed road environment features are also represented, A step of predicting object actors present within the field of view of one or more sensors of the system, wherein the output of the prediction includes the predicted presence of the object actors and their respective classifications and positional states related to sensor data, the received sensor data being associated with one or more cameras, lidars, or radar sensors, the prediction being based on the output of one or more machine learning models predicting the presence, position, category type, and geometric dimensions of the object actors, and the predicted object actor output being sent to the system relative coordinate reference map as a predicted object actor artifact. A step of determining the existence of two or more artifacts loaded into the system relative coordinate reference map corresponding to the same object actor in a road environment, wherein at least one of the two or more artifacts is associated with BSM / CAM data, and thereafter one or more algorithms and / or machine learning models establish the existence of a match between the two or more object actor representative artifacts. The matched set of artifacts constitutes the ground truth object actor in the system reference map, and if sensor-related object actor artifacts are not identified as matches to BSM / CAM artifacts, the unmatched BSM / CAM forms the basis for constructing the ground truth perception object actor, and after the determined ground truth object actor artifacts are constructed, precursor artifacts used for matching are removed from the reference map, and the result of the matched artifacts and map updates is the runtime ground truth perception, and the system relative coordinate reference map is updated to reflect the resulting state for subsequent system runtime use and / or external transmission of ground truth data. The steps include: persisting the matched object actor artifact as a ground truth object actor data artifact in the system relative coordinate map, which is then logged with cross-matched and matched data associated with basic safety messages / collaborative recognition messages and road environment sensor perception data; A system that is programmed or configured to perform a certain action.

2. The one or more processors described above are: The system according to claim 1, further programmed or configured to perform the step of determining the ground truth velocity of an object actor based on the progression of a series of established runtime ground truth perception data.

3. The one or more processors described above are: The system according to claim 1, further programmed or configured to perform the step of determining the ground truth acceleration of an object actor based on the progression of a series of established runtime ground truth perception data.

4. The one or more processors described above are: The system according to claim 1, further programmed or configured to perform the step of constructing labeled data for post-run training of a machine learning model, wherein ground truth perception data determined at runtime and raw data from one or more sensors of the system are logged and timestamped, and the logged and timestamped ground truth perception data generated at runtime includes labeled features associated with the logged sensor data.

5. Steps include receiving data related to road environment object actor sensor-based detection and classification prediction, and IEEE 1609 standard security credential management system (SCMS) cryptographic signature SAE J2735 standard basic safety message (BSM) and / or cooperative recognition message (CAM), The steps include determining runtime ground truth perception based on matched and coordinated data artifacts constructed in a system relative coordinate reference map associated with BSM / CAM reported object actor data and predicted object actor data artifacts, Programmed or configured to do, The steps to determine runtime ground truth perception are: A step of extracting object actor-related data from received SCMS cryptographic signature SAE J2735 participant broadcast basic security message / cooperative recognition message, and then determining the orientation of the object actor, wherein the received data includes, but is not limited to, the spatial dimensions of the reported object actor, including real-time kinematic GNSS corrected position, length, width and / or height, and the category type classification of the object actor, and the BSM / CAM extracted data is converted into a format for storing and retrieving the object actor represented as an artifact in a system relative coordinate reference map, A step of constructing representative object actor artifacts in a system relative coordinate reference map, wherein the coordinate system includes a data structure, and persistent and / or transient data representing road environment features and object actors are related to the host base system in a normalized reference unit format related to the current position of the sensing system and / or the reference frame, and the object actors represented in the reference map are data artifacts generated from sensor-based predictive perception-related output data or BSM / CAM-based message data, which include position and occupation attributes normalized with respect to the base system reference coordinate system, and in addition to applicable object actor artifacts, a priori and / or runtime mapped and sensed road environment features are also represented, A step of predicting object actors present within the field of view of one or more sensors of the system, wherein the output of the prediction includes the predicted presence of the object actors and their respective classifications and positional states related to sensor data, the received sensor data being associated with one or more cameras, lidars, or radar sensors, the prediction being based on the output of one or more machine learning models predicting the presence, position, category type, and geometric dimensions of the object actors, and the predicted object actor output being sent to the system relative coordinate reference map as a predicted object actor artifact. A step of determining the existence of two or more artifacts loaded into the system relative coordinate reference map corresponding to the same object actor in a road environment, wherein at least one of the two or more artifacts is associated with BSM / CAM data, in which one or more algorithms and / or machine learning models establish the existence of a match between the two or more object actor representative artifacts. The matched set of artifacts constitutes the ground truth object actor in the system reference map, and if sensor-related object actor artifacts are not identified as matches to BSM / CAM artifacts, the unmatched BSM / CAM forms the basis for constructing the ground truth perception object actor, and after the determined ground truth object actor artifacts are constructed, precursor artifacts used for matching are removed from the reference map, and the result of the matched artifacts and map updates is the runtime ground truth perception, and the system relative coordinate reference map is updated to reflect the resulting state for subsequent system runtime use and / or external transmission of ground truth data. The steps include: persisting the matched object actor artifact as a ground truth object actor data artifact in the system relative coordinate map, which is then logged with associated data as cross-matched and matched data associated with basic safety messages / collaborative recognition messages and road environment sensor perception data; Methods that include...

6. The method according to claim 5, further comprising the step of determining the ground truth velocity of an object actor based on the progression of a series of established runtime ground truth perception data.

7. The method according to claim 5, further comprising the step of determining the ground truth acceleration of an object actor based on the progression of a series of established runtime ground truth perception data.

8. The method according to claim 5, further comprising the step of constructing labeled data for post-run training of a machine learning model, wherein ground truth perception data determined at runtime and raw data from one or more sensors of the system are logged and timestamped, and the logged and timestamped ground truth perception data generated at runtime includes labeled features associated with the logged sensor data.

9. A non-temporary computer-readable medium storing at least one computer program product which includes one or more instructions causing at least one processor to perform an operation, wherein the operation is Steps include receiving data related to road environment object actor sensor-based detection and classification prediction, and IEEE 1609 standard security credential management system (SCMS) cryptographic signature SAE J2735 standard basic safety message (BSM) and / or cooperative recognition message (CAM), The steps include determining runtime ground truth perception based on matched and coordinated data artifacts constructed in a system relative coordinate reference map associated with BSM / CAM reported object actor data and predicted object actor data artifacts, This includes doing, The steps to determine runtime ground truth perception are: A step of extracting object actor-related data from received SCMS cryptographic signature SAE J2735 participant broadcast basic security message / cooperative recognition message, and then determining the orientation of the object actor, wherein the received data includes, but is not limited to, the spatial dimensions of the reported object actor, including real-time kinematic GNSS corrected position, length, width and / or height, and the category type classification of the object actor, and the BSM / CAM extracted data is converted into a format for storing and retrieving the object actor represented as an artifact in a system relative coordinate reference map, A step of constructing representative object actor artifacts in a system relative coordinate reference map, wherein the coordinate system includes a data structure, and persistent and / or transient data representing road environment features and object actors are related to the host base system in a normalized reference unit format related to the current position of the sensing system and / or the reference frame, and the object actors represented in the reference map are data artifacts generated from sensor-based predictive perception-related output data or BSM / CAM-based message data, which include position and occupation attributes normalized with respect to the base system reference coordinate system, and in addition to applicable object actor artifacts, a priori and / or runtime mapped and sensed road environment features are also represented, A step of predicting object actors present within the field of view of one or more sensors of the system, wherein the output of the prediction includes the predicted presence of the object actors and their respective classifications and positional states related to sensor data, the received sensor data being associated with one or more cameras, lidars, or radar sensors, the prediction being based on the output of one or more machine learning models predicting the presence, position, category type, and geometric dimensions of the object actors, and the predicted object actor output being sent to the system relative coordinate reference map as a predicted object actor artifact. A step of determining the existence of two or more artifacts loaded into the system relative coordinate reference map corresponding to the same object actor in a road environment, wherein at least one of the two or more artifacts is associated with BSM / CAM data, and thereafter one or more algorithms and / or machine learning models establish the existence of a match between the two or more object actor representative artifacts. The matched set of artifacts constitutes the ground truth object actor in the system reference map, and if sensor-related object actor artifacts are not identified as matches to BSM / CAM artifacts, the unmatched BSM / CAM forms the basis for constructing the ground truth perception object actor, and after the determined ground truth object actor artifacts are constructed, precursor artifacts used for matching are removed from the reference map, and the result of the matched artifacts and map updates is the runtime ground truth perception, and the system relative coordinate reference map is updated to reflect the resulting state for subsequent system runtime use and / or external transmission of ground truth data. The matching object actor artifact pairs stored in the system relative coordinate map are then maintained and stored in a matched and unified format based on the combined associated data, as cross-matched and collaborating data associated with basic safety messages / collaborative recognition messages and road environment sensor perception data. Computer-readable media, including [specific text / data].

10. One or more instructions causing the at least one processor to determine runtime ground truth perception are, A computer-readable medium according to claim 9, which causes the medium to perform an operation that includes the step of determining the ground truth velocity of an object actor based on the progression of a series of established runtime ground truth perception data.

11. One or more instructions causing the at least one processor to determine runtime ground truth perception are, A computer-readable medium according to claim 9, which causes the medium to perform an operation that includes the step of determining the ground truth acceleration of an object actor based on the progression of a series of established runtime ground truth perception data.

12. One or more instructions causing the at least one processor to determine runtime ground truth perception are, Computer-readable media according to claim 9, comprising the step of constructing labeled data for post-training of a machine learning model, wherein ground truth perception data determined at runtime and raw data from one or more sensors of the system are logged and timestamped, and the logged and timestamped ground truth perception data generated at runtime includes labeled features associated with the logged sensor data.

Images