Autonomous Vehicle Environment Perception Software Management

Abstract

A method for operating a computer to construct a customized training dataset for training an artificial intelligence (AI) platform includes a) obtaining (i) a specification of training data for training the AI ​​platform and (ii) data indicative of a proposed use of the training data; b) querying a database of training datasets, each associated with usage rights, to identify a candidate set of training datasets that match the specification; and c) approving release of a subset of training datasets in the candidate set of training datasets based on the data indicative of the proposed use of the AI ​​training data and the usage rights associated with the training datasets in the candidate set of training datasets.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 303,363, filed January 26, 2022, which is incorporated herein by reference.

[0002] The present disclosure relates to managing data components used to train artificial intelligence (AI) platforms for use in environmental perception systems, adapt such AI platforms to different application scopes, and the like. [Background technology]

[0003] The automotive industry is undergoing disruptive change, accentuated by electrification and ride-hailing as well as the entry of autonomous vehicles (AVs). Although autonomy has made considerable progress over the past few years, current solutions are still neither safe enough nor reliable enough for commercial deployment in all but the most limited scenarios.

[0004] To successfully navigate common driving scenarios, AVs are equipped with sensing modules that enable them to identify features of the environment in which they are driving. However, the ability to detect the dynamic driving environment surrounding the car, such as other vehicles, pedestrians, and small obstacles on the road, while maintaining the affordability of the car, is a challenging task. Moreover, various driving environments pose unique challenges for accurate autonomous vehicle environmental perception.

[0005] The industry is therefore turning to artificial intelligence (AI) in an effort to improve the speed and accuracy of environmental perception. However, it is becoming increasingly clear that optimal training of AI platforms across a wide range of driving scenarios can only yield results using vast amounts of data originating from many different sources. This poses challenges in terms of managing, controlling and distributing data, both for training and for using AI platforms, particularly in the context of environmental perception for vehicle navigation, whether on land, sea or in the air. Summary of the Invention

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify essential aspects of the claimed subject matter.

[0007] Broadly speaking, in some embodiments, the disclosure relates to a method of operating a computer to construct a customized training dataset for training an artificial intelligence (AI) platform, the method including obtaining (i) a specification of the training data for training the AI ​​platform and (ii) data indicative of a proposed use of the training data, querying a database of training datasets, each associated with usage rights, to identify a candidate set of training datasets that match the specification, and approving release of a subset of the training datasets in the candidate set of training datasets based on the data indicative of the proposed use of the AI ​​training data and the usage rights associated with the training datasets in the candidate set of training datasets.

[0008] In another embodiment, the present disclosure relates to a non-transitory computer-readable storage medium storing instructions that, when executed by a processing entity of a computing device, cause the computing device to perform a method of constructing a customized training dataset for training an artificial intelligence (AI) platform, the method including obtaining (i) a specification of the training data for training the AI ​​platform and (ii) data indicative of a proposed use of the training data, querying a database of training datasets, each associated with usage rights, to identify a candidate set of training datasets that match the specification, and approving release of a subset of the training datasets in the candidate set of training datasets based on the data indicative of the proposed use of the AI ​​training data and the usage rights associated with the training datasets in the candidate set of training datasets.

[0009] In yet another embodiment, the present disclosure relates to a non-transitory computer-readable storage medium including instructions that, when executed by a computer, cause the computer to perform a method including receiving a media unit from a user; creating a training data unit based on a group of media units, each training data unit including a plurality of data blocks, the plurality of data blocks for a given training data unit including: (i) a first data block including media content from a respective one of the group of media units, (ii) a second data block indicating usage rights for the given training data unit, (iii) a third data block indicating media metadata corresponding to the media content in the first data block, and (iv) a fourth data block indicating a user identifier unique to the given training data unit; and storing the training data unit on the non-transitory storage medium or outputting the training data unit over a data network.

[0010] In yet another embodiment, the present disclosure relates to a method for operating a computer to construct a customized training dataset for training an artificial intelligence (AI) platform, the method including: receiving media units from a user; creating a training data unit based on a group of media units, each training data unit including a plurality of data blocks, the plurality of data blocks for a given training data unit including: (i) a first data block including media content from a respective one of the group of media units, (ii) a second data block indicating usage rights for the given training data unit, (iii) a third data block indicating media metadata corresponding to the media content in the first data block, and (iv) a fourth data block indicating a user identifier unique to the given training data unit; and storing the training data unit on a non-transitory storage medium or outputting the training data unit via a data network.

[0011] Features of exemplary embodiments described in this disclosure that are not mutually exclusive may be combined with each other. Elements of one embodiment may be utilized in other embodiments without further recitation. Also, all elements described in connection with a particular embodiment need not be present in all implementations or realizations of that embodiment. Other aspects and features of the present invention will become apparent to those skilled in the art upon review of the following description of specific embodiments in conjunction with the accompanying drawings.

[0012] A detailed description of certain exemplary embodiments is provided herein below with reference to the accompanying drawings. [Brief description of the drawings]

[0013] [Figure 1]FIG. 1 is a block diagram of a computer-based infrastructure for generating customized application-specific datasets for training an AI platform, according to a non-limiting embodiment. [Diagram 2] FIG. 2 is a more detailed block diagram of a user workstation and an administrative server, showing the major functional blocks of the workstation and the major functional blocks of the server, according to a non-limiting embodiment. [Figure 3A] 1 is a graphical user interface (GUI) flow illustrating different GUI controls that allow a user to specify parameters of an application-specific training data set on a workstation implementing the GUI, according to a non-limiting embodiment. [Figure 3B] 1 is a graphical user interface (GUI) flow illustrating different GUI controls that allow a user to specify parameters of an application-specific training data set on a workstation implementing the GUI, according to a non-limiting embodiment. [Figure 3C] 1 is a graphical user interface (GUI) flow illustrating different GUI controls that allow a user to specify parameters of an application-specific training data set on a workstation implementing the GUI, according to a non-limiting embodiment. [Figure 4] FIG. 2 illustrates a format of a training data unit, according to a non-limiting embodiment. [Diagram 5] 1 is a table conceptually illustrating information conveyed in a user rights block of a training data unit, according to a non-limiting embodiment. [Figure 6] 1 is a table conceptually illustrating information conveyed in a metadata block of a training data unit, according to a non-limiting embodiment. [Figure 7] FIG. 1 is a block diagram of an exemplary AI platform, according to a non-limiting embodiment. [Figure 8] 1 is a semantic environment representation output by an AI platform, according to a non-limiting embodiment. [Figure 9] 1 is a flowchart of a process for extracting an application-specific training data set from a training data collection, according to a non-limiting embodiment. [Figure 10] 10 is the flow chart of FIG. 9 according to a variant. [Figure 11] 3 shows the block diagram of FIG. 2 according to a variant. [Figure 12] 12 shows the block diagram of FIG. 11 according to a variant. [Figure 13] 1 is a flowchart of a process for uploading media to a management server for use in generating training data units, according to a non-limiting embodiment. [Figure 14] 1 is a flowchart of a process for generating a training data package from an application-specific training data set, according to a non-limiting embodiment. [Figure 15] FIG. 3 is a block diagram of the system shown in FIG. 2 according to a variant. [Figure 16] FIG. 16 is a block diagram of the system shown in FIG. 15 according to a variant. [Figure 17] FIG. 17 is a block diagram of the system shown in FIG. 16 according to yet another variation. [Figure 18] 1 is a flowchart of a process for generating a training data package, according to a non-limiting embodiment. [Figure 19] FIG. 17 is a block diagram similar to that of FIG. 16 according to yet another variation. [Figure 20] 1 illustrates a data structure for a learned parameter package, according to a non-limiting embodiment. [Figure 21] 1 is a flowchart of a process for generating a learned parameter package, according to a non-limiting embodiment. [Figure 22] 11A, 11B, 11C, and 11D are GUI screens illustrating user controls for selecting parameters in a learned parameter package, according to non-limiting embodiments. [Figure 23] FIG. 1 is a block diagram of a system for performing updates of learned parameters of an AI platform, according to a non-limiting embodiment. [Figure 24] 11A, 11B, and 11C are GUI screens illustrating selectable user options for customizing the operation of an environmental perception system, according to non-limiting embodiments. [Diagram 25] 1 is a flowchart of a process for updating learned parameters of an AI platform, in accordance with a non-limiting embodiment. [Figure 26] 1 is a mapping between information obtained from a user of an AI platform and training data units, according to a non-limiting embodiment. [Figure 27] FIG. 2 is a block diagram of a computing system that can be used to implement various aspects of the management server. [Figure 28] A training data package is conceptually illustrated as a data structure made up of various fields, some of which represent training data units as shown in FIG. 4, according to a non-limiting embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The drawings are illustrative and are not intended to be limiting.

[0015] The present technique will be described in further detail below. This description is not intended to be a detailed list of all the different ways in which the present technique can be implemented or all the features that can be added to the technique. For example, features shown with respect to one embodiment may be incorporated into other embodiments, and features shown with respect to a particular embodiment may be omitted from that embodiment. Furthermore, numerous modifications and additions to the various embodiments suggested herein will be apparent to those skilled in the art upon review of this disclosure, but these modifications and additions do not depart from the present technique. Thus, the following description is intended to illustrate some embodiments of the present technique, but is not intended to exhaustively specify all permutations, combinations, and variations thereof.

[0016] FIG. 1 is a block diagram of a system that allows a user to build customized training data sets for training an AI platform (described in the following paragraphs) to perform environmental perception. The system 10 has different building blocks and functional elements implemented in hardware or software to perform individual tasks. The system 10 includes a set of workstations 12 that communicate with a training data set collection management server 14 via a data network 1. The management server 14, in turn, manages access to a training data set collection stored in a database 16. Based on specifications provided to the workstations 12 that define the desired performance or characteristics of the environmental perception system, the management server 14 builds customized training data collections for the AI ​​platform to train the AI ​​platform to achieve such environmental perception performance or characteristics.

[0017] The main components of an exemplary AI platform that can be trained by a customized training dataset are shown in FIG. 7, with it being understood that a range of different AI platform architectures can be used without departing from the spirit of the present invention. More specifically, the AI ​​platform 70 includes a sensor arrangement 72 that outputs an electronic representation of the environment. The sensor arrangement 72 operates based on a single sensor modality or multiple sensor modalities. For example, the sensor arrangement 72 can use one or more image sensors to output high-resolution camera images. When using multiple sensor modalities, the sensor arrangement 72 can use LIDAR sensors, RADAR sensors, and / or ultrasonic sensors that output a depth map of the environment in addition to the image sensors. The outputs of the different sensors are combined by a raw data fusion functional block 74 that generates a high-resolution 3D map of the environment. Typically, the 3D map may represent the environment with pixels that provide RGB color information. Furthermore, a distance value is associated with each pixel, indicating the distance from a reference point to an object in the environment, a portion or segment of which is visually represented by the pixel. Typically, the reference point is the sensor arrangement 72.

[0018] The high resolution 3D map is then input to a trained neural network 76, which can be a convolutional neural network to perform detection of the objects that the neural network 76 is trained to recognize in the 3D map. The output of the neural network 76 is then fed to a classifier 78 that performs classification of the detected objects. The classifier 78 outputs a semantic representation of the environment depicted by the 3D map. For example, the semantic representation may include an identification in the 3D map of objects of interest, a timestamp and an identification of the speed (velocity + direction of motion) of the detected objects. Figure 8 is an example of a semantic representation, where the top image shows the detected objects with bounding boxes and the bottom image is a velocity map showing the speed of the detected objects.

[0019] For further details regarding suitable, non-limiting AI platforms that may be used as AI platform 70, see WO2021092702A1, which is incorporated by reference herein.

[0020] In an autonomous or semi-autonomous vehicle application, the semantic environment representation is typically input to a driving policy module, which then calculates a vehicle path according to the 3D representation, which is used by the control module to direct the vehicle according to the calculated path. In addition to the 3D representation, the driving policy module may also use additional inputs, such as a current position generated by a GPS module or other map data, to calculate the vehicle path.

[0021] 2, which is a more detailed block diagram of the workstation 12 and the management server 14, the workstation 12 includes a graphical user interface (GUI) functional block 20, which operates to implement a GUI to allow a user to input specifications for the environmental perception system. In a specific example of implementation, the GUI may be implemented via a web browser under the control of the management server 14. Thus, in this form of implementation, the workstation 12 establishes a communication session with the management server 14 over a data network 1 (such as the Internet, or a local area network) and receives scripts or other commands from the management server 14 to implement the GUI, accept user input at the GUI, and receive user messages via the GUI.

[0022] An example of a GUI-mediated workflow that allows a user to define the specifications of a desired environmental perception system is shown in Figures 3A-3C, it being understood that such a workflow is exemplary and many variations are possible.

[0023] A workflow is defined by a series of GUI pages that provide controls for the user through which the user inputs information about the environmental perception system application. User input on one GUI page conditions subsequent GUI pages such that the user is logically directed toward valid input options.

[0024] As an initial GUI page, the user sees page 300 where the user can indicate, via a series of checkboxes or other input mechanisms, a global use class for the environmental perception system. In this example, the global use classes are defined in terms of terrestrial use (ground vehicles), aerial use (drones or other), and maritime use (marine vessels), although other definitions or categories are possible.

[0025] Once the user selects a ground application, a GUI page 302 is displayed requesting input for the particular ground application. Examples of ground applications include, but are not limited to, commercial trucks, commercial buses, robotaxis, shuttles, and agricultural vehicles. In the illustrated example, the user selects "commercial truck." After a selection is made, a GUI page 304 is displayed allowing the user to specify the degree of autonomy level, although it should be understood that the intensity of AI training may be different for higher autonomy levels than for lower autonomy levels where a human driver is responsible for controlling the vehicle at all times.

[0026] In GUI page 306, the user can specify add-on features that allow for automating certain vehicle operations. Again, the selections available in page 306 depend on previous selections made by the user; that is, the system is configured so that the options presented are valid options for the intended use. In this example, "back up to loading dock" is selected. This selection requires the environment perception system to detect the loading dock as an object in the 3D map so that the driving policy module can calculate the backup maneuver. Therefore, the training data needs to be customized for this add-on feature by providing a loading dock image that allows the AI ​​platform to reliably recognize the loading dock.

[0027] Another option in terms of commercial truck applications is automatic trailer free space detection, as described in U.S. Publication US20200062273A1, which is incorporated herein by reference, where the environmental perception system is trained to detect the presence of a trailer attached to the truck, and the size of the trailer, so that the vehicle's free space parameters can be adjusted during autonomous driving.

[0028] In the next GUI page 308, the user specifies the AI ​​platform to be trained. It should be understood that the training data depends on the capabilities and characteristics of the AI ​​platform. For example, an AI platform that uses only a camera to sense the environment may require only camera images for training, while an AI platform that uses sensors of different modalities may require training data generated by a range of sensors. Although not shown in the drawings, it is desirable that the selection list of GUI page 308 is dynamically updated as new AI platforms become available on the market to generate training data sets for those platforms. For example, the management server 14 maintains a database of AI platforms, associating with each entry the hardware and / or software specifications of the platform that influence the selection of training data. As this list of AI platforms is updated in the database, the selections available in GUI page 308 are also updated. More specifically, the AI ​​platforms can be characterized based on the following parameters, which can be specified by the user in GUI page 308: a. The type of sensor (modality), specifically camera, LIDAR, RADAR, and acoustic sensors. b. the number of sensors for each modality; c. The location of the sensors for each modality on the vehicle; d. The capabilities and characteristics of the computing hardware, such as distributed or centralized.

[0029] GUI page 310 allows the user to specify the primary operating environment of the AI ​​platform, again using training data to condition the environment perception performance to adapt to this operating environment. Examples of operating environments include day, night, rain, snow, etc. For example, for a commercial truck intended for night-time only operation, training data should be selected accordingly and will be different from training data for day-time only operation.

[0030] GUI page 312 allows the user to select the region in which the AI ​​platform will operate. The selection of the region may affect the selection of training data from a media content perspective, but is also a useful input from a regulatory compliance perspective. For example, data privacy laws may not allow training data collected within a particular region to be used to train an AI platform operated within a different region. Thus, the selection made on GUI page 312 may be used to ensure regulatory compliance, since only training data approved for the intended use may be identified within the training data collection. It should be appreciated that GUI page 312 provides a general region input that may be customized at the country level on GUI page 314, and may provide further levels of granularity at the state or city level, if desired. Another non-limiting example level of granularity may be based on a division between locations / countries that drive vehicles on the right side of the road and locations / countries that drive vehicles on the left side of the road.

[0031] GUI page 316 of Fig. 3C allows the operator to specify further parameters of the training data. Also, to include some additional objects in the training data that may not be included in the set derived from the previous specification, a GUI page such as that shown in page 318 can be provided. For example, the AI ​​platform to be trained may be operated in an environment where railroad crossings need to be detected with high reliability, and therefore that object needs to be specified.

[0032] 2, in addition to the GUI functionality 20, the workstation 12 also implements a rights management agent 22. The rights management agent is a software program designed to interact with and respond to license challenges from the management server 14 to ensure that the workstation 12 is properly licensed to interface with the management server 14 to build a customized training data set.

[0033] The management server 14 has several functional blocks or modules implemented in software, hardware, or a combination of software and hardware. More specifically, the management server 14 includes a user interface module 24, which is a software component that implements the above-mentioned GUI interface 20 by a web browser. The user interface module 24 establishes a communication session with the workstation 12 and dynamically updates the GUI page when the user provides the specification of the environmental perception system. The GUI hierarchy, options, and input flow of the specification can be stored in a machine-readable storage of the management server 14 and can be referenced by the user interface module 24 to update the GUI page when the user provides the user input, so that these options selectable by the GUI are always valid.

[0034] In addition to managing the GUI pages, the user interface module 24 collects user input and constructs queries that allow for extraction of customized training data sets from the training data collection that match the query specifications.

[0035] The management server 14 also includes a filtering module 26 that interfaces with a database 28 that stores the training data collection. Thus, the user interface module 24 forwards queries to the filtering module 26, which executes the queries to extract training data sets that match the queries from the training data collection in the database 28. The extraction operation is described in further detail below.

[0036] The management server 14 further includes a rights management module 29 that manages the use of the training data in the training data collection to ensure that the use complies with rights and regulatory compliance, as described in more detail below.

[0037] 4, the training data is stored in database 28 in the form of discrete training data units 40. Each training data unit 40 is organized into data blocks (or fields) as follows: a. Training media content blocks 42 include media items with training media used for the actual training of an AI platform, such as the AI ​​platform 70. Such training media can be still camera images with annotations to identify objects of interest in the images. Alternatively, training media can include continuous images, such as videos, with or without audio information. Alternatively or additionally, training media includes environmental representations, such as depth maps, generated by other sensors, such as LIDAR, RADAR, and ultrasonic sensors. The size of the training media content blocks 42 in the training data unit 40 can vary. As shown in FIG. 4, a training media content block 42 is a block that includes several media items 44. A media item 44 can be a single camera or a video frame or a depth map frame. The number of media items 44 per training media content block 42 can vary. Note that in most applications, the media items 44 are of the same sensor modality, such as all camera frames, all LIDAR depth maps, etc. However, it is possible to bundle media items 44 of different sensor modalities into a single training data unit 40, such as 3D map frames obtained by fusing the output of a camera sensor and the output of a LIDAR, RADAR, or ultrasonic sensor. b. Encryption information block 46. The encryption information block 46 stores an encryption key identifier that allows the management server 14 to enable decryption of the information in the training data unit 40, a process that is described below. c. The usage rights information block 48 defines under what circumstances the training media in the training media content block 42 can be used to perform AI platform training. An example of usage rights is shown in the table of FIG. 5. In this example, there are three categories of permissions or rights: applications for the AI ​​platform, areas where the AI ​​platform trained with the training data can be used, and the specific AI platform that can be trained with the training data. This example shows that the training data can be used for terrestrial applications, but not for air or maritime applications, since air and maritime are not listed. It should be understood that the licensed uses can be defined at a finer granularity to distinguish between terrestrial, maritime, or airborne uses. Note that the level of granularity defining the licensed uses in the usage rights information block 48 should match the level of granularity of entering the uses via the GUI at the workstation 12, so that a compliance check can be done by comparing the two. The example of FIG. 5 further specifies the licensed area, which in this case is North America. Thus, the training data cannot be used for AI platforms operating outside of North America. The licensed areas can be defined at a finer granularity, for example, by country. Usage rights information block 48 further specifies the identity of the AI ​​platform(s) that can be trained with this training data. In this example, the licensed AI platforms are defined generically, but it is possible to use a narrower definition, for example by providing a serial number for each AI platform and listing all approved platforms in the Licensed AI Platforms field. This feature is useful when the training data is owned by a third party and needs to be used to train only AI platforms belonging to the third party.In such cases, the Licensed AI Platforms field will only list the third-party AI platforms, so other AI platforms cannot use such third-party specific data. Note that additional usage rights or permissions can be specified, such as an expiration date. Another example is to specify a particular environment in terms of a particular use, such as a military vehicle dataset that can only be used for the Ministry of Defense of a particular country. Another example is to remove certain information in the training media so that the training media is "clean" from objectionable or restricted data. Restricted data may include license plate information or human faces. Yet another example is to specify the type or condition of roads, such as paved roads, unpaved roads, highways or streets, snowy roads, rain-wet roads, etc. d. The metadata block 50 characterizes the actual media content for AI training purposes and enables filtering of training data based on queries. The metadata block 50 characterizes the media with parameters similar to the environmental perception specifications in the GUIs shown in Figures 3A-3C. For example, as shown in Figure 6 in an exemplary table, the characteristics of the training data are based on a use range that may include a list of objects depicted in the media item 44, suitability of the media item 44 in terms of global use (ground), specific use (commercial trucks and buses), and environmental conditions. The list of objects can be generated as a result of human observation of the media item 44, such as an individual looking at an image and identifying and annotating objects appearing in the image. Alternatively, the list of objects can be generated by machine processing of the image, such as processing the image by an AI system that outputs a list of objects and annotations. Optionally, the machine-generated list can be reviewed by a human operator for accuracy. Also, the media content and the list of objects and annotations can be mostly synthetically generated, i.e., generated by a simulation system that creates a virtual environment with virtual objects and renders it as a series of photorealistic images with object lists and annotations, which can be used for AI training in combination with traditional training data material. It is also possible in the metadata block 50 associated with a particular training data (within the training media content block 42) to specify whether the training data is synthetically generated or generated as a result of imaging a real-world environment. Such a distinction can be useful when providing training standards that specify a minimum amount of training data to be obtained from imaging a real-world environment.For example, a specification may stipulate that at least X% of the training data must come from real-world imagery, and at most (100-X)% can be synthetically generated, where X can be 10%, 25%, 50%, 75% or any other percentage. e. A unique identifier block 52 for this training data unit 40 allows this training data unit 40 to be distinguished from other training data units in the collection stored in database 28. The unique identifier block 52 is optional, for example if there is some other way to uniquely identify the training data unit 40. Furthermore, from a traceability perspective, the training data units can be encoded by a blockchain algorithm to ensure that they are not altered and that their origin or source is clearly defined.

[0038] Thus, creation of training data units (such as training data unit 40, also referred to as a training data set) can be accomplished by executing computer-readable instructions stored on a non-transitory computer-readable storage medium. When executed by a computer, such instructions cause the computer to perform a method that includes receiving a media unit from a user, creating a training data unit based on a group of media units, each training data unit including a plurality of data blocks, and storing the training data unit on the non-transitory storage medium or outputting the training data unit via a data network. The plurality of data blocks of a given training data unit include a first data block including media content from a respective one of the group of media units, a second data block indicating usage rights for the given training data unit, and a third data block indicating media metadata corresponding to the media content in the first data block.

[0039] In some embodiments, the plurality of data blocks of a given training data unit further comprises a fourth data block indicating a user identifier unique to the given training data unit.

[0040] In some embodiments, the method further includes grouping the media units into groups of media units.

[0041] In some embodiments, the grouping includes grouping the media units based on objects depicted in the media units.

[0042] In some embodiments, the method further includes performing image processing to identify objects depicted in the media units, and the grouping is performed such as to group together media units that depict at least a portion of the same object.

[0043] In some embodiments, the media metadata in the third data block of a given training data unit includes information related to objects depicted in each group of media units.

[0044] In some embodiments, the method further includes receiving a list of objects depicted in the media units, and the grouping is performed to group together media units that depict at least some of the objects in the list.

[0045] In some embodiments, the media metadata in the third data block of a given training data unit indicates whether the media content of a respective group of media units is derived from real-world imaging or synthetically derived.

[0046] In some embodiments, the method further comprises validating the identity of the user prior to receiving.

[0047] In some embodiments, the storing includes encrypting at least a portion of the training data unit.

[0048] In some embodiments, the encrypting includes encrypting only the first data block of each of the training data units.

[0049] In some embodiments, the encrypting is performed using an encryption key stored in memory.

[0050] In some embodiments, once stored, the media units within the first data block of each of the training data units are decodable by a decryption key.

[0051] In some embodiments, the decryption key and the encryption key are the same.

[0052] In some embodiments, the decryption key is uniquely associated with, but distinct from, the encryption key.

[0053] In some embodiments, the plurality of data blocks further includes a fifth data block indicating how to determine the decryption key.

[0054] In some embodiments, the usage rights for a given training data unit include an indication of the scope of uses licensed for the given training data unit.

[0055] In some embodiments, the usage rights for a given training data unit include an indication of the geographic region licensed for the given training data unit.

[0056] In some embodiments, the usage rights for a given training data unit include an indication of the AI ​​platform that has been licensed to the given training data unit.

[0057] In some embodiments, the method further includes determining usage rights for the given training data unit and storing the usage rights in a second data block of the given training data unit.

[0058] In some embodiments the media unit includes 2D camera images, while in other embodiments the media unit includes LiDAR or RADAR data, and in still other embodiments the media unit includes an ultrasound depth map.

[0059] To further aid in visualizing the relationship between the information requested from the user via the GUI (FIGS. 3A-3C) and the information stored in the training data unit (FIG. 4), reference is made to FIG. 26, which is a non-limiting diagram of how user input (designated 2610) is mapped to a training data unit 40. Here, the user input received from the user via the GUI may be categorized into specifications 2620 and proposed uses 2630, each with their own subfields, while the training data unit 40 has its own organization and subfields, as discussed above with reference to FIG. 4. Specifically: a. The specification 2620 may include: i. a designated list of object subfields 2620-A that maps to a list of object subfields 2670-A in the metadata block 50 of the training data unit 40; ii. a designated application subfield 2620-B that maps to a licensed application subfield 2660-A of the usage rights block 48 of the training data unit 40; iii. A specified minimum % of real-world image subfield 2620-C that maps to a % of real-world image subfield 2670-B of metadata block 50 of training data unit 40. b. Suggested uses 2630 may include: i. A designated identifier of the AI ​​platform subfield 2630-A that maps to a licensed AI platform subfield 2660-B of the usage rights block 48 of the training data unit 50. ii. A proposed geographic location subfield 2630-B that maps to a licensed geographic location subfield 2660-C of the usage rights block 48 of the training data unit 50.

[0060] FIG. 9 is a flowchart illustrating steps in an exemplary process for building a customized training dataset for an AI platform, such as AI platform 70.

[0061] The process begins at 90. In step 92, a user at workstation 12 logs into the management server 14. In step 94, the management server 14, and in particular the rights management module 29, issues a license challenge to the workstation 12 over the data network 1 which is processed by the rights management agent 22. If the rights management agent 22 provides an appropriate response, the management server 14 in step 96 opens a training dataset building session and initiates the environmental perception specification workflow by presenting to the user at workstation 12 the GUI pages shown in Figures 3A-3C.

[0062] In step 98, user interface module 24 collects input entered by the user via the GUI page and constructs a query accordingly. In step 100, the query is passed to filtering module 26, which executes the query to extract a subset of training data units from database 28 that match the query. In particular, filtering module 26 extracts those training data units that have metadata blocks 50 that match the query parameters (see also FIG. 26).

[0063] In step 100, the filtering module 26 outputs a candidate training set, which in step 102 is processed by the rights management module 29 for usage rights compliance. In particular, the rights management module 29 compares the usage rights associated with each training data unit in the candidate data set to user-supplied parameters (again, see also FIG. 26) to determine whether the predicted usage is in compliance. For example, the rights management module 29 compares the forward-looking AI platform application to the licensed application, the forward-looking use domain to the licensed domain, and the forward-looking AI platform to the licensed AI platform. If any discrepancies are found, the training data unit is removed from the candidate set, leaving only the compliant training data units, as shown in step 106.

[0064] The rights management module 29 may then send a message to the user of the workstation 12 informing him of the number of training data units excluded from the candidate set that are not in compliance with the usage rights. This indicates to the user when too many training data units are excluded and too few training data units are left to perform proper training. If the non-compliance can be remedied in some way, the rights management module 29 may trigger such a process. For example, if additional licenses can be obtained, to achieve compliance with the payment, the rights management module 29 calculates the fee to be paid and shows it to the user by means of a GUI page. If the user makes the payment, the rights management module 29 may return the excluded training data units to the original list by permanently or temporarily modifying the license parameters in the usage rights block 48 of the excluded training data units.

[0065] In some embodiments, the training media content blocks 42 of the training data unit 40 are encrypted. This is done to control the use of the media items 44 in the training media content blocks 42. Use of the media items 44 in the training media content blocks 42 is enabled by decrypting the training media content blocks 42. The encryption / decryption process uses data stored in the encryption information block 46. As indicated above, the encryption information block 46 stores an encryption key identifier that maps to an encryption key in the encryption key database 110 (see FIG. 11 ). In order to decrypt the training media content blocks 42 and enable access to the media items 44 that comprise the training media, access to an appropriately identified encryption key in the encryption key database 110 is required, and such access is achieved as described below.

[0066] In a first possible implementation option, the training media content block 42 and some of the other information blocks (such as the usage rights block 48 and the metadata block 50) are encrypted. In this first implementation option, the metadata and user rights information must be accessed to perform decryption to determine whether a particular training data unit matches the query and can be used for a specified application. The decryption process is illustrated in the flow chart shown in Figure 10, which is a variation of the flow chart of Figure 9 and shows an additional decryption step 99. The flow chart of Figure 10 should be read in conjunction with Figure 11, which is a variation of Figure 2 and shows the encryption / decryption functionality of the management server 14.

[0067] 11, the rights management module 29 interfaces with an encryption key database 110 that stores encryption / decryption keys and maps the encryption / decryption keys to respective key identifiers and populates the encryption information block 46 of each of the various training data units 40. For the relevant portion of the encrypted training data unit to be decrypted, the decryption key is extracted from the encryption key database 110-0 and a decryption process is performed. This process is shown in more detail in the flow chart of FIG. 10, where step 99 indicates the additional step of decrypting the relevant portion of the training data unit 40 before executing the query. For each such training data unit to be decrypted, the decryption process requires reading the key identifier in the encryption information block 46, extracting the corresponding decryption key from the encryption key database 110, and decrypting the data in the training data unit 40 (e.g. data in the training data content block 42, the usage rights block 48, and the metadata block 50), so that step 100 can determine whether the training data unit 40 matches the query parameters.

[0068] Before releasing the key from the encryption / decryption key database 110, the rights management module 29 may determine whether the user is permitted to access the training data unit 40 for the purpose of building a training data set. In particular, a distinction is made between the usage rights of the AI ​​platform application, which authorize the use of a particular media to train a particular AI platform, on the one hand, and which allow the user to build a training data set using a training data management tool and use the set to train the AI ​​platform, on the other hand. These two are different processes, and a user building a training data set may require a separate license (build and train license) to build a customized training data set using the data management tool and the training data set collection, and then train the AI ​​platform using the customized training data set. If such a separate license is required, before the decryption key is released from the encryption / decryption key database 110, the rights management module 29 performs the necessary compliance verification to determine whether the user is properly licensed to build a training data set. The compliance verification involves communication between the rights management module 29 and the rights management agent 22, such as establishing the scope of the license granted to the user and determining whether the scope of the license encompasses the anticipated use.

[0069] An example may better explain this. Suppose company A wants to develop an AI platform and needs a suitable training dataset. Company A employs a commercial service that builds a customized training dataset using the infrastructure shown in FIG. 11. However, company A has its own training dataset and wants to bundle this customer-owned training dataset with the training dataset of the commercial service. Company A can therefore provide its own training dataset to the database 28 to integrate the customer-owned dataset into the overall process, provided that it is not used for any other projects involving the commercial service. Therefore, a licensing mechanism is required to ensure that access to the customer-owned training dataset is only allowed for work performed by that customer. Thus, the license rights for the use of the customer-owned training data are set in such a way that access to the training data for the construction of the customized training dataset is only allowed for customer-related projects and not for other projects. Thus, the license rights are set in the rights management agent 22 in the user workstation, and the scope of these rights is established by the rights management module before decrypting the training data unit 40.

[0070] In a second possible implementation, only the training media content block 42 of the training data unit 40 is encrypted. In this form of implementation, the usage rights information (in the usage rights block 48) and the metadata information (in the metadata block 50) are always readable without the need for decryption, while the training media itself remains encrypted, allowing a customized data training set to be constructed.

[0071] FIG. 12 is a variation of FIG. 11 and shows an additional feature, the provision of a rights issuer module 120, which allows third parties to contribute to the training data collection while controlling the use of the training data provided to them. Reference numeral 122 identifies a third party workstation that loads the third party training data. The training data can be organized as a series of media items 44, such as annotated camera images, annotated LIDAR, RADAR and ultrasound depth maps, among others. The third party workstation 122 transmits the training data (i.e., the media items 44) to the management server 14 for integration into the training data collection stored in the database 28. The rights issuer module 120 can package the media items 44 into searchable and manageable training data units 40, associate usage rights with those training data units 40 that the third party can control, and so on.

[0072] Communications between the third party workstations 122 and the management server 14 are handled via a user interface module 24, which may provide a GUI to the third party workstations 122 via a web browser, or any other equivalent arrangement, enabling the user workstations 122 to upload media items 44. The GUI, not shown in the drawings, is made up of controls that enable a user of the workstation 122 to: a. Grouping individual media items 44 together logically, for example based on the object they depict. b. Generating metadata. c. Generate usage rights that specify the Licensed Application, the Licensed Territory, and the Licensed AI Platform.

[0073] Upon input of information items 1, 2 and 3, the rights management module 29 processes the uploaded media items 44 and packages them into training data blocks 42, and creates an encrypted information block 46, a usage rights block 48, a metadata block 50 and a unique identifier block 52 for each training data block 42. The result is a series of created training data units 40, which are then stored in the database 28.

[0074] FIG. 13 is a flow chart illustrating the process of uploading and integrating third party training data into the training data collection.

[0075] The process begins at 130. In step 132, the workstation 122 communicates with the management server 14 and establishes a session, which may require providing user credentials to validate the user identity at the workstation 122. In step 134, the user interface module 24 implements a GUI on the workstation 122 via a web browser with controls that allow the user of the workstation 122 to upload its media items (which are third party training data from the perspective of the management server 14). In step 136, the user of the workstation 122 specifies, via the GUI, the media items 44 to be uploaded, how such media items 44 are grouped into training data units, and, for each such training data unit, the corresponding metadata and usage rights. In step 137, the rights management module 29 processes the uploaded information and generates training data units 40. In step 138, the rights management module 29 may encrypt the training data unit 40 (according to one of the options described above, i.e. either only the media item is encrypted or the metadata and usage rights are also encrypted), and in step 139 the rights management module 29 optionally stores the encrypted training data unit 40 in the training data database 28.

[0076] 12, the process of constructing an application-specific training data set generates a collection of training data units 40, which is a subset of the collection of training data units 40 in database 28. That subset itself constitutes a digital product, and is processed by rights issuer module 120 of management server 14, for example to appropriately package the product (referred to as a "training data package") and associate usage rights with the product, thereby enabling the owner of the digital product to control how the digital product is used downstream.

[0077] With reference to the flow chart of FIG. 14, and further with reference to FIG. 28, the process of creating a training data package 2800 as a product in itself will now be described in more detail.

[0078] The process begins at step 140. In step 142, the rights issuer module 120 receives the application specific training data set 2804 generated by running a query on the database 28. By "receive" it is meant that the rights issuer module 120 has access to the training data units 2804A, 2804B, ... 2804C selected by running the query. For example, the rights issuer module 120 may be provided with an index in the database 28 to point to entries in the database 28 that correspond to the selected training data units 2804A, 2804B, ... 2804C, thus enabling the rights issuer module 120 to fetch the training data units 2804A, 2804B, ... 2804C in sequence from the database 28 and process them.

[0079] In step 144, the rights issuer module 120 accesses the media items 44 in each training data unit 2804A, 2804B, ... 2804C in the application-specific training data set 2804. This may require decrypting the media items based on a decryption key obtained from information in the encrypted information block 46 of each training data unit 2804A, 2804B, ... 2804C.

[0080] In step 145, the rights issuer module 120 generates usage rights associated with the training data package (generally designated 2800) so as to be able to manage use of the training data package 2800 for training the AI ​​platform. The usage rights are located in a usage rights data block 2848 (or field) of the training data package 2800.

[0081] In a specific implementation example, the rights issuer module 120 generates usage rights for the entire training data package 2800 based on the usage rights of the individual training data units 2804A, 2804B, ... 2804C it contains. It is conceivable that the application-specific training data set 2804 may hold training data units 2804A, 2804B, ... 2804C owned by different entities and associated with different usage rights. Thus, the generation of usage rights for the training data package 2800 needs to take into account the individual usage rights. There are two possibilities: The usage rights of every training data unit 2804A, 2804B, ... 2804C in the application-specific training data set 2804 are identical. In such an example, the same usage rights are replicated in the training data package 2800. That is, the training data package 2800 inherits the same usage rights (with respect to the licensed use of the AI ​​platform, the licensed domain, and the licensed AI platform) as any individual training data unit 2804A, 2804B, ... 2804C in the application-specific training data set 2804. b. The usage rights in the training data units 2804A, 2804B, ... 2804C of the application-specific training data set 2804 are not identical. Although the usage rights under each training data unit 2804A, 2804B, ... 2804C are consistent with compliance with the environmental perception system specification specified by the user, the usage rights may not be the same across the training data units 2804A, 2804B, ... 2804C. For example, the training data unit 2804A may have usage rights specifying an application licensed to the AI ​​platform for ground and air use, while the training data unit 2804B may only permit ground use. In such a case, the air use license is revoked and only the ground use license is carried over to the training data package 2800. In such a way, the collective usage rights under the training data package 2800 remain consistent with the environmental perception system specification. In this example, the usage rights for the entire training data package 2800 (e.g., with respect to the licensed applications, licensed areas of the AI ​​platform, and the licensed AI platform) are common to each of the individual training data units.

[0082] Step 146 generates metadata information data blocks 2850 (or fields) of the training data package 2800. In a specific implementation, the metadata of the training data package 2800 is a union or union of the metadata of the individual training data units 2804A, 2804B, ... 2804C. For example, the metadata of the training data package 2800 lists all objects that appear in the media items of the training data package. Optionally, the metadata of the training data package 2800 may also provide statistical information about the entire training data package 2800, such as the total number of objects, the relative distribution between the objects (20% bicycles, 40% cars, 18% pedestrians), the total number of media items in the package, etc. More specifically, the rights issuer module 120 extracts metadata from the metadata blocks 50 of every training data unit 2804A, 2804B, ... 2804C in the application-specific dataset 2804, performs a statistical analysis, and stores the results in the metadata information block 2850 of the training data package 2800.

[0083] In step 147, the rights issuer module 120 generates a unique identifier for the training data package 2800 and places it in a unique identifier data block 2852 (or field). Optionally, in step 148, the rights issuer module 120 can generate an encryption information block 2846 (which identifies a key stored in the encryption key database 110), and finally, in step 149, the portion of the training data package 2800 is encrypted.

[0084] Once the training data package 2800 is completed, it is stored in the training data package database 150, as shown in FIG. 15, which is a variation of FIG.

[0085] FIG. 16, a variation of FIG. 15, illustrates another example of an implementation of management server 14 that includes functionality for using training data packages (such as training database package 2800) stored in training data package database 150 to perform training of AI platform 162. More specifically, management server 14 includes AI platform training module 160, which receives training data packages from training data package database 150 and performs a training process based on media content in the training data packages. In addition to the training data packages, AI platform training module 160 also receives a model of AI platform 162 to train. The output of AI platform training module 160 is a set of learned AI parameters of AI platform 162 that are generated as a result of training the model with the training data. Examples of learnable parameters are weights and biases. The particular weights and biases generated by the training process are learned parameters of AI platform 162. For more information regarding AI platforms and systems, reference may be had to Artificial Intelligence: Fundamentals and Applications, Cherry Bhargava and Pradeep Kumar Sharma (Eds.), ISBN 9780367559700 (published by CRC Press on July 29, 2021), which is incorporated herein by reference.

[0086] Before the AI ​​platform training module 160 can access and use a training data package to perform training operations, the training data package may need to be decrypted, and such decryption is subject to compliance verification against the usage rights attached to the training data package.

[0087] Consider a scenario in which a user of a workstation 12 communicates with the management server 14 to control a training operation by the AI ​​platform training module 160. As in the previous example of interaction with the management server 14, the user works through a GUI implemented by the user interface module 24. The user of the workstation 12 provides input that enables the rights management module 29 to determine whether the use of the training data package 2800 is in compliance with the usage rights associated with the training data package. For example, the user specifies, through the GUI, the intended application of the AI ​​platform 162 to be trained, the intended domain in which the AI ​​platform 162 will be used, and provides an AI platform identifier. The rights management module 29 then compares the intended use specifications provided by the user through the GUI with the usage rights in the usage rights block 2848 of the training data package 2800, and if they match, the AI ​​platform training module 160 may proceed to execute the training operation.

[0088] In a variant, the AI ​​platform model on which training is being performed is encoded with a platform ID, which can be read directly by the rights management module 29, eliminating the need for the user of the workstation 12 to provide this information.

[0089] Thus, what is provided is a method for operating a computer to construct a customized training dataset for training an artificial intelligence (AI) platform, the method including obtaining (i) a specification of training data for training the AI ​​platform and (ii) data indicative of a proposed use of the training data, querying a database of training datasets, each associated with usage rights, to identify a candidate set of training datasets that match the specification, and authorizing release of a subset of the training datasets in the candidate set of training datasets based on the data indicative of the proposed use of the AI ​​training data and the usage rights associated with the training datasets in the candidate set of training datasets.

[0090] In some embodiments, the specifications include an application scope, the application scope including at least one of (i) global application and (ii) specific application, and may further include a list of objects, and may include a minimum percentage of training data derived from real-world imaging or a maximum percentage of training data derived synthetically.

[0091] In some embodiments, the data indicative of the proposed use of the training data includes one or more of: (i) an identifier of the AI ​​platform; and (ii) a proposed geographic region.

[0092] In some embodiments, the method further comprises implementing a graphical user interface to obtain the specification, the data being obtained from a user.

[0093] In some embodiments, approving includes creating a subset of the candidate set of training datasets, the subset including those training datasets in the candidate set whose usage rights include the proposed use of the AI ​​training data.

[0094] In some embodiments, the subset of the candidate set of training datasets excludes any training datasets in the candidate set whose usage rights do not include the proposed use of the AI ​​training data.

[0095] In some embodiments, the specifications and data indicating a proposed use of the training data are obtained from a user, and the method further includes determining whether the user has a license to train the AI ​​platform.

[0096] In some embodiments, at least a portion of the training data set in the database is encrypted, and if it is determined that the user has a license to train the AI ​​platform, the query includes decrypting at least a portion of the training data set in the database.

[0097] In some embodiments, each of the training data sets has a common format that includes a plurality of data blocks, the plurality of data blocks including a first data block including training media for the respective training data set, a second data block indicating usage rights for the respective training data set, a third data block indicating media metadata corresponding to the training media in the first data block, and optionally a fourth data block indicating a user identifier unique to the respective training data set.

[0098] In some embodiments, the plurality of data blocks further includes a fifth data block indicating encryption information associated with the first data block.

[0099] In some embodiments, the media metadata corresponding to the training media in the first data block includes a list of objects depicted in the training media.

[0100] In some embodiments, the media metadata corresponding to the training media in the first data block further indicates whether the training media is derived from real-world imaging or synthetically derived.

[0101] In some embodiments, the media metadata corresponding to the training media in the first data block further includes annotations.

[0102] In some embodiments, to identify a candidate set of training data sets that match the specification, the method includes comparing the specification to media metadata in the third data block of each of the training data sets, the candidate set of training data sets including those training data sets for which the media metadata in the third data block of each training data set matches the specification.

[0103] In some embodiments, the specification includes a user-provided use scope, while in other embodiments, the specification includes a user-provided list of objects. The specification may also include a user-specified use scope, and the usage rights in the second data block include information related to the licensed use scope for each training data set, and to identify a candidate set of training data sets matching the specification, the method includes comparing the user-specified use scope with the licensed use scope, and the candidate set of training data sets includes those training data sets for which the licensed use scope in the second data block of each training data set includes the user-specified use scope.

[0104] In some embodiments, the specification includes a user-specified list of objects, and the media metadata corresponding to the training media in the first data block includes information relating to the list of objects depicted in the training media in the first data block, and to identify a candidate set of training datasets that match the specification, the method further includes comparing the user-specified list of objects to a list of objects in a third data block of each of the training datasets, and the candidate set of training datasets excludes those training datasets where the list of objects in the third data block of each of the training datasets does not include any of the objects in the user-specified list of objects.

[0105] In some embodiments, approving includes creating a subset of the candidate set of training data sets, the subset including in the candidate set those training data sets where the usage rights in the second data block of the respective training data set include the proposed use of the AI ​​training data.

[0106] In some embodiments, the proposed use of the AI ​​training data includes an identifier of the AI ​​platform, and the usage rights in the second data block of each of the training datasets identify at least one licensed AI platform, and the approving includes creating a subset of a candidate set of training datasets, the subset including in the candidate set those training datasets where at least one licensed AI platform identified by the second data block of the respective training dataset includes an identifier of the AI ​​platform.

[0107] In some embodiments, the proposed use of the AI ​​training data includes a proposed geographic region and the usage rights in the second data block of each of the training datasets further include a licensed geographic region, and the approving includes creating a subset of a candidate set of training datasets, the subset including in the candidate set those training datasets where the licensed geographic region in the second data block of each training dataset includes the proposed geographic region.

[0108] In some embodiments, the method further includes determining usage rights for the subset of the training data set and storing the determined usage rights in a database associated with the subset of the training data set.

[0109] In some embodiments, the method further includes setting usage rights for the subset of the training data set based on usage rights associated with each of the training data sets in the subset of the training data set.

[0110] In some embodiments, if the usage rights associated with each of the training datasets in the subset of training datasets are identical, then the usage rights of the subset of training datasets are set to the same usage rights associated with each of the training datasets in the subset of training datasets.

[0111] In some embodiments, if the usage rights associated with each of the training datasets in the subset of the training dataset are not identical, then the usage rights of the subset of the training dataset are set to those usage rights that are common to each of the training datasets in the subset of the training dataset.

[0112] In some embodiments, the training data sets each have a common format including a plurality of data blocks, the plurality of data blocks including a first data block including training media for the respective training data set, a second data block indicating usage rights for the respective training data set, a third data block indicating media metadata corresponding to the training media in the first data block, and optionally a fourth data block indicating a user identifier unique to the respective training data set, and the method further includes grouping subsets of the training data sets into training data set packages and storing the training data set packages on a non-transitory medium.

[0113] In some embodiments, the method further includes providing the training dataset package with a usage rights field that is populated based on the usage rights stored in a second data block of the training dataset in the subset of the training dataset.

[0114] In some embodiments, the content of the usage rights field of the training data set package is the intersection of the usage rights stored in the second data block of the training data set within the subset of the training data set.

[0115] In some embodiments, the method further includes providing the training dataset package with metadata fields that are populated based on metadata stored in a third data block of the training dataset in the subset of the training dataset.

[0116] In some embodiments, the content of the metadata field of the training dataset package is a union of the metadata stored in the third data block of the training dataset in the subset of the training dataset.

[0117] FIG. 17 illustrates yet another variation in which the training of the AI ​​platform 162 is performed at a remote location from the management server 14. FIG. 17 illustrates a scenario in which the training must be performed at a specified location to meet compliance with certain requirements, such as governing regulations governing the use of training data and / or the use of learned parameters generated from the training data. For example, such regulations may stipulate that training data collected in a particular region or country cannot be exported or otherwise used outside of that country or region. Thus, to meet compliance with such regulations, the training data must be stored in the region and country, and the training data must be processed to perform the AI ​​training within such country or region. Additionally, the training of the AI ​​platform 162 may require the use of additional training data that resides outside of the region or country. The system of FIG. 17 is configured to enable the transfer of training data that is not subject to storage and / or usage restrictions to a remote site to augment local training data that is subject to storage and / or usage restrictions in order to provide an effective training dataset for training the AI ​​platform.

[0118] More specifically, the system includes a second managed server 14A, similar in structure and function to managed server 14, residing in a country or region that imposes restrictions on the training data and / or learned parameters. Specifically, managed server 14A has a user interface module 24A (similar to user interface module 24), a rights issuer module 120A (similar to rights issuer module 120), a filtering module 26A (similar to filtering module 26), a rights management module 29A (similar to rights management module 29), and an AI platform training module 160A (similar to AI platform training module 160). Additionally, managed server 14A communicates with training data collections stored in database 28A (similar to database 28), encryption key database 110A (similar to encryption key database 110), and training data package database 150A (similar to training data package database 150).

[0119] The operation of the management server 14A is similar to that of the management server 14 in that the management server 14A can receive input from a workstation (such as workstation 12) to specify the environmental perception system, execute queries via the filtering module 26A to generate a customized training data set, generate a corresponding training data package, and perform AI platform training using the training data package.

[0120] The filtering module 26A is configured to have visibility across both databases 28 and 28A. Thus, upon execution of a query, the filtering module 26A may flag training data units in one or both of the databases 28, 28A as relevant. Such remote visibility is possible when no restrictions are imposed on the training data units stored in the database 28, allowing external entities, which may be entities residing outside the region or country, to read metadata associated with such training data units. The reverse approach is also possible, i.e., allowing the filtering module 26 to have visibility into the training data stored in the database 28A, provided that regulations governing the use and / or storage of the training data units in the database 28A allow access to the corresponding metadata.

[0121] The process for building a training data package in the management server 14A is illustrated by the flowchart of Figure 18, which is a variation of the flowchart of Figure 9 illustrating the extraction of application-specific training data sets by the management server 14. Steps that are similar in both flowcharts are designated with the same reference numbers.

[0122] At step 92, an operator of a workstation 12 (in this case communicating with management server 14A) logs in using suitable credentials, such as a username and password, among others.

[0123] In step 94, the rights management module 29A issues a license challenge to the workstation 12 to determine the license granted to the workstation / user and makes available to the workstation / user services consistent with the scope of the license. It should be noted that in this implementation, the rights management module 29A that handles the license validation is implemented in the management server 14A. In a possible variant, the license validation operation can be performed by the rights management module 29 of the management server 14. In other words, the rights management module that performs step 94 can be a central license manager tool that serves various remote management servers and handles license validation services for all transactions on the remote management servers.

[0124] In step 96, the management server 14A initiates the environmental perception specification workflow as described above, where the user inputs the specifications for the environmental perception system.

[0125] In step 98, a query is constructed.

[0126] In step 180, a process is performed to establish data visibility across all remote training data collection databases (in this case databases 28, 28A). The purpose of this part of the process is to establish in the management server 14A a global list of all training data units stored in the remote databases so that queries can be performed in the global list to identify relevant training data units within the entire training data collection stored across multiple databases that satisfy the query parameters. In practice, although not necessarily, this can be performed by storing in each training data collection database 28, 28A a global list of training data units within the entire collection, each entry in the list corresponding to a training data unit, as shown in step 100, including metadata and optionally other information blocks, but omitting media units, so that queries can be performed.

[0127] Steps 102, 104 and 106 are then performed on the entries in the global list to generate a final set of training data units that satisfy the parameters of the query. For the sake of this example, let us assume that 80% of the training data units so selected reside in database 28A and 20% reside in the (remote) database 28. To access the media units of the training data units stored locally in database 28A, the rights management module 29A performs the necessary license validation and fetches a decryption key from the encryption key database 110A, as described above. The decryption key can then be used to decrypt the media items in the training data units stored in the local database 28A. With respect to the data training units stored in the remote database 28, a similar operation is performed, but this time involving the rights management module 29 of the management server 14. More specifically, a request is sent from the management server 14A over the data network 1 to the management server 14, requesting the media units and the decryption key associated with a certain number of the training data units stored in the database 28. The rights management module 29 performs a validation operation to determine whether the intended use is in compliance with the license (e.g., to determine, among other things, whether the license allows for export of the training data to a different site), and if so, the training data unit is extracted from the database 28 and the decryption key is fetched from the encryption key database 110, and both are relayed back to the management server 14A, and in particular the rights management module 29A, so that now the 20% media of the training data unit can be decrypted.

[0128] It should be noted that the above arrangement allows for the use of training data stored in a distributed manner at sites subject to different legal / regulatory scopes. An advantage of this system is that it provides an efficient and flexible mechanism for managing the use of training data, including maintaining legal / regulatory compliance. For example, training data that is subject to an export ban is blocked from being sent to a remote site because the usage rights associated with the training data authorize local use only, and therefore the rights management module blocks the export of the data and does not provide a decryption key to the remote site. However, for training data that is exportable to a remote site, if the usage rights associated with the training data authorize use of the training data at the remote site, the rights management module 29A can use the media to transfer the data and provide a decryption key to the remote site.

[0129] In step 182, the decoded training data is assembled into a training data package, in step 184 stored in training data package database 150A, and in step 186 provided to AI training module 160A for performing training operations.

[0130] The AI ​​platform training modules 160 and 160A include high-speed data processors, such as high performance graphics processing units, to process the media in the training data packages and generate learned parameters for the AI ​​platform. In a specific example, the learned parameters are weights and biases of a neural network. The learned parameters can then be loaded into the AI ​​platform and used to perform environmental perception. For example, the learned parameters can be loaded into an AI platform implemented in a vehicle, such as to provide environmental perception, and then used for autonomous or semi-autonomous operation of the vehicle.

[0131] In a specific example, the learned parameters are generated at a particular site and loaded into individual AI platforms in a fleet, which may include hundreds or even thousands of units. As previously mentioned, the AI ​​platforms are typically vehicles, but can be implemented in any type of automated equipment that requires environmental perception to provide autonomous capabilities, including land, air, and sea vehicles for a variety of applications, including agriculture, mining, industrial, commercial delivery, and human transportation.

[0132] FIG. 19 is a block diagram, a variation of FIG. 16, including a module 190 for performing distribution of learned parameters to multiple AI platforms. In a specific implementation example, the AI ​​platform can be a vehicle. Distribution can include an initial loading of learned parameters on the AI ​​platform. That is, the AI ​​platform is initially non-functional in the sense that it is not yet programmed to perform environmental perception. That is typically the case when a vehicle is manufactured and hardware implementing the AI ​​platform is installed in the vehicle. The initial loading of learned parameters, which can be performed on the production line, provides the AI ​​platform with environmental perception capabilities, such as to enable autonomous or semi-autonomous functionality.

[0133] The distribution of learned parameters by the system of Figure 19 can also be performed in the context of an update, in other words, existing learned parameters in the AI ​​platform are replaced with new learned parameters to provide different or enhanced environmental perception capabilities.

[0134] The learned parameter delivery module 190 communicates with the AI ​​platform training module 160 to receive from the AI ​​training module 160 learned parameters generated as a result of a training operation. The learned parameter delivery module 190 also communicates with the rights management module 29 and with the rights issuer module 120 to create learned parameter packages associated with usage rights, for example, to be able to manage the use of the learned parameters within the AI ​​platform.

[0135] The AI ​​platform fleet is generally indicated at 194 in FIG. 19. The fleet has a number of individual AI platforms that can receive the trained parameter package by over-the-air (OTA) updates or by any other suitable communication method. For example, each AI platform can be provided with communication capabilities over a data network 1, such as the Internet, and each AI platform can periodically communicate with a management server 14 to receive the trained parameter package and then install the trained parameter package, enabling the AI ​​platform to perform environmental perception.

[0136] 20 illustrates a general data structure of a learned parameter package 200. Note that the data structure of the learned parameter package 200 has a similar structure to the structure of the training data unit 400. In particular, the learned parameter package 200 includes a learned parameter data block 202, an encryption information data block 204, a usage rights data block 206, and a unique identifier data block 208.

[0137] The learned parameters data block 202 holds the weights and biases used by the AI ​​platform to provide environmental perception, the encryption information block 204 holds encryption information, the usage rights data block 206 stores information regarding the permitted use of the learned parameters, and the unique identifier 208 uniquely identifies this learned parameters package 200 among other packages.

[0138] The flowchart in FIG. 21 provides an example of a process for generating a learned parameter package 200.

[0139] The process begins at 210. In step 212, the AI ​​platform training module 160 executes a training process and generates learned parameters. In step 214, the rights issuer module 120 generates a usage rights data block 206 that determines the permitted uses of the learned parameters. In a specific implementation, the usage rights of the learned parameter package 200 are derived fully or partially from usage rights associated with the upstream training data, i.e., the training data package 2800 and / or the training data units 40. In a specific implementation, the usage rights associated with the learned parameter package are inherited from usage rights associated with the training data package, which are at least partially inherited from usage rights associated with the individual training data units. In a specific implementation, assume that the training data package has usage rights including: a. Applications licensed for ground use only; b. Licensed territory limited to Canada and the US; c. Licensed AI Platform limited to Alpha and Delta.

[0140] The rights issuer module 120 copies the same usage rights into the usage rights data block 206. The rights issuer module 120 then generates a unique identifier in step 216, generates an encrypted data block in step 218, and finally encrypts the learned parameter package in step 220.

[0141] In a variant, the user (operator) is offered the possibility to modify the usage rights of the learned parameter package, modifying them and adapting them to a particular situation, via a user interface. The user interface is a graphical user interface, displaying information to the user and offering various controls that enable the user to enter information to modify the default usage rights or to specify certain parameters of the usage rights.

[0142] 22a is an example of a GUI 2200 that allows a user to view the default usage rights for a learned parameter package 200, which may be inherited from the usage rights of a training data package, and also provides the user with the possibility to make changes to these usage rights. In this example, the GUI 2200 has a series of tabs that correspond to usage rights categories in order to present information to the user in a logical and easy to understand manner. There is an "Applications" tab, a "Domain" tab, and an "AI Platform" tab. However, it should be understood that other ways of organizing information may be used without departing from the spirit of the invention.

[0143] The Applications tab 222 shows three major categories of applications that identify those inherited from the training data package. Because this is the major category of usage rights, the application usage rights may not be user selectable, and the tab 222 may be provided only to inform the user of the application usage rights that apply. That is, the GUI is configured such that user input to change the selection from "Ground" to another selection, e.g., Air or Sea, is not accepted by the GUI.

[0144] In the variation shown in FIG. 22b, the application tab 222 can be used to define the licensed applications at a finer granularity under a particular category that is not fixed and selectable. For example, for a licensed ground application usage, the tab 222 shows the range of subcategories that the user can select. In this way, if global usage rights are inherited in relation to a particular category, the user can limit the scope of downstream rights to a particular subcategory. In this example, the global ground application category has three subcategories: (1) robotaxis, (2) commercial trucks, and (3) shuttle vehicles. A selection mechanism is enabled on the tab 224, allowing the user to select a single subcategory or multiple subcategories and then "save" the selection(s).

[0145] 22c, the "Area" category on tab 226 has been selected by a user to show several sub-areas within the global "North America" ​​region. The subcategories presented to the user include "Alaska," "Eastern US," and "Western US," among many other possibilities. The user can select the appropriate sub-category within the "Area" category and save the selection(s).

[0146] FIG. 22d illustrates the “AI Platform” category 228, which includes finer granularity controls provided to the user to select an AI platform. In this example, the inherited usage rights extend to the AI ​​platform category “Alpha”, i.e., each subcategory of the “Alpha” category is licensed to use the learned parameter package. The user is provided with a list of selectable options, each corresponding to a particular subcategory, i.e., D-9653, K-789, and Z-679. The user can select one or more of those subcategories, and such selection operates to restrict the usage rights to only the selected categories. In other words, the subcategories of the “Alpha” category that are not selected are not licensed to use the learned parameter package, even if the training data package has a blanket license for the “Alpha” category. The user selection in tab 228 is then saved.

[0147] After the user makes the desired inputs in the GUI shown in Figures 22a-22d, the selections are sent to the rights issuer module 120, which constructs the usage rights data block 206 accordingly, i.e., adds any constraints or permissions derived from the user inputs to the listed usage rights.

[0148] It should be appreciated that once assembled, the trained parameter packages 200 are stored in the trained parameter package database 192, and that each trained parameter package may be associated with metadata to provide a description of the parameters of the trained parameter package. Although not shown in FIG. 20, the trained parameter packages 200 may include additional data blocks for storing such metadata about the trained parameter packages 200.

[0149] 23 shows a block diagram of a system for remotely updating the learned parameters of an AI platform. The system includes the management server 14 and an AI platform 230 described above, where the AI ​​platform may be integrated into a vehicle, such as a ground vehicle, an air vehicle, or a marine vehicle. For purposes of this example, the vehicle is a ground vehicle.

[0150] The AI ​​platform 230 includes a sensor set 242, which may include a variety of sensors of a single modality or different modalities, as described above. The output of the sensor set 242 is provided to a perception stack 240, which includes software that processes the output of the sensor set to provide environmental perception and generate a 3D map of the environment. The output of the perception stack 240 is provided to a driving policy stack 238, which includes software for determining vehicle motion.

[0151] The perception stack 240 uses the learned parameters to perform environmental perception. The system of Figure 23 is designed to provide remote updates of the learned parameters, either to improve environmental perception performance or to adapt it to provide additional customer selectable features not previously included or new features not previously available.

[0152] The GUI module 234 allows a user to control the parameters of the update process. The GUI module 234 communicates with a communications manager 232, which includes hardware and software elements that allow the AI ​​platform 230 to communicate with the management server 14. In a specific implementation, the communications manager 232 is configured to enable wireless communication with the management server 14, for example, via a cellular network or equivalent. The AI ​​platform 230 also includes a rights management agent 236, which controls how the AI ​​platform 230 is permitted to use the learned parameters downloaded from the management server 14. The rights management agent 236 communicates with the rights management module 29, so that the learned parameters downloaded from the management server 14 can be decoded and used by the AI ​​platform 230 if properly licensed for such use, as established by the rights management agent 236.

[0153] The GUI module 234 provides the user with several selectable options, such as: Customize the learned parameter download process: For example, the GUI module 234 includes a control that allows a user to select between enabling automatic downloading of updates or disabling automatic downloading. b. Customizing Environmental Perception. The user is provided with options to adapt the environmental performance to his or her needs, such as adding optional features to the basic environmental perception capabilities. In a specific implementation example, the basic environmental perception capabilities are configured to detect a minimum range of objects that the AI ​​platform can see to provide a desired level of automation. In terms of autonomous driving in cities, the minimum range of dynamic objects may include cars, trucks, pedestrians, bicyclists, traffic signs, fire hydrants, and construction markers, such as construction cones. Such a basic range of objects enables basic automation capabilities, such as driving in urban environments, including urban streets and highways. However, users of the AI ​​platform may wish to add further automation capabilities to such basic automation capabilities, such as the ability to provide autonomous or semi-autonomous operation in different environments. Examples of such different environments are provided below: i. For commercial vehicles such as trucks, the ability to operate within a particular environment or situation. A first example is a truck backing up to a loading dock. To enable an automated loading dock approach, the environmental perception system needs to be able to detect the loading dock itself, i.e. the learned parameters need to be configured to identify the loading dock as an object in the 3D map. A second example concerns a vehicle operating partly in an industrial environment and partly in an urban environment. An industrial environment may be quite different from an urban environment, as it includes objects not typically found in urban environments. Take the example of a truck transporting containers from a port to a destination within a city. While a basic range of detectable objects may be sufficient for driving in a city, additional object detection capabilities are required to enable autonomous operation in a port environment where unique objects need to be detected. ii. For personal transport vehicles, the ability to park in a complex environment, such as an underground parking lot, where the vehicle needs to navigate around many obstacles, such as pillars. To provide such an automatic parking function, the environment perception software needs to be configured to detect the obstacles. Another example from the perspective of personal transport vehicles, which may also be relevant for commercial vehicles, is the additional function to automatically sense the boundaries of the entire vehicle, such as when a trailer is coupled to the vehicle. In this case, the environment perception system is configured to "see" the trailer and output the trailer as a dynamic object that the driving stack can take into account to calculate "free space" parameters for driving, which basically include vehicle dimensions and trailer dimensions. iii. Taking into account the domain, such as updating the learned parameters based on the vehicle's current location. Updated parameters for a new domain in which the vehicle needs to operate may include (1) updating the license to extend usage rights that were previously limited to a particular domain to the new domain. In such cases, the learned parameters are not changed, but the usage rights are updated to authorize operation within the new domain.

[0154] Examples of GUIs implemented by GUI module 234 that allow a user to customize the environment perception system are shown in Fig. 24a, Fig. 24b, and Fig. 24c. GUI 300 has a number of screens arranged as tabs corresponding to different parameters. In this example, there are four tabs: (1) User Authentication, (2) Operation Settings, (3) Customization, and (4) Payment. The description of GUI 2400 is made in conjunction with the flow chart of Fig. 25, which illustrates the different steps of the process for updating the learned parameters of AI platform 230.

[0155] The process begins at 250. In step 252, a user (e.g., a vehicle occupant) logs into the system by providing user credentials, such as a username and password. The communications manager 232 then establishes a communications session with the management server 14, and in particular with the learned parameter delivery module 190. Assuming the user authentication is successful, the learned parameter delivery module 190 in step 254 retrieves a user profile associated with this particular AI platform. The user profile stores information about the user and the AI ​​platform, and uses this information to adapt the selectable options in the GUI 2400 such that only options that are valid for that particular customer and are also valid for the particular AI platform are shown on the GUI page.

[0156] FIG. 24b shows the Operational Settings tab 256. In this example, the user is provided with two controllable settings. One of the settings is the Learned Parameter Auto Update setting, which can be enabled or disabled. The Auto Update setting allows the management server 14 to push automatic updates over the data network 1. Providing such updates typically fixes bugs or improves the reliability of basic object detection. The other controllable setting is the Licensed Area Auto Update setting, which can be enabled or disabled. This setting provides the ability to automatically push updates to allow seamless driving from one licensed area to another. If the user sets this setting to “enabled”, the AI ​​platform 230 will notify the management server 14 if the vehicle is moving from one licensed area and request a learned parameter and / or license update, allowing the current parameters to be used outside of the original licensed area. The Rights Management Agent 236, responsible for the usage tracking function, receives location information from various possible sources, such as a 3D map generated by the environmental perception system, and triggers an automatic update if it determines that the vehicle is operating outside of the licensed area. An automatic update involves the rights management agent 236 generating a message to the management server 14 to indicate the current area in which the vehicle is traveling. In response to this message, the learned parameter delivery module 190 determines what the vehicle needs in terms of learned parameters, extracts them from the database 192, and sends them along with a decryption key to the rights management agent 236 so that they can be decrypted and installed. If only a license update is required, no new learned parameters are pushed to the AI ​​platform 230, and only a license update is generated and sent to the rights management agent 236. Additionally, a transaction event can be triggered if the user needs to be charged for the license extension.

[0157] Referring back to FIG. 25, operating parameter settings related to automatic updating of learned parameters and automatic updating of licensed regions are received at step 258 and stored in a user profile at step 260 .

[0158] Now referring to FIG. 24c, the customization tab 262 will be described in detail. This tab 262 provides a control to the user to allow the user to select dedicated operation capabilities for autonomous or semi-autonomous operation of the vehicle, which requires certain learned parameters. In terms of add-on features, the user is provided with five selectable options, and it should be understood that more or less options can be provided without departing from the spirit of the invention. The options include loading dock operations, port operations, rail yard operations, smelter yard operations, and airport taxiway operations. Tab 262 also provides a licensed region option to configure the AI ​​platform for operation within a particular region. The example shown provides a selection between Eastern US and Western US.

[0159] As shown in the flowchart of FIG. 25, after receiving the customization options selected by the user by the learned parameter delivery module 190 in step 264, the customization options are stored in the user profile in step 266. It should be understood that in step 268, the learned parameter delivery module 190 retrieves the learned parameter package corresponding to the user's selection from the database 192, and the learned parameter delivery module 190 applies a filter to extract only the learned parameter packages that are compatible with and / or licensed to the AI ​​platform. That is, the learned parameter packages that are not compatible with or approved for use with the AI ​​platform are not selected. The filtering operation is performed based on the usage rights block 206 in the learned parameter package that lists the acceptable AI platforms.

[0160] The retrieved learned parameter package can then be displayed to the user via GUI 2400, allowing the user to make a selection. Optionally, pricing information can be presented to the user if use of the selected learned parameter package is subject to payment. Payment can be made via a payment tab in GUI 2400. In step 270, following payment processing, the learned parameter package is pushed to the AI ​​platform 230, including the decryption key, allowing the rights management agent 236 to decrypt the learned parameter package and provide it to the perceptual stack 240 for installation and subsequent use.

[0161] Aspects of the present disclosure may be implemented by any combination of hardware and software. When implemented as a computer-implemented device, such aspects may be implemented using means for performing all of the steps and functions described above. Aspects of the present disclosure may also be included in an article of manufacture (e.g., one or more computer program products), e.g., having a computer usable medium. The medium has embodied therein, e.g., computer readable program code means for providing and facilitating the mechanisms of the present invention. The article of manufacture may be included as part of a computer system or sold separately.

[0162] For purposes of illustration, application programs and other executable program components are illustrated herein as separate blocks, with the understanding that such programs and components reside at various times in different storage components of the computing system and are executed by the computer's data processor(s). An implementation of the content control software may be stored or transmitted across some form of computer readable media. Any of the methods of the present disclosure may be performed by computer readable instructions embodied on a computer readable medium. A computer readable medium may be any available medium that can be accessed by a computer. By way of example and not meant to be limiting, computer readable media may include "computer storage media" and communication media. "Computer storage media" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Exemplary computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information that can be accessed by a computer.

[0163] The methods and systems can employ artificial intelligence techniques such as machine learning and iterative learning. Examples of such techniques include, but are not limited to, expert systems, case-based reasoning, Bayesian networks, behavior-based AI, neural networks, fuzzy systems, evolutionary computing (e.g., genetic algorithms), swarm intelligence (e.g., ant algorithms), and hybrid intelligent systems (e.g., expert inference rules generated by neural networks, or production rules from statistical learning).

[0164] As a non-limiting example, referring to FIG. 27, the management server 14, 14A (and also the AI ​​platform 230) can include a computer-readable storage medium 2722, which can be a tangible device that can store program instructions for use by the processor 2728. The computer-readable storage medium 2722 can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples of the computer-readable storage medium 2722 include, but are not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a punch card or a mechanically encoded device such as a ridge structure with instructions stored in a groove, and any suitable combination of the foregoing. As used herein, computer-readable storage media does not include transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0165] The program instructions can be downloaded from an external computer or external storage device to the computer readable storage medium 2722 via the data network 1. A network adapter card or network interface 2724 in the management server 14, 14A receives the program instructions via the data network 1 and transfers them to the computer readable storage medium 2722 for storage and execution by the processor 2728. Execution of the program instructions by the processor 28 causes the management server 14 to perform various processes as described above.

[0166] The user interface 2726 may also include various input and / or output devices coupled to the processor 2728, as well as program instructions for interacting with the various input and / or output devices to elicit input from the user 60 and provide output to the user 60 via the input and / or output devices. The user interface 2726 may be a graphical user interface for interfacing with a user. The bus architecture 2723 may interconnect the user interface 2726, the processor 2728, the memory 2722, and the network interface 2724.

[0167] The various program instructions referred to above may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. In some embodiments, electronic circuitry including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can execute the program instructions and perform aspects of the present disclosure by utilizing state information to personalize the electronic circuitry.

[0168] Aspects of the present disclosure are described herein with reference to flowcharts and block diagrams of methods and apparatus (systems) according to various embodiments. It will be understood that each block of the flowcharts and block diagrams, and combinations of such blocks, can be implemented by program instructions. That is, the program instructions read and processed by the processor 2728 direct the processor 2728 to implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. Examples of such functions are those implemented by the user interface module 24, 24A, the rights issuer module 120, 120A, the filtering module 26, 26A, the rights management module 29, 29A, the AI ​​platform training module 160, 160A, the learned parameter distribution module 190, the communication manager 232, the GUI 234, the perception stack 240, the driving policy stack 238, and the rights management agent 236, to name a few non-limiting possibilities. It should also be noted that each block of the flowcharts and / or block diagrams, and combinations of such blocks, can be implemented by dedicated hardware-based systems that perform the specified functions or acts, or perform a combination of dedicated hardware and computer instructions.

[0169] The flowcharts and block diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, apparatus, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing a specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0170] The description of various embodiments of the present disclosure is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0171] It should be understood that throughout this specification, discussions using terms such as "processing," "computing," "calculating," "determining," "analyzing," and the like may refer to operations and / or processes of a computer or computing system, or similar electronic computing device, that manipulates and / or transforms data represented as physical quantities, such as electronic quantities, into other data also represented as physical quantities.

[0172] As used herein, unless otherwise indicated, the use of ordinal adjectives such as "first," "second," "third," etc. to describe a common object or step is intended merely to indicate that different instances of a similar object or step are being referred to and does not imply that the objects or steps so described must be in a given order in time, space, precedence, or otherwise.

[0173] It should be noted that various individual features may be described in terms of only one embodiment. A particular selection to describe in terms of a single embodiment herein should not be construed as a limitation that a particular feature is applicable only to the embodiment in which it is described. Various features described in terms of one embodiment described herein may be equally applicable, additional, or interchangeable to other embodiments described herein, and in various combinations, groupings, or arrangements. In particular, the use of a single reference number herein to illustrate, define, or describe a particular feature does not imply that the feature cannot be associated with or considered equivalent to another feature in another drawing or description.

[0174] Also, where the phrase "at least one of C and D" is used, the phrase is intended and defined to select C or D, or both C and D, and is similar to the word "and / or." When there are more than one variable in such a phrase, the phrase is defined to include only one of the variables, any one of the variables, any combination of the variables, and all of the variables.

[0175] Certain additional elements that may be necessary for the operation of some embodiments are not described or shown, as they are within the purview of those skilled in the art. Furthermore, certain embodiments may not include, lack, and / or function without any element not specifically disclosed herein.

[0176] The foregoing description and accompanying drawings illustrate the principles and modes of operation of certain embodiments. However, these embodiments should not be considered limiting. Further variations of the above-described embodiments will be apparent to those skilled in the art, and the above-described embodiments should be considered illustrative rather than limiting. It should therefore be understood that those skilled in the art can make changes to these embodiments without departing from the scope of the present invention. [Prior art documents] [Patent documents]

[0177] [Patent Document 1] International Publication No. WO2021 / 092702 [Patent Document 2] US Patent Publication No. 20200062273

Claims

1. 1. A method of operating a computer to construct a customized training data set for training an artificial intelligence (AI) platform, comprising: (i) obtaining a specification of training data for training said AI platform, and (ii) data indicative of a proposed use of said training data; querying a database of training data sets, each associated with a usage right, to identify a candidate set of training data sets that match said specification; approve the release of a subset of the training dataset within the candidate set of training datasets based on the data indicative of the proposed use of the AI ​​training data and the usage rights associated with the training dataset within the candidate set of training datasets; A method for providing the above.

2. The method of claim 1 , wherein the specifications include an application range.

3. The method of claim 2 , wherein the application scope includes at least one of: (i) a global application; and (ii) a specific application.

4. The method of claim 1 , wherein the specification further comprises a list of objects.

5. The method of claim 1 , wherein the specifications include a minimum percentage of training data derived from real-world imaging or a maximum percentage of training data derived synthetically.

6. 2. The method of claim 1 , wherein the data indicative of the proposed use of the training data includes one or more of: (i) an identifier of the AI ​​platform; and (ii) a proposed geographic region.

7. The method of claim 1 , further comprising implementing a graphical user interface to obtain the specifications, the data being obtained from a user.

8. 2. The method of claim 1 , wherein the approving includes creating a subset of a candidate set of the training datasets, the subset including those training datasets in the candidate set for which the usage rights include the proposed use of the AI ​​training data.

9. 9. The method of claim 8, wherein the subset of the candidate set of training datasets excludes any training datasets in the candidate set for which the usage rights do not include the proposed use of the AI ​​training data.

10. 2. The method of claim 1 , wherein the specifications and the data indicating the proposed use of the training data are obtained from a user, the method further comprising determining whether the user has a license to train the AI ​​platform.

11. 11. The method of claim 10, wherein at least a portion of the training data set in the database is encrypted, and wherein if it is determined that the user has the license to train the AI ​​platform, the querying includes decrypting the at least a portion of the training data set in the database.

12. 2. The method of claim 1 , wherein each of the training data sets has a common format including a plurality of data blocks, the plurality of data blocks including a first data block including training media for the respective training data set, a second data block indicating the usage rights for the respective training data set, a third data block indicating media metadata corresponding to the training media in the first data block, and optionally a fourth data block indicating a user identifier unique to the respective training data set.

13. The method of claim 12 , wherein the plurality of data blocks further includes a fifth data block indicating cryptographic information associated with the first data block.

14. The method of claim 12 , wherein the media metadata corresponding to the training media in the first data block includes a list of objects depicted in the training media.

15. The method of claim 12 , wherein the media metadata corresponding to the training media in the first data block further indicates whether the training media is derived from real-world imaging or synthetically derived.

16. The method of claim 12 , wherein the media metadata corresponding to the training media in the first data block further includes annotations.

17. 2. The method of claim 1 , wherein to identify a candidate set of training data sets that match the specification, the method comprises comparing the specification to the media metadata in the third data block of each of the training data sets, the candidate set of training data sets comprising those training data sets in which the media metadata in the third data block of the respective training data set matches the specification.

18. The method of claim 1 , wherein the specifications include a user-provided application scope.

19. The method of claim 1 , wherein the specification includes a user-provided list of objects.

20. 11. The method of claim 10, wherein the specification includes a user-specified use range and the usage rights in the second data block include information related to a licensed use range for the respective training data set, and wherein to identify a candidate set of training data sets matching the specification, the method comprises comparing the user-specified use range with the licensed use range, the candidate set of training data sets including those training data sets for which the licensed use range in the second data block of the respective training data set includes the user-specified use range.

21. 11. The method of claim 10, wherein the specification includes a user-specified list of objects, and the media metadata corresponding to the training media in the first data block includes information related to a list of objects depicted in the training media in the first data block, and to identify a candidate set of training data sets that match the specification, the method further comprises comparing the user-specified list of objects to the list of objects in the third data block of each of the training data sets, the candidate set of training data sets excluding those training data sets where the list of objects in the third data block of the respective training data set does not include any of the objects in the user-specified list of objects.

22. 13. The method of claim 12, wherein the approving includes creating the subset of a candidate set of the training datasets, the subset including in the candidate set those training datasets in which the usage rights include the proposed use of the AI ​​training data within the second data block of the respective training dataset.

23. 23. The method of claim 22, wherein the proposed use of the AI ​​training data includes an identifier of the AI ​​platform, and the usage rights in the second data block of each of the training datasets identify at least one licensed AI platform, and the approving includes creating the subset of a candidate set of the training datasets, the subset including in the candidate set those training datasets where the at least one licensed AI platform identified by the second data block of the respective training dataset includes the identifier of the AI ​​platform.

24. 23. The method of claim 22, wherein the proposed use of the AI ​​training data includes a proposed geographic region, and the usage rights in the second data block of each of the training datasets further include a licensed geographic region, and the approving includes creating the subset of a candidate set of the training datasets, the subset including in the candidate set those training datasets where the licensed geographic region in the second data block of the respective training dataset includes the proposed geographic region.

25. The method of claim 1 , further comprising determining usage rights for the subset of the training data set and storing the determined usage rights in the database associated with the subset of the training data set.

26. 26. The method of claim 25, further comprising setting the usage rights of the subset of training data sets based on the usage rights associated with each of the training data sets in the subset of training data sets.

27. 27. The method of claim 26, wherein if the usage rights associated with each of the training datasets in the subset of training datasets are identical, then the usage rights of the subset of training datasets are set to the same usage rights associated with each of the training datasets in the subset of training datasets.

28. 27. The method of claim 26, wherein if the usage rights associated with each of the training datasets in the subset of training datasets are not identical, then the usage rights of the subset of training datasets are set to those usage rights that are common to each of the training datasets in the subset of training datasets.

29. 2. The method of claim 1 , wherein the training data sets each have a common format including a plurality of data blocks, the plurality of data blocks including a first data block including training media for the respective training data set, a second data block indicating the usage rights for the respective training data set, a third data block indicating media metadata corresponding to the training media in the first data block, and optionally a fourth data block indicating a user identifier unique to the respective training data set, the method further comprising grouping subsets of the training data sets into training data set packages and storing the training data set packages on a non-transitory medium.

30. 30. The method of claim 29, further comprising providing the training data set package with a usage rights field that is populated based on the usage rights stored in the second data block of the training data set within the subset of the training data set.

31. 31. The method of claim 30, wherein the content of the usage rights field of the training data set package is an intersection of the usage rights stored in the second data block of the training data set within a subset of the training data set.

32. 31. The method of claim 30, further comprising providing the training dataset package with metadata fields that are populated based on the metadata stored in the third data block of the training dataset in the subset of the training dataset.

33. 33. The method of claim 32, wherein the content of the metadata field of the training data set package is a union of the metadata stored in the third data block of the training data sets in the subsets of the training data set.

34. 1. A non-transitory computer-readable storage medium storing instructions that, when executed by a processing entity of a computing device, cause the computing device to perform a method of constructing a customized training data set for training an artificial intelligence (AI) platform; The method comprises: (i) obtaining a specification of training data for training said AI platform, and (ii) data indicative of a proposed use of said training data; querying a database of training data sets, each associated with a usage right, to identify a candidate set of training data sets that match said specification; approve the release of a subset of the training dataset within the candidate set of training datasets based on the data indicative of the proposed use of the AI ​​training data and the usage rights associated with the training dataset within the candidate set of training datasets; 1. A non-transitory computer-readable storage medium comprising:

35. 1. A computer system for constructing a customized training data set for training an artificial intelligence (AI) platform, comprising: A user interface module configured to obtain (i) a specification of training data for training the AI ​​platform, and (ii) data indicative of a proposed use of the training data; a filtering module configured to query a database of training data sets, each associated with a usage right, to identify a candidate set of training data sets that match the specification; a rights management module configured to approve the release of a subset of the training dataset within the candidate set of training datasets based on the data indicative of the proposed use of the AI ​​training data and the usage rights associated with the training dataset within the candidate set of training datasets; A computer system comprising:

36. A non-transitory computer-readable storage medium containing instructions that, when executed by a computer, receiving a media unit from a user; creating training data units based on the group of media units, each training data unit comprising a plurality of data blocks, the plurality of data blocks of a given training data unit comprising: a first data block including media content from a respective one of the group of media units; a second data block indicating usage rights for the given training data unit; and a third data block indicating media metadata corresponding to the media content in the first data block; said creating comprising: storing the training data unit on a non-transitory storage medium or outputting the training data unit via a data network; A non-transitory computer-readable storage medium for causing a computer to perform a method comprising:

37. 37. The computer-readable medium of claim 36, wherein the plurality of data blocks for the given training data unit further comprises a fourth data block indicating a user identifier unique to the given training data unit.

38. 37. The computer-readable medium of claim 36, further comprising grouping the media units into groups of the media units.

39. 37. The computer-readable medium of claim 36, wherein the grouping comprises grouping the media units based on objects depicted in the media units.

40. 37. The computer-readable medium of claim 36, further comprising performing image processing to identify objects depicted in the media units, and wherein the grouping is performed to group together media units that depict at least a portion of the same object.

41. 41. The method of claim 40, wherein the media metadata in the third data block of the given training data unit includes information related to the objects depicted in each group of the media units.

42. 37. The computer-readable medium of claim 36, further comprising receiving a list of objects depicted in the media units, and wherein the grouping is performed to group together media units that depict at least some of the objects in the list.

43. 37. The method of claim 36, wherein the media metadata in the third data block of the given training data unit indicates whether the media content of each group of the media units is derived from real-world imaging or synthetically derived.

44. 37. The computer-readable medium of claim 36, further comprising validating an identity of the user prior to said receiving.

45. 37. The computer-readable medium of claim 36, wherein the storing comprises encrypting at least a portion of the training data unit.

46. 46. ​​The computer-readable medium of claim 45, wherein the encrypting comprises encrypting only the first data block of each of the training data units.

47. 46. ​​The computer-readable medium of claim 45, wherein the encrypting is performed using an encryption key stored in memory.

48. 48. The computer-readable medium of claim 47, wherein, when stored, the media units within the first data block of each of the training data units are decodable by a decryption key.

49. 49. The computer readable medium of claim 48, wherein the decryption key and the encryption key are the same.

50. 49. The computer readable medium of claim 48, wherein the decryption key is uniquely associated with, but distinct from, the encryption key.

51. 37. The method of claim 36, wherein the plurality of data blocks further comprises a fifth data block indicating how to determine the decryption key.

52. 37. The method of claim 36, wherein the usage rights for the given training data unit include an indication of a range of uses licensed for the given training data unit.

53. 37. The method of claim 36, wherein the usage rights for the given training data unit include an indication of a geographic region licensed for the given training data unit.

54. 37. The method of claim 36, wherein the usage rights for the given training data unit include an indication of an AI platform licensed to the given training data unit.

55. 37. The method of claim 36, further comprising determining the usage rights for the given training data unit and storing the usage rights in the second data block of the given training data unit.

56. The method of claim 36 , wherein the media unit comprises a 2D camera image.

57. The method of claim 36 , wherein the media unit comprises LiDAR or RADAR data.

58. The method of claim 36 , wherein the media unit includes an ultrasound depth map.

59. 1. A method of operating a computer, comprising: receiving a media unit from a user; creating training data units based on the group of media units, each training data unit comprising a plurality of data blocks, the plurality of data blocks of a given training data unit comprising: a first data block including media content from a respective one of the group of media units; a second data block indicating usage rights for the given training data unit; and a third data block indicating media metadata corresponding to the media content in the first data block; said creating comprising: storing the training data unit on a non-transitory storage medium or outputting the training data unit via a data network; A method for providing the above.

Images