Autonomous vehicle environment aware software management
By constructing a customized training data set, the problem of insufficient environmental perception accuracy of autonomous vehicles in complex environments is solved, data management is simplified, and the environment perception capability of the AI platform is improved.
Patent Information
- Application Number
- CN202380029047.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-01-26
- Filing Date
- 2023-01-25
- Publication Date
- 2024-11-05
AI Technical Summary
The current lack of accuracy and reliability of environmental perception in complex driving environments makes it difficult to achieve effective autonomous navigation in a wide range of driving scenarios, and there are challenges in managing and distributing training data for AI platforms.
By building a customized training data set, including obtaining training data specifications and data indicating usage, checking the database associated with usage permissions, identifying matching training data set candidate sets, and releasing subsets based on permission authorization, realizing customization of the AI platform Training dataset construction.
It improves the accuracy and reliability of the environment perception of the AI platform in complex environments, simplifies the data management and distribution process, and adapts to the needs of different driving scenarios.
Smart Images

Figure CN118901071A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 303,363, filed on January 26, 2022, which is hereby incorporated by reference into this document. Technical Field
[0003] The present disclosure relates to the management of data components used to train artificial intelligence (AI) platforms used in environmental perception systems, such as to adapt such AI platforms to a range of different applications. Background Art
[0004] The automotive industry is in the midst of a disruptive change highlighted by the advent of autonomous vehicles (AVs) and electrification and ride-hailing. While autonomous driving has made considerable progress over the past few years, current solutions are still not safe or reliable enough for commercial deployment except in the most limited scenarios.
[0005] To successfully navigate in typical driving scenarios, AVs are equipped with sensing modules that enable them to identify features of the environment while driving. However, being able to detect the dynamic driving environment around the car (such as other vehicles, pedestrians, and small obstacles on the road) while maintaining the affordability of the car is a difficult task. The diversity of driving environments also poses challenges in terms of accurate autonomous vehicle environment perception.
[0006] As a result, the industry has turned to artificial intelligence (AI) to improve the speed and accuracy of environmental perception. However, it is becoming increasingly clear that optimal training of AI platforms across a wide range of driving scenarios can only be achieved when using huge amounts of data from a large number of different sources. This brings challenges in managing, controlling and distributing data used for both training and use of AI platforms, especially in the context of environmental perception for vehicle navigation (whether on land, at sea or in the air). Summary of the invention
[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify essential aspects of the claimed subject matter.
[0008] Broadly speaking, in some embodiments, the present disclosure relates to a method of operating a computer to build a customized training dataset for training an artificial intelligence (AI) platform, the method comprising: obtaining (i) a specification of training data for training the AI platform and (ii) data indicating proposed uses of the training data; consulting a database of training datasets, each associated with usage rights, to identify a candidate set of training datasets that matches the specification; and authorizing the release of a subset of training datasets in a candidate set of training datasets based on the data indicating the proposed uses of the AI training data and the usage rights associated with the training datasets in the candidate set of training datasets.
[0009] In other embodiments, the present disclosure relates to a non-transitory computer-readable storage medium storing instructions that, when executed by a processing entity of a computing device, cause the computing device to perform a method of constructing a customized training data set for training an artificial intelligence (AI) platform, the method comprising: obtaining (i) a specification of training data for training the AI platform and (ii) data indicating proposed use of the training data; consulting a database of training data sets each associated with a usage right to identify a candidate set of training data sets that matches the specification; and authorizing release of a subset of the training data sets in the candidate set of training data sets based on the data indicating the proposed use of the AI training data and the usage rights associated with the training data sets in the candidate set of training data sets.
[0010] In other embodiments, the present disclosure relates to a non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform a method comprising: receiving a media unit from a user; creating a training data unit based on a group of media units, each training data unit comprising a plurality of data blocks, wherein the plurality of data blocks for a given training data unit comprises: (i) a first data block containing media content from a corresponding one of the groups of media units; (ii) a second data block indicating usage rights for the given training data unit; (iii) a third data block indicating media metadata corresponding to the media content in the first data block; and (iv) a fourth data block indicating a unique user identifier for the given training data unit; and storing the training data unit in a non-transitory storage medium or outputting the training data unit via a data network.
[0011] In yet other embodiments, the present disclosure relates to a method of operating a computer to construct a customized training data set for training an artificial intelligence (AI) platform, the method comprising: receiving a media unit from a user; creating a training data unit based on a group of media units, each training data unit comprising a plurality of data blocks, wherein the plurality of data blocks for a given training data unit comprises: (i) a first data block containing media content from a corresponding one of the groups of media units; (ii) a second data block indicating usage rights for the given training data unit; (iii) a third data block indicating media metadata corresponding to the media content in the first data block; and (iv) a fourth data block indicating a unique user identifier for the given training data unit; and storing the training data unit in a non-transitory storage medium or outputting the training data unit via a data network.
[0012] The features of the exemplary embodiments described in this disclosure and not mutually exclusive can be combined with each other. The elements of an embodiment can be used in other embodiments without further mentioning. In addition, not all elements described in association with a particular embodiment need to appear in all implementations or realizations of the embodiment. After reading the description of the specific embodiments below in conjunction with the accompanying drawings, those of ordinary skill in the art will be aware of other aspects and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] A detailed description of specific exemplary embodiments is provided below with reference to the accompanying drawings, in which:
[0014] Figure 1 is a block diagram of a computer-based architecture for generating customized application-specific data sets for training an AI platform, according to a non-limiting embodiment.
[0015] Figure 2 is a more detailed block diagram of a user workstation and an administrative server according to a non-limiting embodiment, showing the main functional blocks of the workstation and the main functional blocks of the server.
[0016] FIG. 3A to FIG. 3C is a graphical user interface (GUI) flow illustrating various GUI controls that allow a user to specify parameters for an application-specific training data set at a workstation implementing the GUI, according to a non-limiting embodiment.
[0017] Figure 4 is a diagram illustrating the format of a training data unit according to a non-limiting embodiment.
[0018] Figure 5 is a table conceptually illustrating information conveyed in a user rights block of a training data unit, according to a non-limiting embodiment.
[0019] Figure 6 is a table conceptually illustrating information conveyed in a metadata block of a training data unit, according to a non-limiting embodiment.
[0020] Figure 7 is a block diagram of an exemplary AI platform according to non-limiting embodiments.
[0021] Figure 8 is a semantic environment representation output by the AI platform according to a non-limiting implementation scheme.
[0022] Fig. 9 is a flow chart of a process of extracting an application-specific training data set from a training data set, according to a non-limiting embodiment.
[0023] Fig.10 It is based on the variant Fig. 9 Flowchart of the process.
[0024] Fig.11 Shown according to the variant Figure 2 Block diagram of the .
[0025] Fig.12 Shown according to the variant Fig.11 Block diagram of the .
[0026] Fig.13 is a flow diagram of a process for uploading media to a management server for generating training data units, according to a non-limiting embodiment.
[0027] Fig.14 is a flow chart of a process for generating a training data packet from an application-specific training data set, according to a non-limiting embodiment.
[0028] Fig.15 It is based on the variant Figure 2 A block diagram of the system is shown.
[0029] Fig.16 Based on the variant Fig.15 A block diagram of the system is shown.
[0030] Fig.17 According to another variation Fig.16 A block diagram of the system is shown.
[0031] Fig.18 is a flow chart of a process for generating a training data packet according to a non-limiting embodiment.
[0032] Fig.19 is based on another variant of Fig.16 A block diagram similar to the block diagram of .
[0033] Fig. 20A data structure of a learned parameter package according to a non-limiting embodiment is shown.
[0034] Fig.21 is a flow chart of a process for generating a learned parameter package according to a non-limiting embodiment.
[0035] Fig.22a , 22b , 22c and 22d are GUI screens showing user controls for selecting parameters of a learned parameter package according to a non-limiting embodiment.
[0036] Fig.23 is a block diagram of a system for performing an update of learned parameters of an AI platform, according to a non-limiting embodiment.
[0037] Fig.24a , 24b 24c are GUI screens illustrating selectable user options for customizing the operation of the context-aware system, according to a non-limiting embodiment.
[0038] Fig.25 is a flow diagram of a process for updating learned parameters of an AI platform according to a non-limiting embodiment.
[0039] Fig.26 is a mapping between information obtained from users of the AI platform and training data units according to a non-limiting embodiment.
[0040] Fig. 27 is a block diagram of a computing system that can be used to implement various aspects of a management server.
[0041] Fig.28 A training data packet is conceptually illustrated as a data structure consisting of various fields, some of which represent, for example, Figure 4 The training data unit shown.
[0042] The drawings illustrate examples and are not intended to be limiting. DETAILED DESCRIPTION
[0043] The present technology is explained in more detail below. This specific embodiment is not intended to be an exhaustive list of all the different ways in which the technology can be implemented or all the features that can be added to the present technology. For example, the features shown for one embodiment can be incorporated into other embodiments, and the features shown for a specific embodiment can be deleted from the embodiment. In addition, considering which changes and additions of the present disclosure do not deviate from the present technology, those skilled in the art will be aware of the various changes and additions of the various embodiments suggested herein. Therefore, the following description is intended to illustrate some embodiments of the technology, rather than to specify exhaustively all its permutations, combinations and variations.
[0044] Figure 1 1 is a block diagram of a system that allows a user to build a customized training data set in order to train an AI platform (to be discussed in the next paragraph) to perform environmental perception. The system 10 has different building blocks and functional elements implemented in hardware or software to perform individual tasks. The system 10 includes a series of workstations 12 that communicate with a training data set collection management server 14 via a data network 1. In turn, the management server 14 manages access to the training data set collection stored in the database 16. Based on the specifications that define the desired performance or characteristics of the environmental perception system provided at the workstation 12, the management server 14 will build a customized collection of training data for the AI platform in order to train the AI platform to achieve such environmental perception performance or characteristics.
[0045] Figure 7 7 shows the main components of an exemplary AI platform that can be trained with a custom training data set, and it should be understood that a series of different AI platform architectures can be used without departing from the spirit of the present invention. More specifically, the AI platform 70 includes a sensor component 72 that outputs an electronic representation of the environment. The sensor component 72 operates based on a single sensor modality or multiple sensor modalities. For example, the sensor component 72 can use one or more image sensors to output a high-resolution camera image. If the sensor component 72 uses multiple sensor modalities, in addition to the image sensor, the sensor component can also use a LIDAR sensor, a radar sensor, and / or an ultrasonic sensor that outputs a depth map of the environment. The outputs of different sensors are combined by a raw data fusion function block 74 that generates a high-resolution 3D map of the environment. In general, a 3D map represents the environment with pixels that provide RGB color information. In addition, a distance value is associated with each pixel, indicating the distance from a reference point to an object in the environment, and a portion or fragment of the object is visually represented by the pixel. In general, the reference point will be the sensor component 72.
[0046] The high-resolution 3D map is then input into a trained neural network 76, which may be a convolutional neural network, to perform detection on objects that the neural network 76 has been trained to recognize in the 3D map. The output of the neural network 76 is then fed to a classifier 78, which performs classification on the detected objects. The classifier 78 outputs a semantic representation of the environment depicted by the 3D map. For example, the semantic representation may include an identification of an object of interest in the 3D map, a timestamp, and an identification of the speed (velocity + direction of motion) of the detected object. Figure 8 is an example of a semantic representation, where the top image shows a detected object with a bounding box and the bottom image is a speed map showing the speed of the detected object.
[0047] For further details on suitable non-limiting AI platforms that may be used as AI platform 70, reference should be made to WO2021092702A1, which is hereby incorporated herein by reference.
[0048] In autonomous or semi-autonomous vehicle applications, the semantic environment representation will generally be input into a driving strategy module, which will calculate a vehicle path based on the 3D representation, which in turn will be used by a control module to guide the vehicle according to the calculated path. In addition to the 3D representation, the driving strategy module may also use additional inputs (such as the current position, map data, etc. generated by a GPS module or other modules) to calculate the vehicle path.
[0049] refer to Figure 2 , which is a more detailed block diagram of a workstation 12 and a management server 14, the workstation 12 includes a graphical user interface (GUI) functional block 20, which operates to implement a GUI that allows user input to the specifications of the environment perception system. In a specific implementation example, the GUI can be implemented via a web browser under the control of the management server 14. Therefore, in this implementation form, the workstation 12 establishes a communication session with the management server 14 via a data network 1 (such as the Internet, or a local area network) and receives scripts or other commands from the management server 14 to implement the GUI, accept user input at the GUI, and receive user messages via the GUI.
[0050] FIG. 3A to FIG. 3C An example of a workflow that enables a user to define the specifications of a desired context-aware system via a GUI is shown, it being understood that such a workflow is exemplary and that numerous variations are possible.
[0051] The workflow is defined by a series of GUI pages that provide controls for the user to enter information about the context-aware system application. User input at one GUI page conditions subsequent GUI pages so that the user is logically directed to valid input options.
[0052] As an initial GUI page, the user is presented with page 300, in which the user can indicate global application categories for the context awareness system via a series of check boxes or other input mechanisms. In this example, the global application categories are defined in terms of land applications (land vehicles), air applications (drones or other), and marine applications (marine vessels), but other definitions or categories are possible.
[0053] When the user selects a land application, a GUI page 302 is displayed requesting input regarding the specific land application. Examples of land applications include, but are not limited to, commercial trucks, commercial buses, self-driving taxis, shuttles, and agricultural vehicles. In the example shown, the user has selected "commercial trucks." After making the selection, a GUI page 304 is displayed in which the user can specify the degree of autonomy level, it being understood that the intensity of AI training for higher autonomy levels may be different than for lower levels (where a human driver is always responsible for controlling the vehicle).
[0054] At GUI page 306, the user can specify additional functionality to automate certain vehicle maneuvers. In addition, the selections available at page 306 depend on previous selections made by the user. That is, the system is configured so that the options presented are valid options for the intended application. In this example, "Back up to loading dock" has been selected. This selection requires the environmental perception system to detect the loading dock as an object in the 3D map so that the driving strategy module can calculate the backing maneuver. Therefore, it is necessary to customize training data for this additional functionality by providing images of the loading dock so that the AI platform can reliably identify the loading dock.
[0055] Another option in the context of commercial truck applications is automatic trailer free space detection, as described in U.S. Publication US20200062273A1, which is hereby incorporated by reference herein. In this case, the environment perception system is trained to detect the presence and size of a trailer attached to the truck so that the free space parameters of the vehicle can be adjusted during autonomous driving.
[0056] At the next GUI page 308, the user specifies the AI platform to be trained. It should be recognized that the training data depends on the capabilities and characteristics of the AI platform. For example, an AI platform that uses only a camera to sense the environment only requires camera images for training, while an AI platform that uses sensors of different modalities may require training data generated by a series of sensors. Although not shown in the accompanying drawings, as new AI platforms appear on the market and it is expected that training data sets can be generated for these platforms, the selection list of GUI page 308 is dynamically updated. For example, the management server 14 maintains a database of AI platforms, associating each entry with the hardware and / or software specifications of the platform that affect the selection of training data. As this list of AI platforms in the database is updated, the available choices at GUI page 308 are also updated. More specifically, the AI platform can be characterized based on the following parameters that can be specified by the user at GUI page 308:
[0057] a. Type of sensor (modality), such as camera, LIDAR, radar, and acoustic sensor.
[0058] b. The number of sensors for each modality,
[0059] c. The location of the sensors for each modality on the vehicle,
[0060] d. Computing hardware capabilities and characteristics, such as distributed, centralized, etc.
[0061] GUI page 310 allows the user to specify the primary operating environment of the AI platform, which will again condition the training data to be used to adapt the environmental perception performance to this operating environment. Examples of operating environments include day, night, rain, snow, etc. For example, for a commercial truck designed for night operation, the training data should be selected accordingly and will be different from the training data in the case of dedicated daytime operation.
[0062] GUI page 312 allows the user to select the region in which the AI platform will operate. The region selection may affect the selection of training data in terms of media content, but is also a useful input in terms of regulatory compliance. For example, data privacy laws may not allow training data collected in a certain region to be used to train an AI platform operating in a different region. Therefore, the selection made at GUI page 312 can be used to identify only training data that is authorized for intended use in the training data set, thereby ensuring regulatory compliance. GUI page 312 provides general regional inputs that can be customized at the country level at GUI page 314, and it should be understood that further levels of granularity can also be provided at the state or city level if necessary. Another non-limiting example level of granularity can be based on: separating locations / countries where vehicles drive on the right side of the road from locations / countries where vehicles drive on the left hand side of the road.
[0063] Figure 3C GUI page 316 at allows the operator to specify further parameters of the training data. In addition, GUI pages such as the one shown in page 318 may be provided to include some additional objects in the training data that may not be included in the set derived from the previous specification. For example, the AI platform to be trained may be operated in an environment where a railroad crossing needs to be detected with high reliability, so the objects need to be specified.
[0064] Return to reference Figure 2 In addition to the GUI functionality 20, the workstation 12 also implements a rights management agent 22, which is a software program designed to interact with the management server 14 in response to permission challenges from the management server 14 in order to confirm that the workstation 12 is properly permitted to interact with the management server 14 to build a customized training data set.
[0065] The management server 14 has several functional blocks or modules implemented in software, hardware, or a combination of software and hardware. More specifically, the management server 14 includes a user interface module 24, which is a software component that implements the GUI interface 20 described earlier via a web browser. The user interface module 24 establishes a communication session with the workstation 12 and dynamically updates the GUI page as the user provides specifications for the context-aware system. The GUI hierarchy, options, and specification input stream can be stored in a machine-readable storage device of the management server 14 and referenced by the user interface module 24 to update the GUI page as the user performs user input so that the options selectable via the GUI are always valid.
[0066] In addition to managing the GUI pages, the user interface module 24 also collects user input and constructs queries that will allow for the extraction of customized training data sets from the training data set that match the specifications of the query.
[0067] The management server 14 also includes a filtering module 26, which is an interface to a database 28 storing training data sets. Thus, the user interface module 24 passes a query to the filtering module 26, which runs the query to extract training data sets that match the query from the training data sets in the database 28. The extraction operation will be described in more detail later.
[0068] The management server 14 also includes a rights management module 29 that manages the use of the training data in the training data set to ensure that the use complies with rights and regulations, as will be described in more detail later.
[0069] refer to Figure 4 , the training data is stored in the database 28 in the form of discrete training data units 40. Each training data unit 40 is organized into data blocks (or fields) as follows:
[0070] a. Training media content blocks 42 contain media items that contain training media for actual training of an AI platform, such as AI platform 70. Such training media may be still camera images with annotations identifying objects of interest in the image. Alternatively, the training media may include continuous images, such as video with or without audio information. Alternatively or in addition, the training media includes representations of the environment, such as depth maps, generated by other sensors, such as LIDAR, radar, and ultrasonic sensors, etc. The size of training media content blocks 42 in training data units 40 may vary. For example, Figure 4As shown, a training media content block 42 is a block that includes several media items 44. A media item 44 can be a single camera or video frame or a depth map frame. The number of media items 44 per training media content block 42 can vary. Note that in most applications, the media items 44 (such as all camera frames, all LIDAR depth maps, etc.) will have the same sensor modality. However, media items 44 of different sensor modalities (such as 3D map frames obtained by fusing the output of a camera sensor with the output of a LIDAR, radar, or ultrasonic sensor) can be bundled into a single training data unit 40.
[0071] b. Encryption information block 46. Encryption information block 46 stores an encryption key identifier that will allow management server 14 to decrypt the information in training data unit 40. This process will be described later.
[0072] c. Usage permission information block 48, which defines under what circumstances the training media in the training media content block 42 can be used to perform AI platform training. An example of usage permission is shown in Figure 5 In this example, there are three categories of permissions or rights, namely, applications for the AI platform, areas where the AI platform trained with the training data can be used, and specific AI platforms that can be trained with the training data. The example shows that the training data can be used for land applications, but not for air or sea applications because air and sea are not listed. It should be understood that the permitted applications can be defined with finer granularity to distinguish between a range of land, sea, or air uses. It should be noted that the level of granularity at which the permitted applications are defined in the usage permission information block 48 should match the level of granularity at which the applications are entered via the GUI at the workstation 12 so that compliance checks can be performed by comparing the two. Figure 5The example also specifies a permitted region, which in this case is North America. Therefore, the training data cannot be used for AI smart platforms operating outside North America. The permitted region can be defined with finer granularity, for example on a country-by-country basis. The usage permission information block 48 also specifies the identity of the AI platform that can be trained with this training data. In the example, the licensed AI platform is generally defined, but a narrower definition can be used, such as by providing a serial number for each AI platform and listing all authorized platforms in the permitted AI platform field. This feature is useful in the case where the training data belongs to a third party, and it should be used to train only AI platforms belonging to the third party. In this case, the permitted AI platform field will only list third-party AI platforms, so that other AI platforms cannot use such third-party specific data. It should be noted that additional usage permissions or permissions, such as expiration dates, can be specified. Another example is to specify a specific environment in the context of a certain application (such as a data set that can only be used for military vehicles of the Ministry of Defense of a specific country). Another example would be to delete specific information in the training media, so that unacceptable or restricted data is "cleared" from the training media. Restricted data can include license plate information or faces. Yet another example is to specify road types or conditions, such as paved road, unpaved road, highway or street, snow covered road, rain wet road, etc.
[0073] d. Metadata block 50, which characterizes the actual media content used for AI training purposes to allow for query-based training data filtering. FIG. 3A to FIG. 3C The context-aware specification at the GUI shown in the figure is similar to the parameters used to characterize the media. Figure 6As shown in the exemplary table, the characteristics of the training data are based on the scope of application, which can include a list of objects depicted in the media item 44, the suitability of the media item 44 in terms of global application (land), specific application (commercial trucks and buses), and environmental conditions. The object list can be generated by human observation of the media item 44 (such as an individual viewing an image and identifying objects appearing in the image and making annotations). Alternatively, the object list can be generated by machine processing of the image (such as processing the image with an AI system that outputs a list of objects and annotations). Optionally, the machine-generated list can be checked by a human operator to determine accuracy. In addition, the media content and the object list and annotations can be mostly synthetically generated, that is, generated via a simulation system that creates a virtual environment with virtual objects and renders the environment and objects as a series of realistic images with object lists and annotations, which can be combined with traditional training data materials for AI training. It can also be specified in the metadata block 50 associated with the specific training data (in the training media content block 42) whether the training data is synthetically generated or generated by imaging the real world environment. This distinction can be useful in providing training criteria that specify a minimum amount of training data obtained from imaging a real-world environment. For example, a specification may state that at least X% of the training data must come from real-world imaging, and at most (100-X)% may be generated synthetically, where X may be 10%, 25%, 50%, 75%, or any other percentage.
[0074] e. A unique identifier block 52 for this training data unit 40, which allows this training data unit 40 to be distinguished from other training data units in the collection stored in the database 28. The unique identifier block 52 is optional, for example, if there is some other way to uniquely identify the training data unit 40. In addition, from a traceability perspective, the training data units can be encoded by a blockchain algorithm to ensure that they cannot be altered and their origin or source is clear.
[0075] Thus, the creation of training data units (such as training data unit 40, also referred to as training data sets) can be achieved by executing computer-readable instructions stored on a non-transitory computer-readable storage medium. When executed by a computer, such instructions cause the computer to perform a method comprising: receiving a media unit from a user; creating a training data unit based on a group of media units, each training data unit comprising a plurality of data blocks; and storing the training data unit in a non-transitory storage medium or outputting the training data unit via a data network. The plurality of data blocks for a given training data unit comprises: a first data block containing media content from a corresponding one of the groups of media units; a second data block indicating usage rights for the given training data unit; and a third data block indicating media metadata corresponding to the media content in the first data block.
[0076] In some embodiments, the plurality of data blocks for a given training data unit further includes a fourth data block indicating a unique user identifier for the given training data unit.
[0077] In some embodiments, the method further comprises grouping the media units into groups of the media units.
[0078] In some embodiments, grouping includes grouping the media units based on objects depicted in the media units.
[0079] In some embodiments, the method further comprises performing image processing to identify objects depicted in the media units, and wherein the grouping is performed so as to group together media units depicting at least some of the same objects.
[0080] In some embodiments, the media metadata in the third data block of a given training data unit includes information related to objects depicted in the corresponding group of media units.
[0081] In some embodiments, the method further comprises receiving a list of objects depicted in the media units, and wherein the grouping is performed so as to group together media units depicting at least some of the objects in the list.
[0082] In some embodiments, the media metadata in the third data block of a given training data unit indicates whether the corresponding set of media content of the media unit was derived from real-world imagery or was derived synthetically.
[0083] In some embodiments, the method further comprises verifying the identity of the user prior to receiving.
[0084] In some embodiments, storing includes encrypting at least a portion of the training data unit.
[0085] In some embodiments, encrypting includes encrypting only the first data block of each of the training data units.
[0086] In some embodiments, encryption is performed using an encryption key stored in memory.
[0087] In some implementations, when stored, the media units in the first data block of each of the training data units are decryptable by a decryption key.
[0088] In some embodiments, the decryption key and the encryption key are the same.
[0089] In some embodiments, the decryption key is uniquely associated with but different from the encryption key.
[0090] In some embodiments, the plurality of data blocks further include a fifth data block indicating how to determine the decryption key.
[0091] In some embodiments, the usage rights for a given training data unit include an indication of a permitted scope of application for the given training data unit.
[0092] In some embodiments, the usage rights for a given training data unit include an indication of a permitted geographic region for the given training data unit.
[0093] In some embodiments, the usage rights for a given training data unit include an indication of a licensed AI platform for the given training data unit.
[0094] In some embodiments, the method further includes determining usage rights for a given training data unit, and storing the usage rights in a second data block for the given training data unit.
[0095] In some embodiments, the media units include 2D camera images, while in other embodiments, the media units include LiDAR or radar data, and in yet other embodiments, the media units include ultrasonic depth maps.
[0096] To further assist in making the information requested via the GUI available to the user ( FIG. 3A to FIG. 3C ) and the information stored in the training data unit ( Figure 4 ) to visualize the relationship between Fig.26 , which is a non-limiting illustration of how user input (denoted as 2610) is mapped to training data units 40. Here, user input received from a user via a GUI can be categorized into specifications 2620 and proposed uses 2630, each with its own subfields, while training data units 40 have their own organization and subfields, as described above with reference to Figure 4Specifically:
[0097] a. Specification 2620 may include:
[0098] i. specifying the object list subfield 2620-A, which maps to the object list subfield 2670-A of the metadata block 50 of the training data unit 40;
[0099] ii. The specified application subfield 2620-B is mapped to the permitted application subfield 2660-A of the usage rights block 48 of the training data unit 40;
[0100] iii. The real-world image specifies the minimum % subfield 2620 -C, which maps to the real-world image % subfield 2670 -B of the metadata block 50 of the training data unit 40 .
[0101] b. Proposed uses of 2630 may include:
[0102] i. The AI platform designated identifier subfield 2630 -A is mapped to the permitted AI platform subfield 2660 -B of the usage rights block 48 of the training data unit 50 .
[0103] ii. Proposed geographic location subfield 2630 -B, which maps to permitted geographic location subfield 2660 -C of usage rights block 48 of training data unit 50 .
[0104] Fig. 9 is a flow chart illustrating steps in an example process for building a custom training data set for an AI platform such as AI platform 70.
[0105] The process starts at 90. At step 92, a user at a workstation 12 logs in at the management server 14. At step 94, the management server 14 (particularly the rights management module 29) issues a permission challenge to the workstation 12 via the data network 1, which is processed by the rights management agent 22. If the rights management agent 22 supplies a sufficient response, the management server 14 opens a training data set construction session at step 96 and presents the user at the workstation 12 with the training data set. FIG. 3A to FIG. 3C The GUI page shown is used to initiate the context-aware specification workflow.
[0106] At step 98, user interface module 24 collects input entered by the user via the GUI page and builds a query accordingly. At step 100, the query is passed to filter module 26, which runs the query to extract a subset of training data units that match the query from database 28. In particular, filter module 26 extracts those training data units that have metadata blocks 50 that match the query parameters (see also Fig.26 ).
[0107] At step 100, the filtering module 26 outputs a candidate training set, which is processed by the rights management module 29 at step 102 for usage rights compliance. In particular, the rights management module 29 compares the usage rights associated with each training data unit in the candidate training set with the parameters supplied by the user (see again Fig.26 ) to determine whether the expected use is compliant. For example, the rights management module 29 compares the expected AI platform application with the permitted application, the expected usage area with the permitted area, and the expected AI platform with the permitted AI platform. If any differences are found, the training data unit is removed from the candidate set, leaving only the compliant training data unit, as shown at step 106.
[0108] The rights management module 29 may then send a message to the user at the workstation 12 to inform the user of the number of training data units that have been excluded from the candidate set due to non-compliance with the usage rights. This indicates to the user whether too many training data units have been excluded and too few training data units remain to perform adequate training. The rights management module 29 may trigger such a process if the non-compliance can be resolved in some way. For example, if additional permissions can be obtained to achieve compliance for payment, the rights management module 29 calculates the fees payable and indicates them to the user via the GUI page. If the user completes the payment, the rights management module 29 permanently or temporarily changes the licensing parameters in the usage rights block 48 of the excluded training data units so that they can be put back into the original list.
[0109] In some embodiments, the training media content blocks 42 of the training data unit 40 are encrypted. This is done to control the use of the media items 44 in the training media content blocks 42. The use of the media items 44 in the training media content blocks 42 is achieved by decrypting the training media content blocks 42. The encryption / decryption process uses data stored in the encryption information blocks 46. As previously indicated, the encryption information blocks 46 store data mapped to the encryption key database 110 (see Fig.11 ) In order to decrypt the training media content block 42 and allow access to the media item 44 containing the training media, access to the appropriately identified encryption key in the encryption key database 110 is required, and such access is achieved as described below.
[0110] In a first possible implementation option, the training media content block 42 and some of the other information blocks (such as the usage rights block 48 and the metadata block 50) are encrypted. In this first implementation option, decryption must be performed to access the metadata and user rights information to determine whether a particular training data unit matches the query and can be used for a specified application. The decryption process is shown in Fig.10 The flowchart shown in FIG. Fig. 9 A variation of the flow chart in , showing an additional decryption step 99. Fig.10 The flowchart should be Fig.11 Combined with reading, the following picture is Figure 2 A variation of , illustrating the encryption / decryption functionality of the management server 14.
[0111] First, refer to Fig.11 , the rights management module 29 interfaces with an encryption key database 110 storing encryption / decryption keys mapped to corresponding key identifiers that populate the encrypted information block 46 of each corresponding training data unit 40. For the relevant portion of the encrypted training data unit to be decrypted, the decryption key is extracted from the encryption key database 110 and the decryption process is performed. Fig.10 The process is shown in more detail in the flowchart of FIG. 1 , where step 99 shows the additional step of decrypting the relevant portion of the training data unit 40 before running the query. For each training data unit to be decrypted, the decryption process will require reading the key identifier in the encryption information block 46, extracting the corresponding decryption key from the encryption key database 110 and decrypting the data in the training data unit 40 (e.g., the data in the training data content block 42, the usage rights block 48, and the metadata block 50) so that step 100 can determine whether the training data unit 40 matches the query parameters.
[0112] Before releasing the key from the encryption key database 110, the rights management module 29 can determine whether the user is allowed to access the training data unit 40 in order to build a training data set. In detail, the following distinction is made between the usage permissions for the AI platform application: one is to authorize the use of certain media to train a specific AI platform; the other is to allow the user to use the training data management tool to build a training data set and use the set to train the AI platform. The two are different processes, and the user who builds the training data set may need a separate license (build and train license) to use the data management tool and the training data set set to build a customized training data set, and then use the customized training data set to train the AI platform. If such a separate license is required, the rights management module 29 will perform the necessary compliance checks before releasing the decryption key from the encryption / decryption key database 110 to determine whether the user has obtained the appropriate permission to build the training data set. Compliance verification involves communication between the rights management module 29 and the rights management agent 22, such as to establish the scope of the permission granted to the user and determine whether the scope of the permission covers the estimated use.
[0113] A practical example can better illustrate this point. Suppose Company A wants to develop an AI platform and needs a suitable training dataset. Company A hires a business services company that uses Fig.11 The architecture shown is used to build a custom training data set. However, Company A has its own training data set and wishes to bundle this customer-owned training data set with the training data set of the business service company. Therefore, Company A will contribute its own training data set to the database 28 so that the training data set can be integrated into the overall process, provided that the customer-owned data set is not used for any other project involving the business service company. Therefore, a licensing mechanism will be required to ensure that access to the customer-owned training data set is allowed only for work performed for the customer. Therefore, the licensing rights for the use of the customer-owned training data are set so that the training data used to build the custom training data set is accessed only for customer-related projects and not other projects. Therefore, the licensing rights are set at the rights management agent 22 in the user workstation, and the scope of these rights is established by the rights management module before decrypting the training data unit 40.
[0114] In a second possible implementation, only the training media content block 42 of the training data unit 40 is encrypted. In this implementation, the usage rights information (in the usage rights block 48) and the metadata information (in the metadata block 50) are always readable, and construction of the customized data training set is achieved without decryption, while the training media itself remains encrypted.
[0115] Fig.12 yes Fig.11 , and illustrates additional features provided by a rights issuing module 120 to enable third parties to contribute to a training data set while controlling the use of the contributed training data. Reference numeral 122 identifies a third party workstation into which the third party training data is loaded. The training data may be organized into a series of media items 44, such as annotated camera images, annotated LIDAR, radar, and ultrasonic depth maps, and the like. The third party workstation 122 sends the training data (i.e., media items 44) to the management server 14 for integration into a training data set stored at the database 28. The rights issuing module 120 is enabled to package the media items 44 into searchable and manageable training data units 40, and also associate these training data units 40 with usage permissions that the third party can control.
[0116] The communication between the third party workstation 122 and the management server 14 is handled via the user interface module 24, which may provide a GUI at the third party workstation 122 via a web browser or via any other equivalent component to allow a user at the workstation 122 to upload media items 44. The GUI, not shown in the figure, is configured with controls that allow a user at the workstation 122 to:
[0117] a. Logically grouping individual media items 44 together, for example, based on the objects depicted.
[0118] b. Generate metadata.
[0119] c. Generate usage permissions, such as specifying permitted applications, permitted areas, and permitted AI platforms.
[0120] Upon inputting information items 1, 2, 3, the rights management module 29 processes the uploaded media items 44 and packages them into training data chunks 42, and also creates an encryption information chunk 46, a usage rights chunk 48, a metadata chunk 50, and a unique identifier chunk 52 for each training data chunk 42. The result is a series of created training data units 40 that are subsequently stored in the database 28.
[0121] Fig.13 is a flow chart illustrating a process of uploading and integrating third-party training data into a training data set.
[0122] The process begins at 130. At step 132, the workstation 122 communicates with the management server 14 and establishes a session, which may require supplying user credentials to verify the identity of the user at the workstation 122. At step 134, the user interface module 24 implements a GUI with controls at the workstation 122 via a web browser, which is used to allow the user of the workstation 122 to upload its media items (which are third-party training data from the perspective of the management server 14). At step 136, the user of the workstation 122 specifies via the GUI the media items 44 to be uploaded, the manner in which these media items 44 are grouped into training data units, and for each such training data unit, the corresponding metadata and usage rights. At step 137, the rights management module 29 processes the uploaded information and generates training data units 40. At step 138, the rights management module 29 may encrypt the training data units 40 (according to one of the aforementioned options, i.e., encrypt only the media items or also encrypt the metadata and usage rights), and at step 139, the rights management module 29 stores the optionally encrypted training data units 40 in the training data database 28.
[0123] Return to reference Fig.12 , the process of building an application-specific training data set generates a set of training data units 40 that is a subset of the set of training data units 40 in the database 28. The subset itself constitutes a digital product that is processed by the rights issuance module 120 of the management server 14, such as to properly package the product (referred to as a "training data package") and associate it with usage rights, thereby allowing the owner of the digital product to control how the digital product is used downstream.
[0124] refer to Fig.14 See the flowchart in Fig.28 The process of creating a training data package 2800 that is itself a product will now be described in more detail.
[0125] The process starts at step 140. At step 142, the rights issuing module 120 receives the application-specific training data set 2804 generated by running a query on the database 28. "Receive" means that the rights issuing module 120 has access to the training data units 2804A, 2804B, ..., 2804C selected by running the query. For example, the rights issuing module 120 can be provided with an index in the database 28 to point to the entry in the database 28 corresponding to the selected training data unit 2804A, 2804B, ..., 2804C, thereby allowing the rights issuing module 120 to sequentially obtain the training data units 2804A, 2804B, ..., 2804C from the database 28 and process them.
[0126] At step 144, the rights issuing module 120 accesses the media item 44 in each training data unit 2804A, 2804B, ..., 2804C in the application-specific training data set 2804. This may require decrypting the media item based on a decryption key obtained from the information in the encrypted information block 46 of each training data unit 2804A, 2804B, ..., 2804C.
[0127] At step 145, the rights issuing module 120 generates usage rights associated with the training data package (as a whole, represented as 2800) so that the use of the training data package 2800 to train the AI platform can be managed. The usage rights are located in the usage rights data block 2848 (or field) of the training data package 2800.
[0128] In a specific implementation example, the permission issuing module 120 will generate the usage rights for the entire training data packet 2800 based on the usage rights of the individual training data units 2804A, 2804B, ..., 2804C contained therein. It is conceivable that the application-specific training data set 2804 will store training data units 2804A, 2804B, ..., 2804C owned by different entities and associated with different usage rights. Therefore, generating the usage rights for the training data packet 2800 needs to consider the individual usage rights. There are two possibilities:
[0129] a. The usage rights of each training data unit 2804A, 2804B, ..., 2804C in the application-specific training data set 2804 are the same. In this case, the same usage rights are replicated for the training data package 2800. That is, the training data package 2800 will inherit the same usage rights (with respect to the permitted applications, permitted regions, and permitted AI platforms of the AI platform) as any individual training data unit 2804A, 2804B, ..., 2804C in the application-specific training data set 2804.
[0130] b. The usage rights in the training data units 2804A, 2804B, ..., 2804C of the application-specific training data set 2804 are not the same. Although the usage rights under each training data unit 2804A, 2804B, ..., 2804C are consistent with the environmental perception system specifications as specified by the user, the usage rights between the training data units 2804A, 2804B, ..., 2804C may be different. For example, the training data unit 2804A may have usage rights that specify permitted applications for the AI platform for use on land and in the air, while the training data unit 2804B may only allow land use. In this case, the air use permission is stripped, and only the land permission is brought into the training data packet 2800. In this way, the collective usage rights under the training data packet 2800 are still consistent with the environmental perception system specifications. In this example, the usage rights for the entire training data packet 2800 (e.g., permitted applications, permitted areas, and permitted AI platforms for the AI platform) are the usage rights shared by each individual training data unit.
[0131] At step 146, a metadata information data block 2850 (or field) of the training data packet 2800 is generated. In a specific implementation example, the metadata of the training data packet 2800 is a combination or union of the metadata of the individual training data units 2804A, 2804B, ..., 2804C. For example, the metadata of the training data packet 2800 lists all objects that appear in the media items of the training data packet. As an option, the metadata of the training data packet 2800 can also provide statistical information about the entire training data packet 2800, such as the total number of objects, the relative distribution between objects (20% bicycles, 40% cars, 18% pedestrians), the total number of media items in the package, etc. More specifically, the rights issuing module 120 extracts metadata from the metadata block 50 of each training data unit 2804A, 2804B, ..., 2804C in the application-specific data set 2804, performs statistical analysis, and stores the results in the metadata information block 2850 of the training data packet 2800.
[0132] At step 147, the rights issuing module 120 generates a unique identifier for the training data packet 2800 and places the unique identifier in a unique identifier data block 2852 (or field). Optionally, at step 148, the rights issuing module 120 may generate an encryption information block 2846 (which identifies a key stored in the encryption key database 110), and finally at step 149, encrypt portions of the training data packet 2800.
[0133] Once the training data packet 2800 is finalized, it is stored in the training data packet database 150, such as Fig.15 As shown, the figure is Fig.12 variants.
[0134] As Fig.15 variant Fig.16 Another implementation example of the management server 14 is shown, which includes the function of using training data packets (such as training database packet 2800) stored in the training data packet database 150 to perform training of the AI platform 162. More specifically, the management server 14 includes an AI platform training module 160, which receives training data packets from the training data packet database 150 and performs a training process based on the media content in the training data packets. In addition to the training data packets, the AI platform training module 160 also receives a model of the AI platform 162 to be trained. The output of the AI platform training module 160 is a set of learned AI parameters of the AI platform 162, which are generated by training the model with training data. Examples of learnable parameters are weights and biases. The specific weight and bias values generated by the training process are learned parameters of the AI platform 162. For more information about AI platforms and systems, see Artificial Intelligence: Fundamentals and Applications, Cherry Bhargava and Pradeep Kumar Sharma (eds.), ISBN 9780367559700, published by CRC Press on July 29, 2021, which is hereby incorporated by reference into this article.
[0135] Before the AI platform training module 160 can access the training data packet and use it to perform training operations, the training data packet may have to be decrypted, and such decryption needs to be checked for compliance with the usage permissions attached to the training data packet.
[0136] Consider a scenario in which a user at workstation 12 communicates with management server 14 to control training operations performed by AI platform training module 160. As with the previous example of interacting with management server 14, the user works through a GUI implemented by user interface module 24. The user at workstation 12 provides input to allow rights management module 29 to determine whether the use of training data packet 2800 complies with the usage rights associated with the training data packet. For example, the user specifies via the GUI the intended application for the AI platform 162 to be trained, the intended area in which the AI platform 162 will be used, and provides an AI platform identifier. Then, rights management module 29 compares the specification of the intended use provided by the user via the GUI with the usage rights in the usage rights block 2848 of the training data packet 2800, and if they match, the AI platform training module 160 can continue to perform the training operation.
[0137] In a variation, the AI platform model on which the training is performed is encoded with a platform ID, which can be read directly by the rights management module 29, thereby eliminating the need for the user to supply this information at the workstation 12.
[0138] Therefore, a method of operating a computer to construct a customized training data set for training an artificial intelligence (AI) platform is provided. The method includes: obtaining (i) a specification of training data for training the AI platform and (ii) data indicating proposed uses of the training data; consulting a database of training data sets each associated with a usage right to identify a candidate set of training data sets that matches the specification; and authorizing release of a subset of training data sets in the candidate set of training data sets based on the data indicating the proposed uses of the AI training data and the usage rights associated with the training data sets in the candidate set of training data sets.
[0139] In some embodiments, the specification includes an application scope, and the application scope includes at least one of (i) a global application and (ii) a specific application; the specification may also include a list of objects. The specification may also include a minimum percentage of training data derived from real-world imaging or a maximum percentage of training data derived synthetically.
[0140] In some embodiments, the data indicative of the proposed use of the training data includes one or more of (i) an identifier of the AI platform and (ii) a proposed geographic region.
[0141] In some embodiments, the method further includes implementing a graphical user interface for obtaining the specification, and the data is obtained from a user.
[0142] In some embodiments, the authorization includes creating a subset of the candidate set of training datasets, the subset including those training datasets in the candidate set whose usage rights include the proposed use of the AI training data.
[0143] In some embodiments, the subset of the candidate set of training datasets excludes any training datasets in the candidate set whose usage rights do not include the proposed use of AI training data.
[0144] In some embodiments, the specification and data indicative of proposed uses of the training data are obtained from a user, and the method further includes determining whether the user has permission to train the AI platform.
[0145] In some embodiments, at least some of the training data sets in the database are encrypted, and wherein if it is determined that the user has permission to train the AI platform, the query includes a step of decrypting at least some of the training data sets in the database.
[0146] In some embodiments, the training data sets each have a common format, comprising multiple data blocks, wherein the multiple data blocks include: a first data block containing training media for the corresponding training data set; a second data block indicating usage rights for the corresponding training data set; a third data block indicating media metadata corresponding to the training media in the first data block; and, optionally, a fourth data block indicating a unique user identifier for the corresponding training data set.
[0147] In some embodiments, the plurality of data blocks further includes a fifth data block indicating encryption information related to the first data block.
[0148] In some embodiments, the media metadata corresponding to the training media in the first data block includes a list of objects depicted in the training media.
[0149] In some embodiments, the media metadata corresponding to the training media in the first data block also indicates whether the training media was derived from real-world imagery or was synthetically derived.
[0150] In some embodiments, the media metadata corresponding to the training media in the first data block also includes annotations.
[0151] In some embodiments, to identify a candidate set of training datasets that match a specification, the method includes comparing the specification with media metadata in a third data block of each training dataset in the training datasets, the candidate set of training datasets including those training datasets whose media metadata in the third data block of their corresponding training datasets matches the specification.
[0152] In some embodiments, the specification includes a user-provided application scope, while in other embodiments, the specification includes a user-provided list of objects. The specification may also include a user-specified application scope, wherein the usage rights in the second data block include information related to a permitted application scope of the corresponding training data set, wherein to identify a candidate set of training data sets that matches the specification, the method includes comparing the user-specified application scope with the permitted application scope, the candidate set of training data sets including those training data sets whose permitted application scope in the second data block of their corresponding training data sets includes the user-specified application scope.
[0153] In some embodiments, the specification includes a user-specified list of objects, wherein the media metadata corresponding to the training media in the first data block includes information related to the list of objects depicted in the training media in the first data block, and wherein to identify a candidate set of training datasets that match the specification, the method further includes comparing the user-specified list of objects with a list of objects in a third data block of each of the training datasets, the candidate set of training datasets excluding those training datasets whose list of objects in the third data block of their corresponding training dataset does not contain any objects in the user-specified list of objects.
[0154] In some embodiments, the authorization includes creating a subset of the candidate set of training datasets, the subset including those training datasets in the candidate set whose usage rights in the second data block of their corresponding training datasets include the proposed use of the AI training data.
[0155] In some embodiments, the proposed use of the AI training data includes an identifier of the AI platform, wherein the use rights in the second data block of each of the training data sets identify at least one permitted AI platform, wherein the authorization includes creating a subset of the candidate set of training data sets, the subset including those training data sets in the candidate set for which the at least one permitted AI platform identified by the second data block of the corresponding training data set includes the identifier of the AI platform.
[0156] In some embodiments, the proposed use of the AI training data includes a proposed geographic area, wherein the usage rights in the second data block of each training data set in the training data set also include a permitted geographic area, wherein the authorization includes creating a subset of a candidate set of training data sets, the subset including those training data sets in the candidate set whose permitted geographic area in the second data block of their corresponding training data sets includes the proposed geographic area.
[0157] In some embodiments, the method further includes determining usage rights for the subset of the training data set, and storing the determined usage rights in a database in association with the subset of the training data set.
[0158] In some embodiments, the method further includes setting usage permissions for the subset of training data sets based on usage permissions associated with each of the training data sets in the subset of training data sets.
[0159] In some embodiments, when the usage rights associated with each of the training datasets in the training dataset subset are the same, the usage rights for the training dataset subset are set to the same usage rights associated with each of the training datasets in the training dataset subset.
[0160] In some embodiments, when the usage rights associated with each of the training datasets in the training dataset subset are not the same, the usage rights for the training dataset subset are set to those usage rights that are common to each of the training datasets in the training dataset subset.
[0161] In some embodiments, the training data sets each have a common format, comprising a plurality of data blocks, wherein the plurality of data blocks include: a first data block containing training media for a corresponding training data set; a second data block indicating usage rights for the corresponding training data set; a third data block indicating media metadata corresponding to the training media in the first data block; and optionally, a fourth data block indicating a unique user identifier for the corresponding training data set, the method further comprising grouping subsets of the training data sets into training data set packages, and storing the training data set packages in a non-transitory medium.
[0162] In some embodiments, the method further includes providing a usage rights field to the training data set package, wherein the usage rights field is populated based on usage rights stored in a second data block of the training data set in the training data set subset.
[0163] In some embodiments, the content of the usage rights field of the training data set packet is the intersection of the usage rights stored in the second data block of the training data set in the training data set subset.
[0164] In some embodiments, the method further includes providing a metadata field to the training data set package, the metadata field being populated based on metadata stored in a third data block of the training data set in the training data set subset.
[0165] In some embodiments, the content of the metadata field of the training data set packet is the union of the metadata stored in the third data blocks of the training data sets in the training data set subset.
[0166] Fig.17 Yet another variation is shown in which training of the AI platform 162 is performed at a remote location from the management server 14 . Fig.17A scenario is shown where training needs to be performed at a specified location to comply with specific requirements, such as government regulations governing the use of training data and / or the use of learned parameters generated from the training data. For example, such regulations may specify that training data collected in a particular region or country may not be exported outside of the country or region or otherwise used outside of the country or region. Therefore, in order to comply with such regulations, the training data needs to be stored in the region and country and processed to perform AI training in this country or region. However, training of the AI platform 162 may require the use of additional training data that resides outside of the region or country. Fig.17 The system is configured to allow training data that is not subject to storage and / or usage restrictions to be delivered to a remote site to augment local training data that is subject to storage and / or usage restrictions in order to provide an effective training data set for training the AI platform.
[0167] More specifically, the system includes a second management server 14A residing in a country or region that imposes restrictions on training data and / or learned parameters, and the second management server is similar in structure and function to the management server 14. Specifically, the management server 14A has a user interface module 24A (similar to the user interface module 24), a permission issuing module 120A (similar to the permission issuing module 120), a filtering module 26A (similar to the filtering module 26), a permission management module 29A (similar to the permission management module 29), and an AI platform training module 160A (similar to the AI platform training module 160). In addition, the management server 14A communicates with a training data set stored in a database 28A (similar to the database 28), an encryption key database 110A (similar to the encryption key database 110), and a training data packet database 150A (similar to the training data packet database 150).
[0168] The operation of management server 14A is similar to that of management server 14, in that management server 14A can receive input from a workstation (such as workstation 12) to define specifications for the environmental perception system, run queries via filtering module 26A to generate a customized training data set, generate a corresponding training data packet, and use the training data packet to perform AI platform training.
[0169] Filter module 26A is configured such that it has visibility to both databases 28 and 28A. Thus, when filter module 26A runs a query, it may flag training data units in either or both of databases 28, 28A as relevant. Such remote visibility is possible if the training data units stored in database 28 have no restrictions attached thereto, and allows external entities (which may be entities residing outside the region or country) to read metadata associated with such training data units. The reverse approach is also possible, i.e., allowing filter module 26 to have visibility to training data stored in database 28A, assuming that regulations governing the use and / or storage of training data units in database 28A allow access to the corresponding metadata.
[0170] The process for constructing a training data package at management server 14A is depicted in Fig.18 , which is a flowchart describing the process of extracting an application-specific training data set by the management server 14 Fig. 9 The steps similar to the two flowcharts are marked with the same reference numerals.
[0171] At step 92, the operator logs in at the workstation 12 (which in this case is in communication with the management server 14A) with appropriate credentials (such as a username and password, etc.).
[0172] At step 94, the rights management module 29A issues a license query to the workstation 12 to determine what licenses have been granted to the workstation / user and to make services available to the workstation / user consistent with the scope of the licenses. Note that in this implementation, the rights management module 29A that handles the license verification is implemented at the management server 14A. In a possible variation, the license verification operation can be performed by the rights management module 29 of the management server 14. In other words, the rights management module that performs step 94 can be a central license manager tool that provides services to various remote management servers and handles the license verification service for all transactions on the remote management servers.
[0173] At step 96, the management server 14A initiates the context-aware specification workflow as described earlier, in which the user will input specifications for the context-aware system.
[0174] At step 98, a query is created.
[0175] At step 180, a process is run to establish data visibility to all remote training data collection databases (in this case, databases 28, 28A). The purpose of this part of the process is to establish a global list of all training data units stored in the remote databases at the management server 14A, so that queries can be run on the global list to identify relevant training data units that meet the query parameters in the entire training data collection stored across multiple databases. In practice, but not necessarily, this can be performed by storing a global list of training data units in the entire collection at each training data collection database 28, 28A, where each entry in the list corresponds to a training data unit and includes metadata and optionally other information blocks, but omits media units, so that queries can be run, as shown in step 100.
[0176] Steps 102, 104 and 106 are then performed for the entries in the global list, which produces a final set of training data units that satisfy the parameters of the query. For the sake of this example, assume that 80% of the training data units so selected reside at database 28A, while 20% reside at the (remote) database 28. In order to access the media units of the training data units stored locally at database 28A, the rights management module 29A will perform the necessary license checks (as described earlier) and obtain the decryption key from the encryption key database 110A. The decryption key can then be used to decrypt the media items in the training data units stored in the local database 28A. For the data training units stored in the remote database 28, similar operations are performed, but this time the operations involve the rights management module 29 at the management server 14. More specifically, a request is transmitted from the management server 14A to the management server 14 via the data network 1 to request the media units and decryption keys associated with a specific number of training data units stored in the database 28. The rights management module 29 will perform a verification operation to determine whether the intended use is in compliance with the license (in particular, to determine whether the license permits the export of the training data to a different site), and if so, extract the training data unit from the database 28 and obtain the decryption key from the encryption key database 110, and relay both the training data unit and the decryption key back to the management server 14A, and in particular back to the rights management module 29A, so that the media of 20% of the training data units can now be decrypted.
[0177] It should be noted that the above arrangement allows the use of training data stored in a distributed manner at sites subject to a range of different laws / regulations. An advantage of the system is that it provides an efficient and flexible mechanism for managing the use of training data, such as to maintain compliance with laws / regulations. For example, training data that is subject to an export ban will be blocked from being sent to a remote site because the usage rights associated with the training data only authorize local use, so the rights management module will block the data from being exported and will not provide a decryption key to the remote site. However, for training data that can be exported to a remote site, where the usage rights associated with the training data authorize the use of the training data at the remote site, the rights management module 29A will pass the data and provide a decryption key so that the media can be used at the remote site.
[0178] At step 182, the decrypted training data is assembled into a training data packet, which is stored in the training data packet database 150A at step 184 and provided to the AI training module 160A at step 186 to perform a training operation.
[0179] The AI platform training modules 160 and 160A include a high-speed data processor, such as a high-performance graphics processing unit, to process the media in the training data packets and generate learned parameters for the AI platform. In a specific example, the learned parameters are weights and biases of the neural network. The learned parameters can then be loaded into the AI platform and used to perform environmental perception. For example, the learned parameters can be loaded into an AI platform implemented in a vehicle, such as to provide environmental perception, which is then used for autonomous or semi-autonomous operation of the vehicle.
[0180] In a specific example, learned parameters are generated at a specific site and loaded into individual AI platforms in a fleet, which may include hundreds or even thousands of units. As indicated previously, the AI platforms will typically be vehicles, but they may be implemented in any kind of automated equipment that requires environmental perception to provide autonomous functionality, including land, air, and sea vehicles for a variety of applications including agriculture, mining, industry, commercial delivery, and personnel transportation.
[0181] Fig.19 is a block diagram, which is Fig.16A variation of includes a module 190 for performing distribution of learned parameters to multiple AI platforms. In a specific implementation example, the AI platform may be a vehicle. The distribution may include initially loading the learned parameters into the AI platform. That is, the AI platform is initially non-functional because it has not yet been programmed to perform environmental perception. This will typically be the case when: a vehicle is manufactured and hardware implementing the AI platform is installed in the vehicle. The initial loading of learned parameters, which may be performed at the production line, will provide the AI platform with environmental perception functionality, such as to enable autonomous or semi-autonomous functionality.
[0182] Depend on Fig.19 The system distribution of learned parameters can also be performed in the context of updates. In other words, the existing learned parameters in the AI platform are replaced with new learned parameters to provide different or enhanced environmental perception functions.
[0183] The learned parameter distribution module 190 communicates with the AI platform training module 160 and receives learned parameters generated by the training operation from the AI training module 160. In addition, the learned parameter distribution module 190 communicates with the permission management module 29 and with the permission issuing module 120, such as to create a learned parameter package associated with the usage permission so as to be able to manage the use of the learned parameters in the AI platform.
[0184] Fig.19 A fleet of AI platforms is shown at 194. The fleet has multiple individual AI platforms that can receive learned parameter packages via over-the-air (OTA) updates or via any other suitable communication method. For example, each AI platform can have communication capabilities via a data network 1 such as the Internet and periodically communicate with a management server 14 to receive learned parameter packages and then install the learned parameter packages to enable the AI platform to perform environmental perception.
[0185] Fig. 20 is a diagram showing a general data structure of a learned parameter package 200. It should be noted that the data structure of the learned parameter package 200 has a structure similar to that of the training data unit 400. In particular, the learned parameter package 200 includes a learned parameter data block 202, an encryption information data block 204, a usage rights data block 206, and a unique identifier data block 208.
[0186] The learned parameter data block 202 stores the weights and biases used by the AI platform to provide environmental perception. The encrypted information block 204 stores encrypted information, the usage permission data block 206 stores information about the permitted use of the learned parameters, and the unique identifier 208 uniquely identifies this learned parameter package 200 among other packages.
[0187] Fig.21 An example of a process for generating learned parameter package 200 is provided by the flowchart in .
[0188] The process starts at 210. At step 212, the AI platform training module 160 performs the training process and generates learned parameters. At step 214, the permission issuing module 120 generates a usage permission data block 206, which will determine the permitted use of the learned parameters. In a specific implementation example, the usage rights for the learned parameter package 200 are derived in whole or in part from the usage rights associated with the training data upstream, i.e., the training data packet 2800 and / or the training data unit 40. In a specific example, the usage rights associated with the learned parameter package inherit from the usage rights associated with the training data packet, and the latter usage rights are then at least partially inherited from the usage rights associated with the individual training data units. In a specific example, it is assumed that the training data packet has the following usage rights:
[0189] a. permitted applications limited to terrestrial use,
[0190] b. Limited to permitted areas in Canada and the United States,
[0191] c. Limited to licensed AI platforms of alpha and delta.
[0192] The rights issuing module 120 copies the same usage rights into the usage rights data block 206. Then, the rights issuing module 120 generates a unique identifier at step 216, generates an encrypted data block at step 218, and finally encrypts the learned parameter package at step 220.
[0193] In a variant, the user (operator) is provided with the opportunity to change the usage rights for the learned parameter package via a user interface in order to change these rights and adapt them to specific circumstances. The user interface is a graphical user interface that provides a series of controls that display information to the user and allow the user to enter information to change the default usage rights or specify specific parameters of the usage rights.
[0194] Fig.22a 2200, which allows the user to view the default usage permissions for the learned parameter package 200 that may be inherited from the usage permissions of the training data package, and also provides the user with the opportunity to make changes to these usage permissions. In this example, the GUI 2200 has a series of tabs corresponding to the types of usage permissions so as to present information to the user in a logical and easy to understand manner. There is an "Application" tab, an "Area" tab, and an "AI Platform" tab. However, it should be understood that other ways of organizing information can be used without departing from the spirit of the present invention.
[0195] Applications tab 222 shows three main categories of applications, identifying one application inherited from the training data package. Since this is a main category of usage rights, application usage rights may not be user selectable, and tab 222 may be provided only to inform the user of the application usage rights that will apply. That is, the GUI is configured so that the GUI does not accept user input that changes the selection from "land" to another selection (such as "air" or "sea").
[0196] exist Figure 22b In the variation shown, the Applications tab 222 can be used to define permitted applications under a particular category with finer granularity, where the particular category is fixed and non-selectable. For example, for permitted land application usage, tab 222 shows a series of subcategories that the user can select. In this way, when global usage permissions related to a particular category have been inherited, the user can limit the scope of downstream permissions to a particular subcategory. In this example, the global Land Applications category has three subcategories, namely (1) Self-Driving Taxis, (2) Commercial Trucks, and (3) Shuttle Vehicles. The selection mechanism is enabled on tab 224, and the user can select a single subcategory or multiple subcategories and then "save" the selection.
[0197] refer to Fig.22c , the user has selected the "Region" category on the tab page 226, showing multiple sub-regions in the global "North America" region. The sub-categories provided to the user include "Alaska", "Eastern United States", and "Western United States", etc. The user can select an applicable sub-category from the sub-categories in the "Region" category and save the selection.
[0198] Fig.22d The "AI Platform" category 228 is shown, which includes more fine-grained controls provided to the user to select an AI platform. In this example, the inherited usage rights extend to the AI platform category "α", that is, each subcategory of the "α" category is permitted to use the learned parameter package. The user is provided with a list of selectable options, each corresponding to a specific subcategory, namely D-9653, K-789, and Z-679. The user can select one or more of these subcategories, where such selection will work to limit the usage rights to only the selected category. In other words, unselected subcategories of the "α" category will not be permitted to use the learned parameter package, even if the training data package has a comprehensive license for the "α" category. The user selection at tab 228 is then saved.
[0199] When the user is already Figures 22a to 22dOnce the desired input is made at the GUI shown, the selection will be sent to the rights issuing module 120, which will construct the usage rights data block 206 accordingly, that is, the rights issuing module will add the constraints or permissions derived from the user input to the listed usage rights.
[0200] After the learned parameter packages 200 are assembled, they are stored in the learned parameter package database 192. It should be understood that each learned parameter package may be associated with metadata to provide a description of the parameters of the learned parameter package. Fig. 20 Not shown in the figure, but the learned parameter package 200 may include a further data block for storing such metadata related to the learned parameter package 200 .
[0201] Fig.23 A block diagram of a system for remotely updating learned parameters of an AI platform is shown. The system includes the management server 14 described previously and an AI platform 230, which can be integrated in a vehicle such as a land vehicle, an air vehicle, or a sea vehicle. For the purpose of this example, the vehicle is a land vehicle.
[0202] The AI platform 230 includes a sensor set 242, which as previously indicated may include a series of single-modal or different-modal sensors. The output of the sensor set 242 is provided to a perception stack 240, which includes software that processes the sensor set output to provide environmental perception and generate a 3D map of the environment. The output of the perception stack 240 is supplied to a driving strategy stack 238, which includes software for determining vehicle movement.
[0203] The perception stack 240 uses the learned parameters to perform environmental perception. Fig.23 The system is designed to provide remote updating of learned parameters in order to improve environmental awareness performance or to adapt it to provide customer selectable additional features not previously included or new features not previously available.
[0204] The GUI module 234 allows the user to control the parameters of the update process. The GUI module 234 communicates with the communication manager 232, which includes hardware and software elements that allow the AI platform 230 to communicate with the management server 14. In a specific implementation example, the communication manager 232 is configured to implement wireless communication with the management server 14, such as via a cellular network or an equivalent network. The AI platform 230 also includes a rights management agent 236, which controls how the AI platform 230 is allowed to use the learned parameters downloaded from the management server 14. The rights management agent 236 communicates with the rights management module 29 so that, as established by the rights management agent 236, if the AI platform 230 obtains the appropriate permission for such use, the learned parameters downloaded from the management server 14 can be decrypted and used by the AI platform.
[0205] The GUI module 234 provides the user with a number of selectable options regarding:
[0206] a. Customizing the download process of learned parameters. For example, the GUI module 234 includes a control that allows the user to select between enabling automatic downloading of updates or disabling automatic downloading.
[0207] b. Customized environmental perception. Provide the user with the option of adapting environmental performance to his or her needs, such as adding optional features to the basic environmental perception function. In a specific implementation example, the basic environmental perception function is configured to detect the minimum range of objects that the AI platform should be able to see to provide the desired level of automation. In the context of automated urban driving, the minimum range of dynamic objects may include cars, trucks, pedestrians, cyclists, traffic signs, fire hydrants, and construction signs (such as construction cones), etc. This basic range of objects implements basic automation functions, such as driving in an urban environment including city streets and highways. However, users of the AI platform may desire to add additional automation features to such basic automation functions, such as the ability to provide autonomous or semi-autonomous operation in different environments. Examples of such different environments are provided below:
[0208] i. For commercial vehicles such as trucks, the ability to maneuver in specific surrounding environments or conditions. A first example would be backing a truck into a loading dock. In order to achieve automatic dock approach, the environmental perception system should be able to detect the loading dock itself, that is, the learned parameters need to be configured to recognize the loading dock as an object in the 3D map. A second example involves a vehicle that operates partially in an industrial environment and partially in an urban environment. An industrial environment can be quite different from an urban environment because it includes objects that are not found in a typical urban environment. Take a truck that transports containers from a port to a destination in a city. While a basic range of detectable objects may be sufficient to meet the needs of urban driving, additional object detection capabilities will be required to achieve autonomous operation in a port environment where unique objects need to be detected.
[0209] ii. For personal transport vehicles, the ability to park in complex environments such as underground parking lots where the vehicle needs to navigate around many obstacles such as pillars. In order to provide this automated parking functionality, the environmental perception software needs to be configured to detect obstacles. Another example of the case of a personal transport vehicle (which may also be relevant to commercial vehicles) is the added functionality of automatically sensing the entire vehicle boundary, such as when a trailer is connected to the vehicle. In this case, the environmental perception system is configured to "see" the trailer and output the trailer as a dynamic object, which can then be taken into account by the driving stack to calculate a "free space" parameter, which basically includes the vehicle dimensions for the driving plus the trailer dimensions.
[0210] iii. Regional considerations, such as updating learned parameters based on the vehicle's current location. Updated parameters for a new region in which the vehicle is to operate may include (1) permission updates to extend usage rights previously limited to a specific region to the new region. In this case, the learned parameters will not change, but the usage rights will be updated to authorize operation in the new region.
[0211] Fig.24a , 24b An example of a GUI implemented by GUI module 234 that will enable a user to customize the context-aware system is shown at 24c and 24d. GUI 300 has multiple screens arranged as tabs corresponding to different parameters. In this example, there are four tabs, namely (1) User Authentication, (2) Operation Settings, (3) Customization, and (4) Payment. The description of GUI 2400 will be combined with Fig.25 The flowchart of FIG. 20 describes the different steps of the process for updating the learned parameters of the AI platform 230.
[0212] The process starts at 250. At step 252, a user (e.g., a vehicle occupant) logs into the system by providing user credentials (such as a user name and password). The communication manager 232 then establishes a communication session with the management server 14, and in particular, the learned parameter distribution module 190. Assuming that the user authentication is successful, the learned parameter distribution module 190 retrieves the user profile associated with this particular AI platform at step 254. The user profile stores information about the user and the AI platform, which is used to adjust the selectable options at the GUI 2400 so that only options that are valid for the particular customer and also valid for the particular AI platform are shown on the GUI page.
[0213] Figure 24b An operation settings tab 256 is shown. In this example, two controllable settings are provided to the user. One of the settings is a learned parameter automatic update setting that can be enabled or disabled. The automatic update setting allows the management server 14 to push automatic updates via the data network 1. Such updates are usually provided to fix vulnerabilities or improve the reliability of basic object detection. Another controllable setting is a permitted area automatic update setting that can be enabled or disabled. This setting provides the ability to automatically push updates to achieve seamless driving from one permitted area to another permitted area. When the user sets this setting to "enable", the AI platform 230 will notify the management server 14 when the vehicle starts moving from an permitted area, and request learned parameters and / or permission updates to enable the use of current parameters outside the original permitted area. The permission management agent 236 responsible for usage tracking receives location information from a range of possible sources, such as a 3D map generated by an environmental perception system, and triggers automatic updates when it is determined that the vehicle is operating outside the permitted area. Automatic updates include the permission management agent 236 generating a message to the management server 14 to indicate the current area in which the vehicle is traveling. In response to this message, the learned parameter distribution module 190 determines what the vehicle needs in terms of learned parameters, extracts these parameters from the database 192, and sends them to the rights management agent 236 along with the decryption key so that they can be decrypted and installed. If only a license update is required, the new learned parameters will not be pushed to the AI platform 230, but only a license update will be generated and sent to the rights management agent 236. In addition, if the license extension requires billing the user, a transaction event can be triggered.
[0214] Return to reference Fig.25 , at step 258 , settings of operating parameters related to automatic updating of learned parameters and automatic updating of permitted areas are received and at step 260 , stored in the user profile.
[0215] Reference now Fig.24c, the customization tab 262 will be discussed in detail. This tab 262 provides controls to the user to allow the user to select specific operating capabilities for autonomous or semi-autonomous operation of the vehicle, which in turn requires specific learned parameters. In terms of additional features, five selectable options are provided to the user, and it should be understood that more or fewer options may be provided without departing from the spirit of the invention. Options include loading dock operations, port operations, rail yard operations, refinery yard operations, and airport taxiway operations. Tab 262 also provides a permitted area option to set the AI platform for operation within a specific area. The example shown provides a choice between the eastern United States and the western United States.
[0216] like Fig.25 As shown in the flowchart at , after the customization option selected by the user is received by the learned parameter distribution module 190 at step 264, the customization option is stored in the user profile at step 266. At step 268, the learned parameter distribution module 190 retrieves the learned parameter package corresponding to the user's selection from the database 192. It should be understood that the learned parameter distribution module 190 applies a filter to extract only the learned parameter packages that are compatible with the AI platform and / or licensed for the AI platform. In other words, the learned parameter packages that are not compatible with the AI platform or are not authorized for use with it will not be selected. The filtering operation is performed based on the usage permission block 206 that lists the allowed AI platforms in the learned parameter package.
[0217] The retrieved learned parameter packages may then be displayed to the user via GUI 2400, allowing the user to make a selection. If payment is required to use the selected learned parameter package, pricing information may optionally be presented to the user. Payment may be made via the payment tab of GUI 2400. At step 270, after payment processing, the learned parameter package including the decryption key is pushed to the AI platform 230 to enable the rights management agent 236 to decrypt the learned parameter package and provide it to the perception stack 240 for installation and subsequent use.
[0218] Aspects of the present disclosure may be implemented with any combination of hardware and software. If implemented as a computer-implemented device, components for performing all of the steps and functions described above may be used to implement such aspects. Aspects of the present disclosure may also be included in an article (e.g., one or more computer program products) having, for example, a computer-usable medium. The medium has embodied therein, for example, a computer-readable program code component for providing and facilitating the mechanisms of the present invention. The article may be included as part of a computer system or sold separately.
[0219] For the purpose of illustration, application programs and other executable program components are shown herein as discrete blocks, but it should be recognized that such programs and components reside in different storage components of the computing system at different times and are executed by the data processor of the computer. The implementation of the content control software can be stored on or transmitted across some form of computer-readable medium. Any disclosed method can be executed by computer-readable instructions embodied on a computer-readable medium. Computer-readable media can be any available medium that can be accessed by a computer. By way of example and not intended to be limiting, computer-readable media can include "computer storage media" and communication media. "Computer storage media" includes volatile and non-volatile media, mobile and non-removable media implemented in any method or technology to store information such as computer-readable instructions, data structures, program modules or other data. Exemplary computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, magnetic cassette, tape, disk storage device or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer.
[0220] Methods and systems can employ artificial intelligence techniques such as machine learning and iterative learning. Examples of such techniques include, but are not limited to, expert systems, case-based reasoning, Bayesian networks, behavior-based AI, neural networks, fuzzy systems, evolutionary computing (e.g., genetic algorithms), swarm intelligence (e.g., ant algorithms), and hybrid intelligent systems (e.g., expert inference rules generated by neural networks or production rules from statistical learning).
[0221] For example, and with reference to non-limiting embodiments Fig. 27, the management server 14, 14A (and the AI platform 230) may include a computer-readable storage medium 2722, which may be a tangible device capable of storing program instructions for use by the processor 2728. The computer-readable storage medium 2722 may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium 2722 includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device (such as a punched card or a raised structure in a groove with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, computer-readable storage media itself does not include transient signals such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.
[0222] The program instructions may be downloaded from an external computer or external storage device to the computer-readable storage medium 2722 via the data network 1. The network adapter card or network interface 2724 in the management server 14, 14A receives the program instructions via the data network 1 and forwards them to the computer-readable storage medium 2722 for storage and execution by the processor 2728. The processor 28 executes the program instructions causing the management server 14 to perform various processes as described above.
[0223] The user interface 2726 is also connected to the processor 2728 and may include various input and / or output devices and program instructions that interact with the various input and / or output devices to elicit input from the user 60 and provide output to the user 60 via the input and / or output devices. The user interface 2726 may be a graphical user interface for interacting with the user. The bus architecture 2723 may interconnect the user interface 2726, the processor 2728, the memory 2722, and the network interface 2724.
[0224] The various program instructions mentioned above can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages (including: object-oriented programming languages, such as Smalltalk, C++, etc.; and conventional procedural programming languages, such as "C" programming language or similar programming languages). In some embodiments, electronic circuits (including, for example, programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs)) can execute program instructions by the following to perform aspects of the present disclosure: using state information to personalize the electronic circuit.
[0225] Various aspects of the present disclosure are described herein with reference to the flowchart and block diagram of the method and device (system) according to the embodiments of the present disclosure.It should be understood that each frame of the flowchart and block diagram and the combination of such frames can be realized by executing computer program instructions.That is, the program instruction command processor 2728 read and processed by processor 2728 realizes the function / action specified in one or more flowcharts and / or block diagram frames.The example of such function is realized by user interface module 24, 24A, permission release module 120, 120A, filtering module 26, 26A, permission management module 29, 29A, AI platform training module 160, 160A, learning parameter distribution module 190, communication manager 232, GUI 234, perception stack 240, driving strategy stack 238 and permission management agent 236, just to name a few non-limiting possibilities.It should also be noted that each frame of the flowchart and / or block diagram and the combination of such frames can also be realized by a dedicated hardware-based system that performs a specific function or action or performs a combination of dedicated hardware and computer instructions.
[0226] Flowcharts and block diagrams illustrate the architecture, functions and operations of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each frame in a flow chart or block diagram may represent a module, a fragment or a portion of an instruction, and the module, a fragment or a portion includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions mentioned in the frame may not appear in the order mentioned in the accompanying drawings. For example, two frames shown in succession may actually be performed substantially simultaneously, or, depending on the functions involved, frames may sometimes be performed in reverse order.
[0227] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical application or technical improvement of technologies existing in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
[0228] It should be appreciated that throughout the specification, discussions utilizing terms such as "processing," "computing," "calculating," "determining," "analyzing," and the like may refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or transform data represented as physical (such as electronic) quantities into other data similarly represented as physical quantities.
[0229] As used herein, unless otherwise specified, the use of ordinal adjectives "first," "second," "third," etc. to describe common objects or steps merely indicates that different instances of similar objects or steps are referred to, and is not intended to imply that the objects or steps so described must be in a given order, whether temporally, spatially, in ranking, or in any other manner.
[0230] It should be noted that various individual features may be described in the context of only one embodiment. The specific selections used to describe a single embodiment herein should not be considered to limit the specific features to only the embodiments in which the specific features are described. The various features described in the context of one embodiment described herein may be equally applicable to, added to, or interchanged with other embodiments described herein, and may be in various combinations, groupings, or arrangements. In particular, the use of a single reference numeral herein to illustrate, define, or describe a specific feature does not mean that the feature may not be associated or equivalent with another feature in another figure or description.
[0231] Additionally, when the phrase "at least one of C and D" is used, this phrase is intended and is hereby defined as selecting either C or D or both C and D, similar to the phrase "and / or." When there are more than two variables in such a phrase, this phrase is hereby defined to include only one of the variables, any one of the variables, any combination of any variables, and all of the variables.
[0232] Certain additional elements that may be required to operate some embodiments are not described or illustrated because they are considered to be within the scope of one of ordinary skill in the art. In addition, certain embodiments may not have any elements not specifically disclosed herein, may lack these elements, and / or may function without these elements.
[0233] The foregoing description and accompanying drawings illustrate the principles and modes of operation of certain embodiments. However, these embodiments should not be considered to be limiting. Additional changes to the embodiments discussed above will be recognized by those skilled in the art, and the embodiments described above should be considered to be illustrative rather than restrictive. Therefore, it should be appreciated that those skilled in the art can make changes to those embodiments without departing from the scope of the present invention.
Claims
1. A method of operating a computer to construct a customized training data set for training an artificial intelligence (AI) platform, the method comprising: Obtaining (i) a specification of training data for training the AI platform and (ii) data indicating a proposed use of the training data; consulting a database of training data sets each associated with a usage right to identify a candidate set of training data sets that matches the specification; as well as The release of a subset of the training datasets in the candidate set of training datasets is authorized based on the data indicating the proposed use of AI training data and the usage rights associated with the training datasets in the candidate set of training datasets.
2. A method as defined in claim 1, wherein the specification includes an application scope.
3. The method defined in claim 2, wherein the application scope comprises at least one of (i) a global application and (ii) a specific application.
4. A method as defined in claim 1, wherein the specification further comprises a list of objects.
5. The method defined in claim 1, wherein the specification comprises a minimum percentage of training data derived from real-world imaging or a maximum percentage of training data derived synthetically.
6. The method defined in claim 1, wherein the data indicative of a proposed use of the training data comprises one or more of (i) an identifier of the AI platform and (ii) a proposed geographic region.
7. The method defined in claim 1 further comprising implementing a graphical user interface for obtaining the specification, and the data is obtained from a user.
8. A method as defined in claim 1, wherein the authorizing comprises creating a subset of the candidate set of training datasets, the subset comprising those training datasets in the candidate set whose usage rights include the proposed use of the AI training data.
9. The method defined in claim 8, wherein the subset of the candidate set of training data sets excludes any training data sets in the candidate set whose usage rights do not include the proposed use of the AI training data.
10. The method defined in claim 1, wherein the specification and the data indicating the proposed use of the training data are obtained from a user, the method further comprising determining whether the user has permission to train the AI platform.
11. A method as defined in claim 10, wherein at least some of the training data sets in the database are encrypted, and wherein if it is determined that the user has permission to train the AI platform, the reviewing includes the step of decrypting at least some of the training data sets in the database.
12. The method according to claim 1, wherein the training data sets each have a common format, including a plurality of data blocks, wherein the plurality of data blocks include: a first data block containing training media for a corresponding training data set; A second data block indicating the usage rights for the corresponding training data set; a third data block indicating media metadata corresponding to the training media in the first data block; and optionally, a fourth data block indicating a unique user identifier for the corresponding training data set.
13. The method defined in claim 12, wherein the plurality of data blocks further comprises a fifth data block indicating encryption information associated with the first data block.
14. The method defined in claim 12, wherein the media metadata corresponding to the training media in the first data block includes a list of objects depicted in the training media.
15. The method defined in claim 12, wherein the media metadata corresponding to the training media in the first data block further indicates whether the training media is derived from real-world imaging or is synthetically derived.
16. The method defined in claim 12, wherein the media metadata corresponding to the training media in the first data block further includes annotations.
17. A method as defined in claim 1, wherein in order to identify a candidate set of training data sets that matches the specification, the method includes comparing the specification with the media metadata in the third data block of each training data set in the training data sets, and the candidate set of training data sets includes those training data sets whose media metadata in the third data block of their corresponding training data sets matches the specification.
18. A method as defined in claim 1 wherein the specification includes a user provided application scope.
19. A method as defined in claim 1, wherein the specification comprises a user-provided list of objects.
20. A method according to claim 10, wherein the specification includes a user-specified application scope, wherein the usage rights in the second data block include information related to the permitted application scope of the corresponding training data set, wherein in order to identify a candidate set of training data sets that matches the specification, the method includes comparing the user-specified application scope with the permitted application scope, the candidate set of training data sets including those training data sets whose permitted application scope in the second data block of their corresponding training data sets includes the user-specified application scope.
21. A method as defined in claim 10, wherein the specification includes a user-specified list of objects, wherein the media metadata corresponding to the training media in the first data block includes information related to the list of objects depicted in the training media in the first data block, and wherein in order to identify a candidate set of training data sets that matches the specification, the method also includes comparing the user-specified list of objects with the list of objects in the third data block of each of the training data sets, the candidate set of training data sets excluding those training data sets whose list of objects in the third data block of the corresponding training data sets does not contain any objects in the user-specified list of objects.
22. A method as defined in claim 12, wherein the authorization comprises creating the subset of the candidate set of training datasets, the subset comprising those training datasets in the candidate set whose usage rights in the second data blocks of their corresponding training datasets include the proposed use of the AI training data.
23. The method as defined in claim 22, wherein the proposed use of the AI training data comprises an identifier of the AI platform, wherein the usage rights in the second data block of each of the training data sets identify at least one permitted AI platform, wherein the authorization comprises creating the subset of the candidate set of training data sets, the subset comprising those training data sets in the candidate set for which the at least one permitted AI platform identified by the second data block of the respective training data set comprises the identifier of the AI platform.
24. A method as defined in claim 22, wherein the proposed use of the AI training data includes a proposed geographic area, wherein the usage rights in the second data block of each training data set in the training data sets also include a permitted geographic area, wherein the authorization includes creating the subset of the candidate set of training data sets, the subset including those training data sets in the candidate set whose permitted geographic area in the second data block of their corresponding training data sets includes the proposed geographic area.
25. The method defined in claim 1 further comprising determining usage rights for a subset of the training data set, and storing the determined usage rights in the database in association with the subset of the training data set.
26. The method as defined in claim 25, further comprising setting the usage rights for the subset of training data sets based on the usage rights associated with each of the training data sets in the subset of training data sets.
27. A method as defined in claim 26, wherein when the usage rights associated with each of the training data sets in the training data set subset are the same, the usage rights for the training data set subset are set to the same usage rights associated with each of the training data sets in the training data set subset.
28. A method as defined in claim 26, wherein when the usage rights associated with each of the training data sets in the training data set subset are not the same, the usage rights for the training data set subset are set to those usage rights that are common to each of the training data sets in the training data set subset.
29. The method as defined in claim 1, wherein the training data sets each have a common format, comprising a plurality of data blocks, wherein the plurality of data blocks comprises: a first data block containing training media for a corresponding training data set; A second data block indicating the usage rights for the corresponding training data set; a third data block indicating media metadata corresponding to the training media in the first data block; and optionally, a fourth data block indicating a unique user identifier for the corresponding training data set, the method further comprising grouping the training data set subsets into training data set packages, and storing the training data set packages in a non-transitory medium.
30. A method as defined in claim 29, further comprising providing a usage rights field to the training data set package, the usage rights field being populated based on the usage rights stored in the second data block of the training data set in the training data set subset.
31. The method defined in claim 30, wherein the content of the usage rights field of the training data set packet is an intersection of the usage rights stored in the second data block of the training data set in the training data set subset.
32. A method as defined in claim 30, further comprising providing a metadata field to the training data set package, the metadata field being populated based on the metadata stored in the third data block of the training data set in the training data set subset.
33. A method as defined in claim 32, wherein the content of the metadata field of the training data set packet is a union of the metadata stored in the third data block of the training data set in the training data set subset.
34. A non-transitory computer-readable storage medium storing instructions that, when executed by a processing entity of a computing device, cause the computing device to perform a method of constructing a customized training data set for training an artificial intelligence (AI) platform, the method comprising: Obtaining (i) a specification of training data for training the AI platform and (ii) data indicating a proposed use of the training data; consulting a database of training data sets each associated with a usage right to identify a candidate set of training data sets that matches the specification; as well as The release of a subset of the training datasets in the candidate set of training datasets is authorized based on the data indicating the proposed use of AI training data and the usage rights associated with the training datasets in the candidate set of training datasets.
35. A computer system for constructing a customized training data set for training an artificial intelligence (AI) platform, the computer system comprising: a user interface module configured to obtain (i) a specification of training data for training the AI platform and (ii) data indicative of a proposed use of the training data; a filtering module configured to consult a database of training data sets each associated with a usage right to identify a candidate set of training data sets that matches the specification; as well as A rights management module configured to authorize release of a subset of the training datasets in the candidate set of training datasets based on the data indicating the proposed use of AI training data and the usage rights associated with the training datasets in the candidate set of training datasets.
36. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform a method comprising: - receiving a media unit from a user; - creating training data units based on the group of media units, each training data unit comprising a plurality of data blocks, wherein the plurality of data blocks of a given training data unit comprises: - a first data block containing media content from a respective one of the groups of media units; - a second data block indicating usage rights for said given training data unit; and - a third data block indicating media metadata corresponding to the media content in the first data block; and - Storing the training data unit in a non-transitory storage medium or outputting the training data unit via a data network.
37. A computer-readable medium as defined in claim 36, wherein the plurality of data blocks for the given training data unit further comprises a fourth data block indicating a unique user identifier for the given training data unit.
38. A computer readable medium as defined in claim 36 further comprising grouping the media units into groups of the media units.
39. A computer-readable medium as defined in claim 36, wherein grouping comprises grouping the media units based on objects depicted in the media units.
40. A computer readable medium as defined in claim 36 further comprising performing image processing to identify objects depicted in the media units, and wherein grouping is performed to group together media units that depict at least some of the same objects.
41. A method as defined in claim 40, wherein the media metadata in the third data block of the given training data unit includes information related to the objects depicted in the corresponding group of media units.
42. A computer readable medium as defined in claim 36, further comprising receiving a list of objects depicted in the media units, and wherein the grouping is performed to group together media units depicting at least some of the objects in the list.
43. A method as defined in claim 36, wherein the media metadata in the third data block of the given training data unit indicates whether the media content of the corresponding group of media units is derived from real-world imagery or is synthetically derived.
44. A computer-readable medium as defined in claim 36, further comprising verifying the identity of the user prior to said receiving.
45. A computer-readable medium as defined in claim 36, wherein said storing comprises encrypting at least a portion of said training data unit.
46. A computer-readable medium as defined in claim 45, wherein the encrypting comprises encrypting only the first data block of each of the training data units.
47. A computer readable medium as defined in claim 45, wherein the encrypting is performed using an encryption key stored in memory.
48. A computer-readable medium as defined in claim 47, wherein when stored, the media unit in the first data block of each of the training data units is decryptable by a decryption key.
49. A computer readable medium as defined in claim 48, wherein the decryption key and the encryption key are the same.
50. A computer readable medium as defined in claim 48, wherein the decryption key is uniquely associated with but different from the encryption key.
51. A method as defined in claim 36, wherein the plurality of data blocks further includes a fifth data block indicating how a decryption key is determined.
52. A method as defined in claim 36, wherein the usage rights for the given training data unit include an indication of a permitted scope of application of the given training data unit.
53. A method as defined in claim 36, wherein the usage rights for the given training data unit include an indication of a permitted geographic region for the given training data unit.
54. A method as defined in claim 36, wherein the usage rights for the given training data unit include an indication of a permitted AI platform for the given training data unit.
55. A method as defined in claim 36, further comprising determining the usage rights for the given training data unit and storing the usage rights in the second data block of the given training data unit.
56. A method as defined in claim 36, wherein the media unit comprises a 2D camera image.
57. A method as defined in claim 36, wherein the media unit comprises LiDAR or radar map data.
58. A method as defined in claim 36 wherein the media element comprises an ultrasonic depth map.
59. A method of operating a computer, the method comprising: - receiving a media unit from a user; - creating training data units based on the group of media units, each training data unit comprising a plurality of data blocks, The plurality of data blocks of a given training data unit include: - a first data block containing media content from a respective one of the groups of media units; - a second data block indicating usage rights for said given training data unit; and - a third data block indicating media metadata corresponding to the media content in the first data block; - Storing the training data unit in a non-transitory storage medium or outputting the training data unit via a data network.
