Computer-implemented data structure for processing sensor data

Abstract

The invention relates to a computer-implemented data structure (DS) for processing sensor data (SD) with a model (M) based on artificial intelligence, comprising - at least one pre-processing program code (PRE_PR) for pre-processing the sensor data (SD), - at least one post-processing program code (POST_PR) for post-processing the sensor data (SD), - the model (M) for processing the sensor data (SD) with o at least one training program code (TR_C) for training the model (M), o at least one inference program code (IN_C) for applying the sensor data (SD) to the model (M), o at least one artefact (ART) which describes the result of the executed at least one training program code (TR_C) and / or at least one inference program code (IN_C), - at least one business logic link (BL) relating to the at least one artefact (ART), - a processing configuration (CONF) which has at least one parameter for the execution of pre-processing (PRE_PR), post-processing (POST_PR) and / or processing of the data (TR_C, IN_C), - a dependency configuration (SW_PKG) which has at least one item of information on the dependency of program code modules and program code libraries.

Description

[0001] Computer-implemented data structure for processing sensor data

[0002] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identity are included.

[0003] The invention relates to a computer-implemented data structure for processing sensor data with a model based on artificial intelligence.

[0004] The invention further relates to a computer-implemented method and a system for operating a technical device with a model based on artificial intelligence.

[0005] The invention also relates to a computer program product.

[0006] For more and more industrial customers, it is becoming a challenge to develop individual use cases individually due to a shortage of skilled workers and the high cost of development.

[0007] In production plants, visual quality control is often required, which can theoretically be easily automated.

[0008] However, this requires an enormous amount of effort in terms of individual adjustment of the algorithms, because, for example, different lighting conditions, placement of machines, age and operating hours of the machines require a high degree of adaptation, which can adversely affect the prediction quality of individual Kl models.

[0009] In the current state of the art, such adjustments are carried out manually or a solution is developed individually.

[0010] Solutions with artificial intelligence (AI) based on machine learning (ML) are currently manually adjusted or parameterized. Such adjustments are also necessary for the AI ​​algorithms used.

[0011] The object of the invention is to simplify the application of an AI model and to allow even unprofessional personnel to easily compile AI applications, execute them with sensor data, and also operate a technical device.

[0012] The object of the invention is achieved by a computer-implemented data structure for processing sensor data with a model based on artificial intelligence, comprising at least one pre-processing program code for pre-processing the sensor data, at least one post-processing program code for post-processing the sensor data,

[0013] - the model for processing the sensor data with o at least one training program code for training the model, o at least one inference program code for applying the sensor data to the model, o at least one artifact which describes the result of the executed at least one training program code and / or at least one inference program code, at least one business logic link relating to the at least one artifact,

[0014] - a processing configuration which has at least one parameter for carrying out pre-processing, post-processing and / or processing of the data,

[0015] - a dependency configuration that contains at least information about the dependencies of program code modules and program code libraries. This ensures that the application of AI models is very simple to implement, is very easily scalable, and simplifies the application of AI solutions overall, since no special expertise in ML or statistics is required.

[0016] Furthermore, a new type of portability is achieved for such K0 tasks; it is therefore possible to capture a K1 task from a first computing device and have it calculated by a second computing device, for example to achieve a better resource distribution in the system.

[0017] In other words, the data structure can be viewed as an interface between spatially or logically separated system units or corresponding computing devices.

[0018] The creation of a corresponding data structure as a "solution package" offers the decisive advantage that a software bundle can be put together in a "modular" manner, which can then be processed in a simple manner by a corresponding computing device, which is different from a "generating" computing device, which in turn defines the Kl task and / or carries out the corresponding control with the calculated results of the Kl task of a device.

[0019] A training program code for training the model is used to train a model with a data set, for example, to use data sets with data labeled as "good" and "bad" to create an accurate and realistic model.

[0020] An inference program code for applying the sensor data to the model is used to apply the previously trained model and, for example, to predict the operating behavior of a technical device based on a current data set.

[0021] An artifact is a machine learning term used to describe the output produced by the training process. The output can be a fully trained model, a model checkpoint, or a file created during the training process.

[0022] AI artifacts describe all digital products used in an AI tool. They can be the input, the output, or an intermediate result processed by tools.

[0023] Essentially, six types of artifacts can be specified, which correspond to the steps in a pipeline: data, knowledge, model, application, algorithm, and benchmark.

[0024] The object according to the invention is achieved by a computer-implemented method for operating a technical device with a model based on artificial intelligence, wherein the following steps are carried out: a) Acquisition of sensor data by an acquisition means and provision to a first computing device, b) Provision of application data to the first computing device, comprising at least one pre-processing program code for pre-processing the sensor data, at least one post-processing program code for post-processing the sensor data, at least one model, comprising o at least one training program code, o at least one inference program code, o at least one artifact which represents the result of the executed at least one training program code and / or at least one inference

[0025] Program code describes at least one business logic link which generates an operation for the operation of the technical device from the at least one artifact,

[0026] - a processing configuration,

[0027] - a dependency configuration, c) creating a data structure according to the preceding claim with the sensor data from step a) and the application data from step b) as an application package by the first computing device, and transmitting the application package to a second computing device, d) executing the application package by the second computing device, e) generating an operating data record for controlling the technical device from the application package by the second computing device, and transmitting the operating data record to the first computing device, f) operating the technical device with the operating data record by the first computing device.

[0028] The method can be used to apply the data structure mentioned to control and operate a technical device in a simple manner and without specific ML / AI expertise.

[0029] The data structure can improve the internal processes when processing the relevant data.

[0030] For example, any second computing device can be used to calculate the AI ​​task, independently of the first computing device, which determines the inference and controls the device accordingly. This makes it possible to perform the processing on a second computing device without the need for a skilled operator.

[0031] This provides additional flexibility for the efficient processing of the AI ​​task, for example regarding the availability of the first inference computing device.

[0032] Furthermore, the first computing device may have a lower performance than the second computing device.

[0033] Overall, an improved power distribution can be achieved when performing computing tasks in the system.

[0034] The data structure therefore represents a simple and efficient means of transporting the relevant data of a class task.

[0035] The steps d) and e) are thus not carried out by the first computing device, but by the second computing device, which is not formed by the first computing device.

[0036] The first and the second computing device are comprised in a system which is designed to carry out the method according to the invention.

[0037] For example, the system is a client-server system, and the first computing device is comprised by a first client and the second computing device is comprised by the server or another client.

[0038] In a further development of the invention, it is provided that when providing in step b) the respective elements are selected from predefined lists.

[0039] This allows, for example, a model and / or program code to be provided from one server to multiple clients in the sense of federated learning. Federated learning describes a technique in the field of machine learning in which a model is trained on multiple devices.

[0040] Each participating device has its own local data set that is not exchanged with other participants.

[0041] However, it is advantageous if a generic central model is provided by a server to clients, and the clients each develop an individual, local model.

[0042] In contrast to federated learning, traditional machine learning has a central dataset.

[0043] Through federated learning, multiple clients can build and centrally manage a common robust model, taking into account critical issues such as privacy, data security, access rights and access to heterogeneous data.

[0044] In a further development of the invention, it is provided that the at least one model further comprises a filter program code, preferably a noise removal filter program code, which is based on an auto-encoder.

[0045] This makes it easy to filter, for example, noise in a digital image.

[0046] The object of the invention is achieved by a system for operating a technical device with a model based on artificial intelligence, comprising a first computing device, a second computing device, and a connected means for detecting sensor data, wherein the computing device is set up to carry out the method according to the invention. In a development of the invention, a system is provided, comprising a server with a first processor and a first memory and at least one client, each with a second processor and a second memory, wherein the at least one client is set up to carry out step a) and the server is set up to carry out step b).

[0047] In other words, the second computing device may be formed by the server, for example to improve the resource distribution in the system.

[0048] This also makes it possible for the server to provide modules and / or program code to the client in the form of a central library.

[0049] In a further development of the invention, a system is provided, comprising a first client and at least one second client, wherein the first client is configured to carry out step a) and the at least one second client is configured to carry out step d).

[0050] In other words, the second computing device can be formed by the at least one second client in order to, for example, improve the resource distribution and availability in the system.

[0051] This also makes it possible to select clients with different resources in terms of computing power or memory in order to execute process components in a distributed manner according to an optimal resource utilization.

[0052] The object of the invention is achieved by a computer program product having machine-readable instructions stored therein which, when executed by a processing unit, cause the processing unit to carry out the method according to one of the preceding claims. The invention is explained in more detail below with reference to an exemplary embodiment shown in the accompanying drawings. In the drawings:

[0053] Fig. 1 shows an example of a client-server system architecture,

[0054] Fig. 2 a general example of an "AI solution package" concept,

[0055] Fig . 3 an example of a production of pastries,

[0056] Fig . 4 an example of "good" pastry,

[0057] Fig . 5 an example of "defective" pastry,

[0058] Fig . 6 an example of an "AI solution package" ,

[0059] Fig. 7 shows an example of processing by an auto-encoder,

[0060] Fig. 8 shows an embodiment of the method according to the invention.

[0061] Fig. 1 shows an example of a client-server system architecture with a server S and three connected clients C1-C3, each having computing devices with a processor and a memory, wherein the clients C1-C3 are each connected to a technical device TD1-TD3.

[0062] Fig. 2 shows the general concept of an “AI solution package” using an example of such a solution package SP.

[0063] An "AI solution package" allows the integration of AI models, for example, for a production application, without the need for AI / ML expert knowledge.

[0064] An AI solution package is easy to use because the solution package provides a simple and intuitive user interface for integrating an AI model into a production environment.

[0065] Predefined AI models can be easily selected via the user interface, for example from models that are already tailored and trained for specific use cases, such as object recognition or image segmentation.

[0066] Furthermore, a user of the system can easily upload or download data, such as images from a camera system, and process this data using a visual quality engineering system (VQE for short).

[0067] The VQE system can automatically train the AI ​​model M with the data and, for example, perform parameter optimizations without the user requiring relevant specialist knowledge, such as in the areas of ML or statistics.

[0068] The VQE system can also automatically validate the AI ​​model M with the data to provide the user with feedback on the operations performed.

[0069] An "AI solution package" also allows for the integration of the generated AI model into a production environment, as well as different deployment modes, such as a cloud or an on-premises solution.

[0070] An "AI solution package" also allows observation of the AI ​​model by providing metrics or model states to the user, for example.

[0071] A use case includes software packages, pre-processing program code PRE_PR and post-processing program code POST_PR.

[0072] Furthermore, the use case includes program code for a model M, which includes a training code TR_C, an inference code IN_C, and at least one artifact ART. Furthermore, the use case includes business logic BL.

[0073] A GONE configuration describes the connections of the previously mentioned packages and codes that make up a software

[0074] Packet SW_PKG can be formed, which in turn can be provided to an Al-Inf erence server AUS and further processed there.

[0075] In Fig . 3 an example of a production of pastries is shown .

[0076] Three different images from respective camera sensors show pastries on a conveyor belt from different angles in an overview.

[0077] For example, a pastry may be "defective," meaning it does not meet predetermined quality parameters and should be automatically sorted out.

[0078] Fig. 4 shows an example of a pastry which is of order in shape and is sufficiently spatially separated from other pastries.

[0079] Fig. 5 shows an example of a pastry which is considered to be defective because the separation from other adjacent pastries is insufficient.

[0080] Fig. 6 shows an example of an AI solution package with input operations applied to sensor image data.

[0081] Operation OPI can include, for example, image cropping and image size scaling and is performed by the pre-processing PRE_PR .

[0082] For example, operation 0P2 can be a detection of good and defective pastries and is performed by model M and its processing. Operation OP3 can be a viewing angle adjustment and is performed by the post-processing POST_PR.

[0083] For example, operation 0P4 can calculate coordinates for a gripper of a transport robot and is executed by the business logic BL.

[0084] Overall, a solution package can help improve the development time of AI models and their application, such as in production.

[0085] Fig. 7 shows an example of improved processing of an AI model M by an auto-encoder, which comprises an encoder ENG, a latent space LS and a decoder DEC.

[0086] The auto-encoder allows learning a representation of data in an efficient way, such as dimension reduction or feature extraction.

[0087] The auto-encoder has an input layer which provides input data to a network.

[0088] Furthermore, the auto-encoder includes a hidden layer with the encoder ENG and the decoder DEC to process the input data by applying weights, bias and activation functions.

[0089] The auto-encoder also has an output layer that corresponds to the input neurons.

[0090] The auto-encoder is implemented by a denoising filter program code, which forms a denoising filter DF, with the denoised images being provided to the Solution Package SP.

[0091] The noise removal filter program code can optionally be included in the model M. Fig. 8 shows an exemplary embodiment of the method according to the invention, which is executed on the system according to Fig. 1 and takes into account the data structure DS according to Fig. 2.

[0092] The following steps are carried out: a) Acquisition of sensor data SD in the form of image data by a recording device in the form of an inspection

[0093] Camera CAM and providing to a computing device formed by the client CI, b) providing application data to the computing device CI, comprising

[0094] - Pre-processing program codes PRE_PR for pre-processing the sensor data SD,

[0095] - Post-processing program codes POST_PR for post-processing the sensor data SD,

[0096] - a model M, comprising o training program codes TR_C, o inference program codes IN_C, o artifacts ART , which describe the result of the executed training program codes TR_C and / or inference program codes IN_C ,

[0097] - Business logic links BL, which generates operations from the artifacts ART for the operation of the technical device TD1,

[0098] - a processing configuration CONE,

[0099] - a dependency configuration SW_PKG . c ) Creating a data structure DS according to the previous explanations with the sensor data SD from step a ) and the application data from step b ) as an application package SP, by the first computing device CI , and transmitting the application package SP to a second computing device d) Executing the application package SP by the second computing device, e ) Creating an operating data record for controlling the technical device TD1 from the application package SP, by the second computing device, and transmitting the operating data record to the first computing device CI , f ) Operating the technical device TD1 with the operating data record by the first computing device Gl .

[0100] When providing in step b ), the respective elements can be selected from predefined lists provided by the server S .

[0101] In steps e) and f), the model M is applied to the device TD1, for example by calculating prediction data from which the operating data set is determined.

[0102] Optionally, the Model M may further comprise a noise removal filter program code based on an auto-encoder.

[0103] The method can be carried out on the system according to Fig. 1, which comprises a computing device GI with a connected acquisition means CAM for the sensor data SD.

[0104] The system according to Fig. 1 comprises a server S with a first processor and a first memory and three clients C1-C3, each with a second processor and a second memory.

[0105] A client CI can be configured to perform step a) and the server S can be configured to

[0106] Carry out step b).

[0107] In other words, the second computing device may be formed by the server S.

[0108] The system according to Fig. 1 comprises a first client CI and further second clients C2, C3. The first client CI can be configured to perform step a), and a second client C2, C3 can perform step d).

[0109] In other words, the second computing device may alternatively be formed by the second client C2 or C3.

[0110] The steps d) and e) are therefore not carried out by the first computing device, but by the second computing device, which is not formed by the first computing device, and therefore does not correspond to the first client CI and not to the server S.

[0111] List of reference symbols:

[0112] AUS AI Inference Server

[0113] ART Artifact

[0114] BL Business Logic

[0115] C1-C3 Client

[0116] CAM sensor, camera

[0117] CONE configuration

[0118] DEC decoder

[0119] DF noise removal filter

[0120] DS data structure

[0121] ENG Encoder

[0122] IN_C inference code

[0123] LS latent space

[0124] M model

[0125] OP1-OP4 operations

[0126] POST_PR Post-Processing

[0127] PRE_PR Pre-Processing

[0128] S Server

[0129] SD sensor data

[0130] SP solution package

[0131] SW_PKG Software Package

[0132] TD1-TD3 technical device

[0133] TR_C training code

Claims

Patent claims 1. Computer-implemented data structure (DS) for processing sensor data (SD) with a model (M) based on artificial intelligence, comprising at least one pre-processing program code (PRE_PR) for pre-processing the sensor data (SD), at least one post-processing program code (POST_PR) for post-processing the sensor data (SD), - the model (M) for processing the sensor data (SD) with o at least one training program code (TR_C) for training the model (M), o at least one inference program code (IN_C) for applying the sensor data (SD) to the model (M), o at least one artifact (ART) which describes the result of the executed at least one training program code (TR_C) and / or at least one inference program code (IN_C), at least one business logic link (BL) relating to the at least one artifact (ART), - a processing configuration (GONE) which has at least one parameter for executing preprocessing (PRE_PR), postprocessing (POST_PR) and / or processing of the data (TR_C, IN_C), - a dependency configuration (SW_PKG) which contains at least information on the dependency of program code modules and program code libraries.

2. Computer-implemented method for operating a technical device (TD1-TD3) with a model (M) based on artificial intelligence, wherein the following steps are carried out: a) Acquiring sensor data (SD) by a capturing means (CAM) and providing it to a first computing device (C1-C3), b) providing application data to the first computing device (C1-C3), comprising at least one pre-processing program code (PRE_PR) for pre-processing the sensor data (SD), at least one post-processing Program codes (POST_PR) for post-processing the sensor data (SD), at least one model (M), comprising o at least one training program code (TR_C), o at least one inference program code (IN_C), o at least one artifact (ART) which describes the result of the executed at least one training program code (TR_C) and / or at least one inference program code (IN_C), at least one business logic link (BL) which generates an operation for operating the technical device from the at least one artifact (ART), - a processing configuration (GONE) , - a dependency configuration (SW_PKG). c) Creating a data structure (DS) according to the preceding claim with the sensor data (SD) from step a) and the application data from step b) as an application package (SP) by the first computing device (C1-C3), and transmitting the application package (SP) to a second computing device, d) Executing the application package (SP) by the second computing device, e) Generating an operating data set for controlling the technical device (TD1-TD3) from the application package (SP) by the second computing device (C1-C3), and transmitting the operating data record to the first Computing device, f) operating the technical device (TD1-TD3) with the operating data set by the first computing device (GIGS).

3. Method according to the preceding claim, wherein during the provision in step b) the respective elements are selected from predefined lists.

4. Method according to one of the preceding claims, wherein the at least one model (M) further comprises a filter program code, preferably a noise removal filter program code, which is based on an auto-encoder.

5. System for operating a technical device with a model (M) based on artificial intelligence, comprising a first computing device (C1-C3), a second computing device, and a connected acquisition means (CAM) for sensor data (SD), wherein the computing device (C1-C3) is configured to carry out the method according to one of the preceding claims.

6. System according to the preceding claim, comprising a server (S) with a first processor and a first memory and at least one client (C1-C3) each with a second processor and a second memory, wherein the at least one client (C1-C3) is configured to carry out step a) and the server (S) is configured to carry out step b).

7. System according to the preceding claim, comprising a first client (Gl) and at least one second client (C2, C3), wherein the first client (Gl) is configured to carry out step a) and the at least one second client (C2, C3) is configured to carry out step d).

8. A computer program product having machine-readable instructions stored therein which, when executed by a processing unit, cause the processing unit to carry out the method according to any one of the preceding claims.

Images