Petri net-based modeling and identification of operational disturbances in sensor systems

The accessibility diagram is generated through the Petri network model to identify the operating interference of the sensor system in the new environment, solve the problem of not being able to identify system interference in the existing technology, realize the estimation of the operating status of the sensor system and real-time interference detection, and improve the System stability.

CN119089720APending Publication Date: 2024-12-06ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410725020.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-05
Filing Date
2024-06-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art cannot effectively identify operational interference and failures of sensor systems when running in new environments, especially when processing new data sets, and it is impossible to determine whether system interference will be caused.

Method used

The Petri net model is used to generate the accessibility diagram. By initializing the Petri net and meeting the required conditions along the propagation step, it reflects the operating status of the sensor system, generates an accessibility diagram containing the operating status information, and uses this diagram to identify system interference. .

Benefits of technology

It can estimate all possible operating status of the sensor system without actually running it, identify system interference in real time, prevent equipment functions from being affected, and improve system stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089720A_ABST
    Figure CN119089720A_ABST
Patent Text Reader

Abstract

Petri net-based modeling and identification of operation interference in a sensor system. One aspect of the present disclosure relates to a method for generating a reachability map for identifying operational disturbances of a sensor system, in which the sensor system can be operated with software designed to perform a plurality of tasks in a specific order. The method includes generating a Petri net containing information about the number and order of tasks of software of the sensor system, where each task is described by a respective transition of the Petri net, which transition is performed by the software if one or more required conditions for that task are met. The Petri net includes an input library containing information about required conditions for performing the task. The method includes initializing the Petri net by an initial flag in a manner that corresponding conditions for executing at least one or more tasks of the software are set to be satisfied.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for identifying operational disturbances of a sensor system using a reachability diagram, a corresponding module, a sensor system and a computer program. Background Art

[0002] Sensor systems (e.g. based on cameras, radars, lidars or other sensors) are increasingly being used in various technical devices in order to ensure the detection of objects and structures and to enable the execution of the functions of such devices. In many cases, for example when the sensor systems are used for semi-autonomous, autonomous or assisted driving of vehicles, classical image processing algorithms are combined with methods of artificial intelligence in order to ensure robust object recognition, for example, which is detected by a camera. For example, the sensor systems include machine learning systems (e.g. such systems using artificial neural networks), which are extensively trained and / or validated on the basis of known data sets in order to provide reasonable output data (i.e. unknown output data) for any given input data (e.g. in the context of image processing). In some cases of the prior art, for example in conjunction with driver assistance systems, partially automated or automated driving, the functional behavior of the device using the sensor system (e.g. the specific behavior of the ABS emergency brake function) can be verified by means of a large validation data set in a hardware-in-the-loop simulation (“HiL” simulation), which validation data set has usually been recorded in advance (e.g. video stream, measured radar sensor reflections or lidar point cloud recordings).

[0003] On the other hand, the stability or operating state of the sensor system is also evaluated in conjunction with the devices coupled to the sensor system, usually by stress testing the system, for example using expert estimates for specific situations, such as driving scenarios, which result in the highest possible system computing load (e.g. CPU utilization, RAM bandwidth transfer, runtime / utilization of individual system components).

[0004] However, some prior art methods do not make any statement about the extent to which the operating state of the sensor system (or in other words its internal state) is covered by a validation data set previously created and subsequently processed by the sensor system (e.g. for training an artificial neural network). Furthermore, some prior art methods cannot determine whether a new data set generated by the sensor system (e.g. new video data detected in real time by a vehicle camera during driving in a new environment) may cause a disturbance in the operation of the sensor system.

[0005] Therefore, there is a need to develop new techniques for identifying operational disturbances in sensor systems that can address some or all of the above-mentioned issues. Summary of the invention

[0006] A first general aspect of the disclosure relates to a method for generating a reachability graph for identifying operational disturbances of a sensor system, wherein the sensor system can be operated using software, the software being designed to perform a plurality of tasks in a specific order. The method comprises generating a Petri net, the Petri net containing information about the number of tasks of the software of the sensor system and their order. Each task of the first aspect is described by a corresponding transition of the Petri net, which is executed by the software if one or more required conditions for the task are satisfied. For this purpose, the execution of a task is at least one required condition for executing a subsequent task. The Petri net of the first aspect comprises one or more input places, the input places containing information about the required conditions for executing a task, wherein the one or more input places are connected to the transitions of the Petri net by one or more input edges, respectively. Furthermore, the method comprises initializing the Petri net by an initial marking, in such a way that the corresponding conditions for executing at least one or more of the plurality of tasks from the software are set to be satisfied. Furthermore, the method comprises propagating along the Petri net starting from the initial marking by determining the tasks of the software for which the required conditions are satisfied for each corresponding propagation step. Finally, the method includes generating a reachability graph, the reachability graph including a plurality of nodes, wherein each node includes information about the required conditions satisfied for executing a task for a corresponding propagation step along the Petri net and reflects the operating state of the sensor system. The information contained in the node includes the label of the Petri net for the corresponding propagation step, wherein the information contained in the first and last nodes includes the initial label of the Petri net.

[0007] A second general aspect of the present disclosure relates to a method for identifying operational disturbances of a sensor system using a reachability graph. The method includes receiving a reachability graph for identifying operational disturbances of a sensor system generated according to the first aspect. In addition, the method includes processing a validation data set through the received reachability graph.

[0008] A third general aspect of the present disclosure relates to a module configured to execute the method for generating a reachability graph for identifying operational disturbances of a sensor system according to the first general aspect of the present disclosure.

[0009] A fourth general aspect of the present disclosure relates to a sensor system designed for generating a validation data set suitable for being processed in a method according to the second aspect.

[0010] A fifth general aspect of the present disclosure relates to a module, which is designed to receive a reachability graph from a module according to the third aspect. In addition, the module of the fifth aspect is designed to receive a validation data set from a sensor system according to the fourth aspect and to perform a method according to the second aspect for identifying operational disturbances of a sensor system.

[0011] A sixth general aspect of the present disclosure also comprises a computer program designed to execute the method of the present disclosure.

[0012] The techniques of the first to sixth general aspects may have one or more of the following advantages.

[0013] First, compared to some prior art techniques, the techniques of the present disclosure provide the following possibilities: estimating which operating states of a sensor system can be realized (e.g., all possible operating states, including those operating states that may occur when the sensor system is put into operation in a new environment). In particular, this knowledge can be obtained with the aid of software that runs the sensor system, without having to put the sensor system into operation.

[0014] Secondly, the present technique may enable identification of operational disturbances of the sensor system (or in other words, the risk of a possible malfunction of the sensor system) based on knowledge about the operational state of the sensor system, for example when the sensor system is operated in a new environment.

[0015] Thirdly, in the present technology, knowledge about the operating state can be directly implemented into the sensor system, so that (for example, when the detected image data is processed in real time by the sensor system) it is possible to efficiently detect in real time when an operational disturbance of the sensor system may occur. In particular, the present technology provides the possibility of preventing the functions of devices assigned to the sensor system from being affected by the sensor system when the sensor system has an operational disturbance.

[0016] Certain terms are used in this disclosure in the following manner:

[0017] The term "sensor system" is understood to be any system that can contain one or more sensors, such as cameras, radars, lidars, ultrasonic or thermal sensors as modules. As a non-exhaustive list of other modules of the sensor system, CPU cores, system-on-chip (SoC) hardware, working memory, hard disks, processing modules (such as image processing modules), and communication interfaces with other devices can be mentioned. The above-mentioned sensors can detect their environment and generate their environment, for example, in the form of image data or data series. For example, the sensor system can be used in the scope of partial autonomy, autonomous or assisted driving of a vehicle to provide functionality (such as safety functions, implementation of driver information and comfort functions, such as intelligent headlight control and traffic sign information) for the vehicle. In other examples, it is conceivable that the sensor system can be used for monitoring tasks (such as manufacturing processes and / or examples for quality assurance) (for example, for monitoring the environment of at least partially autonomous robots).

[0018] The sensor system may include a camera (e.g., a front camera and / or a rear camera of a vehicle) and a system on chip (SoC), which includes, for example, a microprocessor for an image processing algorithm. A camera-based sensor system may use one or more technical paths for image processing. The first path of such a sensor system may allow the typical appearance of an object class, such as a vehicle, a cyclist, or a lane marking to be recognized by a programmed algorithm. In the case of a second path, a camera-based sensor system may use optical flow and so-called motion structure (SfM) to identify prominent lane limiting objects, such as curbstones, scraping edges, or guardrails. Closely related image points may be tracked in terms of their motion. Thus, a three-dimensional structure may be estimated from a two-dimensional camera image. The third path may rely on artificial intelligence (e.g., based on a computer-based machine learning system) by classifying objects such as vehicles parked on the side of the road and distinguishing lane surfaces from lane edges.

[0019] The above-described exemplary sensor system of a vehicle can communicate via corresponding interfaces with other systems in the vehicle, such as a vehicle computer (English: “Vehicle Computer”) or a control unit (English: “ECU” or “electronic control unit”), and with systems outside the vehicle, such as a cloud system.

[0020] In the following, the term "software" refers to software that can run the sensor system. That is, the software can manage various system resources of the sensor system, such as CPU core, working memory, hard disk, processing modules (e.g. imaging processing modules), input and output devices, and provide said system resources for applications of the sensor system and / or for various tasks (e.g. associated with image processing). For example, the software can be the operating system (English "operating system" or "OS system") of the sensor system.

[0021] Correspondingly, the term "task of software" is a schedulable unit managed by software (e.g., by an operating system). A task can be a part of code or a function that performs a specific activity when the function is allowed to be executed. A task of software associated with image processing of a camera can be one of the following tasks called in a sensor system, such as "read in a new image", "image preprocessing", "execute a neural network" (e.g., a convolutional neural network or "CNN" for short), for example, classification or regression tasks (e.g., for object recognition such as pedestrians, traffic signs, etc.), "calculate optical flow", "3D lane surface estimation", "lane marking detection", or others. The software can decide which tasks can be run on the CPU core of the sensor system and when. The application of the sensor system can also run in the context of a task. The "task planner" (English: "Scheduler") of the software can control the execution of tasks according to task priorities and task scheduling strategies. In some cases, the "task planner" can be part of the software (e.g., an operating system).

[0022] Within the scope of the present disclosure, a task may need to end the execution of one or more other tasks in order to begin its execution. In some cases, when these required tasks have completed execution, a "software event" (English "event") can be generated by the software (e.g., one "software event" for each completed task). Alternatively or additionally, other "software events" that are necessary for the execution of the task and are not associated with the previously completed tasks can be generated. In this sense, when one or more specific software events occur (more about "required conditions" in the following paragraphs), the task can be executed (or continue its execution). Software interrupts (English "interruptus"), locks (English "Lock"), spin locks or any combination thereof belong to other processes that may be required for the execution of tasks in combination with software.

[0023] In the present disclosure, a "Petri net" refers to a model for modeling various processes in software (e.g. in an operating system) that runs a sensor system (e.g. a camera). As already mentioned above, a process may include a non-exhaustive list of software tasks, software events, interrupts, locks or software spin locks. A Petri net may be referred to as a special type of directed graph built from three types of objects. These objects are places (input or output places), transitions and directed edges (hereinafter also referred to as "edges") that connect places to transitions and vice versa, see e.g. https: / / doi.org / 10.1109 / 41.334574 For more details. In a graphical manner, places may be represented by circles 14a-f, transitions may be represented by boxes 12a-d, and directed edges may be represented by arrows 16a-f, see Figures 2a to 2e As discussed further below. If there is a directed edge connecting a place to a transition (more specifically: directed edge 16a goes in the direction from place 14a to transition 12a), then the place is an input place for the transition. If there is a directed edge connecting a transition to a place (more specifically: directed edge 16c goes in the direction from transition 12a to place 14c), then the place is an output place for the transition.

[0024] A "transition" in the sense of the present disclosure may describe a task of software, and an "input place" connected to the transition by an "input edge" may describe a "required condition" for executing the task associated with the transition. In some cases, a transition may be connected to only a single input place. In other cases, a transition may be connected to more than one input place (e.g., two or more, three or more input places), which describe the corresponding required conditions. A non-exhaustive list of "required conditions" includes the occurrence of any combination of one or more events of software, one or more interrupts of software, one or more locks of software, or one or more spin locks of software required to execute the task under consideration (in other words, each of the required conditions must contribute to the execution of the task accordingly). In some cases, another (previously executed) task of the software that executes another transition assigned to the Petri net may belong to the required condition for the task to be executed subsequently. The existence of the required condition (in other words, if the condition is met) is indicated by a token (in English "token") in the input place (i.e., the input place is filled with a token), see Figures 2a to 2eThe bold points 18a-d in FIG. 18b. An input place may be filled with no tokens (if the required conditions assigned to the place are not met), with one token, with two tokens, with three tokens or more. As explained in more detail further below, in order to generate a Petri net and propagate along the net, it is not necessary to perform the actual task of the software (e.g. a task requiring image processing): rather, this involves determining the conditions required for performing the task while propagating along the Petri net.

[0025] If each input library place of the transition contains at least the number of tokens corresponding to the weight of the input edge connected to the transition, then this "transition" of the Petri net is switchable (in other words, it is considered to be activated / switchable). Switching an activated transition may mean that the task assigned to the transition is executed. In addition, a switching activated transition can remove a number of tokens corresponding to the weight of the input edge connecting the input library place to the transition from each input library place. A number of tokens corresponding to the weight of the output edge connecting the transition to the output library place can also be stored in each output library place. For each propagation step (in other words, at the corresponding time point), the distribution of tokens on the library place, the so-called Petri net marking, can define the "operating state of the sensor system" (see more about this further described below). The initial marking corresponds to the initial distribution of tokens on the library places of the Petri net: starting from the initial distribution and assuming that one or more transitions are switchable in the sense defined above, the propagation along the Petri net can be carried out step by step. This means that such a propagation can be regarded as a switching sequence of transitions and a discrete movement of tokens within the Petri net (the distribution of tokens for each propagation step).

[0026] The term "reachability graph" relates to a graph comprising nodes which contain information about Petri net markings during propagation (step by step) along the Petri net (a node may describe exactly one Petri net marking for one propagation step). In one case, during propagation (step by step) along the Petri net, a set of nodes may appear in a specific order, as in another case another set of nodes may appear (in both cases the first node may describe the initial marking). Each of these sequences determines a corresponding step-by-step propagation through the Petri net, and such sequences are referred to below as "paths of the reachability graph". A "reachability graph" may include all possible paths (see below for other parameters of the reachability graph).

[0027] In the present disclosure, the term "validation data set" is understood as a collection of multiple image data or data series, which can be used to identify operational disturbances of the sensor system, as further explained in more detail below. The validation data set may include image sequences, video data, data sequences (e.g., time series) or any combination thereof. In some cases, the validation data set may also include metadata (e.g., timestamps or other metadata) characterizing the data sequences listed above. The image data or data series can be generated with the help of various sensors of the sensor system (e.g., as already mentioned above, cameras, radars, lidars, ultrasonic or thermal sensors). For the purpose of identifying operational disturbances of the sensor system, the image data or data series can be processed in real time, for example during video recording by a camera of the sensor system or during measurement by a thermal sensor (e.g., in the case of a moving vehicle). In some cases, the detected data can first be detected by a sensor and then processed in the sense mentioned above at the sensor system including these sensors. Furthermore, these data can be generated by an external device (e.g. by another sensor system that can be run by the same software) and / or synthetic image data can be generated, which are then used at the sensor system to identify operational disturbances of the sensor system, for which further information see below.

[0028] The term "operational disturbance" of a sensor system means any disturbance of the system, including such disturbances that may affect the executableness of one or more functions of a device that uses information from the sensor system to ensure these functions. For example, if the sensor system is used in a vehicle, the term "function" may include control-based functions of the vehicle, such as functions of a driving or parking assistance system, functions for autonomous or partially autonomous driving, or functions in combination with an air conditioning system and / or an electronic system for controlling functions of the interior space (or any combination of the functions mentioned above). The term "operational disturbance" will also include "potential operational disturbances", i.e. disturbances that have not yet occurred but may still occur (the prerequisites of which are further explained below).

[0029] Accordingly, the operating states of the modules of the sensor system in their entirety may define the "operating state of the sensor system". For example, a task of the software that provides / controls the calling of one or more modules so that they perform corresponding tasks may change the operating state of these modules and, therefore, the operating state of the sensor system. As explained in more detail below, in some cases, such changes in the operating state of the sensor system may lead to "operational disturbances". It can be said that a "Petri net" (or "reachability diagram") reflects the "operating state of the sensor system", because the net may contain information about the tasks of the software, more on which is described further below.

[0030] The term "vehicle" includes any device designed for transporting passengers and / or cargo. A vehicle may be a motor vehicle (e.g. an at least partially autonomously operated / assisted motor vehicle, in particular a passenger car or a truck). However, a vehicle may also be a ship, a train, an aircraft or a spacecraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1a is a flow chart illustrating an example of a method for generating a reachability map for identifying operational disturbances of a sensor system according to the first aspect. Figure 1b is a flow chart showing an example of a method according to a second aspect for identifying operational disturbances of a sensor system using a reachability graph.

[0032] Figures 2a to 2e Schematically, an exemplary structure of a Petri net 10 is shown, which has four transitions (small boxes 12a-d), six places (circles 14a-f) and twelve edges (arrows 16a-f). In these figures, an exemplary distribution of tokens at places during propagation along the Petri net is also shown (bold points 18a-d).

[0033] Figure 3 The exemplary structure of the reachability graph 50 is schematically shown. Figures 2a to 2e 52a, b, c and e of the reachability graph, and their sequence ("path") is indicated by solid arrows. Possible but not implemented markings with corresponding parts of the path that also belong to the reachability graph are illustrated with dashed arrows and dashed small boxes 52d.

[0034] Figure 4 An exemplary number of reached nodes 62 , 70 of a reachability graph is schematically shown as a function of time 61 for a validation dataset (time interval 71 ) and an enhanced validation dataset (time intervals 71 and 72 ). DETAILED DESCRIPTION

[0035] First, according to Figure 1a A technique for generating a reachability graph for identifying operational disturbances in a sensor system is described. Figures 2a to 2e Discuss the exemplary structure of a Petri net and the propagation along the Petri net. Next, combine Figure 3 An example of the structure of a reachability graph is presented. Figure 1b and Figure 4 Techniques for identifying operational disturbances of a sensor system using reachability graphs are discussed.

[0036] As in Figure 1a As outlined in , a first general aspect relates to a method (e.g., a computer-implemented method) for generating a reachability map 50 for identifying operational disturbances of a sensor system. An operational disturbance of a sensor system (e.g., as mentioned above with respect to an operational state of the sensor system) may be, for example, an overload of the sensor system (e.g., an overload of a CPU core or insufficient memory capacity), delayed communication within a module or between different modules of the sensor system (e.g., between a sensor of the sensor system and its memory or its SoC hardware) or between the sensor system and a module external to the sensor system (e.g., a vehicle computer), abnormal termination of one or more applications of the sensor system (e.g., with respect to an image processing algorithm), or other operational disturbances in the sense defined above. In some cases, an operational disturbance of the sensor system may occur because image data (e.g., video data) is processed by the sensor system (e.g., in real time during image recording) containing features that caused one or more of the above-mentioned operational disturbances of the sensor system during the image processing. In the example of a vehicle-integrated sensor system, features on the image that are related to a street scene recorded thereon are conceivable, the road scene comprising one or more objects, such as traffic signs, lanes or other road or pedestrian zone markings, trees, buildings, traffic participants such as pedestrians, cyclists or other vehicles. In some cases, image processing of one or more of the above-mentioned features by a neural network, such as image classification such as semantic segmentation of the image (e.g., region-by-region and / or pixel-by-pixel classification of features of the image), may cause one or more of the operational disturbances listed above.

[0037] In the present disclosure, a sensor system (which, for example, includes one or more sensors, such as cameras, radars, lidars, ultrasonic or thermal sensors) can be operated using software that is designed to perform multiple tasks in a specific order (for example, regarding image processing, such as described above).

[0038] The first step of the method comprises generating 100 a Petri net 10, which contains information about the number of tasks of the software of the sensor system and their order. For example, the number of tasks may be five or more, ten or more, fifty or more, one hundred or more. The Petri net may include a plurality of transitions 12a-d, a plurality of places 14a-f, and a plurality of edges 16a-f, such as in Figures 2a to 2e Each task of the present technology is described by a corresponding transition 12a-d of the Petri net, and when one or more required conditions for the task are met (the term "required conditions" has been introduced above), the transition is executed by the software. Figure 2aIn the example of , a Petri net 10 with four transitions is shown, which are drawn as small boxes 12a-d in the figure. As already described above, each transition can be assigned to a specific task: for example, transition 12a can be assigned to the task "read in a new image", transition 12b can be assigned to the task "compute optical flow", and transition 12c can be assigned to the task "execute CNN".

[0039] In addition, in the present disclosure, executing the task is a necessary condition for executing at least one subsequent task. Figure 2a In the example of , tasks 12b and 12c can only be started before output 12a has been (completely) executed, and task 12d can only be started when both tasks 12b and 12c have been executed.

[0040] In the technology of the present disclosure, the Petri net includes one or more input places 14a-f, which contain information about the conditions required for performing a task, such as the task "reading in a new image" that may be associated with transition 12a, the task "calculating optical flow" associated with transition 12b, or another task. Here, one or more input places are connected to the transitions of the Petri net via one or more input edges 16a-f, respectively. In addition, an edge may connect just one place to one transition or one transition to one place, wherein each transition may be connected to one or more input places in a plurality of places via one or more input edges in a plurality of edges, respectively. Figure 2a and 2b In the example of , transition 12a is connected to a single input place 14a via an input edge 16a, while transition 12d is connected to three input places 14d, e, f via corresponding three input edges 16d, e, f. In some cases, one or more transitions may be connected to one or more output places of a plurality of places via one or more output edges of a plurality of edges, respectively (see also the definition further above). Furthermore, a place may be an output place for one transition while the same place is an input place for another transition. Return Figure 2a and 2b Example: three places 14d, e, f are input places for transition 12d, while two of the places 14d, e are output places for transition 12b, and one of the places 14f is an output place for transition 12c.

[0041] The next step of the method comprises initializing 200 the Petri net by initial marking in such a way that the corresponding conditions for executing at least one or more of the plurality of tasks from the software are set to be satisfied. It may be that the required conditions for the remaining tasks from the plurality of tasks are not (wholly or partially) satisfied and therefore these tasks cannot be executed (at least in the propagation step, if the task for which the required conditions are satisfied is executed, more on this further below). Here, each of the one or more input places describing the transition of the task may be filled with a token 18a-d if the required condition assigned to the input place is satisfied. In Figure 2a In FIG. 1 , an initialized Petri net 10 is shown by way of example with tokens which fill the input place 14a with tokens. In this example, the required conditions for the remaining tasks 12b-d are not met and therefore the corresponding input places 14b-f of the transitions 12b, c, d to which these remaining tasks are assigned are not filled with tokens. Figure 2a In the example of , is the initial marking of the Petri net. This initial marking can simulate the following situation: in which the image has, for example, reached the corresponding module of the sensor system and thus the required conditions for executing the task "read in new image" have been met. (As already explained above, for the purpose of generating a reachability graph, actual execution is not necessary.) In another example (not shown in the figure), another initial marking can be selected, in which, for example, the three input repositories 14d, e, f for the transition 12d are filled with tokens respectively, which will mean that the three required conditions for the task associated with the transition 12d are met.

[0042] In the technology of the present disclosure, initializing the Petri net may include filling each of the input places of one or more transitions describing at least one or more tasks with the required number of tokens that provide the conditions required for performing these tasks. If, for example, the task "read in two consecutive images" is assigned to transition 12a, the task is only executed when the input place 14a of the transition is filled with two tokens (for each image), which means that the required conditions for the task "read in two consecutive images" are met. In a non-limiting example, if the sensor system includes a camera, the initial marking of the Petri net can be created with the help of a "task planner" of the software, which can, for example, achieve the filling of the places with the desired number of tokens by specifying the corresponding delays for performing the tasks (for example, the task planner table can be created by the task planner).

[0043] In addition, the present technology includes propagating 300 along the Petri net starting from an initial marking by determining a task of the software, for which the required conditions are satisfied at each corresponding propagation step. In some cases, if all the required conditions for performing the task are satisfied, then one or more input edges, respectively, can control the switching of the transition. (This "satisfaction" is given by filling the input library with tokens, as discussed above.) In addition, each of the one or more input edges, respectively, of the transition can be described by a weight. If the number of tokens in each of the one or more input library places of the transition is at least equal to the weight of the corresponding input edge in the one or more input edges connecting the input library place to the transition, then the transition can be switchable.

[0044] This procedure can be found in Figures 2a to 2e As explained above, Figure 2a For example, an initial marking can make it possible to execute the task assigned to the transition 12a in the first propagation step. In this non-limiting case, the input edge 16a of this transition can be described by a weight of "one" and control the switching of the transition 12a, because the input library place 14a contains a "one" token. In another case, if for example the weight of the input edge 16a is equal to "two", the transition 12a with a single token 18a in the input library place 14a cannot lead to the switching of this transition 12a: in this case, the initial marking may not be applicable, because the further propagation along the Petri net is blocked (i.e. the further propagation leads to a "deadlock"). In order to avoid this problem, the input library place 14a can be occupied, for example, with two tokens. Alternatively, the transition 12a can be kept filled with one token 18a at the input library place 14a, but at least one of the subsequent library places 14b, c can be occupied with a token in addition. Thus, the propagation can continue along the Petri net and, on a second traversal, the place 14a can be filled with the second (missing) token (see also further discussion). This situation may occur in one of the above examples, where the transition 12a is assigned the task "read in two consecutive images". The weights of the input places can take the values ​​one, two, three, four, five or more.

[0045] In the technology of the present disclosure, each of the one or more output edges of a transition can be described by a weight. In addition, after switching a transition, one or more tokens can be removed from the multiple tokens of the input library of the transition, the number of which is equal to the weight of the corresponding input edge of the transition. In addition, one or more tokens can be added to the corresponding output library of the transition, the number of which is equal to the weight of the corresponding output edge of the transition.

[0046] Return to Figures 2a to 2eExample: After transition 12a has been switched (in other words, execution of the task associated with transition 12a has been released), token 18a may be removed from input place 14a (whose weight is exemplarily "one"). In a non-limiting case, the weights of the two outgoing edges 16b, 16c of transition 12a are equal to "one". Therefore, tokens may be added to each of the output places 14b, 14c, as in Figure 2b This distribution of tokens over the places (two tokens at places 14b, 14c and no tokens at the remaining places 14a, d, e, f) represents the Petri net notation for the second propagation step. If, for example, the input edges of transitions 12b and 12c ( Figure 2b In the example of FIG. 1 , if each transition 12b, 12c has one input edge with a weight of “one” (i.e. “all required conditions” for the tasks assigned to these transitions are satisfied in the above sense), then the transitions 12b and 12c may now be switchable. In one example, the transition 12b may be switched first, wherein as long as the weight of the output edge of the transition is, for example, equal to “one”, a token is removed from the input place 14b of the transition 12b and a token is added at each of the two output places 14d, e of the transition (see Figure 2c ).

[0047] In the next propagation step, since one of the three input places 14d, e, f is not occupied by a token, in order to Figure 2c The marking shown in can only switch transition 12c. The next propagation step after switching transition 12c can be Figure 2d (In this non-limiting example, the output edge of transition 12c has a weight equal to "one".) It should be noted that in another example, transitions 12b, c can be run in the opposite order, i.e. first transition 12c and then transition 12b (not shown in this figure). Now the last propagation step in the first cycle can be completed by the Petri net (the three input places 14d, e, f are filled with tokens respectively), because in this example the three input edges 16d, e, f of transition 12d have weights equal to "one". Finally, tokens can be removed from these places and tokens can be added at the output place 18a of transition 12d (in this example, the weight of the output edge of transition 12d is equal to "one"). The final marking of the Petri net after the first propagation cycle is consistent with its initial marking, as shown in Figure 2e As shown in (see Figure 2a ).

[0048] In this way, the second (or each subsequent) propagation cycle can also be executed along the Petri net (for example, the above-mentioned transitions 12b, c can be run in the opposite order in the second propagation cycle). It should be noted that, in another example, the input edge of the transition 12c can have a weight of "two" (not shown in the figure). In this case, since the input library place 14c can only be filled with one token in the first propagation cycle, the transition 12c cannot be switched at all during the first propagation cycle: only when the input library place 14c is occupied by the second token, for example, the transition 12c can be switched in the second propagation cycle along the Petri net (in other words: when traversing the Petri net for the second time). Similar to what has been described in conjunction with the transition 12a (in the example where the task "reading in two consecutive images" is assigned to the transition 12a), in this case, the initial marking for the Petri net can contain the token at the library place 14f (the output library place of the transition 12c) in order to ensure that the propagation along the Petri net is to the end without switching the transition 12c within the first propagation cycle. For example, if the transition 12c is assigned the task "compute optical flow for at least two images", a scenario may occur where two consecutive images are required to compute the optical flow, so that the task cannot be started after the first image has been received.

[0049] The next step of the method comprises generating 400 a reachability graph comprising a plurality of nodes 52a-e, wherein each node comprises information about the required conditions satisfied for performing a task for a corresponding propagation step along the Petri net and reflects the operating state of the sensor system (see also the definition further above). In the present technique, the information contained in the node comprises the required conditions satisfied for the corresponding propagation step (e.g. for the task in the Petri net). Figures 2a to 2e In addition, the information contained in the first and last nodes contains the initial marking of the Petri net (see e.g. Figure 2a and Figure 2e ). In addition, nodes in the reachability graph can be represented by multiple identifiers ID 1 -ID 6 , where each identifier is assigned to an input place or an output place of a corresponding transition and describes a plurality of tokens in the input place or the output place (as already explained above, an input place for one transition can in some cases be considered as an output place for another transition). As explained in detail above, the number of tokens can change during the propagation along the Petri net (see the combination Figures 2a to 2e for example).

[0050] The above method steps for generating a reachability graph can be performed with the aid of Figure 3 To illustrate, show that according to Figures 2a to 2eDuring the exemplary propagation along the Petri net in five steps, there are five labeled nodes (the fact that the first and fifth nodes contain the initial label is indicated by the arrow connecting the fourth and fifth nodes to each other). 1 -ID 6 exist Figures 2a to 2e and Figure 3 The first node 52a is shown in FIG. 1 by a plurality of indicators (1, 0, 0, 0, 0, 0) for example (there is a token at the library 18a, and there is no token at the other library, see Figure 2a ), and the third node 52c is given by multiple indicators (0, 0, 1, 1, 1, 0) (there are tokens at three library places 18b, c, d, and no tokens at the remaining library places, see Figure 2c ).

[0051] As already described above, the operating state of the sensor system before executing a task of the plurality of tasks may be different from another operating state of the sensor system after executing the task. It can be said that the nodes reflect the operating state of the sensor system, because, for example, the node 52b contains the marking (0, 1, 1, 0, 0, 0) of the Petri net for the propagation step before executing the corresponding task 12b, wherein the subsequent node 52c contains another marking (0, 0, 1, 1, 1, 0) of the Petri net for the next propagation step after the already executed task 12b.

[0052] In the disclosed techniques, a reachability graph may include multiple paths, where each path contains a set of nodes that appear in a particular order during propagation along the Petri net. In some cases, a portion of a path may overlap with a portion of another path. Figure 3 In the example in Figure 1, such a path in the reachability graph is shown by solid boxes (“nodes”) and solid arrows connecting these boxes (“sequences”). This path accurately describes the propagation during the first traversal along the Petri net, which has been combined with Figures 2a to 2e The nodes (0, 1, 0, 0, 0, 1) do not appear in this particular order, because they correspond to the reverse switching sequence of transitions 12b, c, i.e. if transition 12c is switched first and then transition 12b (see Figures 2b to 2d Therefore, for this exemplary Petri net 10, there may still be a second path, which belongs to the reachability graph and contains the node (0, 1, 0, 0, 0, 1) instead of the node (0, 0, 1, 1, 1, 0) in its node group (see Figure 3 The first and second paths partially overlap, and the non-overlapping portions of the two paths are shown in the example Figure 3As already mentioned above, the second path of the reachability graph may occur in the second (or another other) traversal of the Petri net.

[0053] In the present technology, the reachability graph may include five or more, ten or more, twenty or more, thirty or more, fifty or more, one hundred or more nodes. In some cases, to generate the reachability graph, one or more, two or more, three or more, five or more, ten or more, fifty or more, one hundred or more traversals (in other words: propagation loops) may be performed along the Petri net.

[0054] In the technology of the present disclosure, in some cases, multiple reachability graphs (two or more, three or more, ten or more) can be generated in the same manner using corresponding initial identifications (for example, one reachability graph for each initial identification).

[0055] A second general aspect of the present disclosure relates to a method (e.g., a computer-implemented method) for identifying operational disturbances of a sensor system using a reachability graph. The method includes receiving 500 a reachability graph for identifying operational disturbances of the sensor system generated according to the first aspect. Furthermore, the method includes processing 600 a validation dataset (the definition of the validation dataset has been further described above) by means of the received reachability graph.

[0056] In the disclosed techniques, processing 600 a received validation data set through a reachability graph may include identifying a plurality of reached nodes (in the sense discussed above) in a reachability graph with respect to the validation data set reflecting a plurality of covered operating states of the sensor system. To this end, the method step "processing" may include identifying a plurality of reached paths in a reachability graph with respect to the validation data set.

[0057] Returning to the non-limiting example of a sensor system comprising a camera (e.g. a vehicle-integrated camera such as a third generation multi-function camera or, for short, an MPC3 camera): the software running the sensor system can, for example, collect information during video recording via the camera (e.g. in the case of a moving vehicle), which information can be useful for identifying the nodes reached. In practice, this can occur, for example, in such a way that each of the tasks performed (e.g. in conjunction with image processing) sends a signal at the end in the sensor system. This signal can contain information (e.g. an ID), for example in a universal asynchronous receiver / transmitter (UART) debug interface of the sensor system, which information clearly identifies the task (e.g. the task "read in a new image", "execute a CNN", or another task). This information can then be used to identify the corresponding switching sequence in the Petri net, the node in the reachability graph, the path in the reachability graph, or any combination thereof. In some cases, this information can constitute part of the validation data set.

[0058] In the present technology, it can be checked 700 whether the number of nodes reached meets a predetermined coverage criterion. In addition, the method may include classifying 800 the verification data set as a data set that does not identify operational interference with the sensor system (in other words, operational interference may potentially occur) if the number of nodes reached meets the predetermined coverage criterion. Otherwise, if the number of nodes reached does not meet the predetermined coverage criterion, the method may include classifying the verification data set as a data set that identifies operational interference with the sensor system (e.g., operational interference that may potentially occur). In one example, the predetermined coverage criterion may include: the number of nodes reached falls below a predefined threshold (e.g., if the number of nodes reached of all nodes included in the reachability graph falls below 90% or less, 80% or less, or 50% or less).

[0059] This scene can be seen from Figure 4 This is apparent from the example of which schematically shows an exemplary number of reached nodes 62, 70 of a reachability graph as a function of time 61 (time interval 71) for verifying a data set. In this case, time can mean a number of verification hours, for example in the form of consumed reproductions of the video data included in the verification data set. Figure 4In a non-limiting example, the number of nodes reached at a time point 63 (e.g. 100 hours) corresponds to a value 64 (e.g. 30%), which falls below a predefined threshold (the threshold is, for example, 90%). In this case, the validation dataset (more precisely, the part of it that has been processed until the time point 63) can be classified as a dataset in which no operational disturbances of the sensor system (e.g. operational disturbances that could potentially occur) were identified. In other words, the validation dataset does not contain enough data in the sense that the resulting coverage of the reachability graph is too low (i.e. the number of reached nodes is below a threshold). In this case, one or more paths of the reachability graph that were not determined when processing the validation dataset (but which are contained in the reachability graph) may result in operational disturbances of the sensor system (see the above discussion on this). On the other hand, in the same Figure 3 , the number of nodes that arrived at the time point 65 (e.g. 1000 hours) corresponds to the value 65 (e.g. 92%), which exceeds a predefined threshold value (e.g. 90%). The validation data set (which has been processed up to the time point 65, i.e. over the entire time interval 71) can now be classified as a data set for identifying operational disturbances of the sensor system.

[0060] Furthermore, the method may include enhancing the validation dataset by adding another validation dataset. Returning to the example of a sensor system for use in a vehicle (e.g. for use in a driver assistance system, partial or autonomous driving): the validation dataset may in some cases include video data detected by the sensor system of the moving vehicle in different regions. Figure 4 An exemplary number of reached nodes 62, 70 of a reachability graph as a function of time 61 (time interval 71) for a validation dataset is discussed. Video data for new regions can now be collected in the same way, which may be understood as "other validation datasets" in the following.

[0061] A next step of the method may include processing the enhanced validation data set through the received reachability graph. In the present technology, processing the enhanced validation data set through the received reachability graph may include identifying a plurality of reached nodes in the reachability graph with respect to the enhanced validation data set (e.g., in the same manner as already discussed in connection with the validation data set). To this end, the method step "processing" may include identifying a plurality of reached paths in the reachability graph with respect to the enhanced validation data set. Figure 4 Schematically shown in FIG. 1 is an exemplary number of reached nodes 62, 70 of a reachability graph as a function of time 61 (time intervals 71 and 72) for an enhanced validation dataset. In addition to the curve for the validation dataset, the curve 70 also contains a continuation of the time interval 72 (the continuation corresponds to another added validation dataset).

[0062] In addition, the present technique may include comparing the number of nodes about the validation data set and the number of nodes about the enhanced validation data set. For this purpose, for example, the number of nodes reached at the end of the corresponding time interval may be compared (in Figure 4 In the example of , the time point 65 at the end of the time interval 71 and the time point 67 at the end of the time intervals 71 and 72). In addition, it can be checked whether the comparison result meets a predetermined criterion. In addition, the method can include classifying the enhanced verification data set as a data set that does not identify additional operational disturbances of the sensor system (for example, operational disturbances that can be identified when using the verification data set) if the comparison result meets the predetermined criterion. Otherwise, if the comparison result does not meet the predetermined criterion, the method can include classifying the enhanced verification data set as a data set that identifies additional operational disturbances of the sensor system (for example, in addition to operational disturbances that can be identified when using the verification data set). In this case, one or more paths of the reachability graph that were not determined when processing the enhanced verification data set (but the paths are included in the reachability graph) may lead to operational disturbances of the sensor system.

[0063] In some cases, the method step "comparing" may include calculating a ratio between the number of validation data sets and the number of enhanced validation data sets, wherein the comparison result includes the ratio. For example, the predetermined criterion may include the ratio falling below a predefined threshold (e.g., 0.95 or less, 0.9 or less, or 0.8 or less). Figure 4 In a non-limiting example of , the number of nodes reached in the reachability graph for the validation dataset and the enhanced validation dataset is equal (the value of which is indicated in the figure by reference numeral 66). Therefore, the ratio has a value of "one". In this case, the enhanced validation dataset can be classified as a dataset for which no additional operational disturbances of the sensor system were identified (in other words: no additional operational disturbances have been analyzed based on the validation dataset). This can be interpreted as adding other validation datasets to the validation dataset (e.g. video data for a new region in the above example) without causing Figure 4 The new reached node in the reachability graph in the example of : Therefore, disturbance of the operation of the sensor system is not expected.

[0064] A third general aspect of the present disclosure relates to a module, which is designed to perform the method for generating a reachability graph for identifying operational disturbances of a sensor system according to the first general aspect of the present disclosure. In one example, the module can be a module external to the sensor system, i.e., the module is located outside the sensor system (e.g., a module of a computing system). The software of the first aspect can be installed on the module so that the reachability graph of the first aspect can be generated without, for example, having to put the sensor system into operation.

[0065] A fourth general aspect of the present disclosure relates to a sensor system designed for generating a validation data set (eg as already described above for the example with a camera), the validation data set being suitable for being processed in a method according to the second aspect.

[0066] In one case, the sensor system may be designed to send the generated validation data set to a module (eg, to a module outside the sensor system, such as a computing system or a data cloud system).

[0067] In another case, the sensor system of the fourth aspect can be designed to receive a reachability graph from a module according to the third aspect. (For example, the sensor system can include a reachability graph in compiled form.) To this end, the sensor system can be designed to process the generated verification data set through the reachability graph (for example, if the sensor system is in operation, the verification data set can be generated and processed in real time). To this end, the sensor system can be designed to process the generated verification data set through the reachability graph so as to identify multiple reached paths in the reachability graph with respect to the verification data set. In addition, the sensor system can be designed to receive a known number of reached paths according to the second aspect. The known number of reached paths can, for example, be the number of reached paths in the reachability graph with respect to another verification data set (for example, a verification data set that has been generated by the sensor system of the fourth aspect at an earlier point in time or by another sensor system).

[0068] To this end, the sensor system of the fourth aspect can be designed to verify whether the known number of reached paths includes each path of the number of reached paths. In addition, the sensor system can be designed to determine a plurality of unknown paths if at least one of the plurality of reached paths is not included in the known number of reached paths. In some cases, the sensor system can further be designed to send the plurality of unknown paths to a module (e.g., to a module outside the sensor system, such as a module of a computing system or a data cloud system). Alternatively or additionally, the sensor system can further be designed to deactivate communication with a device that uses information from the sensor system in order to provide one or more functions of the device (e.g., the functions further described above when the sensor system is used in a vehicle) if the number of unknown paths exceeds a predefined threshold (e.g., one or more, two or more, five or more, ten or more, fifty or more unknown paths).

[0069] A fifth general aspect of the present disclosure relates to a module designed to receive a reachability graph from a module according to the third aspect. (The module of the fifth aspect may, for example, include a reachability graph in compiled form.) In addition, the module is designed to receive a verification data set from a sensor system according to the fourth aspect and to perform operations according to the second aspect for identifying operational disturbances of the sensor system using the reachability graph. In one example, the module may be a module external to the sensor system, i.e., the module is located outside the sensor system (e.g., a module of a computing system). In one case, the module of the third aspect and the module of the fifth aspect may be modules of independent systems. In another case, the module of the third aspect may be a module of a distributed system. In the last case, the modules may communicate, for example, via a network (e.g., the Internet).

[0070] A sixth general aspect of the present disclosure also relates to a computer program, the computer program being designed to perform the method of the present disclosure. The present disclosure also relates to a computer-readable medium (e.g., a machine-readable storage medium, such as an optical storage medium or a read-only memory, such as a flash memory) and a signal, the signal storing or encoding the computer program of the present disclosure.

Claims

1. A method for generating a reachability diagram (50) for identifying operational disturbances of a sensor system, wherein the sensor system is capable of being operated using software designed to perform multiple tasks in a specific sequence, The method comprises: generating (100) a Petri net (10) containing information about the number of tasks of the software of the sensor system and their sequence, wherein each task is described by a corresponding transition (12a-d) of said Petri net, said transition being executed by software if one or more required conditions for the task are met, and wherein executing a task is at least one required condition for executing a subsequent task, wherein the Petri net comprises one or more input places (14a-f), the input places containing information about conditions required for performing a task, wherein the one or more input places are connected to transitions of the Petri net via one or more input edges (16a-f) respectively; Initializing (200) the Petri net by initializing marking in such a way that corresponding conditions for executing at least one or more tasks from a plurality of tasks from the software are set to be satisfied; Propagation is performed along the Petri net starting from the initial marking by determining the tasks of the software for which the required conditions are satisfied for each corresponding propagation step (300), generating (400) a reachability graph, the reachability graph comprising a plurality of nodes (52a-e), wherein each node comprises information about a required condition satisfied for executing a task for a corresponding propagation step along the Petri net and reflects an operating state of the sensor system, The information contained in the nodes contains the label of the Petri net for the corresponding propagation step, and the information contained in the first and last nodes includes the initial label of the Petri net.

2. The method according to claim 1, wherein the Petri net comprises a plurality of transitions (12a-d), a plurality of places (14a-f) and a plurality of edges (16a-f), One of the edges connects exactly one place to one transition or one transition to one place. Each transition is connected to one or more input places in the plurality of places through one or more input edges in the plurality of edges, wherein each of one or more input places describing the transition of the task is filled with a token (18a-d) if the required conditions assigned to the input place are met, Wherein if all the required conditions for executing the task are met, one or more input edges control the switching of the transition, and One or more transitions are connected to one or more output places in the plurality of places through one or more start edges in the plurality of edges, respectively. Optionally one of the places is an output place for one transition, and the same place is an input place for another transition.

3. The method of claim 2, wherein initializing the Petri net comprises filling each input place describing one or more transitions of the at least one or more tasks with a required number of tokens that provide conditions required to be met for performing these tasks.

4. The method according to claim 2 or 3, wherein each of the one or more input edges of the transition is described by a weight, A transition is switchable if the number of tokens in each of one or more input places of the transition is at least equal to the weight of the corresponding input edge in one or more input edges connecting the input place to the transition.

5. The method according to claim 4, wherein each of the one or more starting edges of the transition is described by a weight, wherein after the switching transition, one or more tokens are removed from a plurality of tokens in the input place of the transition, the number of tokens being equal to the weight of the corresponding input edge of the transition, And wherein one or more tokens are added to the corresponding output library of the transition, and the number of the tokens is equal to the weight of the corresponding output edge of the transition.

6. The method according to any one of claims 2 to 5, wherein the nodes of the reachability graph are described by a plurality of identifiers (ID1-ID6), wherein each identifier is assigned to an input place or an output place of a corresponding transition and describes a plurality of tokens in the input place or the output place, The number of tokens changes during propagation along the Petri net.

7. The method according to any one of claims 1 to 6, wherein the reachability graph comprises a plurality of paths, wherein each path comprises a node group of a plurality of nodes that appear in a specific order during propagation along the Petri net, Optionally a portion of one of the paths overlaps a portion of the other path.

8. A method for identifying operational disturbances of a sensor system using a reachability diagram: Receiving (500) a reachability map for identifying operational disturbances of a sensor system generated according to the method of any one of the preceding claims 1 to 7; and The validation data set is processed (600) through the received reachability graph.

9. The method of claim 8, wherein processing (600) a validation data set through the received reachability graph comprises identifying a plurality of reached nodes in the reachability graph with respect to the validation data set reflecting a plurality of covered operating states of the sensor system.

10. The method according to claim 9, further comprising: Checking (700) whether the number of nodes reached satisfies a predetermined coverage criterion; If the plurality of reached nodes satisfy a predetermined coverage criterion, the validation set is classified (800) as a data set in which no operational interference of the sensor system is identified, And otherwise if the number of reached nodes does not satisfy the predetermined coverage criterion, the validation set is classified as a data set for identifying operational disturbances of the sensor system, Optionally, the predetermined coverage criterion includes the number of reached nodes falling below a predefined threshold.

11. The method according to any one of claims 8 to 10, wherein processing (600) a validation data set through the received reachability graph comprises identifying a plurality of reached paths in the reachability graph with respect to the validation data set generated by the method according to claim 7. 12 . A module designed to carry out the method according to claim 1 for generating a reachability diagram for identifying operational disturbances of a sensor system.

13. A sensor system designed to generate a validation data set suitable for being processed in a method according to any of the preceding claims 8 to 11.

14. The sensor system according to claim 13, wherein the sensor system is designed to: receiving a reachability graph from a module according to claim 12; and The resulting validation dataset is processed through the reachability graph.

15. A module, the module being designed to: receiving a reachability graph from a module according to claim 12; receiving a validation data set from a sensor system according to claim 13; and A method for detecting operational disturbances of a sensor system using a reachability diagram is carried out according to any of the preceding claims 8 to 11 .