Method and system for processing multimodal input signals

Abstract

Method in a fusion unit (26) of a system (40) for processing multimodal input signals, comprising the steps: - Receiving (S5) instruction messages, each indicating an instruction that can be executed by an application (28; 30; 32) that can be coupled to the fusion unit (26); - for at least two of the instruction messages: creating (S6) a candidate object for the instruction message, wherein the candidate object designates the instruction and includes a validity time specification of the candidate object, which specifies a validity time within which an instruction designated by the candidate object can compete with other instructions designated by other candidate objects, - Merging (S10) two of the candidate objects if the validity period information of these candidate objects specifies overlapping validity periods, and - Forwarding (S12) a statement to the application, which statement is designated by a candidate object resulting from the merging step.

Description

[0001] The present invention relates to a method and a system for processing multimodal input signals.

[0002] Multimodal human-machine interfaces (“MMI”) allow a user to input instructions to an application in various ways via specific, predefined input signals, for example via contact-based input using a keyboard, mouse, touchpad or the like, via voice input, via gestures of the body, hand or individual fingers of a hand or via captured eye movement (so-called “eye tracking”).

[0003] Each instruction can be assigned a predefined input signal for each modality. For example, an instruction to "turn the page" in a document or menu, which can be used to operate an application for reading an e-book or a vehicle assistance system, can be assigned the voice input signal "Next" with regard to voice input, while with regard to gesture input, the instruction can be linked to a swipe movement in a predefined direction.

[0004] Using such multimodal human-machine interfaces, various computer-based systems can be operated or controlled, for example, a driver assistance system in a motor vehicle.

[0005] Input signals from different modalities can be entered sequentially and processed into a single piece of information. This is called alternating multimodality. More interesting—and technically more challenging—is processing input signals from different modalities into a single piece of information when they are entered in parallel or simultaneously, for example, a gesture accompanied by speech input. This is called synergistic multimodality.

[0006] US Patent 2007 / 0260972A1 describes a method and system for accepting multimodal inputs and deriving synchronized and processed information. A reusable multimodal application is deployed on the mobile device. A user transmits a multimodal command to the multimodal platform via the mobile network. The input one or more communication modes are transmitted via the mobile network to the multimodal platform(s) and subsequently processed on the multimodal platform. The synchronized and processed information is then transmitted back to the multimodal application.

[0007] DE 101 33 945 A1 describes a method and a device for exchanging and jointly processing object data between sensors and a processing unit, wherein position information and / or speed information and / or other object attributes (size, identification, marks) of sensor objects and fusion objects are transmitted and processed.

[0008] The invention is based on the objective of proposing a method and a system which allow input signals entered via different modalities to be processed reliably and with high robustness, even in parallel or simultaneously.

[0009] This task is solved by a method and a fusion unit with the features of independent claims.

[0010] A preferred embodiment of the invention comprises a method in a fusion unit of a system for processing multimodal input signals.

[0011] The fusion unit receives instruction messages from at least one sensor data evaluation unit. Each instruction message represents an instruction that can be executed by an application connected to the fusion unit. The fusion unit then creates a candidate object for each of at least two of the instruction messages. Each candidate object represents the instruction specified in the instruction message and includes a validity time specification for the candidate object. The fusion unit merges two of the candidate objects if their validity times overlap and forwards the resulting instruction to the application, which is identified by a candidate object created during the merging process.

[0012] In the context of the present invention, the term "merging two candidate objects" is to be interpreted broadly such that this step does not necessarily have to generate a new candidate object resulting from the two merged candidate objects. This is merely one option among others. According to a second option, the two candidate objects may emerge unchanged from the merging step. This is the case if the instructions designated by the respective candidate objects do not conflict. The concept of conflicting instructions is explained in detail below. Finally, according to a third option, it is possible that one or both candidate objects are deleted in the merging step. In other words, it is possible that no candidate object results from a merging step.

[0013] The validity period of a candidate object generally specifies a period of validity during which an instruction designated by the candidate object may compete with other instructions designated by other candidate objects. The validity period can specify one or more points in time, one or more time intervals, or a combination of points in time and time intervals. A first validity period specified by a validity period of a first candidate object overlaps, within the meaning of the present invention, with a second validity period specified by a validity period of a second candidate object if the first validity period and the second validity period overlap in time.A temporal overlap occurs when an initial period of validity encompasses a single point in time, and a second period of validity encompasses a second point in time that is identical to the first. A temporal overlap also occurs when an initial period of validity encompasses a single point in time, and a second period of validity encompasses a time interval, and the point in time lies within that interval. A temporal overlap also occurs when an initial period of validity encompasses a single time interval, and a second period of validity encompasses a second time interval, and the first time interval shares at least one point in time with the second time interval. As a rule, the period of validity of a candidate object will encompass at least one time interval. For the sake of readability, the following text will often refer to a single period of validity for a candidate object.Such a period of validity is to be understood within the meaning of the present invention as the period of validity of the candidate object specified by the validity period of the candidate object.

[0014] A further preferred embodiment of the invention relates to a fusion unit for a system for processing multimodal input signals. The fusion unit is configured to receive instruction messages from at least one sensor data evaluation unit, each of which designates an instruction that can be executed by an application that can be coupled to the fusion unit. Furthermore, the fusion unit is configured to generate a candidate object of the type described above for at least two of the instruction messages, each object designating the respective instruction and including a validity period. Finally, the fusion unit is configured to merge two of the candidate objects if the validity periods of these candidate objects overlap, and to forward an instruction to the application, which instruction is designated by a candidate object resulting from the merging step.

[0015] Creating a candidate object for a received instruction message makes it possible to compare instruction messages relating to input signals acquired in parallel or simultaneously. Competing or contradictory instructions can be identified and, if necessary, deleted. Compatible instructions acquired via different modalities can be combined and processed as a single instruction. In summary, the concept of introducing a candidate object allows instruction messages originating from input signals acquired simultaneously or in parallel to be appropriately merged. This results in robust input signal recognition and a high recognition rate, as described in detail below.

[0016] Further preferred embodiments of the invention result from the other features mentioned in the dependent claims.

[0017] As mentioned, the fusion unit receives multiple instruction messages, specifically a first instruction message and a second instruction message. The second instruction message differs from the first instruction message. A second sensor data evaluation unit, through which the second instruction message was received, can either be the same as the first sensor data evaluation unit through which the first instruction message was received, or it can be different from it.

[0018] As described above in general terms for an instruction message, for example the first instruction message, the fusion unit also generates a second candidate object for the second instruction message, where the second candidate object denotes the second instruction and includes a validity time specification for the second candidate object. The first candidate object and the second candidate object can each additionally include a weighting of the first and second candidate objects, respectively.

[0019] Specifically, the fusion unit can now merge the first candidate object with the second candidate object depending on the weight of the first candidate object and the weight of the second candidate object, if the validity time specification of the first candidate object specifies a first validity time that overlaps with a second validity time specified by the validity time specification of the second candidate object, and if the first statement conflicts with the second statement.

[0020] In general, a first instruction competes with a second instruction within the meaning of the present invention if the first instruction and the second instruction belong to the same instruction class. Instructions of the same instruction class relate to a common technical aspect with respect to an application configured to execute the instructions of that instruction class. Instructions of one instruction class may, for example, be those that control the movement of an input element, such as a cursor ("up," "down," "left," "right"). Another instruction class may, for example, include instructions for navigating a document, a menu, or the like ("page forward or backward"). A third instruction class may, for example, include instructions for actuating an input element ("click," "double-click," sustained "click" (with pressure or duration)).

[0021] As explained below, conflicting instructions can be compatible. This applies when the instructions are essentially identical and can be interpreted as a single, unified instruction. Conversely, conflicting instructions can also be in conflict. This occurs when the instructions are so different that they are incompatible.

[0022] According to one variant, in the step of merging the first candidate object with the second candidate object, the first candidate object, the second candidate object, or both candidate objects can be deleted if the first statement conflicts with the second statement. The candidate object that remains, if any, is then retained unchanged from the merge step.

[0023] In general, for the purposes of the present invention, a first instruction conflicts with a second instruction if the first instruction and the second instruction are incompatible or contradictory. With regard to the above-mentioned class of instructions that involve navigation within a document or menu, the instructions "page forward" and "page back" conflict with each other because they are contradictory and incompatible. Similarly, for example, an instruction to move a cursor "to the right" conflicts with an instruction to move a cursor "to the left".

[0024] A deleted candidate object has no influence on the further procedure; in particular, a deleted candidate object does not affect any of the other candidate objects, neither with regard to a designated instruction nor with regard to a validity period or a weighting.

[0025] In the event of a conflict, the merger process typically deletes at least the candidate object with a lower weight than the other candidate object. However, if, for example, the combined weight of both candidate objects does not exceed a predefined value, it may be necessary to delete both candidate objects.

[0026] According to a second variant, in the merging step, a merged candidate object is formed from the first and second candidate objects if the first and second instructions are compatible. This merged candidate object, resulting from the merging step, then replaces the first and second candidate objects, which can each be considered deleted. Unlike the first variant, however, the first and second candidate objects persist within the merged candidate object and therefore influence the subsequent process.

[0027] In general, for the purposes of the present invention, a first instruction and a second instruction are compatible if the first instruction is identical to the second instruction. A first instruction and a second instruction are also considered compatible if the first instruction is specified by one or more first instruction parameters and the second instruction is specified by one or more second instruction parameters, and the respective first and second instruction parameters lie within a predetermined compatibility tolerance interval.An instruction to actuate an input device at specified X and Y coordinates (such as "Click (x=1000, y=500)") can, for example, be compatible with a corresponding instruction (such as "Click (x=992, y=507)"), provided that the X and Y coordinates, which the respective instructions specify as instruction parameters, are sufficiently close to each other, i.e., within the specified compatibility tolerance interval, which, for example, can have a value of 10 in both the X and Y directions.

[0028] The statement then designated by the merged candidate object is derived from the first statement, or from the second statement, or from both the first and second statements. In particular, the statement designated by the merged candidate object can correspond to the first statement if the first statement corresponds to the second statement.

[0029] In the case where the first instruction is specified by one or more first instruction parameters and the second instruction is specified by one or more second instruction parameters, and where the respective first and second instruction parameters lie within a specified similarity tolerance interval, which is usually smaller than the aforementioned compatibility tolerance interval (and, with reference to the previous example, only allows a deviation of 5 instead of 10), the instruction designated by the merged candidate object can correspond to either the first instruction or the second instruction.

[0030] In the event that the second instruction does not substantially correspond to the first instruction—that is, the respective instruction parameters lie within the specified compatibility tolerance interval but not within the specified similarity tolerance interval—the instruction designated by the merged candidate object can be specified by one or more third instruction parameters. These third parameters must lie within the corresponding compatibility tolerance interval with respect to the first and second instruction parameters. Such a third instruction parameter could, for example, be formed by taking an approximate average value between the corresponding first and second instruction parameters (referring to the previous example, "Click (x=996, y=503)").

[0031] The validity period of the merged candidate object is derived from the validity period of the first candidate object, or from the validity period of the second candidate object, or from the validity period of both the first and second candidate objects. As mentioned previously, the validity period of a candidate object generally specifies a time within which a statement designated by the candidate object can compete with other statements. Consequently, the validity period of a candidate object can be determined specifically for each candidate object separately, depending on various parameters.

[0032] Preferably, the validity period of a candidate object, specified by the validity time, is generated depending on the designated instruction and the instruction message for which the candidate object was created. Instructions of different instruction classes generally require different validity periods. For example, instructions for moving an input element, such as a cursor, are transmitted to the fusion unit at very short intervals. Consequently, the corresponding validity periods of these instructions are very short compared to, for example, instructions for navigating within a document, such as turning pages.In other words, the validity period of a candidate object can be generated depending on the modality to which a first sensor device is assigned. This device has acquired sensor data, based on which the first sensor data evaluation device generated the first instruction message. Put another way, depending on whether the first instruction message was entered via voice input, gesture input, user pupil measurement, or touchpad input, the determination of the validity period of the corresponding candidate object generated for that instruction message can vary.

[0033] The validity period of the merged candidate object can, according to one variant, correspond to the validity period of the first candidate object. According to a second variant, the validity period of the merged candidate object can correspond to the validity period of the second candidate object. According to a third variant, the validity period of the merged candidate object can correspond to a validity period encompassed by the intersection of the validity periods of the first and second candidate objects. Finally, according to a fourth variant, the validity period of the merged candidate object can correspond to a validity period encompassed by the union of the validity periods of the first and second candidate objects.

[0034] The weight of the merged candidate object is derived from the weight of the first candidate object, or from the weight of the second candidate object, or from the weights of both the first and second candidate objects. A candidate object's weight generally indicates its importance compared to other candidate objects. A higher weight emphasizes a candidate object's priority over others with correspondingly lower weights. It is understood that, for example, the weight of the first candidate object can be generated based on the first instruction message that identifies the first candidate object. In this way, the priority of a statement over other statements can be expressed in the candidate object's weight.As already described with reference to the validity period, the weighting of the first candidate object can also be generated depending on a modality via which the first instruction message, which designates the first candidate object, was captured.

[0035] The weight of the merged candidate object will generally exceed the weight of both the first and second candidate objects. To determine the weight of the merged candidate object, the corresponding weights of the first and second candidate objects can be summed appropriately or combined mathematically in another suitable manner.

[0036] According to the third variant, briefly explained above, if the instruction designated by the first candidate object does not conflict with the instruction designated by the second candidate object, both candidate objects emerge unchanged from the merger step. The fact that the term "merge" is used here again is primarily due to the simplification of the merger process, as only the validity periods of the respective candidate objects need to be considered when selecting a pair of candidates for merger.

[0037] An instruction message received by the fusion unit from a sensor data evaluation device can include additional parameters besides the instruction name.

[0038] According to one variant, the instruction message can include a sensor device parameter, which identifies a sensor device that provided sensor data on the basis of which the sensor data evaluation device generated the instruction message. In other words, the parameter can, for example, indicate that the instruction message was recognized based on image data that was captured and provided by a camera device.

[0039] According to a second variant, the instruction message can include an application parameter. Such an application parameter specifies one or more applications that are configured or intended to execute the instruction. For example, an application configured to execute instructions for navigating forward and backward in a document or menu could be part of a vehicle assistance system.

[0040] According to a third variant, the instruction message can include a acquisition time parameter. Such a parameter can specify an acquisition time interval within which sensor data has been acquired by a sensor device, on the basis of which the corresponding sensor data evaluation device has generated the instruction message.

[0041] According to a fourth variant, the instruction message can include a detection time parameter, which specifies a detection time at which the sensor data evaluation device has recognized the instruction specified in the instruction message from sensor data received from a sensor device.

[0042] It is understood that the validity period of a candidate object is preferably determined depending on the acquisition time parameter or the recognition time parameter or depending on both of these parameters.

[0043] According to a fifth variant, the instruction message can include a confidence parameter. Such a confidence parameter specifies how confidently the sensor data evaluation unit has recognized the instruction specified in the instruction message from the sensor data received by the sensor device. Using the confidence parameter, any uncertainties regarding the recognition of the instruction can thus be quantified and incorporated into the fusion process. Typically, such a confidence parameter will take a value between 0 and 1 or between 0 and 100, with a value at the upper end of such a range indicating a high degree of confidence in the recognition of the instruction.

[0044] It goes without saying that the weighting of a candidate object can also be determined depending on the corresponding confidence parameter of the instruction message. As a rule, the confidence parameter will be incorporated into the determination of the candidate object's weighting in a predefined manner.

[0045] It goes without saying that the variants described above can be combined in any way.

[0046] As mentioned above, a first sensor device that has acquired sensor data, based on which a first sensor data processing device has generated a first instruction message, can be assigned to a first modality that differs from a second modality assigned to a second sensor device that has acquired sensor data, based on which a second sensor data processing device has generated a second instruction message. In simpler terms, this means that instruction messages acquired and recognized based on different modalities can be processed. Examples of such modalities include speech recognition, gesture recognition, gaze direction detection, mechanical input via a keyboard, mouse, or touchpad, and the like.

[0047] This, in combination with the fact that a first candidate object is merged with a second candidate object when their respective validity periods overlap, implies that the procedure described here is suitable and designed to support synergistic multimodality, i.e., a situation in which input signals of different modalities are entered in parallel or simultaneously and processed into a single piece of information.

[0048] The procedure typically includes an additional step for each candidate object that has not been deleted by the end of its validity period. According to this step, the instruction designated by the candidate object is forwarded to an application coupled with the merger unit that is suitable or intended to execute the instruction. A candidate object that has not been deleted by the end of its validity period has thus prevailed over other, competing, or even conflicting candidate objects. This could be a candidate object originally created for a received instruction message or a merged candidate object, meaning a newly created candidate object as part of the merger process.

[0049] According to a preferred embodiment of the method described above, the first candidate object, or more generally, any candidate object generated by the fusion unit, can include a fusion parameter. Such a fusion parameter specifies whether, and if so, at what point in time, the first candidate object is intended for potential fusion with another candidate object.

[0050] According to one variant, the fusion parameter can specify that the first instruction designated by the first candidate object is immediately forwarded to an application coupled with the fusion unit that is suitable or intended to execute the first instruction. In other words, the instruction designated by the candidate object is forwarded before the candidate object is potentially subjected to a fusion process described above. This can be useful for instructions that are very time-critical or of very high priority.

[0051] The merge parameter can further specify that, after the first instruction designated by the first candidate object is forwarded, the first candidate object is intended for potential merger with further candidate objects that designate a conflicting instruction. In other words, a merger of the candidate object with candidate objects designating compatible instructions is no longer intended. Furthermore, in this case, the merge parameter will preferably specify that the application designated by the first candidate object will no longer be forwarded if the first candidate object has not been deleted by the end of its validity period.

[0052] Alternatively, the merge parameter can also specify that the first candidate object is not intended for potential merging with further candidate objects after the first instruction designated by the first candidate object has been forwarded. In other words, the candidate object is completely excluded from the merge process.

[0053] According to a second variant, the merge parameter can specify that the first instruction designated by the first candidate object is forwarded, after the validity period of the first candidate object has expired, to an application coupled with the merge unit that is suitable or intended to execute the first instruction. This can also apply if the first candidate object is considered deleted after the validity period has expired. According to this variant, only those candidate objects whose instructions conflict with the first instruction are considered for potential merger with the first candidate object.

[0054] In general, communication between a sensor device and a sensor data processing unit, for transmitting sensor data acquired by the sensor device, or communication between a sensor data processing unit and the fusion unit, for transmitting an instruction message, or communication between the fusion unit and an application coupled to the fusion unit, for forwarding an instruction, can be easily accomplished via sockets, preferably TCP sockets. This allows additional sensor units and / or applications to be easily integrated into the system. Other suitable communication methods and protocols are also usable.

[0055] According to a preferred approach, an instruction message is encoded in JSON data format. This standardized data format offers an easily understandable, readable, and readily processable data format that also easily accommodates a varying number of parameters.

[0056] The procedure described above comprises the following steps before the first instruction message is received: An input signal specified by a user of the system is captured by a first sensor device in the form of sensor data. The received sensor data is forwarded to a first sensor data evaluation unit, which recognizes an instruction for an application that can be coupled with the fusion unit from the sensor data. The sensor data evaluation unit then generates a corresponding instruction message and forwards this instruction message to the fusion unit.

[0057] The corresponding steps are usually repeated many times as a result of the user entering further input signals, preferably also using different modalities in parallel or simultaneously.

[0058] A preferred embodiment of a system for processing multimodal input signals with a fusion unit of the type described above further comprises a plurality of sensor devices. These sensor devices are configured to acquire input signals in the form of sensor data. At least two of the sensor devices are assigned to different modalities. For example, one sensor device may be configured as a camera for capturing user gestures or for capturing a user's pupil movement. Another sensor device may be configured for recording spoken language. Finally, one sensor device may also be configured as a touch-sensitive input device, for example, as a touchscreen or touchpad.

[0059] The system also includes at least one sensor data evaluation unit. This unit is configured to receive sensor data from one or more of the sensor devices, to recognize instructions for applications that can be coupled with the fusion unit from the sensor data, and to forward instruction messages to the fusion unit specifying the respective instructions.

[0060] At least one sensor data processing unit can, for example, be configured to extract a user's gestures as input signals from image data captured by a camera and to recognize a corresponding instruction for an application that can be coupled with the fusion unit. This or another sensor data processing unit can also be configured to extract a user's gaze pattern as input signals from image data captured by the camera and to recognize an instruction for applications that can be coupled with the fusion unit. Furthermore, a sensor data processing unit of the aforementioned type can be configured to extract voice commands entered by a user as input signals from recorded speech data and to recognize an instruction for an application that can be coupled with the fusion unit.Finally, a sensor data evaluation device may be present that is set up to extract contact-related input gestures entered by a user from captured touch data, such as tapping, dragging, pressing, swiping or the like, as input signals and to recognize from this an instruction for an application that can be coupled with the fusion unit.

[0061] The system may also include at least one preprocessor configured to preprocess instruction messages received from the at least one sensor data evaluation unit, for example to smooth them, and then forward them to the fusion unit.

[0062] A system as described above may be set up to support the operation or control of a vehicle assistance system, for example a navigation system, of a motor vehicle.

[0063] Unless otherwise stated in individual cases, the various embodiments of the invention mentioned in this application can be advantageously combined with one another.

[0064] The invention is explained below by way of exemplary embodiments with reference to the accompanying drawings. These show: Fig. 1 a preferred embodiment of a system for processing multimodal input signals and Fig. Two steps of a preferred embodiment of a method for processing multimodal input signals.

[0065] Fig. Figure 1 shows a system 40 for processing multimodal input signals. The multimodal input signals can be acquired via various sensor devices 10, 12, 14, 16, which form a first level of the system. One sensor device 10 is configured to acquire speech data. Various camera devices 12, 14 can, on the one hand, capture pupil size and movement of a user, and on the other hand, capture user gestures, in particular gestures of a hand or the fingers of a hand, in the form of image data. A touchpad 16 can capture touch-based input gestures.

[0066] In a second level, a plurality of sensor data evaluation units 18, 20, 22 are available. Each sensor unit 10, 12, 14, 16 is assigned a sensor data evaluation unit 18, 20, 22, which is configured to evaluate the sensor data acquired by the sensor unit 10, 12, 14, 16.

[0067] The sensor data evaluation unit 18, for example, includes a speech recognition function for evaluating speech data. The sensor data evaluation unit 20 is configured to process image data; it includes an image recognition application configured to process various types of image data with regard to recognizable input signals. The sensor data evaluation unit 22 is configured to evaluate input signals specified via the touch-sensitive sensor unit 16.

[0068] The evaluation of the sensor data is generally based on predefined patterns or gestures to be recognized, or a predefined vocabulary to be recognized. For example, the speech recognition unit of the sensor data evaluation unit 18 is capable of recognizing a predefined set of specific input commands and assigning a corresponding instruction to each of these commands. Similarly, for the image recognition unit of the sensor data evaluation unit 20, specific hand or finger gestures are predefined, which are directly assigned to corresponding instructions. For example, a voice command like "Next" can be assigned an instruction to scroll through a document or menu. The same instruction can also be assigned to a predefined hand gesture, such as a quick swipe from right to left.

[0069] The sensor data evaluation units 18, 20, 22 are configured to forward a detected instruction in the form of an instruction message to the fusion unit 26 or to a preprocessor 24 upstream of the fusion unit 26. The fusion unit 26 and the optional preprocessor 24 form a third level of the system 40.

[0070] Preprocessor 24 is configured to preprocess instruction messages. Instruction messages, for example, concerning a user's gaze position and derived from pupil movements detected by sensor 12, are typically sent from sensor data evaluation unit 20 to preprocessor 24 or fusion unit 26 at a high update frequency of approximately 30 Hz. The gaze positions transmitted via these instruction messages do not always provide a smooth or continuous description of the gaze path, but are often rather discontinuous or "jagged" due to the natural characteristics of the human eye. To nevertheless obtain a smooth and continuous path, for example, of a cursor controlled by such a gaze direction, appropriate smoothing can be performed within preprocessor 24.Based on the smoothed gaze coordinate parameters, the corresponding instruction messages are then forwarded by the preprocessor to the fusion unit 26. Various image smoothing algorithms can be used in the aforementioned preprocessor 24, particularly with the use of suitable filters (low-pass). Further input parameters can also be used by such smoothing algorithms, for example, how much the eye position changes between successive instruction messages. If a detected change is very large, it is advantageous to temporarily disable filtering using the low-pass filter, as otherwise the resulting cursor movement will "lag" too much, i.e., exhibit a noticeable and user-disruptive delay.

[0071] In fusion unit 26, the received instruction messages are then processed in the following way with reference to Fig. 2 fused in a manner described in detail. Such instructions that survive the fusion process, that is, that they emerge from a step of the fusion process, are forwarded by the fusion unit 26 to at least one of the various applications 28, 30, 32 coupled to the fusion unit.

[0072] The applications 28, 30, 32, in a fourth level of the system each process the instructions in a specific way, for example to control a graphical user interface (GUI) 34, 36 coupled with the respective application.

[0073] The preprocessor 24 and the fusion unit 26 are combined in a multimodal server 38. Such a multimodal server 38 can, for example, be located in a motor vehicle and serve to process multimodal input signals, which have been acquired via a series of sensors 10, 12, 14, 16 described above, in order to control or operate, for example, a vehicle assistance system.

[0074] Steps of a preferred embodiment of a method for processing multimodal input signals are described below with reference to Fig. 2 are described as examples.

[0075] In step S1, the user of system 40 inputs a signal, for example by speaking an input command, which is detected in step S2 by a corresponding sensor device 10 and recognized in step S3 by a corresponding sensor data evaluation device 18. In step S4, the sensor data evaluation device 18 generates an instruction message, which specifies the corresponding instruction, and forwards this instruction message to the fusion unit 26.

[0076] Further user input can occur simultaneously or at least overlapping, but also at a time interval and not overlapping, via other modalities, for example via input gestures, gaze direction or input on the touchpad 16, as shown in Fig. 2 with reference to steps S1' to S4' and S1" to S4" is indicated.

[0077] In the fusion unit 26, three processing stages can now be distinguished for the further processing of instruction messages.

[0078] In a first stage, the instruction messages are received by the fusion unit in step S5, and in step S6 a candidate object is created for each of the received instruction messages.

[0079] The candidate object refers to the corresponding instruction contained in the instruction message and includes a validity period for the candidate object. This validity period is also referred to as the validity period. Preferably, the candidate object also includes a weighting of the candidate object.

[0080] In addition to the instruction name, an instruction message, such as the one received by the fusion unit in step S5, includes further parameters, for example, a sensor setup parameter which specifies on the basis of which sensor data (speech data, image data, etc.) the instruction message was recognized. An application parameter of the instruction message specifies to which of the applications 28, 30, 32 (compare Fig.1) the application specified in the instruction message. The instruction message may also include a capture time parameter. This parameter describes a time interval within which sensor data was captured by a sensor device, on the basis of which the instruction message was generated. Another time-dependent parameter is the recognition time parameter. This parameter specifies a point in time at which a corresponding sensor data evaluation device recognized the instruction specified in the instruction message from the corresponding sensor data. Based on these latter time-dependent parameters, the fusion unit 26 can determine the validity period of a candidate object.This validity period can be determined, for example, via a minimum lifetime (minTTL), i.e. a first time interval, and a maximum lifetime (maxTTL), i.e. a second time interval, each with reference to a common starting time.

[0081] In addition to the parameters already mentioned, the instruction message typically includes a confidence parameter. This specifies how confidently the sensor data evaluation unit has recognized the designated instruction from the respective sensor data. Fusion unit 26 determines the weight of a candidate object based on this confidence parameter. For example, a weight parameter "weight" between 0 and 1 can be assigned to the candidate object in a first step. This weight parameter "weight" is then multiplied by the confidence parameter "conf". This results in the candidate object's weight "score": score = conf * weight.

[0082] In a second processing stage, generated candidate objects are merged as described below. This involves combining compatible instructions that occur practically simultaneously to prevent unintended multiple executions. Furthermore, the merging process allows for the deletion of conflicting instructions, which may be due to faulty capture or recognition. This ensures robust recognition and processing of input signals, especially those entered in parallel or simultaneously via different modalities.

[0083] In step S7, a first and a second candidate object are selected from the set of those candidate objects whose validity period has not yet expired.

[0084] In step S8, it is checked whether the validity period of the first candidate object overlaps with the validity period of the second candidate object. If this is not the case, new candidate objects are selected in step S7. To simplify the process of finding candidate objects with overlapping validity periods, the candidate objects can, for example, be sorted according to their respective validity periods.

[0085] If the relevant validity periods overlap, step S9 checks whether the first instruction specified by the first candidate object conflicts with the instruction specified by the second candidate object. If this is not the case, both candidate objects emerge unchanged from the merger step. The merger process continues with step S7.

[0086] If, however, there are conflicting instructions, they are merged as indicated with reference to step S10.

[0087] In substep S10.1, it is checked whether the first statement conflicts with the second statement. If so, the candidate object with the lower weight is deleted in substep S10.2. Alternatively, it may be stipulated that both candidate objects are deleted if their respective weights do not exceed a predefined minimum weight. Deletion of both candidate objects may also be stipulated if the difference between the weights assigned to the candidate objects does not exceed a predefined minimum difference.

[0088] The candidate object that has not been deleted continues to participate in the merger process, which continues with step S7, provided its validity period has not yet expired.

[0089] In the case that the first instruction is compatible with the second instruction, a merged candidate object is created in substep S10.3, which replaces the first and the second candidate object.

[0090] Agreed instructions exist, in particular, when the first and second instructions are identical. In this case, the merged application refers to the corresponding application. The validity period and weighting of the merged candidate object are each determined based on the validity periods and weightings of the first and second candidate objects. The validity period of the merged candidate object can, for example, correspond to one of the validity periods of the two merged candidate objects, or be derived from the union or intersection of the corresponding validity periods. Alternatively, the validity period of the merged candidate object can also be derived from at least one of the validity periods of the merged candidate objects.The weighting of the merged candidate object typically exceeds the weighting of the first and second candidate objects. This increases the priority of the merged candidate object. The merged candidate object is then reintroduced into the merger process, starting with step S7. The two merged candidate objects are no longer considered.

[0091] The merger process, i.e. steps S7 to S10, can be carried out continuously in merger unit 26 as long as candidate objects with an active validity period are available.

[0092] In a third processing stage, beginning with step S11, those candidate objects that have not yet been deleted, whose validity period has expired, and which consequently no longer participate in the merger process of steps S7 to S10, are considered. From this set, a candidate object is selected in step S11.

[0093] In step S12, the instruction designated by the candidate object is forwarded to an application 28, 30, 32 suitable or intended for executing the instruction. This forwarding can be controlled, in particular, via the application parameter of the candidate object mentioned above.

[0094] The processing of the instruction by the application itself is now application-specific and no longer the subject of the procedure described here.

[0095] Preferred embodiments of the present invention are given below in the numbered paragraphs: 1. A procedure in a fusion unit of a system for processing multimodal input signals may include the following steps: - Receiving instruction messages, each indicating an instruction that can be executed by an application that can be coupled to the fusion unit; - for at least two of the instruction messages: creating a candidate object for the instruction message, where the candidate object designates the instruction and includes a validity time specification of the candidate object, - Merging two of the candidate objects if the validity period information of these candidate objects specifies overlapping validity periods, and - Forwarding an instruction to the application, which instruction is designated by a candidate object resulting from the merging step. 2. In the procedure under paragraph 1, the candidate objects may emerge unchanged from the merger step if the instructions designated by the respective candidate objects do not conflict. 3. The procedure referred to in paragraph 1 or 2 may include the following steps: - Receiving a first instruction message from a first sensor data evaluation unit, wherein the first instruction message denotes a first instruction that can be executed by an application that can be coupled to the fusion unit; - The fusion unit generates a first candidate object for the first instruction message, wherein the first candidate object designates the first instruction and includes a validity period specification of the first candidate object as well as a weighting of the first candidate object; - Receiving a second instruction message from a second sensor data evaluation unit, wherein the second instruction message denotes a second instruction that can be executed by an application that can be coupled to the fusion unit; - Creating a second candidate object for the second instruction message, wherein the second candidate object denotes the second instruction and includes a validity time specification of the second candidate object as well as a weighting of the second candidate object, and - Merging the first candidate object with the second candidate object depending on the weight of the first candidate object and the weight of the second candidate object, if the validity time specification of the first candidate object specifies a first validity time that overlaps with a second validity time specified by the validity time specification of the second candidate object, and if the first statement conflicts with the second statement. 4. In the procedure according to paragraph 3, in the step of merging the first candidate object with the second candidate object, the first candidate object may be deleted, or the second candidate object may be deleted, or the first candidate object and the second candidate object may be deleted, if the first instruction conflicts with the second instruction. 5. In the procedure according to paragraph 3, in the merging step a merged candidate object may be formed from the first candidate object and the second candidate object if the first instruction and the second instruction are compatible, wherein the instruction designated by the merged candidate object is derived from the first instruction or from the second instruction or from the first instruction and the second instruction, and wherein the validity period of the merged candidate object is derived from the validity period of the first candidate object or from the validity period of the second candidate object or from the validity period of the first candidate object and the validity period of the second candidate object, and wherein the weighting of the merged candidate object is derived from the weighting of the first candidate object or from the weighting of the second candidate object or from the weighting of the first candidate object and the weighting of the second candidate object. 6. In the procedure under paragraph 5, the instruction designated by the merged candidate object may correspond to the first instruction if the first instruction is identical to the second instruction. 7. In the procedure according to paragraph 5, the instruction designated by the merged candidate object may correspond to the first instruction or the second instruction if the first instruction is specified by one or more first instruction parameters and the second instruction is specified by one or more second instruction parameters and the respective first and second instruction parameters lie within a specified similarity tolerance interval. 8. In the procedure according to paragraph 5, if the second instruction does not correspond to the first instruction, wherein the first instruction is specified by one or more first instruction parameters and the second instruction is specified by one or more second instruction parameters and the corresponding first and second instruction parameters are within a specified compatibility tolerance interval, the instruction designated by the merged candidate object may be specified by one or more third instruction parameters which, with reference to the corresponding first and second instruction parameters, are within the corresponding compatibility tolerance interval. 9. In the procedure according to at least one of paragraphs 5 to 8, the validity period of the merged candidate object may correspond to the validity period of the first candidate object, or to the validity period of the second candidate object, or specify a validity period which includes an intersection of the validity period of the first candidate object and the validity period of the second candidate object, or specify a validity period which includes the union of the validity period of the first candidate object and the validity period of the second candidate object. 10. In the procedure under at least one of paragraphs 5 to 9, the weighting of the merged candidate object may exceed the weighting of the first candidate object and the weighting of the second candidate object. 11. In the procedure according to at least one of paragraphs 1 to 10, the validity period of the first candidate object can be generated depending on the first instruction message. 12. In the procedure according to at least one of paragraphs 1 to 11, the validity period of the first candidate object can be generated depending on a modality to which a first sensor device is assigned, which has recorded sensor data on the basis of which the first sensor data evaluation device has generated the first instruction message. 13. In the procedure according to at least one of paragraphs 3 to 12, the weighting of the first candidate object can be generated depending on the first instruction message. 14. In the procedure according to at least one of paragraphs 3 to 13, the weighting of the first candidate object can be generated depending on a modality to which a first sensor device is assigned which has acquired sensor data on the basis of which the first sensor data evaluation device has generated the first instruction message. 15. In the procedure according to at least one of paragraphs 1 to 14, a received instruction message may include a sensor device parameter which designates a sensor device which has provided sensor data on the basis of which a sensor data evaluation device has generated the instruction message. 16. In the procedure according to at least one of paragraphs 1 to 15, a received instruction message may include an application parameter which specifies one or more applications which are set up or intended to execute the instruction. 17. In the procedure according to at least one of paragraphs 1 to 16, a received instruction message may include a capture time parameter which specifies a capture time interval within which sensor data have been acquired by a sensor device, on the basis of which a sensor data evaluation device has generated the instruction message. 18. In the procedure according to at least one of paragraphs 1 to 17, a received instruction message may include a recognition time parameter which specifies a recognition time at which a sensor data evaluation device has recognized the instruction specified in the instruction message from sensor data received from a sensor device. 19. In the procedure according to paragraph 17 or 18, the validity period of a candidate object can be determined depending on the recording time parameter. 20. In the procedure according to paragraphs 17 to 19, the validity period of a candidate object can be determined depending on the recognition time parameter. 21. In the procedure according to at least one of paragraphs 1 to 20, a received instruction message may include a confidence parameter which specifies how confidently a sensor data evaluation device has recognized the instruction specified in the instruction message from sensor data received from a sensor device. 22. In the procedure according to paragraph 21, a weighting of a candidate object may be determined depending on the confidence parameter. 23. In the procedure according to at least one of paragraphs 1 to 22, a first sensor device which has acquired sensor data on the basis of which a first sensor data evaluation device has generated a first instruction message may be assigned to a first modality which is different from a second modality which is assigned to a second sensor device which has acquired sensor data on the basis of which a second sensor data evaluation device has generated a second instruction message. 24. The procedure under at least one of paragraphs 1 to 23 may include the following further step for each candidate object that has not been deleted by the end of its validity period: - Forwarding the instruction designated by the candidate object to an application coupled with the fusion unit that is suitable or intended to execute the instruction. 25. In the procedure according to at least one of paragraphs 1 to 24, a candidate object may further include a merger parameter which indicates whether and, if so, at what time the candidate object is intended for potential merger with another candidate object. 26. In the procedure according to paragraph 25, the fusion parameter may specify that the instruction designated by the candidate object is forwarded directly to an application coupled to the fusion unit which is suitable or intended to execute the instruction. 27. In the procedure according to paragraph 26, the merger parameter may further indicate that, after the instruction designated by the candidate object has been forwarded, the candidate object is intended for potential merger with such further candidate objects which designate a further instruction which conflicts with the instruction designated by the candidate object. 28. In the procedure according to paragraph 26, the merger parameter may further indicate that the candidate object is not intended to be merged with other candidate objects after the instruction designated by the candidate object has been forwarded. 29. In the procedure according to paragraph 25, the merger parameter may specify that, after the expiry of the validity period of the candidate object, the instruction designated by the candidate object is forwarded to an application coupled to the merger unit which is suitable or intended to execute the instruction, and that the first candidate object is intended, until the expiry of its validity period, to be potentially merged with such further candidate objects which designate a further instruction which conflicts with the instruction designated by the candidate object. 30. In the procedure according to at least one of paragraphs 1 to 29, communication between a sensor device and a sensor data evaluation device, for transmitting sensor data acquired by the sensor device, or communication between a sensor data evaluation device and the fusion unit, for transmitting an instruction message, or communication between the fusion unit and an application coupled to the fusion unit, for forwarding an instruction, may take place via sockets. 31. In the procedure according to at least one of paragraphs 1 to 30, the instruction messages may be encoded in JSON data format. 32. The procedure according to at least one of paragraphs 1 to 31 may include the further steps before receiving an initial instruction message: - Capturing an initial input signal in the form of sensor data, entered by a user, by a first sensor device; - Receiving sensor data from the first sensor device and recognizing an instruction for an application that can be coupled with the fusion unit by a first sensor data evaluation device; - Generation of an initial instruction message by the first sensor data evaluation unit; and - Forwarding the first instruction message to the fusion unit. 33. A fusion unit for a system for processing multimodal input signals may be set up, - To receive instruction messages from at least one sensor data evaluation unit, each of which designates an instruction that can be executed by the application that can be coupled to the fusion unit; - to create a candidate object for at least two of the instruction messages, each of which designates the instruction and includes a validity time specification of the candidate object, - to merge two of the candidate objects if the validity period specifications of these candidate objects indicate overlapping validity periods, and - to forward an instruction to the application, which instruction is designated by a candidate object resulting from the merging step. 34. The merger entity referred to in paragraph 33 may be established and designated to carry out the procedure in accordance with at least one of paragraphs 2 to 32. 35. A system for processing multimodal input signals with a fusion unit according to paragraph 33 or 34 may include the following components: - a plurality of sensor devices which are set up to acquire input signals in the form of sensor data, wherein at least two of the sensor devices are assigned to different modalities; - at least one sensor data evaluation unit that is configured to receive sensor data from the sensor devices, to recognize instructions for applications that can be coupled with the fusion unit from the sensor data, and to send instruction messages to the fusion unit specifying the instructions. 36. In the system according to paragraph 35, one of the sensor devices may be designed as a camera. 37. In the system according to paragraph 36, at least one sensor data evaluation device may be installed to extract user gestures as input signals from image data captured by the camera and to recognize instructions for applications that can be coupled with the fusion unit. 38. In the system according to paragraph 36 or 37, at least one sensor data evaluation device may be installed to extract a user's gaze pattern as input signals from image data captured by the camera and to recognize from this an instruction for applications that can be coupled with the fusion unit. 39. In the system according to at least one of paragraphs 35 to 38, one of the sensor devices may be designed to record spoken language. 40. In the system according to paragraph 39, at least one sensor data evaluation device may be installed to extract voice commands entered by a user as input signals from recorded voice data and to recognize from this an instruction for applications that can be coupled with the fusion unit. 41. In the system according to at least one of paragraphs 35 to 40, one of the sensor devices may be designed as a touch-sensitive input device, in particular as a touchscreen or touchpad. 42. In the system according to paragraph 41, at least one sensor data evaluation device may be set up to extract contact-related input gestures entered by a user from touch data captured by means of the input device as input signals and to recognize from this an instruction for applications that can be coupled with the fusion unit. 43. The system according to at least one of paragraphs 35 to 42 may include at least one preprocessor configured to preprocess instruction messages received from the at least one sensor data evaluation unit and forward them to the fusion unit. 44. A motor vehicle may include a system according to at least one of paragraphs 35 to 43, wherein the system is designed to assist in the operation or control of a vehicle assistance system. Reference symbol list 10 Sensor setup (language) 12 Sensor setup (image) 14 Sensor setup (image) 16 Sensor device (touch) 18 Sensor data evaluation unit (speech recognition) 20 Sensor data evaluation unit (image recognition) 22 Sensor data evaluation unit (touch detection) 24 Preprocessor 26 fusion units 28 Application 30 Application 32 Application 34 GUI 36 GUI 38 multimodal servers 40 System S..., S...', S...'' Steps of a procedure for processing multimodal input signals

Claims

[1] Method in a fusion unit (26) of a system (40) for processing multimodal input signals, comprising the steps: - Receiving (S5) instruction messages, each indicating an instruction that can be executed by an application (28; 30; 32) that can be coupled to the fusion unit (26); - for at least two of the instruction messages: Create (S6) a candidate object for the instruction message, wherein the candidate object designates the instruction and includes a validity time specification of the candidate object, which specifies a validity time within which an instruction designated by the candidate object can compete with other instructions designated by other candidate objects, - Merging (S10) two of the candidate objects if the validity period information of these candidate objects specifies overlapping validity periods, and - Forwarding (S12) a statement to the application, which statement is designated by a candidate object resulting from the merging step. [2] Method according to claim 1, characterized by that the candidate objects emerge unchanged from the merge step if the instructions designated by the respective candidate objects do not conflict. [3] Method according to claim 1 or 2, characterized by the steps: - Receiving (S5) a first instruction message from a first sensor data evaluation unit (18), wherein the first instruction message denotes a first instruction that can be executed by an application (28) that can be coupled to the fusion unit (26); - Creating (S6) a first candidate object for the first instruction message, wherein the first candidate object denotes the first instruction and includes a validity time specification of the first candidate object as well as a weighting of the first candidate object; - Receiving (S5) a second instruction message from a second sensor data evaluation unit (20), wherein the second instruction message denotes a second instruction that can be executed by an application that can be coupled to the fusion unit; - Creating (S6) a second candidate object for the second instruction message, wherein the second candidate object denotes the second instruction and includes a validity time specification of the second candidate object as well as a weighting of the second candidate object, and - Merging (S10) the first candidate object with the second candidate object depending on the weight of the first candidate object and the weight of the second candidate object, if the validity time specification of the first candidate object specifies a first validity time that overlaps with a second validity time specified by the validity time specification of the second candidate object (S8), and if the first statement conflicts with the second statement (S9). [4] Method according to claim 3, characterized by , that in the step of merging (S10) the first candidate object with the second candidate object the first candidate object is deleted or the second candidate object is deleted or the first candidate object and the second candidate object are deleted (S10.2), if the first instruction conflicts with the second instruction (S10.1). [5] Method according to claim 3, characterized by, that in the merging step (S10) a merged candidate object is formed from the first candidate object and the second candidate object (S10.2) if the first instruction and the second instruction are compatible (S10.1), wherein the instruction designated by the merged candidate object is derived from the first instruction or from the second instruction or from the first instruction and the second instruction, and wherein the validity period of the merged candidate object is derived from the validity period of the first candidate object, or from the validity period of the second candidate object, or from the validity period of the first candidate object and the validity period of the second candidate object, and wherein The weighting of the merged candidate object is derived from the weighting of the first candidate object, or from the weighting of the second candidate object, or from the weighting of the first candidate object and the weighting of the second candidate object. [6] Method according to at least one of claims 1 to 5, characterized by , that the validity period of a candidate object is generated depending on the corresponding instruction message and / or depending on a modality to which a sensor device (10) is assigned, which has recorded sensor data, on the basis of which a sensor data evaluation device (18) has generated the instruction message. [7] Method according to at least one of claims 3 to 6, characterized by, that the weighting of a candidate object is generated depending on the corresponding instruction message and / or depending on a modality which is assigned to a sensor device (10) which has recorded sensor data on the basis of which a sensor data evaluation device (18) has generated the instruction message. [8] Method according to at least one of claims 1 to 7, characterized by, that a received instruction message includes a capture time parameter which specifies a capture time interval within which sensor data have been captured by a sensor device (10), on the basis of which a sensor data evaluation device (18) has generated the instruction message, and / or that a received instruction message includes a recognition time parameter which specifies a recognition time at which a sensor data evaluation device (18) has recognized the instruction specified in the instruction message from sensor data received by a sensor device (10), wherein the validity time specification of a candidate object is determined as a function of the capture time parameter and / or the recognition time parameter. [9] Method according to at least one of claims 3 to 8, characterized by, that a received instruction message includes a confidence parameter which specifies how confidently a sensor data evaluation device (18) has recognized the instruction specified in the instruction message from sensor data received from a sensor device (10), wherein the weighting of a candidate object is determined depending on the confidence parameter. [10] Method according to at least one of claims 1 to 9, characterized by , that a first sensor device (10), which has acquired sensor data, on the basis of which a first sensor data evaluation device (18) has generated a first instruction message, is assigned to a first modality, which is different from a second modality, which is assigned to a second sensor device (12), which has acquired sensor data, on the basis of which a second sensor data evaluation device (20) has generated a second instruction message. [11] Method according to at least one of claims 1 to 10, characterized by , that a candidate object further includes a merger parameter which specifies in what form and / or at what time the candidate object is intended for potential merger with another candidate object. [12] Method according to at least one of claims 1 to 11, characterized by The next steps before receiving an initial instruction message: - Capture (S1) a first input signal in the form of sensor data entered by a user by a first sensor device (10); - Receiving sensor data from the sensor device and recognizing an instruction (S2) for an application that can be coupled to the fusion unit by a first sensor data evaluation device (18); - Generation (S3) of a first instruction message by the first sensor data evaluation unit (18); and - Forwarding (S4) the first instruction message to the fusion unit (26). [13] Fusion unit (26) for a system (40) for processing multimodal input signals, wherein the fusion unit (26) is configured, - To receive instruction messages, each of which designates an instruction that can be executed by an application (28; 30; 32) that can be coupled to the fusion unit (26); - to create a candidate object for at least two of the instruction messages, wherein the candidate object designates the instruction and includes a validity time specification of the candidate object, which specifies a validity time within which an instruction designated by the candidate object can compete with other instructions designated by other candidate objects, - to merge two of the candidate objects if the validity period specifications of these candidate objects indicate overlapping validity periods, and - to forward an instruction to the application (28; 30; 32), which instruction is designated by a candidate object resulting from the merging step. [14] System (40) for processing multimodal input signals with a fusion unit (26) according to claim 13, further comprising - a plurality of sensor devices (10; 12; 14; 16) which are configured to acquire input signals in the form of sensor data, wherein at least two of the sensor devices (10; 12; 14; 16) are assigned to different modalities, and - at least one sensor data evaluation device (18; 20; 22) which is configured to receive sensor data from the sensor devices (10; 12; 14; 16), to recognize instructions from the sensor data for applications (28; 30; 32) that can be coupled with the fusion unit (26) and to send instruction messages designating the instructions to the fusion unit (26). [15] Motor vehicle with a system (40) according to claim 14, characterized by , that the system (40) is designed to assist in the operation or control of a vehicle assistance system of the motor vehicle.

Images