Device for detecting fraudulent audio communication using local AI-based analysis

Abstract

Device for detecting fraudulent audio communications, comprising: a) a data processing device (10) comprising at least one processor and one memory; b) an audio acquisition unit (20) designed to acquire audio signals from an audio communication and supply them to the data processing unit (10); c) an AI-based detection unit (50) running locally on the data processing device (10), configured to determine fraud indicators from the audio signals or text data derived therefrom and to determine a probability of fraud; d) a warning unit (60) which warns the user when a threshold of the probability of fraud is exceeded; wherein the detection unit (50) is executed entirely locally on the data processing device (10).

Description

I. Technical Field

[0001] The invention relates to a device for the automated detection of fraudulent audio communication, in particular fraudulent telephone calls. The invention relates to a data processing unit that analyzes call content locally and without a connection to external services and warns the user when fraudulent patterns are detected. In a preferred embodiment, the data processing unit is designed as an embedded computer system; in further embodiments, the data processing unit can also be designed as a mobile device, a personal computer with appropriate software, or another type of computing unit. Audio communication includes, in particular, telephone calls, VoIP calls, video calls, messenger-based voice calls, and voice messages.

[0002] The device can be designed as a standalone unit (so-called edge device) that is positioned at the user's terminal device. In other embodiments, the device can also be integrated into an existing telecommunications system, such as a router, a telephone system, a gateway, or another network device. In a further embodiment, the device can be designed as a software application on a mobile device (smartphone, tablet), as an application on a personal computer, or as an extension module (plugin) in an existing communications application.

[0003] The invention belongs to the field of artificial intelligence in the area of ​​communication security, in particular to the field of local AI processing (hereinafter referred to as "edge AI"). For the purposes of this application, "AI-based" means all methods of automated pattern recognition and decision-making, including neural networks, statistical models, rule-based expert systems, probabilistic models, decision tree-based methods, and hybrid approaches that combine several of these methods. II. State of the art

[0004] Telephone fraud, particularly in the form of so-called "grandparent scams," shock calls, impersonations of police officers or bank employees, causes millions of euros in damages annually and disproportionately affects elderly and vulnerable groups. Demographic change is continuously exacerbating this problem.

[0005] Various approaches to combating telephone fraud are known in the current state of the art: Number-based blocking systems: These systems compare incoming phone numbers with blocklists. However, they are ineffective against so-called number spoofing, where fraudsters transmit fake phone numbers that can even imitate real government numbers. Cloud-based speech analytics services: These systems transmit conversation content to external servers for analysis. They pose significant privacy risks, cause latency that makes real-time alerts difficult, and require a permanent internet connection. Network-level solutions from telecommunications providers: These filter calls at the network level but cannot perform content analysis of the conversations. They are limited to connection data (metadata) and known numbers and offer no protection against conversations with unknown or spoofed numbers that are manipulated in terms of content. Device-based spam and fraud detection systems: Recently, approaches have emerged in which end devices, especially smartphones, locally check incoming calls for spam or fraudulent activity. However, these systems rely primarily on metadata (phone number, call frequency, contact book matching) or on pre-trained classifiers that categorize a call as "probably spam" based on connection characteristics. A content-based, meaning-based (semantic) analysis of the actual conversation content—that is, what is said during the call—does not occur with these systems. In particular, a real-time assessment of specific fraud indicators such as demands for money, pressure from authority, or requests for secrecy, as well as a dynamic adjustment of the decision threshold to the risk context recognized during the call, is lacking. Device-based speech analysis of telephone calls: From EP 4 156 665 B1 (Deutsche Telekom AG), a speech analysis unit is known that can be implemented as internal software or as an external hardware adapter. It converts the audio signals of a telephone call into text in order to compare the text elements with predefined speech patterns of potential voice phishing attempts. If a pattern is detected, the user is warned via a background voice. Optionally, analysis results can be transmitted to a service provider for further investigation (geolocation, blacklist comparison). However, this system relies on a rule-based pattern comparison with static keyword tables and predefined thresholds.No AI-based semantic analysis of the conversation, dynamic adjustment of the decision threshold to the risk context recognized in the conversation, acoustic feature analysis of the audio signal, or detection of synthetically generated voices are disclosed. Local AI-based fraud detection on embedded systems: A scientific publication by Lu and Chen ("Edge AI System Using Lightweight Semantic Voting to Detect Segment-Based Voice Scams," MDPI Engineering Proceedings, February 2026) describes a system that captures audio signals using a microcontroller (ESP32), transcribes them locally using a quantized speech recognition model (Whisper), and then embeds the transcribed segments using a locally executed speech model (DeepSeek). Fraud detection is performed by comparing the embeddings' similarity to a local database of known fraud phrases using cosine. An alarm is triggered if a fixed threshold is exceeded in a defined number of consecutive segments.This system demonstrates the fundamental feasibility of local edge AI-based speech analysis, but relies on a static, unchanging threshold without dynamic adaptation to the risk context recognized in the conversation. A multi-stage hybrid recognition processing chain with feature-based ensemble classification, acoustic feature analysis of the audio signal for the detection of synthetically generated voices, or physical coupling to a telecommunications line via galvanically isolated audio transmitters are not disclosed.

[0006] None of the known solutions offer a device that combines an AI-based, meaning-based detection unit with multiple weighted fraud indicators categorized by criticality and dynamically adjusts the detection sensitivity to the risk context recognized in the conversation, particularly to mentioned monetary amounts, while fully protecting data privacy through local processing. This results in a technical security gap at the end device itself. III. Object of the invention

[0007] The invention is based on the objective of providing a device that analyzes audio communication, in particular telephone conversations, at or within the user's terminal device for fraudulent content without transmitting conversation data to external services, and warns the user upon detection of fraudulent patterns. In a preferred embodiment, the analysis and warning take place in real time during the ongoing conversation; in further embodiments, the analysis can also be performed with a time delay, in segments, or after the communication has concluded. IV. Solution to the task

[0008] The problem is solved by a device according to claim 1. An alternative solution is provided by a device according to dependent claim 25. A further alternative solution is provided by a device according to dependent claim 36. A further alternative solution is provided by a device according to dependent claim 40. Advantageous embodiments are the subject of the dependent claims.

[0009] The device according to the invention comprises a data processing unit with an audio acquisition unit, a local AI-based recognition unit, and a warning unit. In a preferred embodiment, the data processing unit is designed as an embedded computer system. The recognition unit is executed entirely locally on the data processing unit and determines fraud indicators and a fraud probability from the acquired audio signals or text data derived therefrom.

[0010] In a preferred embodiment, the recognition unit is configured as a multi-stage hybrid recognition processing pipeline comprising at least one neural language analysis, one feature-based classification, and one linguistic analysis. In a further embodiment, the recognition unit can also be configured as a unified language model that combines the aforementioned analysis functions in a single integrated processing step.

[0011] The key feature of the invention is the combination of an AI-based recognition unit with fully local processing on a data processing device, which analyzes conversation content for meaning and detects fraud attempts without a cloud connection. In a preferred embodiment, the recognition takes place in real time during the ongoing conversation. V. Detailed description of the invention V.1 System architecture (Fig. 1)

[0012] The device comprises the following main components: a) Embedded computer system (10): A computer system comprising at least one processor and memory configured for the local execution of AI models. In a first preferred embodiment, the computer system comprises a single-board computer (SBC) with a multi-core processor, preferably an ARM-based system-on-a-chip (SoC). In a preferred embodiment, an SoC with at least two high-performance cores and several efficiency cores is used, wherein the AI ​​inference is assigned to the high-performance cores with CPU core allocation (CPU affinity). Optionally, an integrated neural processing unit (NPU) can be used to accelerate the inference.In other embodiments, the computer system can also be designed as an integrated single-chip system with application-specific circuitry (ASIC), as a field-programmable logic device (FPGA), as a computer module (English: Compute Module) or as a custom printed circuit board (Custom-PCB) with an integrated processor. b) Audio Acquisition Unit (20): A unit for acquiring audio signals from a telephone conversation. In a first embodiment (analog variant), the audio acquisition unit comprises at least one galvanically isolated audio transformer (21) that passively couples the device to an analog telephone line. Preferably, two 600:600Ω audio transformers are used, enabling bidirectional acquisition of both sides of the conversation. A surge protector (22) protects the electronics from voltage spikes on the telephone line. The coupling is polarity-independent. In a second embodiment (VoIP variant), the computer system registers with an IP telephone system using the SIP protocol (Session Initiation Protocol) and receives the audio data over the network, without requiring an analog line tap.In other embodiments, the audio acquisition unit can also include a DECT base station (Digital Enhanced Cordless Telecommunications), a Bluetooth hands-free interface (Bluetooth Hands-Free Profile, HFP), a USB audio interface, or a software interface to a softphone. In a mobile communication version, the audio acquisition unit includes a GSM / LTE modem that serves as an intermediary between the mobile network and the acquisition unit. c) Analog-to-digital converter (30): The analog audio signal is converted into a digital signal by means of an audio signal encoder / decoder (audio codec) (31). Sampling is performed at a minimum of 16 kHz, single-channel (mono), 16-bit pulse-code modulation (PCM). In one embodiment, a USB audio codec is used; in a production version, an I 2An S-Audio codec is provided, which is located directly on an adapter board (40). For digital audio acquisition methods (VoIP, DECT, Bluetooth, softphone), the analog-to-digital converter can be omitted, since the audio data is already in digital form. d) Detection unit (50): An AI-based unit executed locally on the embedded computer system, configured to determine fraud indicators from the audio signals or text data derived therefrom and to calculate a probability of fraud. In the preferred embodiment, the detection unit is configured as a multi-stage hybrid detection processing pipeline (see Section V.3). In another embodiment, the detection unit can also be configured as a unified language model (Large Language Model, LLM) that combines the functions of fraud indicator detection, probability calculation, and language analysis in an integrated processing step.In a simplified embodiment, the recognition unit is designed in two stages, with a neural language model combining the functions of neural analysis and linguistic analysis, and a downstream decision module determining the probability of fraud. e) Warning unit (60): A multi-channel warning system that informs the user in real time when fraudulent patterns are detected (see section V.5). V.2 Audio preprocessing (Fig. 2)

[0013] Audio preprocessing includes several stages: Line activity detection (51): An energy-based detector with automatic baseline calibration distinguishes between the states of idle, noise, audio signal, and speech. Calibration is performed over a defined period at system startup. Voice Activity Detection (VAD) (52): Voice activity detection with configurable sensitivity level and energy-based fallback mechanism. A pre-buffer stores audio signal segments before speech begins to reliably detect word beginnings. Call state machine (53): A tone recognition system based on fast Fourier transform (FFT) identifies ring tones and hang-up events and controls the transition between call states (idle, ringing, active call, ended).

[0014] The audio signal segments are processed in configurable time slices and temporarily stored in a ring buffer. V.3 Multi-stage hybrid recognition processing chain (Fig. 3)

[0015] In the preferred embodiment, the recognition unit is designed as a three-stage hybrid recognition processing chain (pipeline) that combines different artificial intelligence methods: Level 1 - Neural Speech Analysis (54): A quantized neural speech model (preferably a pre-trained transformer model, executed using an optimized inference runtime) analyzes transcribed speech segments and determines a probability for a plurality of predefined fraud indicators (flags). Additionally, the fraud type is classified. In further embodiments, a different neural network model can be used instead of the transformer model, for example, a convolutional neural network (CNN), a recurrent neural network (RNN), or a state-space model (e.g., Mamba). Level 2 - Feature-Based Ensemble Classification (55): An ensemble model based on gradient boosting receives a large number of extracted features and determines an overall probability of a fraud attempt. The decision threshold is dynamically adjusted to the amount of money mentioned in the conversation: When large amounts are mentioned, the alarm threshold is lowered to enable earlier and more sensitive detection. In further embodiments, a different classifier can be used instead of the gradient boosting model, for example, a random forest, a support vector machine (SVM), or a neural network. Stage 3 - Linguistic Analysis (56) (English: Natural Language Processing, NLP): A machine language processing module performs a detailed linguistic analysis, including: recognition of the caller type (e.g., impersonating an authority figure), identification of typical victim forms of address (e.g., recognition of grandma / grandpa as a grandchild scam indicator), extraction of mentioned amounts of money, and recognition of negation patterns.

[0016] The three stages work in a complementary manner, each providing independent assessments that are combined (fused) to form an overall decision.

[0017] In another embodiment, the recognition processing chain can also be designed in two stages, with a unified language model combining the functions of neural analysis (stage 1) and linguistic analysis (stage 3) in one step, and a downstream decision module determining the overall probability of fraud.

[0018] In another embodiment, the detection unit can be designed as a single large language model (LLM) that performs all analytical functions—fraud indicator detection, probability determination, fraud type classification, and linguistic analysis—in a single integrated processing step. The AI ​​models of the detection unit are preferably designed as interchangeable modules that can be updated locally without any conversation data leaving the device.

[0019] In a further embodiment, the recognition unit (50) additionally or alternatively comprises an acoustic feature analysis component in addition to or as an alternative to text-based analysis. This component derives fraud indicators directly from the audio signal without the intermediate step of transcription. The acoustic feature analysis can include prosodic features (intonation, speech rate, pauses, stress patterns), voice biometric features (voice spectrum, formants, fundamental frequency), emotional features (stress indicators, arousal level), and acoustic anomalies (synthetic voices, voice distortion). The acoustic feature analysis can be implemented as an independent stage in the recognition processing chain, as a parallel analysis path, or as an integrated component of a multimodal recognition model.

[0020] In another embodiment, the detection unit (50) can be configured as an agent-based system in which several specialized AI subcomponents (agents) each perform subtasks of fraud detection and transmit their results to a central decision component. The agents can be specialized, for example, for the detection of specific fraud types, for acoustic analysis, for linguistic analysis, or for contextual evaluation. In a further development, the detection unit can include a knowledge-based retrieval-augmented generation (RAG) component that utilizes local knowledge databases containing known fraud patterns, regional fraud scenarios, or updatable rule sets to improve detection accuracy. The knowledge databases are stored locally on the embedded computer system and can be updated independently of the AI ​​models. V.4 Fraud Indicators and Weighting System

[0021] The detection unit evaluates a number of predefined fraud indicators (markers), which are classified and weighted into criticality categories: Critical indicators (highest weighting): Demand for money, request for remote access to computer or telephone, request for sensitive personal data, demand for advance payments. High indicators: identity fraud, pressure from authority, demand for secrecy, instilling fear, fraudulent activity via third parties. Medium indicators: Artificial time pressure, requests for unusual actions, unrealistic promises. Low indicators: Authentication requests, emotional pressure, callback blockage.

[0022] The overall assessment also takes into account the type of fraud, with the device distinguishing at least the following categories: grandparent scam, fake police officer, tech support fraud, fake bank employee, investment fraud, sweepstakes fraud, and generally suspicious patterns.

[0023] A key feature is the dynamic, money-based adjustment of the decision threshold: If a monetary amount above a configurable threshold is mentioned during a conversation, the detection threshold for the alarm level is automatically lowered, making the device more sensitive to potentially high financial losses. The decision threshold can also be adjusted contextually, for example, depending on the detected caller type, the time of day, or the duration of the call. V.5 Warning mechanism (Fig. 1, reference numeral 60)

[0024] The warning unit includes several channels that can be switched simultaneously or selectively: Acoustic warning (61): Playback of caller-type-specific warning files via a loudspeaker, preferably with two urgency levels and a cooldown period to avoid warning overload. Optical warning (62): An individually controllable multi-color light-emitting diode (RGB-LED) indicates the operating status by color coding: Green for readiness, Blue for active listening, Yellow for ongoing evaluation, Red flashing for detected fraud attempt. Network-based notification (63, optional): A web interface (dashboard) accessible on the local area network (LAN) displays the real-time status and alerts. Future support includes network callbacks (webhooks), instant messaging (SMS / Messenger), and email notifications.

[0025] The device warns the user but does not automatically disconnect the phone connection. The user retains full control. V.6 Physical structure (Fig. 4)

[0026] In a first preferred embodiment (standalone device), the device comprises: Single-board computer (10): An ARM-based single-board computer (SBC) with a general-purpose input / output header (GPIO header). Adapter board (40, so-called HAT board): A circuit board that plugs into the pin header and is populated differently depending on the product variant. The analog variant includes an audio transformer (21), surge protection (22), an audio codec (31), and connectors. The VoIP variant only includes a network controller (Ethernet controller), leaving the analog board area unpopulated. Housing (70): A housing, preferably of multi-part construction. Connections: Analog version: two RJ11 jacks (phone line pass-through), one audio output to the single-board computer, USB-C jack for power. VoIP version: RJ45 jack (network), USB-C jack for power.

[0027] Power is supplied externally via a USB-C power adapter (5V). The device is not powered via the telephone line. V.7 Alternative embodiments of the physical structure

[0028] In a second embodiment (integrated assembly), the computer system (10) and the audio acquisition unit (20) are integrated on a common assembly, for example, on a single custom printed circuit board (PCB) with integrated processor, memory, and audio input stage. In this embodiment, the separate adapter board and the pin header are omitted.

[0029] In a third embodiment (computer module design), the computer system comprises a pluggable computer module (English: Compute Module) which is plugged onto a carrier board which includes the audio acquisition unit and the connections.

[0030] In a fourth embodiment (integrated telecommunications equipment), the device is integrated into an existing telecommunications device, for example, a router, a telephone system, a VoIP gateway, a DECT base station, or a smart home system. In this embodiment, the device uses the existing hardware of the telecommunications equipment as a computer system and accesses the audio signals via an internal software interface.

[0031] In a fifth embodiment (USB add-on device), the device is designed as a compact USB add-on device (dongle) that is connected to a personal computer, a notebook or another terminal device with softphone software and receives the audio signals via a USB audio interface. V.8 Privacy-by-Design

[0032] A key feature of the invention is the completely local implementation of the recognition unit (50). All AI models, speech recognition, linguistic analysis, and decision-making are executed exclusively on the embedded computer system. No conversation content, transcripts, or derived analysis results are transmitted to external servers for processing.

[0033] In a preferred embodiment, the device does not include any means for transmitting call content to external services. An optional network connection is used exclusively for the local web interface, software updates, and – in the VoIP version – the SIP protocol for telephone connectivity.

[0034] In another embodiment, the device can have a network interface used for purposes beyond fraud analysis, such as software updates, anonymized usage statistics, or optional downstream notification of third parties (e.g., family members). In this embodiment as well, the detection unit runs entirely locally; the network interface has no access to the conversation content or transcripts. V.9 Model Management

[0035] The AI ​​models of the recognition unit (50) are preferably designed as interchangeable modules that can be updated independently of one another. The update is performed locally, for example via a USB interface, a secure network connection (over-the-air update, OTA), or manual model replacement. When updating via a network connection, only model data is transferred from the server to the device; no call data or analysis results are transmitted from the device to the server. V.10 Optional hybrid processing architecture

[0036] In a further embodiment, the device can include an optional downstream external processing component in addition to the local detection unit (50). In this embodiment, the local detection unit performs the primary fraud detection entirely on the embedded computer system. Only after completion of the local analysis and only if a positive suspicion result is obtained can anonymized or aggregated analysis results—but not conversation content, transcripts, or audio data—be transmitted to an external service for supplementary post-processing. This supplementary post-processing can include, for example, a refined fraud type classification, a comparison with a central fraud pattern database, or statistical evaluation across multiple devices. The user can configure or disable the transmission option.The primary real-time fraud detection always remains entirely local and is not dependent on the availability of external services. V.11 Acoustic Feature Analysis

[0037] In a further embodiment, the detection unit (50) comprises an acoustic feature analysis component that derives fraud indicators directly from the audio signal. The acoustic feature analysis can be implemented as a standalone stage, as a parallel path to text-based analysis, or as an integrated component of a multimodal model. The analyzed acoustic features include, in particular: prosodic features such as intonation, speaking rate, pause patterns, and stress contours; voice biometric features such as fundamental frequency, formant structure, and voice spectrum; emotion-related features such as stress indicators, arousal level, and vocal tremor; and anomaly detection for synthetically generated voices (deepfake detection), voice distortion, or unnatural acoustic patterns.Acoustic feature analysis enables fraud detection even when a written transcription of the conversation is not possible, not reliable, or not necessary. V.12 Microphone-based audio capture

[0038] In a further embodiment, the audio acquisition unit (20) comprises at least one microphone that acquires audio signals from a telephone conversation or other audio communication from the environment, particularly when the user's terminal device is operated in hands-free mode or an external loudspeaker is used. The microphone-based acquisition can be configured as the sole audio acquisition method or in addition to the wired or network-based acquisition methods described in Sections V.1 and V.6. In this embodiment, the audio preprocessing preferably also includes noise reduction and acoustic echo cancellation to suppress ambient noise and feedback. The microphone can be configured as an integrated component of the device, as an external USB microphone, as a microphone array, or as a component of a hands-free headset. V.13 Multilingual Recognition

[0039] In a further embodiment, the detection unit (50) is configured for processing and analyzing audio communication in multiple languages. For this purpose, the detection unit comprises either language-specific AI models that are loaded depending on the detected language, or a multilingual AI model that detects fraudulent patterns independently of language or in a plurality of languages. Language recognition can be performed automatically, with a language identification component determining the language used in the conversation and activating the corresponding model or model path. The language-specific fraud indicators, forms of address, and fraud patterns are preferably configured as interchangeable language packages that can be updated independently of the core model. The multilingual detection also includes the ability to detect and process language switching within a conversation (code-switching). V.14 Cross-call correlation analysis

[0040] In a further embodiment, the detection unit (50) comprises a cross-call correlation component (58) that analyzes patterns and fraud indicators across multiple successive communication events. For this purpose, the correlation component stores anonymized or pseudonymized features of past calls locally on the data processing unit, in particular voice characteristics of the caller, detected fraud types, temporal patterns, and call histories. The cross-call analysis enables, in particular, the detection of repeated fraud attempts by the same caller, the detection of escalating fraud patterns (e.g., increasing demands for money over several calls), and the correlation of call patterns that indicate coordinated fraudulent activities. The stored correlation data is processed exclusively locally and automatically deleted after a configurable period. V.15 Local learning and adaptation capability

[0041] In another embodiment, the recognition unit (50) is configured to learn from user feedback and local operational data and adapt to the individual usage profile. This adaptation can take the form of local fine-tuning of the AI ​​models, adjustment of decision thresholds and weightings based on user feedback (e.g., confirmation or rejection of warnings), or local reinforcement learning. In a further development, the device can support federated learning, whereby model improvements are exchanged between multiple devices without conversation data or personalized analysis results leaving the individual devices. All learning and adaptation processes are performed locally on the data processing unit. V.16 Active intervention in conversation

[0042] In the preferred embodiment, the device warns the user but does not automatically disconnect the telephone connection; the user retains control. In a further embodiment, however, the device can be configured to actively intervene in the conversation when a configurable, increased threshold of fraud probability is exceeded. This active intervention can include, in particular: automatically disconnecting the connection, inserting a warning message into the audio stream of the conversation, muting the user's microphone to prevent further disclosure of information, or automatically forwarding the call to a trusted person or a counseling service. The active intervention is preferably configurable by the user or an authorized trusted person and can be activated or deactivated. V.17 Multimodal end-to-end detection

[0043] In a further embodiment, the recognition unit (50) is designed as a multimodal end-to-end model that processes audio signals and derived text data simultaneously and in an integrated manner in a single processing step, without a sequential division into separate analysis stages. The multimodal model receives as input both the raw audio signals or derived acoustic feature vectors and the conversation content converted into written form, and determines the fraud indicators and the probability of fraud in an integrated inference step. By processing both modalities simultaneously, the model can detect correlations between linguistic and acoustic features that would not be accessible through sequential processing. V.18 Software Execution Form

[0044] In another embodiment, the device is designed as a software application that runs on a general-purpose data processing device, in particular a smartphone, tablet, personal computer, or notebook. In this embodiment, the general-purpose data processing device constitutes the data processing unit, and the audio acquisition unit (20) is designed as a software interface that acquires audio signals via the device's integrated microphone, via an operating system audio API, or via an interface to a communication application (softphone, messenger, video conferencing software). The recognition unit (50) runs as a software module locally on the processor of the data processing device; optionally, a neural processing unit (NPU) or a graphics processing unit (GPU) present on the device can be used to accelerate the inference process.In this embodiment, the warning unit (60) is configured as a notification function of the software application, which issues visual, acoustic, and / or haptic warnings via the device's user interface. The software application can be configured as a standalone application (app), as an extension module (plugin) for an existing communication application, as a background service of the operating system, as a browser-based application (Progressive Web App or WebAssembly application), as an embeddable software library (Software Development Kit, SDK) for third-party applications, or as a cloud-managed endpoint with a local detection unit. V.19 Context Data-Based Recognition

[0045] In a further embodiment, the detection unit (50) is configured to use additional contextual data as input for fraud probability determination, in addition to the audio signals and the text data derived therefrom. The contextual data may include, in particular: Telephone number information and caller identification data (Caller ID), Connection metadata such as call time, call duration and call frequency, Network information such as country of origin or network type of connection, a comparison with the user's local contact book to distinguish between known and unknown caller, as well as environmental information such as time of day or day of the week. The context data is processed locally on the data processing unit (10) and is incorporated as additional features into the fraud probability determination. without transmitting them to external services. In a further training, the recognition unit can combine the context data with the audio and text features in a multimodal fusion step to improve recognition accuracy.

[0046] In a further embodiment, the recognition unit (50) comprises a deepfake detection component (59) configured to detect synthetically generated or manipulated voices (so-called audio deepfakes). For this purpose, the deepfake detection component analyzes acoustic features characteristic of synthetically generated speech, in particular spectral artifacts, unnatural transitions between phonemes, missing micromodulations of the voice, atypical fundamental frequency responses, and inconsistencies between speech content and prosodic features. The deepfake detection component can be configured as an independent module, as an integrated component of the acoustic feature analysis component (57), or as a sub-function of a multimodal recognition model. The result of the deepfake detection is incorporated into the fraud probability assessment as an additional fraud indicator. V.20 Procedure for detecting fraudulent audio communication

[0047] The aforementioned devices perform a method for detecting fraudulent audio communication, comprising the following steps: (a) capturing audio signals of an audio communication using an audio capture unit (20); (b) feeding the captured audio signals to a data processing unit (10) comprising at least one processor and memory; (c) determining fraud indicators and calculating a fraud probability from the audio signals or text data derived therefrom using an AI-based detection unit (50); and (d) warning the user using a warning unit (60) if the calculated fraud probability exceeds a threshold. In a preferred embodiment, steps (c) and (d) are performed entirely locally on the data processing unit and in real time during the ongoing audio communication.In further embodiments, the method may also include the following steps: dynamically adjusting the decision threshold depending on monetary amounts mentioned in the conversation or other context-dependent parameters; detecting synthetically generated voices or AI-generated audio content directly from the audio signal; correlating fraud indicators across multiple temporally successive communication processes; using contextual data such as phone number information, connection metadata, or a local contact book comparison as additional input variables for fraud probability determination; and updating the AI ​​models via a unidirectional update path in which only model data is transferred to the data processing unit without any conversation data or analysis results leaving the data processing unit. REFERENCE MARK LIST 10 Data processing device (embedded computer system / mobile device / personal computer / custom SoC / single-board computer / SBC / computer module / FPGA / ASIC) 20 audio recording units 21 Audio transformers (transformer, 600:600Ω) 22 Overvoltage protection (TVS diode) 30 Analog-to-Digital Converters 31 Audio codec (USB or I) 2 S) 40 Adapter board (HAT board) / Carrier board 50 Recognition unit (AI-based processing chain / unified language model / multimodal end-to-end model / agent-based system) 51 Line activity detection 52 Voice Activity Detection (VAD) 53 Call status machine 54 Neural language analysis (Level 1) 55 Feature-based ensemble classification (level 2) 56 Linguistic Analysis / NLP (Level 3) 60 warning units 61 Acoustic warning (speaker) 62 Optical warning (RGB LED) 63 Network-based notification (web dashboard) 70 cases 57 Acoustic Feature Analysis Component 58 Cross-call correlation component 59 Deepfake detection component 80 Optional downstream external processing component VI. Character Description

[0048] The invention will now be explained in more detail using exemplary embodiments and with reference to the accompanying drawings. Fig. Figure 1 shows a block diagram of the overall system architecture of the device for detecting fraudulent audio communication. The audio acquisition unit (20) forms the interface to the communication channel and feeds the signals to the data processing unit (10) via an optional analog-to-digital converter (30). The core component of the data processing unit (10) is the AI-based detection unit (50), which in turn includes neural speech analysis (54), ensemble classification (55), linguistic analysis (56), acoustic analysis (57), the correlation component (58), and deepfake detection (59). Upon detection of a fraud pattern, the warning unit (60) is activated, which issues warnings via acoustic (61), visual (62), or network-based channels (63). Fig. Figure 2 illustrates the flowchart of the audio preprocessing. The audio signal sequentially passes through the line activity detection (51) for baseline calibration, the speech activity detection (52) with pre-buffer, and the call state machine (53). The latter controls the transition between the states idle, ringing, active call, and ended based on FFT tone detection. Fig. Figure 3 shows the internal structure of the recognition processing chain (50). In the preferred embodiment, transcribed conversation segments are processed by a three-stage pipeline: Stage 1 (54) determines fraud indicators using quantized transformer models; Stage 2 (55) determines the overall probability using a gradient boosting model; Stage 3 (56) performs the linguistic NLP analysis. A key feature is the dynamic threshold adjustment, which feeds back from the linguistic analysis (56) to the classification (55). Alternative implementations, such as a unified LLM, a multimodal model, or an agent-based system, are also shown. Fig. Figure 4 illustrates the physical setup in various configurations. The analog version uses an adapter board (40) plugged into the single-board computer (10) with audio transformers (21), surge protection (22), and an audio codec (31) for connection to the telephone line. The VoIP version uses an Ethernet controller for network connectivity instead. As a further alternative, the integrated assembly (custom PCB) is shown, in which all components are combined on a single circuit board. Fig. Figure 5 illustrates the hybrid processing architecture for maintaining data sovereignty. Analysis primarily takes place on the local device, with text-based (50) and acoustic feature analyses (57) leading to fusion. Optional downstream processing (80) in an external component receives only anonymized and aggregated data. Model updates occur via a strictly unidirectional path from the server to the device. Fig.Figure 6 shows the block diagram of the software implementation on a general data processing device (10) such as a smartphone or PC. The software application integrates the audio capture (20) via system APIs, the local detection unit (50) as a software module, and the warning unit (60) via the operating system's notification system. The various training formats as a standalone app, plugin, or SDK are indicated at the bottom. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] EP 4 156 665 B1

[0005]

Claims

Device for detecting fraudulent audio communication, comprising: a) a data processing unit (10) with at least one processor and a memory; b) an audio acquisition unit (20) configured to acquire audio signals from an audio communication and supply them to the data processing unit (10); c) an AI-based detection unit (50) executed locally on the data processing unit (10), configured to determine fraud indicators from the audio signals or text data derived therefrom and to calculate a probability of fraud; d) a warning unit (60) that warns the user when a threshold of the probability of fraud is exceeded; wherein the detection unit (50) is executed entirely locally on the data processing unit (10). Device according to claim 1, characterized in that the detection unit (50) and the warning unit (60) are configured such that the detection and warning take place in real time during the ongoing audio communication. Device according to claim 1 or 2, characterized in that the data processing device (10) is designed as an embedded computer system and the device is designed as a standalone device that can be connected to a telecommunications line, an IP telephone system or a communication network. Device according to claim 1 or 2, characterized in that the data processing device (10) is designed as a mobile device, personal computer or tablet and the recognition unit (50) is implemented as a software module, wherein the audio acquisition unit (20) is designed as a software interface that acquires audio signals via an integrated microphone, an audio API of the operating system or an interface to a communication application. Device according to one of the preceding claims, characterized in that the recognition unit (50) is designed as a multi-stage hybrid recognition processing chain comprising at least: (i) a neural speech analysis component (54) for determining fraud indicators from conversation segments converted into written form, (ii) a feature-based classification component (55) for determining an overall fraud probability from a plurality of extracted features, and (iii) a linguistic analysis component (56) for meaning-related detailed analysis of the conversation content. Device according to one of claims 1 to 4, characterized in that the recognition unit (50) is designed as a unified language model which is configured to perform the determination of fraud indicators, the determination of the probability of fraud and the linguistic analysis in an integrated processing step. Device according to one of claims 1 to 4, characterized in that the recognition unit (50) is designed as a multimodal end-to-end model that simultaneously processes audio signals and derived text data in an integrated processing step and determines the fraud indicators and the fraud probability from this. Device according to one of the preceding claims, characterized in that the detection unit (50) is configured to dynamically adjust the decision threshold to at least one context-dependent parameter, wherein the context-dependent parameter comprises at least one of the following: a monetary amount mentioned in the conversation, a detected caller type, a conversation duration, a time of day or a combination of several fraud indicators, and wherein the alarm threshold is automatically lowered in the presence of an increased risk context. Device according to one of the preceding claims, characterized in that the detection unit (50) is configured to evaluate a plurality of predefined fraud indicators, which are classified into criticality categories with different weightings, comprising at least the categories critical, high, medium and low, and combines the weighted indicators into an overall assessment. Device according to one of the preceding claims, characterized in that the recognition unit (50) is configured to recognize typical familiar forms of address as an indicator of a grandchild scam attempt and to extract amounts of money mentioned in the conversation. Device according to one of the preceding claims, characterized in that the audio acquisition unit (20) comprises at least one audio transformer (21) with galvanic isolation, which couples the device passively and polarity-independently to an analog telephone line. Device according to one of claims 1 to 10, characterized in that the audio acquisition unit (20) comprises a SIP client which is configured to register with an IP telephone system and receive audio data via a network protocol. Device according to one of claims 1 to 10, characterized in that the audio acquisition unit (20) comprises a Bluetooth hands-free interface, a DECT interface, a USB audio interface or a microphone for acquiring audio signals from the environment. Device according to one of the preceding claims, characterized in that the data processing unit (10) and the audio recording unit (20) are integrated on a common assembly. Device according to claim 14, characterized in that the data processing device (10) comprises a single-board computer with an ARM-based single-chip system, wherein an adapter board (40) is plugged onto a general-purpose input / output pin header of the single-board computer and is equipped with components for analog or VoIP-based audio acquisition, depending on the product variant. Device according to one of the preceding claims, characterized in that the warning unit (60) comprises at least two of the following warning channels: an acoustic warning (61) with caller type-specific warning files and graded urgency, an optical warning (62) by means of a color-coded multi-color LED, a network-based notification (63) via a web interface accessible in the local network, and a notification via the user interface of a software application. Device according to one of the preceding claims, characterized in that the warning unit (60) is configured such that the type and content of the warning depend on the detected type of fraud, wherein different warning files are stored for different types of fraud. A device according to one of the preceding claims, characterized in that the device is configured to actively intervene in audio communication when a configurable increased threshold of the probability of fraud is exceeded, wherein the active intervention comprises at least one of the following: an automatic disconnection of the connection, the insertion of a warning message into the audio stream, the muting of the user's microphone or an automatic forwarding to a trusted person, and wherein the active intervention is configurable and deactivatable by the user or an authorized trusted person. Device according to one of the preceding claims, characterized in that the device further comprises an audio preprocessing unit configured to perform energy-based line activity detection (51), speech activity detection (52) and FFT-based call state detection (53) before the detected speech segments are fed to the detection unit (50). Device according to claim 5, characterized in that the neural speech analysis component (54) comprises a quantized transformer model that is executed in an optimized inference runtime environment on the processor of the data processing device (10). Device according to claim 5, characterized in that the fraud indicators determined in stage (i), the overall probability determined in stage (ii) and the linguistic evaluation determined in stage (iii) are determined independently of each other and subsequently merged into a fused overall decision. Device according to one of the preceding claims, characterized in that the audio acquisition unit (20) in the analog variant comprises two audio transformers (21) each with 600:600Ω for bidirectional acquisition of both sides of the conversation and at least one overvoltage protection device (22). Device according to one of the preceding claims, characterized in that the AI ​​models of the recognition unit (50) are designed as interchangeable modules that can be updated locally without conversation data or analysis results leaving the data processing unit (10). Device according to one of the preceding claims, characterized in that the recognition unit (50) is configured in such a way that it does not transmit any conversation content, transcripts or directly derived personal analysis results to external services for processing. Device for detecting fraudulent audio communication, comprising: a) a data processing unit (10) with at least one processor and a memory; b) an audio acquisition unit (20) configured to acquire audio signals from an audio communication and supply them to the data processing unit (10); c) an AI-based detection unit (50) executed locally on the data processing unit (10), configured to determine fraud indicators from the audio signals or text data derived therefrom and to calculate a fraud probability, wherein the primary fraud detection is performed entirely locally on the data processing unit (10); d) a warning unit (60) that warns the user when a fraud probability threshold is exceeded;e) an optional downstream external processing component configured to transmit anonymized or aggregated analysis results to an external service for supplementary post-processing after completion of the local analysis, whereby no conversation content, transcripts or audio data are transmitted to the external service and the primary fraud detection is independent of the availability of the external service. Device according to one of the preceding claims, characterized in that the recognition unit (50) is designed as a two-stage recognition processing chain, wherein a neural language model combines the functions of neural language analysis and linguistic analysis in one processing step and a downstream decision module determines the overall probability of fraud from the results of the language model. Device according to one of the preceding claims, characterized in that the detection unit (50) comprises an acoustic feature analysis component configured to derive fraud indicators directly from the audio signal, wherein the acoustic feature analysis component analyzes at least one of the following features: prosodic features comprising melody, speech rate and pause pattern; voice biometric features comprising fundamental frequency and formant structure; emotion-related features comprising stress indicators and arousal level; and acoustic anomalies for the detection of synthetically generated voices. Device according to one of the preceding claims, characterized in that the data processing device (10) is designed as an application-specific single-chip system (SoC / ASIC) with integrated audio input stage on a single printed circuit board, wherein the audio acquisition unit (20), the processor and the memory are arranged on the same printed circuit board. Device according to one of the preceding claims, characterized in that the detection unit (50) comprises a knowledge-based lookup component configured to utilize local knowledge databases containing fraud patterns, regional fraud scenarios, or updatable rule sets to improve detection accuracy, wherein the knowledge databases are stored locally on the data processing unit (10) and can be updated independently of the AI ​​models. Device according to one of the preceding claims, characterized in that the detection unit (50) is designed as an agent-based system with a plurality of specialized AI subcomponents, wherein each AI subcomponent performs a subtask of fraud detection and the results of the subcomponents are transmitted to a central decision component. Device according to claim 25, characterized in that the downstream external processing component comprises a comparison with a central fraud pattern database, a refined fraud type classification or a statistical evaluation across multiple devices, and wherein the transmission is configurable or deactivatable by the user. Device according to one of the preceding claims, characterized in that the detection unit (50) comprises a call-spanning correlation component (58) configured to locally store anonymized or pseudonymized features of past communication processes and analyze patterns across multiple successive communication processes in order to detect repeated fraud attempts, escalating fraud patterns or coordinated fraudulent activities. Device according to one of the preceding claims, characterized in that the recognition unit (50) is configured for processing and analyzing audio communication in multiple languages, wherein the recognition unit comprises language-specific AI models or a multilingual AI model and a language identification component automatically determines the language used in the conversation. Device according to one of the preceding claims, characterized in that the recognition unit (50) is configured to learn locally from user feedback and to adapt at least the decision thresholds, weightings or model parameters on the basis of the feedback, wherein all learning and adaptation processes are performed locally on the data processing device (10). Device according to one of the preceding claims, characterized in that the audio communication comprises at least one of the following: an analog telephone call, a VoIP call, a messenger-based voice call, a video call, a voice message or a call via video conferencing software. Device for detecting fraudulent audio communication, comprising: a) a general data processing device, in particular a smartphone, tablet, personal computer or notebook, which forms the data processing unit (10); b) an audio acquisition unit (20) configured as a software interface, which acquires audio signals of an audio communication via an integrated microphone of the device, via an audio API of the operating system or via an interface to a communication application and supplies them to the data processing unit (10); c) an AI-based detection unit (50) executed locally as a software module on the data processing unit (10), which is configured to determine fraud indicators from the audio signals or text data derived therefrom and to calculate a probability of fraud;d) a warning unit (60) designed as a notification function of a software application which warns the user via the device's user interface when a threshold of the probability of fraud is exceeded; wherein the detection unit (50) is executed entirely locally on the data processing device (10) and the software application may be designed as a standalone application, as an extension module for an existing communication application, as a background service of the operating system, as a browser-based application or as an embeddable software library. Device according to one of the preceding claims, characterized in that the recognition unit (50) comprises a deepfake recognition component (59) configured to recognize synthetically generated or manipulated voices, wherein the deepfake recognition component analyzes acoustic features characteristic of synthetically generated speech, in particular spectral artifacts, unnatural transitions between phonemes, missing micromodulations of the voice, atypical fundamental frequency responses and inconsistencies between speech content and prosodic features, and wherein the result of the deepfake recognition is incorporated into the fraud probability determination as an additional fraud indicator. Device according to one of the preceding claims, characterized in that the recognition unit (50) is configured to use, in addition to the audio signals and the text data derived therefrom, further context data as input variables for fraud probability determination, wherein the context data comprises at least one of the following: telephone number information and caller identification data, connection metadata comprising call time, call duration and call frequency, network information comprising country of origin or network type of the connection, a comparison with a local contact book of the user to distinguish between known and unknown callers, and environment information comprising time of day or day of the week, wherein the context data is processed locally on the data processing device (10). Device according to one of the preceding claims, characterized in that the device comprises an update unit configured to receive AI models, knowledge databases or rule sets of the recognition unit (50) via a unidirectional update path, wherein the update path is configured such that only update data from an external source is transferred to the data processing unit (10) and no conversation data, transcripts, analysis results or other user-related data are transmitted from the data processing unit (10) to the external source. Device for detecting fraudulent audio communication, comprising: a) a data processing unit (10) with at least one processor and a memory; b) an audio acquisition unit (20) configured to acquire audio signals from an audio communication and supply them to the data processing unit (10); c) an AI-based detection unit (50) executed on the data processing unit (10), configured to determine a plurality of predefined fraud indicators from the audio signals or text data derived therefrom and to calculate a fraud probability from the weighted fraud indicators, wherein the detection unit (50) is configured to dynamically adjust the decision threshold to at least one context-dependent parameter; d) a warning unit (60) that warns the user when the decision threshold is exceeded. Use of a device according to any one of claims 1 to 40 for detecting fraudulent audio communication, in particular for detecting grandparent scam fraud, shock calls, impersonating police officers, fake bank employee fraud, technical assistance fraud, investment fraud, sweepstakes fraud or fraud using synthetically generated voices.

Images