Report generation system, report generation method, and program

Abstract

The present invention provides a system that, in response to a detection signal related to an abnormal event, can align time-series data obtained from multiple data sources based on a single time reference and generate structured report data including time reference information related to the alignment. [Solution] Equipped with a report generation unit, The report generation unit, in response to a detection signal related to an abnormal event, acquires time-series data from multiple data sources, aligns the time-series data based on a single time criterion, and generates structured report data including time reference information related to the alignment. Report generation system.

Description

Technical Field

[0001] The present invention relates to information processing technology and system monitoring technology, and particularly to a reporting generation system, a reporting generation method, and a program that, in response to a detection signal regarding an abnormal event, acquire time-series data obtained from a plurality of data acquisition sources, align the time-series data based on a single time reference, and generate structured report data including time reference information related to the alignment.

Background Art

[0002] In a computer system, a control system, etc., when an abnormal event such as a failure, an abnormality, an accident, or an operational deviation occurs, in order to analyze the cause of the abnormal event, prevent recurrence, conduct an audit, or notify an external system, it is required to generate a report regarding the abnormality.

[0003] This type of report is not limited to those based on a single information source, and is often created by referring to time-series data obtained from a plurality of data acquisition sources (for example, a plurality of sensors, device logs, application logs, or acquired data from external devices, etc.).

[0004] On the other hand, the time-series data obtained from a plurality of data acquisition sources may have different time systems for each acquisition source, different acquisition timings and delays, and different data formats, etc., making it not easy to appropriately extract the sections related to the abnormal event and associate them with each other to summarize them as a report.

[0005] For example, if it is necessary to manually adjust the time shift between a plurality of acquisition sources or perform a separate resynchronization process, the burden of report generation increases, and the reproducibility and traceability of post-mortem analysis may also decrease.

[0006] If the report created after the detection of an abnormal event does not sufficiently include reference information regarding time, it may become difficult to track and reproduce the states of each data acquisition source before and after the abnormal event time.

[0007] Furthermore, systems that generate reports in response to abnormal events vary depending on the monitored target and operating environment, and it is sometimes desirable to output the generated reports in a certain structured format so that they can be reused in other systems or verified during audits.

[0008] Techniques for generating reports in response to the detection of abnormal events are known (for example, Patent Document 1).

[0009] Patent Document 1 discloses, for example, a technology that integrates notifications from multiple partitions, etc., in response to the detection of a fault and generates a fault report.

[0010] However, aligning time-series data obtained from multiple data sources based on a single time criterion and generating structured report data that includes time reference information related to that alignment remains a valuable challenge. [Prior art documents] [Patent Documents]

[0011] [Patent Document 1] Japanese Patent Publication No. 2008-165347 [Overview of the Initiative] [Problems that the invention aims to solve]

[0012] When creating reports on abnormal events using time-series data obtained from multiple data sources, it can be difficult to match the time-series data due to time differences between data sources, which can make report creation and subsequent tracking and verification time-consuming. Furthermore, if the generated report is not structured in a way that makes it easily reusable in other systems, it can be difficult to use it in a way that is suitable for auditing and external collaboration. Therefore, the present invention aims to enable the generation of structured report data that includes time reference information related to the matching process by aligning multiple time-series data based on a single time criterion in response to detection signals related to abnormal events. [Means for solving the problem]

[0013] Equipped with a report generation unit, The report generation unit, in response to a detection signal related to an abnormal event, acquires time-series data from multiple data sources, aligns the time-series data based on a single time criterion, and generates structured report data including time reference information related to the alignment. Report generation system.

[0014] The report generation system further includes an output unit that outputs the structured report data generated by the report generation unit to an external device or records it to a recording medium. Report generation system.

[0015] In the report generation system, the time reference information includes at least one of the following: start time information indicating the start of the time series data alignment interval, end time information indicating the end of the alignment interval, window identification information identifying the interval or analysis window used for alignment, data acquisition source identification information identifying the data acquisition source, and time offset information for each data acquisition source. Report generation system.

[0016] In the report generation system, the time reference information further includes a consistency error index that represents the consistency error between the multiple data acquisition sources. Report generation system.

[0017] In the report generation system, the time reference information further includes synchronization source identification information that identifies the synchronization source that provides the single time reference, and at least one of the switching history of the synchronization source. Report generation system.

[0018] In the report generation system, the time reference information further includes time quality metadata representing at least one of jitter or drift. Report generation system.

[0019] In the report generation system, the time quality metadata includes distribution statistics regarding at least one of the time deviation, the jitter, or the drift among the plurality of data acquisition sources, and the distribution statistics include at least one of a quantile, a maximum value, and a representative value. Report generation system.

[0020] In the report generation system, the time reference information further includes compensation information for compensating the time deviation for each of the plurality of data acquisition sources. Report generation system.

[0021] In the report generation system, the report generation unit divides the aligned time-series data into a plurality of analysis windows, assigns window identification information for identifying each analysis window as the time reference information, and the analysis window is at least one of a fixed-length window, a quasi-fixed-length window, and an event-driven variable-length window based on the abnormal event. Report generation system.

[0022] In the report generation system, the report generation unit obtains or estimates a stimulus series regarding an external stimulus and a response series regarding a response to the external stimulus based on the time-series data obtained from the plurality of data acquisition sources, calculates a degree-of-agreement index representing the degree of agreement between the stimulus series and the response series, the structured report data includes the degree-of-agreement index, the process of calculating the degree-of-agreement index includes at least one of a process based on cross-correlation in the time domain, a process based on coherence in the frequency domain, and a process based on correlation in time-frequency analysis. Report generation system.

[0023] In the report generation system, when obtaining the time-series data, the report generation unit assigns a time tag based on the single time reference. Report generation system.

[0024] In the report generation system, the report generation unit calculates an input quality index for the aligned time-series data. Report generation system.

[0025] In the report generation system, the input quality index is a composite index based on at least two of the evaluation values ​​related to signal noise, loss, and gate fit. Report generation system.

[0026] In the report generation system, the report generation unit does not use data that has been determined to be of low quality based on the input quality index for generating the structured report data. Report generation system.

[0027] In the report generation system, the structured report data includes at least one of the input quality index and exclusion reason information corresponding to the data determined to be of low quality. Report generation system.

[0028] In the report generation system, the multiple data sources include information belonging to at least two groups from among movement, behavior, consciousness, autonomous response, gaze, voice, operation, and environment. Report generation system.

[0029] In the report generation system, the report generation unit extracts features from the aligned time-series data, and the structured report data includes the features. Report generation system.

[0030] In the report generation system, the report generation unit calculates an anomaly probability index for the anomaly event based on the aligned time-series data or the feature quantities, and the structured report data includes the anomaly probability index. Report generation system.

[0031] In the report generation system, the anomaly likelihood index is a value obtained by a calculation process that includes a monotonically non-decreasing aggregation process in the direction of increasing anomaly likelihood, and a monotonic transformation process on the result of the aggregation process, and belongs to the anomaly likelihood index genus, which treats anomaly likelihood indices with different expressions due to differences in the monotonic transformation process as equivalent anomaly likelihood indices. Report generation system.

[0032] In the report generation system, the monotonic transformation process includes at least one of the transformations corresponding to rescaling, probabilistic calibration, and reparameterization. Report generation system.

[0033] In the report generation system, the calculation of the anomaly index is performed by at least one of the following methods: calculation using a machine learning model, calculation using rule-based processing, and calculation using a combination thereof. Report generation system.

[0034] In the report generation system, the report generation unit calculates the anomaly index based on at least some of the available data from the multiple data acquisition sources. Report generation system.

[0035] In the report generation system, the structured report data includes at least one of the reason code associated with the anomaly likelihood index, or a note relating to the anomaly likelihood index. Report generation system.

[0036] In the report generation system, The structured reporting data includes, as source information for calculating the anomaly probability index, at least one of the following: model identification information, rule set identification information, threshold or parameter version control information, learning date and time information, and applied profile identification information. Report generation system.

[0037] In the report generation system, the report generation unit extracts the time-series data for a predetermined interval related to the time of the abnormal event associated with the detection signal, specifically the interval before and after the time of the abnormal event, and includes it in the structured report data. Report generation system.

[0038] In the report generation system, the report generation unit sets a plurality of candidate intervals as the predetermined interval, and selects an interval to be adopted from among the plurality of candidate intervals based on an evaluation of at least one of the input quality, stability, or degree of missing data for each candidate interval. Report generation system.

[0039] In the report generation system, the report generation unit aligns at least one of the start time and end time of the extracted interval with a delimiter boundary (including frame boundary, encoding boundary, or data chunk boundary) of the time series data or media data related to the time series data. Report generation system.

[0040] In the report generation system, The structured reporting data includes reference information to at least one waveform data or media data corresponding to the aligned time-series data, the reference information is associated with at least one time reference information or anomaly index, The aforementioned reference information includes a plurality of reference formats that can be selected according to the capabilities or configuration policy of the receiving party, and the plurality of reference formats include at least one of the reference formats corresponding to Media, DocumentReference, and Binary in FHIR. Report generation system.

[0041] In the report generation system, The structured report data further includes readable summary document data generated based on the information contained in the structured report data, The aforementioned summary document data includes at least one format, such as PDF or HTML, and has language identification information including language tags, and the presentation language is switched according to the settings of the delivery destination or recipient. Report generation system.

[0042] In the report generation system, the output unit delivers the structured report data via a network, and the delivery is protected by encryption and authentication of the recipient. Report generation system.

[0043] In the report generation system, the output unit encrypts and temporarily stores the structured report data when delivery is impossible, and resends the structured report data when delivery is possible. Report generation system.

[0044] In the report generation system, the output unit performs retransmission control based on the receipt acknowledgment from the recipient, and if the receipt acknowledgment is not received, it switches to an alternative recipient or alternative delivery route and continues delivery. Report generation system.

[0045] The report generation system further includes a tamper-resistant storage unit that stores at least one of the structured report data, the time reference information, or the log relating to the generation or delivery of the structured report data in a tamper-resistant form, wherein the tamper-resistant form includes at least one of storage on an appendable recording medium or storage using a chained hash structure. Report generation system.

[0046] The report generation system further includes an audit interface that outputs information or results of integrity verification that allows for verification of the integrity of the data stored by the tamper-proof storage unit. Report generation system.

[0047] The report generation system further comprises an anonymization unit that performs anonymization processing on data included in or referenced from the structured report data, wherein the anonymization processing includes at least one of the following: anonymization of video data, anonymization of location information, and anonymization of audio information. Report generation system.

[0048] In the report generation system, the anonymization unit performs the anonymization process based on a profile that defines the anonymization strength, records a log related to the anonymization process, or separates and manages key information that can be used to restore the anonymization process. Report generation system.

[0049] The report generation system further comprises a monitoring unit that detects no response or abnormality in the processing related to the generation of the structured report data, and a notification unit that notifies the system of an unconfirmed abnormality in response to the detection of the no response or abnormality. Report generation system.

[0050] The report generation system further includes a safety operation unit that, in response to notification of an undetermined anomaly, activates a safety operation that transitions the device on which the report generation system is installed or the device controlled by the report generation system to the safety side. Report generation system.

[0051] In the report generation system, the structured report data includes screening information for candidate causes related to the abnormal event, and the screening information includes a value indicating one of the following for each candidate cause: affirmative, negative, or unknown. Report generation system.

[0052] The report generation system further includes a policy profile switching unit that switches at least one of the data items to be included in the structured report data, the intensity of the anonymization process, or the output destination of the structured report data, according to the policies of the jurisdiction or organization. Report generation system.

[0053] The process includes: acquiring time-series data from multiple data sources in response to a detection signal related to an abnormal event; aligning the time-series data based on a single time reference; and generating structured report data including time reference information related to the alignment. Report generation method.

[0054] A program that instructs a computer to execute a report generation method. [Effects of the Invention]

[0055] According to the present invention, in response to a detection signal related to an abnormal event, time-series data obtained from multiple data sources can be aligned based on a single time reference, and structured report data including time reference information related to the alignment can be generated. This facilitates the creation of reports on abnormal events and subsequent tracking and verification, and enables the generation of reports suitable for reuse in other systems and audits. [Brief explanation of the drawing]

[0056] [Figure 1] This block diagram shows the overall configuration of a report generation system that generates structured report data in response to detection signals related to abnormal events. [Figure 2] This is a flowchart showing the main flow of the report generation process. [Figure 3] This flowchart shows the detailed flow of the processes related to the alignment of time-series data, the generation of analysis windows, and the generation of time reference information (S3 and S4 in Figure 2). [Figure 4] This flowchart shows the detailed flow of calculating input quality indicators and handling data judged to be of low quality (S5 in Figure 2). [Figure 5] This flowchart shows the detailed flow of feature extraction and calculation of anomaly probability indices (S6 and S7 in Figure 2). [Figure 6] This flowchart shows the detailed flow of calculating the agreement index based on the stimulus and response sequences (S8 in Figure 2). [Figure 7] This flowchart shows the detailed flow of generating structured reporting data (S9 in Figure 2). [Figure 8] This flowchart shows the detailed flow of outputting, delivering, recording, anonymizing, tamper-proofing, and providing an audit interface for structured reporting data (S10 in Figure 2). [Figure 9] This flowchart shows the detailed flow of the process (S11 in Figure 2) that notifies an unconfirmed anomaly and initiates safe operation when an unresponsive or abnormal process related to the generation of structured report data is detected. [Modes for carrying out the invention]

[0057] In this specification, unless otherwise specified, the following terms have the meanings set forth below. The following definitions and examples are not intended to limit the present invention.

[0058] A report generation system is a system that, in response to a detection signal related to an abnormal event, acquires time-series data from multiple data sources, aligns this time-series data based on a single time reference, and generates structured report data that includes time reference information related to the alignment. A report generation system may be configured as, for example, an in-vehicle device, a mobile terminal, a stationary device, a server device, or a combination thereof.

[0059] The report generation unit refers to a functional unit (hardware, software, or a combination thereof) that performs at least the functions of acquiring time-series data, aligning time-series data, generating time reference information, and generating structured report data. The output unit refers to a functional unit that outputs structured report data to an external device, delivers it via a network, or records it on a recording medium, and may be optionally provided.

[0060] An abnormal event refers to a predetermined abnormal state, abnormal sign, or suspected abnormal event in the monitored system. Abnormal events may include, for example, changes in the driver's or user's condition, changes in operating conditions, changes in environmental conditions, or deviations from sensor values.

[0061] A detection signal is a signal indicating the occurrence, possibility, or suspicion of an abnormal event, and may include, for example, threshold judgment results, classification results, alerts, triggers, or notifications from external systems. A detection signal may be implemented as an analog signal, a digital signal, a message, an interrupt, or an event.

[0062] A data source is an information source that generates or provides time-series data, and may include, for example, sensors, device logs, application logs, network logs, external databases, or data received from other devices. A data source may include information belonging to at least two of the following groups: movement, behavior, consciousness, autonomous responses, gaze, voice, operation, and environment.

[0063] Time-series data refers to a series of data obtained over time, and may include waveforms, numerical sequences, event sequences, frame sequences (images or videos), audio sample sequences, or combinations thereof. Time-series data is not limited to sample sequences acquired at regular intervals; it may also be event-driven data with unequal intervals.

[0064] A single time reference refers to a time reference that serves as a basis for comparing time-series data from multiple data sources on the same time axis. A single time reference can be provided, for example, by PTP (IEEE 1588), PPS+PLL, gPTP (TSN), or GNSS synchronization. Alternatively, it may be provided by, for example, time tagging in an in-vehicle bus (CAN-FD, etc.), time synchronization implemented by software, optical or other physical layer synchronization, or a combination of a highly stable oscillator and temperature compensation.

[0065] A time tag refers to time-related information attached to time-series data or elements of such time-series data (samples, frames, events, chunks, etc.), and may include absolute time, relative time, or equivalent information based on a single time standard. Time tags may be attached at the time of acquisition, or they may be estimated or corrected after acquisition.

[0066] Alignment refers to the process of mapping time-series data obtained from multiple data sources based on a single time criterion. Alignment may include, for example, timestamp-based relocation, interpolation, resampling, delay compensation, missing data compensation, or re-establishment of synchronization.

[0067] An analysis window refers to a time interval defined as a unit for processing or evaluating aligned time-series data, and may include fixed-length windows, semi-fixed-length windows, or event-driven variable-length windows based on abnormal events. The analysis window may be configured as a sliding window with overlap.

[0068] Window identification information refers to information used to identify an analysis window, and may include, for example, a windowId, a serial number, a UUID, or a combination of start and end times.

[0069] Time reference information refers to reference information related to synchronization, which includes at least one of the following: start time information indicating the start of the synchronization target section, end time information indicating the end of the synchronization target section, window identification information identifying the section or analysis window used for synchronization, data acquisition source identification information identifying the data acquisition source, and time offset information for each data acquisition source. Time reference information can be expressed, for example, as utcStart, utcEnd, windowId, sourceId, offsetMs, etc.

[0070] A consistency error index is an index that represents the consistency error between multiple data sources, and may include, for example, the absolute value, mean, maximum value, or quantile of the time difference. The consistency error index may be assigned to each analysis window, or it may be assigned as a statistic for a predetermined period.

[0071] Synchronization source identification information refers to information that identifies a synchronization source that provides a single time reference, and synchronization source switching history refers to the history of synchronization source switching (switching time, identifiers before and after switching, reason for switching, etc.). Synchronization source identification information and switching history may be included in time reference information.

[0072] Time quality metadata refers to metadata that represents the quality of a single time standard or consistency, and may include information representing at least one of jitter or drift. The time quality metadata may include distribution statistics relating to time differences between multiple data sources, at least one of the jitter or drift, and the distribution statistics may include at least one of quantiles (e.g., p50, p95, etc.), maximum value, and / or representative value (e.g., mean, median, or mode, etc.).

[0073] Compensation information refers to information used to compensate for time differences between multiple data sources, and may include, for example, compensation vectors, compensation coefficients, or compensation lookup tables. Compensation information may be included in time reference information.

[0074] An input quality index is an index used to evaluate the quality of consistent time-series data. For example, it may be a composite index based on at least two of the following evaluation values: signal noise, data loss, and gate fit. The input quality index may be calculated for each analysis window, or it may be calculated as a statistic over a predetermined interval.

[0075] Data deemed to be of low quality refers to time-series data or analysis windows that are deemed inappropriate for use in generating structured reporting data, such as when the input quality indicators do not meet the prescribed conditions. Exclusion reason information refers to information indicating the reason for the low quality determination, and may include, for example, excessive missing data, saturation, excessive disturbance, poor exposure, and reduced tracking reliability.

[0076] Features are characteristics extracted from consistent time-series data. Examples of features include statistics, frequency band features, time-frequency features, small-wave features, cepstrum features, graph-structure-based features, or combinations thereof.

[0077] An anomaly probability index is a measure that represents the likelihood of an anomaly event being anomaly, calculated based on consistent time-series data or features. It can be expressed as a score, probability, or equivalent value. The anomaly probability index is not limited to a single value and may be expressed as a set of multiple components.

[0078] Monotonically non-decreasing aggregation refers to aggregation processing configured such that the aggregation result does not decrease when at least a portion of the input data or features change in a direction that increases the likelihood of anomalies. This is achieved by ensuring monotonically non-decreasing properties through methods such as signing input features, imposing constraints on coefficients or rules, or learning with monotonic constraints. Monotonically non-decreasing aggregation processing can be implemented, but is not limited to, non-negative weighted sums, monotonically constrained generalized additive models, linear models with non-negative constraints on coefficients (such as logistic regression or linear SVM), monotonically constrained decision tree models (such as gradient-boosted decision trees), isotonic regression, or neural networks configured or trained to satisfy monotonicity.

[0079] A monotonic transformation is a transformation applied to the result of an aggregation process, and may include transformations equivalent to, for example, rescaling, probabilistic calibration, or reparameterization. Monotonic transformations can be implemented, for example, by logistic maps, isotonic maps, or affine maps.

[0080] An anomaly probability index is a concept in which an anomaly probability index I can be expressed as a value obtained by mapping the output of a monotonically non-decreasing aggregation process g(·) on time-series data or evidence vectors E based on features to a monotonically increasing or monotonically non-decreasing transformation process φ(·). That is, it can be expressed as I=φ(g(E)). Here, if indices I1=φ1(g(E)) and I2=φ2(g(E)) obtained using different transformation processes φ1(·) and φ2(·) on the same aggregation process g(·) exist in a relationship where there exists a monotonically increasing or monotonically non-decreasing mapping ψ that satisfies I2=ψ(I1), then I1 and I2 can be treated as anomaly probability indices with the same meaning to the extent that the ranking or threshold judgment results of anomaly probability coincide (this is the meaning of "equivalent" in this specification). Therefore, anomaly probability indices that differ in expression due to differences equivalent to differences in monotonic transformation processes φ(·), such as rescaling, probabilistic calibration, and reparameterization, can be treated as equivalent anomaly probability indices. Furthermore, even if the representation of feature quantities is replaced due to differences in representation in the time domain, frequency domain, time-frequency domain, small waves, cepstrums, graphs, etc., they can be treated as equivalent as long as the meaning regarding the ranking or thresholding of anomaly probability in the sense described above remains the same. The genus of anomaly probability indices can be applied, for example, to seizure-evidence indices for seizures such as epilepsy, and to the group of equivalent indices for said indices.

[0081] A machine learning model refers to a process that performs inference using a pre-trained model, and may include, for example, neural networks, decision tree models, linear models, or combinations thereof. Rule-based processing refers to a process that generates output based on predetermined rules, conditional branches, or state transitions, and the calculation of anomaly probability index may be performed by machine learning models, rule-based processing, or a combination thereof.

[0082] A reason code is a machine-readable code that expresses the rationale associated with an anomaly index, while a note is a human-readable short sentence or explanatory information about the anomaly index. Reason codes or notes may include key features, key rules, or cautionary content.

[0083] Origin information refers to information related to the calculation of anomaly probability index, and includes at least one of model identification information, rule set identification information, and parameter version control information. Origin information may include, for example, model ID, threshold version, parameter version, training date, or applicable profile identifier.

[0084] A stimulus series refers to a sequence of external stimuli included in time-series data, and may include, for example, fluctuations in ambient light, auditory stimuli, visual patterns, or periodic disturbances. A response series refers to a sequence of responses to a stimulus series, and may include, for example, gaze, blinking, head movements, changes in manipulation, autonomous responses, or changes in vocalizations. Stimulus series and response series may be acquired directly or generated by estimation.

[0085] An agreement index is an index that represents the degree of agreement between a stimulus sequence and a response sequence, and can be calculated by at least one of the following methods: a cross-correlation-based process, a frequency-domain coherence-based process, or a correlation-based process based on time-frequency analysis. The agreement index can be expressed as a score, probability, or normalized value. Furthermore, the agreement index can be used as a Reflex Trigger Index to examine potential causes related to reflexive events or extrinsic stimuli.

[0086] Structured reporting data refers to reporting data that includes time reference information and stores information related to abnormal events as fields or elements according to a predetermined data model or schema. Structured reporting data can be represented, for example, in JSON, XML, CBOR, or database records. Furthermore, in the case of integration with a medical information system, it may be configured as an FHIR R4 / R5 Bundle (type=document or collection) including DiagnosticReport, Observation groups, Media / DocumentReference / Binary, etc., and may also be configured as DICOM-SR. In addition, as a fallback when the predetermined standard is difficult to use depending on the receiving environment, a format such as HL7 v2 ORU^R01 or a readable document format such as PDF / HTML may be included. These reference formats or output formats may be selected from multiple formats depending on the receiving side's capabilities or configuration policy.

[0087] Reference information refers to information indicating a reference to at least one waveform data or media data corresponding to the aligned time-series data, and may include, for example, a URI, identifier, hash, or storage address. The reference information may be associated with at least one time reference information or anomaly index, for example, a time range, window identification information, or index calculation interval.

[0088] Readable summary document data refers to human-readable document data generated based on information contained in structured reporting data, and may include, for example, PDF, HTML, or equivalent formats. Language identification information refers to language information attached to summary document data or structured reporting data, and may include, for example, language tags. Summary document data may include a mechanism that switches the presentation language according to the recipient's settings or policies.

[0089] Delivery refers to the process of transmitting structured reporting data over a network and ensuring it reaches its destination. Encryption refers to the process of encrypting and protecting data during delivery or storage, and authentication refers to the process of verifying the legitimacy of the delivery source or destination. Delivery can be protected by, for example, TLS 1.2 or higher communication, server certificate verification, certificate pinning, or mutual authentication, but is not limited to these.

[0090] Encrypting, temporarily storing, and resending refers to the process of encrypting and temporarily storing structured report data when delivery is impossible, and resending it when delivery is possible (so-called Store & Forward). Receipt acknowledgment refers to a response from the recipient acknowledging receipt (may include ACK, NAK, etc.), and retransmission control refers to the process of controlling whether retransmission is necessary or the scope of retransmission based on the receipt acknowledgment. Switching to an alternative destination or alternative delivery route refers to the process of switching the destination or route under predetermined conditions to continue delivery.

[0091] Storing data in a tamper-resistant manner means storing it in a manner that suppresses or makes detectable data modification after storage, and may include storage on an appendable recording medium or storage using a chained hash structure. An appendable recording medium may include, for example, a WORM medium. A chained hash structure may include, for example, a structure in which each record holds its own hash and the hash of the previous record and is chained together.

[0092] An audit interface refers to an interface for outputting information that allows verification of the integrity of data stored by a tamper-proof storage unit, or for outputting the results of integrity verification. The audit interface may include, for example, outputting a chained hash structure record sequence (record identifier, previous hash, hash, timestamp, data hash, etc.) in a format such as JSON, and including the verification results. Alternatives to tamper resistance may include using TPM signing, remote notarization, or a timestamp authority.

[0093] Anonymization refers to the process of reducing information that could identify an individual or subject in data contained in or referenced from structured reporting data, and may include at least one of the following: anonymization of video data, anonymization of location information, and anonymization of audio information. Anonymization may include, for example, blurring or mosaicing facial areas, rounding or offsetting location coordinates, or deleting or transforming speaker IDs.

[0094] An anonymization strength profile refers to a profile that defines the strength or scope of anonymization, and an anonymization log refers to a log relating to the execution history, scope, or mask area of ​​the anonymization process. Separate management of key information refers to managing key information that can be used to restore the anonymized data in a separate area or a separate management system from the data to be anonymized. The anonymization process may be combined with the separate storage of personally identifiable information (PII) as needed.

[0095] Screening information refers to information that includes the screening results for candidate causes of abnormal events, and may include values ​​indicating affirmative, negative, or unknown for each candidate cause. Screening information may be included in structured reporting data with the addition of time period information, supporting notes, or reference information. In medical applications, screening information may be structured as acute symptomatic screening.

[0096] A policy profile is a set of settings that correspond to the policies of a jurisdiction or organization and can be used to switch at least one of the data items to be included in structured reporting data, the intensity of anonymization processing, or the destination of structured reporting data.

[0097] No response or abnormality refers to a state in which the process for generating structured report data does not respond within a predetermined time or deviates from predetermined conditions, and may include, for example, timeout, NaN value, distribution deviation, synchronization failure, failure to generate calculation results, or model corruption.

[0098] An unconfirmed anomaly refers to the handling of an anomaly state that is assigned to ensure safety even when an anomaly event cannot be confirmed, in the event of no response or detection of an anomaly. A safe operation refers to an operation that transitions a device equipped with a report generation system or a device controlled by a report generation system to a safe state in response to notification of an unconfirmed anomaly, and may include, for example, transition to a restricted mode, stop, alarm, or external notification.

[0099] Referring to Figure 1, the configuration of a report generation system according to one embodiment of the present invention will be described. Figure 1 is a block diagram represented by lines showing the connection relationships between each component, and each block is shown as a functional block.

[0100] The report generation system (100) comprises at least a report generation unit (110). The report generation system (100) may be configured, for example, as an in-vehicle device, a mobile terminal, a stationary device, a server device, or a combination thereof. The report generation system (100) may be implemented in a single device or its functions may be distributed among multiple devices.

[0101] The report generation system (100) can receive detection signals (101) related to abnormal events. The detection signals (101) may be generated internally by the report generation system (100), or they may be input from an external detector, another device, or an external system. The implementation form of the detection signals (101) may be, for example, a message, an event, an interrupt, or a data packet.

[0102] The report generation system (100) can acquire time-series data obtained from multiple data acquisition sources (102). The data acquisition sources (102) are, for example, sources that generate or provide information belonging to at least two groups from among movement, behavior, consciousness, autonomous response, gaze, voice, operation, and environment, and may include sensors, device logs, application logs, external databases, data received from other devices, etc. The data acquisition sources (102) may be built into the report generation system (100), or they may be external devices connected to the report generation system (100) by communication or bus connection, etc.

[0103] The report generation unit (110) responds to a detection signal (101) related to an abnormal event by acquiring time-series data from multiple data acquisition sources (102), aligning the time-series data based on a single time reference, and generating structured report data including time reference information related to the alignment. The report generation unit (110) may be configured as a software function implemented by, for example, a processor and a memory device, but it may also be implemented as a dedicated circuit, a programmable circuit, or a combination thereof.

[0104] The report generation unit (110) can functionally perform at least the following: (i) acquisition of time series data, (ii) setting a single time criterion or using an externally set single time criterion, (iii) consistency of the time series data, (iv) generation of time reference information, and (v) generation of structured report data. Furthermore, the report generation unit (110) may optionally perform at least one of the following: assigning time tags based on the single time criterion, generating an analysis window, calculating an input quality index and excluding low-quality data, extracting features, calculating an anomaly probability index or a similarity index, and generating reason codes, notes, or origin information associated therewith. These functions may be implemented as a single module or divided into multiple submodules.

[0105] Structured reporting data may include at least one time reference information. This time reference information may include, for example, start time information indicating the start of the matching interval, end time information indicating the end of the matching interval, window identification information identifying the analysis window, data source identification information identifying the data source, and time offset information for each data source. The time reference information may also optionally include at least one of the following: matching error index, synchronization source identification information, synchronization source switching history, time quality metadata (such as jitter or drift), and compensation information.

[0106] The report generation system (100) may optionally further include an output unit (120). The output unit (120) may output the structured report data generated by the report generation unit (110) to an external device, deliver it via a network, or record it on a recording medium, at least one of these.

[0107] The output destination of the structured report data output, delivered, or recorded by the output unit (120) may be, for example, an external device / recording medium / delivery destination (900). The external device / recording medium / delivery destination (900) may include, for example, a server device, a cloud service, an electronic recording system, a monitoring center, an analysis terminal, storage, or a mobile terminal.

[0108] Delivery by the output unit (120) can be protected by encryption and authentication of the recipient. The output unit (120) may also have a function to encrypt and temporarily store structured report data when delivery is impossible and resend it when delivery is possible (so-called Store & Forward). Furthermore, the output unit (120) may have a function to control retransmission based on an acknowledgment of receipt from the recipient, and to continue delivery by switching to an alternative recipient or alternative delivery route if an acknowledgment of receipt is not obtained.

[0109] The report generation system (100), although not shown in Figure 1, may further include functional units to realize each arbitrary configuration. For example, the report generation system (100) may include a policy profile switching unit (130) that switches at least one of the data items to be included in the structured report data, the intensity of the anonymization process, or the output destination of the structured report data, depending on the jurisdiction or organization's policy.

[0110] The report generation system (100) may include an anonymization unit (140) that performs anonymization processing on data included in or referenced from structured report data. The anonymization unit (140) may perform at least one of the following: anonymization processing based on a profile that defines the anonymization strength, recording logs related to the anonymization processing, or separating and managing key information that can be used to restore the anonymized data.

[0111] The report generation system (100) may include a tamper-resistant storage unit (150) that stores at least one of structured report data, time reference information, or logs relating to the generation or delivery of structured report data in a tamper-resistant form. The tamper-resistant form may include, for example, storage on an appendable recording medium or storage using a chained hash structure.

[0112] The report generation system (100) may include an audit interface (160) that outputs information or results of integrity verification that allows for verification of the integrity of data stored by the tamper-proof storage unit (150).

[0113] The report generation system (100) may include a monitoring unit (170) that detects no response or abnormality in the processing related to the generation of structured report data, and a notification unit (180) that notifies the system of an unconfirmed abnormality in response to the detection of no response or abnormality. Furthermore, the report generation system (100) may include a safety operation unit (190) that activates a safety operation that transitions the device on which the report generation system (100) is installed or the device controlled by the report generation system (100) to the safety side in response to the notification of an unconfirmed abnormality.

[0114] The division, integration, arrangement, and implementation configuration of each component shown in Figure 1 are merely examples, and the present invention is not limited to these. For example, the report generation unit (110) and the output unit (120) may be configured within the same device, or they may be distributed across separate devices. Furthermore, some functions of the external device / recording medium / delivery destination (900) shown in Figure 1 may be included in the report generation system (100), or some functions of the report generation system (100) may be implemented on the external device side.

[0115] Referring to Figure 2, the main flow of the report generation process according to one embodiment of the present invention will be described. Figure 2 is a flowchart showing an example of a process that can be executed by the report generation system (100), and each step can be executed, for example, by the report generation unit (110). Note that the order of steps, branching conditions, number of repetitions, and the entity executing each step shown in Figure 2 are examples, and the present invention is not limited to these.

[0116] In step S1 of Figure 2, the report generation system (100) receives a detection signal related to an abnormal event (S1). The detection signal may be generated by a detector within the report generation system (100) or may be input from an external device. Upon receipt of the detection signal, the report generation process may be initiated.

[0117] In step S2, the report generation system (100) acquires time-series data obtained from multiple data acquisition sources (102) (S2). The acquisition of time-series data may be initiated in response to a detection signal, or it may be read from data continuously accumulated by a ring buffer or the like to obtain a predetermined range.

[0118] In process S3, the report generation system (100) aligns the acquired time-series data based on a single time criterion (S3). A specific example of alignment can be performed according to the detailed flow (S3-1 to S3-3) shown in Figure 3.

[0119] In step S4, the report generation system (100) generates time reference information related to the consistency (S4). The time reference information may include at least one of the following: start time information, end time information, window identification information, data acquisition source identification information, and time offset information for each data acquisition source. In addition, step S4 may optionally include a process of dividing the consistent time series data into multiple analysis windows and assigning window identification information to identify each analysis window. A specific example of step S4 can be executed according to the detailed flow (S4-1 to S4-3) shown in Figure 3.

[0120] In step S5, the report generation system (100) calculates an input quality index for the consistent time-series data and does not use data determined to be of low quality for generating structured report data (S5). Step S5 is an arbitrary step, and for example, it may be a mode in which the input quality index is not calculated, or a mode in which the calculation results are used only for recording. A specific example of step S5 can be performed according to the detailed flow (S5-1 to S5-3) shown in Figure 4.

[0121] In step S6, the report generation system (100) extracts features from the aligned time-series data (S6). Step S6 is an arbitrary step, and the extracted features can be included in the structured report data. A specific example of step S6 can be performed according to the detailed flow (S6-1) shown in Figure 5.

[0122] In step S7, the report generation system (100) calculates an anomaly probability index for an anomaly event based on the aligned time-series data or features (S7). Step S7 is an arbitrary step, and the anomaly probability index may be obtained by a calculation process that includes a monotonically non-decreasing aggregation process and a monotonically transforming process on the results of the aggregation process. In addition, reason codes, notes, or origin information may be generated in association with the calculated anomaly probability index and included in the structured report data. A specific example of step S7 may be performed according to the detailed flow (S7-1, S7-2) shown in Figure 5.

[0123] In step S8, the report generation system (100) calculates a degree of agreement index (S8) that represents the degree of agreement between the stimulus series relating to an extrinsic stimulus included in the time series data and the response series relating to the response to said extrinsic stimulus. Step S8 is an arbitrary step, and the calculation of the degree of agreement index may include at least one of the following: processing based on cross-correlation in the time domain, processing based on coherence in the frequency domain, and processing based on correlation based on time-frequency analysis. A specific example of step S8 may be performed according to the detailed flow (S8-1, S8-2) shown in Figure 6.

[0124] In step S9, the report generation system (100) generates structured report data (S9). The structured report data may include at least time reference information. The structured report data may also optionally include at least one of the following: input quality index, exclusion reason information, feature quantities, anomaly probability index, match index, reason code or note, origin information, reference information, readable summary document data and language identification information, and screening information. A specific example of step S9 may be performed according to the detailed flow (S9-1 to S9-4) shown in Figure 7.

[0125] In process D1, the report generation system (100) determines whether there is any unresponsiveness or abnormality in the processing related to the generation of structured report data (D1). Unresponsiveness or abnormality may include, for example, a state in which processing is not completed within a predetermined time, a state in which predetermined validity conditions are not met, or a synchronization failure. The determination in process D1 may be performed, for example, by a monitoring unit (170), but is not limited thereto.

[0126] If the result of the determination in step D1 is "no," that is, if no response or abnormality is detected, the report generation system (100) proceeds to step S10. In step S10, the report generation system (100) outputs, delivers, or stores structured report data (S10). Step S10 is an arbitrary step and may include at least one of the following: policy profile switching, anonymization processing, delivery protected by encryption and authentication, encrypted temporary retention and retransmission in case of delivery failure, retransmission control based on acknowledgment of receipt, switching to an alternative delivery destination or alternative route, tamper-proof storage, and provision of an audit interface. A specific example of step S10 may be performed according to the detailed flow (S10-1 to S10-5) shown in Figure 8.

[0127] On the other hand, if the result of the judgment in process D1 is "yes," that is, if no response or an abnormality is detected, the report generation system (100) proceeds to process S11. In process S11, the report generation system (100) notifies the system of an unconfirmed abnormality and, if necessary, activates a safety operation (S11). Process S11 is an arbitrary process and may include, for example, notification of an unconfirmed abnormality by the notification unit (180) and a transition to the safety side by the safety operation unit (190). A specific example of process S11 can be executed according to the detailed flow (S11-1 to S11-3) shown in Figure 9.

[0128] Furthermore, the main flow shown in Figure 2 can also be understood as an example of a report generation method. For example, at least a portion of steps S1 to S11 in Figure 2 can be executed as a report generation method, and can also be provided as a program for a computer to execute this method.

[0129] Referring to Figure 3, the integration of time-series data, generation of an analysis window, and generation of time reference information according to one embodiment of the present invention will be described. Figure 3 is a flowchart showing an example of the detailed flow of steps S3 and S4 in Figure 2. Note that the order of each process, branching conditions, number of repetitions, and execution entity in the following description are examples only, and the present invention is not limited to these.

[0130] In step S3-1 of Figure 3, the report generation system (100) sets a single time reference or uses an externally set single time reference to enable comparison of time-series data obtained from multiple data sources (102) on the same time axis (S3-1). The single time reference may be set by, for example, the report generation unit (110), or it may be a single time reference provided by another device, a time server, or a network, but is not limited to these.

[0131] Setting a single time reference can be achieved, for example, by time synchronization based on PTP (IEEE 1588), time synchronization based on a combination of PPS (1 pulse per second) and PLL (phase-locked loop), time synchronization based on gPTP (TSN), or time synchronization based on GNSS synchronization. Alternatively, a single time reference may be provided based on time tagging in an in-vehicle bus (CAN-FD, etc.), time synchronization implemented by software (so-called software PTP), optical or other physical layer synchronization, or a combination of a highly stable oscillator and temperature compensation.

[0132] In setting a single time reference, synchronization source identification information may be generated or obtained to identify the synchronization source (e.g., a PTP master, GNSS receiver, PPS broadcast source, or time server). This synchronization source identification information may be included in the time reference information described later.

[0133] In step S3-2 of Figure 3, the report generation system (100) may assign time tags based on a single time criterion when acquiring time-series data (S3-2). Step S3-2 is an arbitrary step, and for example, it may be a configuration in which time tags are not assigned at the time of acquisition, or a configuration in which time tags are assigned after estimation or correction.

[0134] Time tags may be assigned to units such as samples, frames, events, or data chunks. Time tags may be expressed as absolute time (e.g., UTC) or as a combination of a reference epoch and a relative offset. Time tags may also include information identifying the clock used to generate the time tag (e.g., clockId or an equivalent identifier).

[0135] In step S3-3 of Figure 3, the report generation system (100) aligns time-series data obtained from multiple data sources (102) based on a single time criterion (S3-3). Alignment can be implemented, for example, as a process of rearranging multiple series onto a common timeline based on time tags.

[0136] Matching may include, for example, at least one of interpolation, extrapolation, resampling, delay compensation, missing data compensation, and synchronization re-establishment. Matching is not limited to sample sequences of a fixed period, but can also be applied to sequences of events or media frames with unequal intervals.

[0137] In matching, errors resulting from time differences between multiple data sources (102) may be evaluated. For example, a matching error index representing the matching error between data sources can be calculated for each analysis window. The matching error index may include, for example, the absolute value, mean, maximum value, or quantile of the time difference.

[0138] In step S4-1 of Figure 3, the report generation system (100) may divide the aligned time-series data into multiple analysis windows and assign window identification information to each analysis window (S4-1). Step S4-1 is an arbitrary step, and may not generate any analysis windows.

[0139] The analysis window may be at least one of a fixed-length window, a semi-fixed-length window, and an event-driven variable-length window based on an anomaly event. The analysis window may be configured, for example, as a sliding window and may have overlap between windows. The window length, overlap rate, or setting rules for variable-length windows can be set according to the operating conditions, and these are merely examples.

[0140] The generation of the analysis window can be triggered, for example, by a synchronization pulse, timer interrupt, or detection signal (101). The boundaries of the analysis window may also be managed as a start time and an end time so that they can be used for subsequent media referencing or interval extraction.

[0141] In step S4-2 of Figure 3, the report generation system (100) generates time reference information related to the alignment (S4-2). The time reference information may include at least one of the following: start time information indicating the start of the alignment target section, end time information indicating the end of the alignment target section, window identification information identifying the section or analysis window used for alignment, data acquisition source identification information identifying the data acquisition source, and time offset information for each data acquisition source.

[0142] The start time information and / or end time information may be expressed as UTC time, for example, but not limited to this. For example, it may be expressed as a combination of a reference epoch and a relative offset. The window identification information may be expressed as a sequence number, UUID, or an identifier based on the start time information and / or end time information. The data acquisition source identification information may be expressed as a sensor ID, channel ID, device ID, or logical source ID. The time offset information may be expressed as a value (e.g., offsetMs) that represents a delay or synchronization misalignment specific to each data acquisition source.

[0143] Time reference information may be assigned per analysis window, per data source, or per predetermined interval to be included in the structured report data. Furthermore, time reference information may be stored in association with each element within the structured report data (e.g., features, anomaly probability indicators, reference information, etc.).

[0144] In step S4-3 of Figure 3, the report generation system (100) may optionally generate at least one of the following as time reference information: a matching error index, synchronization source identification information, synchronization source switching history, time quality metadata, and compensation information (S4-3).

[0145] The consistency error index may be calculated, for example, based on an estimate of the time difference between data sources. The consistency error index may be calculated for each analysis window, or as a statistic for a predetermined period. The statistics may include quantiles (e.g., p50, p95, etc.), maximum values, or representative values.

[0146] The synchronization source switching history may include, for example, the time the synchronization source switch occurred, the synchronization source identifiers before and after the switch, the reason for the switch, or the consistency error index immediately after the switch. Synchronization source identification information and switching history can contribute to facilitating auditing or post-event analysis on the receiving end.

[0147] Time quality metadata is information representing the quality of a single time standard or consistency, and may include information representing at least one of jitter or drift. For example, time quality metadata may include at least one of jitter distribution, drift rate, synchronization source identifier, and switching history. Time quality metadata may also include distribution statistics relating to at least one of time differences, jitter, or drift between multiple data sources, and distribution statistics may include at least one of quantiles, maximum values, and representative values.

[0148] Compensation information is information used to compensate for time differences across multiple data sources, and may include, for example, compensation vectors, compensation coefficients, or compensation lookup tables. Compensation information may be used during alignment, or for post-alignment correction or reanalysis. Compensation information may be included in time reference information and stored in structured reporting data.

[0149] Furthermore, each process from steps S3-1 to S4-3 may be executed within the same device, or it may be distributed and executed across multiple devices. For example, the data acquisition source (102) may assign time tags, and the report generation unit (110) may perform consistency and generate time reference information.

[0150] Furthermore, the selection of time reference information items, granularity (e.g., analysis window units, interval units), and representation format (e.g., field names, units, rounding rules) may be modified depending on the receiving end's capabilities, operational policies, or system requirements. These are illustrative examples, and the present invention is not limited to these specific examples.

[0151] Referring to Figure 4, the calculation of input quality indicators and the handling of data determined to be of low quality according to one embodiment of the present invention will be described. Figure 4 is a flowchart showing an example of the detailed flow of process S5 in Figure 2. Note that the order of each process, branching conditions, number of repetitions, and execution entity in the following description are examples only, and the present invention is not limited thereto.

[0152] In step S5-1 of Figure 4, the report generation system (100) calculates an input quality index for the matched time series data (S5-1). The input quality index is an index that evaluates the quality of the matched time series data and may be a composite index based on at least two of the evaluation values ​​related to signal noise, data loss, and gate fit.

[0153] The evaluation value for signal noise can be expressed, for example, as the signal-to-noise ratio, peak-to-noise ratio, or normalized values ​​thereof in a given bandwidth. The evaluation value for loss can be expressed, for example, as the loss rate, drop rate, or effective sample rate (1 minus loss rate). The evaluation value for gate fit can be expressed, for example, as a value representing the degree to which the time series data fits a predetermined quality gate (e.g., waveform matching conditions, exposure conditions, tracking reliability conditions, or shape conditions).

[0154] Input quality metrics can be calculated, for example, by weighted addition, multiplication, nonlinear combination, or a learning model of multiple evaluation values. For example, the input quality metric IQS can be calculated as IQS = w1·q1 + w2·q2 + w3·q3 (where q1 is an evaluation value related to signal noise, q2 is an evaluation value related to lossiness, q3 is an evaluation value related to gate fit, and w1~w3 are weights), but is not limited to this. The weights and combination methods can be changed depending on the type of data acquisition source (102), the operating environment, or the policy profile, etc.

[0155] The input quality index may be calculated per analysis window, per predetermined interval, or per data source. Furthermore, the input quality index may be included in the structured report data, or it may be used solely for determining the exclusion of low-quality data.

[0156] In the determination process D5 of Figure 4, the report generation system (100) determines whether or not there is data that has been determined to be of low quality based on the input quality index (D5). The conditions for determining low quality may include, but are not limited to, cases where the input quality index falls below a predetermined threshold or where predetermined quality conditions are not met. The threshold and determination rules can be set according to the operating conditions and are merely examples.

[0157] If the result of the judgment process D5 is "yes," that is, if there is data that has been determined to be of low quality, the report generation system (100) does not use the data that has been determined to be of low quality in the generation of structured report data in process S5-2 of Figure 4 (S5-2). For example, the analysis window that has been determined to be of low quality may be excluded from the input for feature extraction, anomaly probability index calculation, or agreement index calculation.

[0158] In step S5-3 of Figure 4, the report generation system (100) may generate exclusion reason information corresponding to the data determined to be of low quality (S5-3). The exclusion reason information may be expressed as a code or short phrase indicating, for example, excessive missing data, saturation, excessive disturbance, poor exposure, reduced traceability, or gate non-conformity. The exclusion reason information may be included in the structured report data or in the log stored by the tamper-proof storage unit (150).

[0159] If the result of the judgment step D5 is "no," that is, if there is no data judged to be of low quality, the report generation system (100) may continue the subsequent processing without excluding low-quality data. Note that the presence or absence of input quality indicators and exclusion reason information, the target of assignment (per analysis window, per interval, etc.), and the format of expression may be changed according to the requirements of the receiving party, operational policies, or policy profiles.

[0160] Next, with reference to Figure 5, a method for extracting features, calculating anomaly-likeness indices belonging to the anomaly-likeness index genus, and generating reason codes or notes and origin information according to one embodiment of the present invention will be described. Figure 5 is a flowchart showing an example of the detailed flow of steps S6 and S7 in Figure 2. Note that the order of each process, branching conditions, number of repetitions, and execution entity in the following description are examples only, and the present invention is not limited thereto.

[0161] In step S6-1 of Figure 5, the report generation system (100) may extract features from the aligned time-series data (S6-1). Feature extraction is an optional step; for example, the time-series data may be used directly to calculate the anomaly index, or the anomaly index may be calculated using only rule-based processing without using features.

[0162] Features can be extracted based on information belonging to at least two groups from among multiple data sources (102), including movement, behavior, consciousness, autonomous response, gaze, voice, operation, and environment. For example, features related to movement, features related to operation, features related to gaze, features related to autonomous response, or features related to voice can be extracted, but are not limited to these.

[0163] Features can be expressed, for example, as time-domain statistics, frequency-domain bandwidth features, time-frequency domain features, low-wave features, cepstrum features, or features based on graph structure. Furthermore, features may include those representing relationships between multiple data sources (correlation, delay, simultaneity, etc.).

[0164] Feature extraction can be performed by signal processing such as filtering, normalization, peak detection, correlation calculation, and feature encoding. Alternatively, feature extraction may be performed by a learning model (e.g., a time-series neural network, a transformer model, or an embedded model) or by rule-based processing.

[0165] In step S7-1 of Figure 5, the report generation system (100) may calculate an anomaly probability index based on the aligned time-series data or features (S7-1). The anomaly probability index is a value that represents the anomaly of an anomaly event and can be expressed, for example, as a score, probability, or a normalized value equivalent thereto.

[0166] The anomaly index can be calculated by a calculation process that includes a monotonically non-decreasing aggregation process that increases the anomaly index, and a monotonic transformation process applied to the result of the aggregation process. For example, the anomaly index I can be expressed as I = φ(g(E)), where E is an evidence vector based on time series data or features, g(·) represents a monotonically non-decreasing aggregation process, and φ(·) represents a monotonic transformation process.

[0167] A monotonically non-decreasing aggregation process g(·) can be constructed such that monotonicity is ensured by methods such as signing input features, constraining coefficients to non-negativity, learning with monotonic constraints, or rule design, so that it remains monotonically non-decreasing in the direction of increasing anomaly probability. For example, it can be implemented using, but is not limited to, non-negative weighted sums, monotonic generalized additive models, linear models with non-negativity constraints on coefficients (such as logistic regression or linear SVM), monotonic decision tree models (such as gradient-boosted decision trees), isotonic regression, or monotonic neural networks.

[0168] A monotonic transformation process φ(·) is a monotonic transformation of the result of an aggregation process, and may include at least one transformation equivalent to, for example, rescaling, probabilistic calibration, or reparameterization. The monotonic transformation process may be implemented by, for example, a logistic map, an isotonic map, or an affine map, etc., but is not limited to these.

[0169] The definition of anomaly probability indexes follows the definition described above. That is, even if an index is obtained based on the same aggregation process g(·), its numerical representation may differ due to differences in monotonic transformation processes such as rescaling, probability calibration, and reparameterization. However, if these differences are mutually interchangeable by monotonically increasing or monotonically non-decreasing mappings, and the ranking or threshold judgment results of anomaly probability match, they can be treated as anomaly probability indexes with the same meaning.

[0170] Furthermore, even if the input representation of an anomaly index is replaced due to differences in the representation format of the features (time domain, frequency domain, time-frequency domain, small wave, cepstrum, graph, etc.), the anomalousness index can be treated as equivalent in the sense described above, as long as the processing after replacement is configured to give the same anomaly ranking or threshold judgment result as before replacement. For example, this includes using mutually convertible representations such as the time waveform and frequency spectrum of vehicle acceleration, or the audio waveform and spectrogram.

[0171] The anomaly index can be calculated by at least one of the following methods: calculation using a machine learning model, calculation using rule-based processing, and calculation using a combination of the two. For example, this may include a method in which the output of a learning model and the output of rule-based processing are combined to form an aggregate processing g(·), or a method in which rule-based processing is added as a pre-processing or post-processing.

[0172] Furthermore, in calculating the anomaly probability index, even if not all of the multiple data sources (102) are available, the anomaly probability index may be calculated based on at least some of the available data. For example, this may include methods of excluding missing or unobtainable data sources and calculating the index based on features or time-series data from the remaining sources, or methods of calculating the index when a predetermined number or more of the multiple candidate feature sets are available (so-called N-of-M methods).

[0173] The anomaly index may be calculated as a single value or as multiple components (e.g., multiple index values, multiple scales, or index values ​​corresponding to multiple time intervals). In the case of multiple components, each component may be associated with separate time reference information or reference information.

[0174] In step S7-2 of Figure 5, the report generation system (100) may generate reason codes or notes associated with the anomaly probability index, and origin information related to the calculation of the anomaly probability index (S7-2). The generation of reason codes or notes and origin information is an optional step and can be included in the structured report data.

[0175] Reason codes are machine-readable information that expresses the basis for an anomaly index, and may be expressed as codes indicating, for example, key features, key rules, key conditional branches, or attention-grabbing content. Notes are human-readable short sentences or explanatory information, and may include, for example, short explanations corresponding to reason codes, or information to aid in the interpretation of an anomaly index. Reason codes and notes may be stored in association with an anomaly index.

[0176] Origin information is information relating to the calculation of anomaly probability index and may include at least one of model identification information, rule set identification information, and parameter version control information. Origin information may include, but is not limited to, model ID, rule set ID, parameter version, threshold version, learning date, or applied profile identifier.

[0177] Reason codes, notes, or origin information may be assigned per analysis window, per predetermined interval, or per indicator component. These may also be stored in association with other elements within the structured reporting data (e.g., reference information, screening information, or audit logs).

[0178] Furthermore, in the feature extraction and calculation of anomaly probability indices, if data judged to be of low quality by the preceding input quality indices is excluded, the impact of such exclusion (e.g., the set of data sources used, data sources that could not be used, or information on the reasons for exclusion) may be recorded in the structured report data. This can facilitate evaluation, auditing, or post-analysis on the receiving end.

[0179] Referring to Figure 6, the calculation of the degree of agreement index according to one embodiment of the present invention will be described. Figure 6 is a flowchart showing an example of the detailed flow of step S8 in Figure 2. Note that the order of each process, branching conditions, number of repetitions, and execution entity in the following description are examples only, and the present invention is not limited thereto.

[0180] In step S8-1 of Figure 6, the report generation system (100) acquires or estimates a stimulus series relating to an extrinsic stimulus included in the time-series data, and a response series relating to the response to the extrinsic stimulus (S8-1). The stimulus series and the response series may be obtained from the same data acquisition source (102), or from different data acquisition sources (102).

[0181] The stimulus sequence can be represented as a time series corresponding to, for example, fluctuations in ambient light (flicker, etc.), the beat of an auditory stimulus, changes in visual patterns, road surface disturbances, or other external stimuli. The stimulus sequence may be obtained directly from environmental sensors, video, audio, or external information (e.g., periodic information of traffic lights, etc.), or it may be estimated from already obtained time series data.

[0182] A response sequence can be represented as a time series corresponding to, for example, gaze, blinking, head movement, body movement, changes in operation, changes in voice, autonomous responses, etc. A response sequence can be generated from information belonging to at least one group from among movement, behavior, consciousness, autonomous responses, gaze, voice, operation, environment, etc.

[0183] Stimulus and response sequences can be generated based on synchronized time-series data. For example, on a synchronized common timeline based on a single time criterion as shown in Figure 3, the stimulus and response sequences may be associated with the same time reference information (start time information, end time information, window identification information, etc.).

[0184] In step S8-2 of Figure 6, the report generation system (100) calculates a concordance index representing the degree of agreement between the stimulus series and the response series (S8-2). The concordance index may, but is not limited to, be expressed as a normalized score or a value corresponding to a probability.

[0185] The calculation of the degree of agreement index may include at least one of the following: processing based on cross-correlation in the time domain, processing based on coherence in the frequency domain, and processing based on correlation based on time-frequency analysis. For example, this may include a method of calculating cross-correlation considering the time delay between stimulus and response sequences, a method of calculating coherence for each frequency band, or a method of calculating correlation in the time-frequency domain by small-wave analysis, etc.

[0186] In calculating the degree of agreement index, the time interval or analysis window to be evaluated may be limited. For example, the degree of agreement index may be calculated for a predetermined interval or analysis window related to the time of the abnormal event, and time reference information may be added to the degree of agreement index.

[0187] The agreement index may be included in the structured reporting data. The agreement index may also be stored in association with anomaly probability indices, features, or screening information, etc.

[0188] The methods for obtaining or estimating stimulus and response series, as well as the method for calculating the degree of agreement index, may be modified depending on the operating environment, the type of data source (102), or the policy profile. These are illustrative examples, and the present invention is not limited to these specific examples.

[0189] Next, with reference to Figure 7, the generation of structured reporting data according to one embodiment of the present invention will be described. Figure 7 is a flowchart showing an example of the detailed flow of step S9 in Figure 2. Note that the order of each process, branching conditions, number of repetitions, and execution entity in the following description are examples only, and the present invention is not limited to these.

[0190] In step S9-1 of Figure 7, the report generation system (100) may extract time-series data for a period before and after the time of the abnormal event as a predetermined interval related to the time of the abnormal event associated with the detection signal (S9-1). The length of the predetermined interval and the range before and after it can be set according to the operating conditions and are merely examples.

[0191] Extraction of a predetermined interval can be implemented, for example, by reading from continuously acquired and stored time-series data (such as a ring buffer), or by extracting from time-series data acquired in response to a detection signal. The data to be extracted may include waveform data, video frame sequences, audio sample sequences, event sequences, or combinations thereof.

[0192] In step S9-2 of Figure 7, the report generation system (100) may set multiple candidate intervals as predetermined intervals and select an adopted interval from among the multiple candidate intervals by evaluation based on at least one of the input quality, stability, or degree of missing data for each candidate interval (S9-2). For example, multiple candidate intervals of ±L seconds may be set as the interval before and after the abnormal event time (L can be at least one such as 5, 10, 20, etc.), and for each candidate interval L, an evaluation score S(L) may be calculated based on at least one of the input quality (e.g., interval average of the input quality index), stability (e.g., stability index), degree of missing data (e.g., missing data rate), etc. within the candidate interval, and the candidate interval with the maximum evaluation score S(L) may be selected as the adopted interval. As an example of an evaluation score, S(L) = α·IQS_bar_L + β·Stability_L - γ·Drop_L (where IQS_bar_L is the mean value of the input quality index within candidate interval L, Stability_L is the stability index within candidate interval L, Drop_L is the missing value rate within candidate interval L, and α, β, and γ are weighting coefficients) may be used, but is not limited to this. The number of candidate intervals, the length of the candidate intervals, the types of indicators used for evaluation, the normalization method, and the setting of weighting coefficients may be changed depending on the operating conditions or policy profile, etc.

[0193] The input quality used to evaluate candidate intervals can be evaluated based on, for example, the input quality index shown in Figure 4. Stability can be evaluated as, for example, the stability of fluctuations in time series data, the stability of sensor operation, or the stability of features. Missing data can be evaluated as, for example, the missing data rate, the drop rate, or the effective sample rate.

[0194] In step S9-3 of Figure 7, the report generation system (100) may align at least one of the start and end times of the extracted interval with a delimiter boundary of the time-series data or media data associated with the time-series data (S9-3). The delimiter boundary may include, for example, a frame boundary, an encoding boundary (e.g., a GOP boundary), or a data chunk boundary.

[0195] Alignment of start and end times to delimiter boundaries can be achieved, for example, by rounding the start time to the previous delimiter boundary (K_prev) and the end time to the next delimiter boundary (K_next) for the start time t_s and end time t_e of the extracted interval, by setting (t_s', t_e') = (K_prev(t_s), K_next(t_e)). Here, the delimiter boundaries can be defined corresponding to frame boundaries, encoding boundaries (e.g., GOP boundaries), or data chunk boundaries. This can contribute to more efficient media referencing, improved playback consistency, or easier chunk management during delivery.

[0196] In step S9-4 of Figure 7, the report generation system (100) generates structured report data (S9-4). The structured report data may include at least the time reference information related to the consistency shown in Figure 3.

[0197] Structured reporting data may optionally further include at least one of the following: input quality metrics, exclusion reason information corresponding to data judged as low quality, features, anomaly probability metrics, agreement metrics, reason codes or notes, origin information, and screening information. This information may be stored per analysis window, per selection interval, or per anomaly event.

[0198] Structured reporting data may include reference information to at least one waveform data or media data corresponding to the aligned time-series data. Reference information may include, for example, a URI, identifier, hash, storage address, or access token. Reference information may be associated with at least one time reference information (start time information, end time information, window identification information, etc.) or anomaly index. Furthermore, the reference information may include multiple reference forms for the same waveform data or media data, or one of the multiple reference forms may be selected and included depending on the receiving end's capabilities or configuration policy.

[0199] Structured reporting data may further include readable summary document data generated based on the information contained in the structured reporting data. The summary document data may be generated in a document format such as PDF or HTML. Furthermore, the summary document data may contain language identification information (e.g., language tags). Additionally, the summary document data may include a mechanism to switch the presentation language depending on the recipient's settings or policies.

[0200] Screening information includes the results of screening for potential causes of abnormal events, and may include a value indicating affirmative, negative, or unknown for each potential cause. Screening information may be included in structured reporting data with the addition of period information, supporting notes, or reference information.

[0201] The format of structured reporting data may include, but is not limited to, JSON, XML, CBOR, or database records.

[0202] As one embodiment of structured reporting data, in a configuration linked with a medical information system, the structured reporting data may be configured as a Bundle (type=document or collection) compliant with FHIR R4 or R5. For example, the Bundle may be configured to include a DiagnosticReport, an Observation group (such as anomaly probability index, feature quantities, input quality index, and screening information), and Media / DocumentReference / Binary (for referencing or storing waveforms, video, audio, etc.).

[0203] One example of a reference relationship in FHIR is a configuration in which a DiagnosticReport references a group of Observations, and an Observation references Media, DocumentReference, or Binary, etc. It may also include a configuration in which at least one of the reference forms such as Media, DocumentReference, or Binary is selected and used based on the capabilities or configuration policy of the receiving side. Furthermore, it may include a configuration in which multiple reference forms for the same waveform data or media data are listed together, and the receiving side selects and resolves the available reference form.

[0204] As another embodiment for integration with a medical information system, structured reporting data may be configured as DICOM-SR. Furthermore, as a fallback for cases where the use of FHIR or DICOM-SR is difficult due to the receiving environment, the output may be in a format such as HL7 v2 ORU^R01, or it may include a readable document (PDF or HTML, etc.) alongside the report. These are examples, and the present invention is not limited to these specific examples.

[0205] The items, granularity, and presentation format included in the structured reporting data may be modified depending on the recipient's requirements, operational policies, or policy profiles.

[0206] Next, with reference to Figure 8, the output, delivery, storage, auditing, anonymization, and policy switching of structured reporting data according to one embodiment of the present invention will be described. Figure 8 is a flowchart showing an example of the detailed flow of step S10 in Figure 2. Note that the order of each process, branching conditions, number of repetitions, and execution entities in the following description are examples only, and the present invention is not limited thereto.

[0207] In step S10-1 of Figure 8, the report generation system (100) may switch at least one of the data items to be included in the structured report data, the intensity of the anonymization process, or the output destination of the structured report data, depending on the jurisdiction or organization's policy (S10-1). For example, this may include a policy profile switching unit (130) that selects an applicable profile and changes the processing conditions of subsequent steps according to that profile.

[0208] The data items subject to switching may include, for example, the presence or absence of reference information, the presence or absence of summary document data, the presence or absence of screening information, or the level of detail of origin information. The output destinations subject to switching may include, for example, a delivery server, external device, recording medium, or region.

[0209] In step S10-2 of Figure 8, the report generation system (100) may perform anonymization processing on data included in or referenced from the structured report data (S10-2). For example, this may include an anonymization unit (140) performing at least one of the following: anonymization of video data, anonymization of location information, and anonymization of audio information.

[0210] Anonymization of video data may include, for example, blurring or mosaic processing of facial regions, processing to distinguish the original image from the anonymized image using identifiers, or logging of masked regions. Anonymization of location information may include, for example, grid-level rounding, offsetting with pseudo-noise, or precision reduction. Anonymization of audio information may include, for example, deletion or transformation of speaker IDs, or feature transformation that makes speaker identification difficult.

[0211] The anonymization unit (140) may perform anonymization processing based on a profile that defines the anonymization strength. The anonymization unit (140) may also record a log related to the anonymization processing, or separate and manage key information that can be used to restore the anonymized data. For example, this may include storing the restoration key in a separate area and separating access control.

[0212] In step S10-3 of Figure 8, the report generation system (100) may output the structured report data to an external device, record it on a recording medium, or deliver it (S10-3). For example, the output unit (120) may output or deliver the structured report data to an external device / recording medium / delivery destination (900).

[0213] Delivery can be protected by encryption and authentication of the recipient. Specific examples of encryption and authentication include, but are not limited to, TLS 1.2 or higher, server certificate verification, certificate pinning, or mutual authentication (mTLS).

[0214] For data delivery, if the data to be delivered is large in volume, the data may be divided into chunks before transmission. Chunking can be applied, for example, to media data, waveform data, or summary document data. The chunk units, division methods, and joining methods are examples and may be changed depending on the operating conditions.

[0215] During transmission, the compression ratio or encoding conditions may be switched depending on conditions such as communication bandwidth or delay. For example, variable control may be performed, such as increasing the compression ratio when bandwidth is low and decreasing it when bandwidth is sufficient.

[0216] In step S10-4 of Figure 8, the report generation system (100) may encrypt and temporarily store the structured report data when delivery is impossible, and resend the structured report data when delivery is possible (S10-4). This process can be implemented as a so-called Store & Forward method. For example, this may include temporarily storing the data in an encrypted buffer (encrypted queue) and resending it after the line is restored.

[0217] The output unit (120) may perform retransmission control based on an acknowledgment of receipt from the recipient. The acknowledgment of receipt may include, for example, an ACK, a NAK, or an equivalent acknowledgment message. The retransmission control may include, for example, a mode in which only the undelivered chunk is retransmitted, a mode in which retransmission is performed up to a certain number of times, or a mode in which a retransmission rule such as exponential backoff is applied.

[0218] The output unit (120) may switch to an alternative destination or alternative delivery route and continue delivery if it does not receive an acknowledgment of receipt. An alternative destination may include, for example, a redundant server, a server in a different region, or a gateway on a different line. An alternative delivery route may include, for example, a different network route, a different relay device, or a different protocol stack.

[0219] In step S10-5 of Figure 8, the report generation system (100) may store at least one of structured report data, time reference information, or logs relating to the generation or delivery of structured report data in a tamper-proof form and provide an audit interface (S10-5). For example, the tamper-proof storage unit (150) may perform at least one of storage on an appendable recording medium or storage using a chained hash structure.

[0220] In storage using a chained hash structure, for example, each record can be recorded sequentially in a structure where each record holds its own hash and the hash of the previous record, creating a chain. Write-once recording media may include, for example, WORM media.

[0221] The audit interface (160) may output information that allows verification of the integrity of the data stored by the tamper-proof storage unit (150), or the results of integrity verification. For example, the audit interface (160) may output a record sequence of chained hash structures (e.g., record identifier, previous hash, hash, timestamp, data hash, etc.) in a format such as JSON, and include the verification results (e.g., match / mismatch).

[0222] Alternative or additional forms of tamper-proof storage may include, for example, ensuring integrity or non-tampering using TPM signatures, remote notarization, or a timestamp authority. These are illustrative examples, and the present invention is not limited to these specific examples.

[0223] The process shown in Figure 8 may be completed and executed within a single device, or it may be distributed and executed across multiple devices. For example, it may include an embodiment in which anonymization processing is performed on the edge side and delivery and tamper-proof storage are performed on the cloud side, or vice versa.

[0224] Furthermore, the execution status, execution order, and processing conditions of each step shown in Figure 8 may be changed depending on the recipient's request, operational policy, policy profile, or available computing resources. These are illustrative examples, and the present invention is not limited to these specific examples.

[0225] Referring to Figure 9, the handling of unresponsive or abnormal processing (so-called fail-closed operation) in the process of generating structured report data according to one embodiment of the present invention will be described. Figure 9 is a flowchart showing an example of the detailed flow of step S11 in Figure 2. Note that the order of each process, branching conditions, number of repetitions, and execution entity in the following description are examples only, and the present invention is not limited thereto.

[0226] In step S11-1 of Figure 9, the report generation system (100) detects a lack of response or an anomaly in the processing related to the generation of structured report data (S11-1). Detection of the lack of response or an anomaly may be performed, for example, by the monitoring unit (170). Specific examples of a lack of response or anomaly may include, but are not limited to, at least one of the following: (i) a timeout in which processing is not completed within a predetermined time; (ii) an invalid value such as NaN or infinity; (iii) an invalid value such as NaN or infinity; (iv) a discrepancy in consistency in the time reference information, where an item required by the applicable profile or policy is missing, or the consistency results contradict those based on the same time standard; (v) a synchronization failure, a consistency error index or time quality metadata that does not meet predetermined conditions; (vi) the inability to obtain the required input; or (vii) an internal error (e.g., memory error, model corruption, exception occurrence, verification failure, etc.).

[0227] In step S11-2 of Figure 9, the report generation system (100) issues a notification as an unconfirmed abnormality (S11-2) in response to the detection of no response or an abnormality. The notification as an unconfirmed abnormality may be performed, for example, by a notification unit (180). The recipient of the notification may include, for example, an external device / recording medium / delivery destination (900), a display unit or alarm unit of a device on which the report generation system (100) is installed, or a monitoring center. The notification of an unconfirmed abnormality may include, for example, a reason code indicating the type of no response or abnormality, information regarding the time of occurrence, and at least one of available time reference information.

[0228] Notifications of unconfirmed anomalies may be generated as messages containing minimal information (e.g., identification information for the detection signal (101), information regarding the time of the anomaly event, and a reason code for no response or anomaly) to ensure that the event is treated as an "unconfirmed anomaly" even if the structured reporting data cannot be fully generated. Notifications of unconfirmed anomalies may also be combined with a Store & Forward process to encrypt, temporarily store, and resend the message if delivery fails.

[0229] In step S11-3 of Figure 9, the report generation system (100), in response to notification of an unconfirmed abnormality, activates a safety action (S11-3) to transition the device on which the report generation system (100) is installed or the device controlled by the report generation system (100) to a safe state, if necessary. For example, if a safety action unit (190) is provided, the safety action may be activated by the safety action unit (190). On the other hand, in configurations without a safety action unit (190) or in configurations where it is determined that activation of a safety action is unnecessary, processing may be terminated after the notification in step S11-2. Specific examples of safety actions may include, but are not limited to, transitioning the target device to a function restriction mode, stopping, outputting an alarm, sending an external notification, or suppressing operation. The activation conditions for safety actions, waiting time, and stepwise escalation (for example, changing the notification intensity or the stage of safety action if recovery is not achieved within a certain period of time) may be set according to the operating conditions or policy profile.

[0230] Alternative or additional forms of fail-closed operation may include, but are not limited to, at least one of the following: (i) failover by a redundant processing system (redundant model or redundant computation path); (ii) backup output by simple rule-based processing or threshold determination; (iii) restart by heartbeat monitoring and watchdog; (iv) re-notification with backoff; and (v) a form of adding a reason code or part of the origin information to the notification of an undetermined anomaly.

[0231] Next, optional embodiments and modifications that supplement the above embodiments will be described. The embodiments shown below are examples of embodiments of the present invention and can be implemented in any combination without contradiction. Furthermore, they do not limit the present invention.

[0232] As one embodiment of structured reporting data, in a configuration linked with a medical information system, the structured reporting data may be configured as a Bundle compliant with FHIR R4 or R5, or as DICOM-SR. For example, the Bundle may be configured to include a DiagnosticReport, an Observation group (such as anomaly probability index, features, input quality index, and screening information), and Media / DocumentReference / Binary (for referencing or storing waveforms, video, audio, etc.).

[0233] Furthermore, the observed items, classification items, or potential causes included in the structured reporting data may be assigned codes (mapped) based on at least one terminology code system such as SNOMED CT, LOINC, or ICD. The mapping is not limited to a single code and may include the simultaneous use of multiple codes (multi-code assignment), the use of a translation table between internal codes and standard codes, or mapping accompanied by version control.

[0234] Depending on the Implementation Guide (IG) or profile referenced by the receiving end, items may be subsetted, constraints applied, or profiles switched. As a fallback when the use of FHIR or DICOM-SR is difficult due to the receiving end's environment, this may include outputting in a format such as HL7 v2 ORU^R01, or including a readable document such as PDF or HTML.

[0235] This method may include classifying abnormal events into multiple categories based on anomaly probability indices and screening information included in structured reporting data. The classification categories may, but are not limited to, non-provoked, acute symptomatic, reflexive, unknown, etc. It may also include probabilistic calibration of the output of the anomaly probability indices (e.g., Pratt scaling, isotonic regression, temperature scaling, etc.). Furthermore, it may include representing the dependencies between multiple modalities as a graph structure and including graph features such as density, cluster coefficients, and connectivity as features.

[0236] As time quality metadata (QoT), distribution statistics (quantiles, maximum value, representative value, etc.) of the matching error index (e.g., |Δt|) may be calculated and recorded per analysis window or per predetermined interval. Furthermore, methods for reducing measurement noise by interpolating or rounding on a sample basis may be included. Methods for maintaining the synchronization source switching history and emphasizing the matching error index immediately after switching, as well as methods for calculating compensation information (compensation vectors, etc.) and including it in the time reference information, may also be included.

[0237] The report generation process may be implemented as a two-stage configuration, where minimal structured report data is generated on the edge side, and coding, profile conformance, signature, or additional formatting processing is performed on the cloud side. This allows for flexible distribution of workload depending on communication bandwidth, latency, or operational policies.

[0238] This may include calibrating or updating parameters of anomaly indicators based on feedback from users, operators, or experts (e.g., confirmed findings, false alarms, or cause estimations). The difference before and after the update may be recorded as origin information or an audit log.

[0239] The update may include a safety mode that does not immediately reflect in online operation, applying only parameters for which offline verification has been completed. Furthermore, it may be combined with A / B testing, shadow updates (parallel model comparison), rollback functionality, time-limited application, or manual approval systems.

[0240] This may include anonymization processing to separate and store personally identifiable information (PII) in a separate area, and to manage the recovery key or information that can be used for recovery separately. By combining this with access control, audit logs, and tamper-proof storage, reproducibility and auditability can be enhanced.

[0241] Next, examples will be described. The following examples are illustrative for the sake of explanation and do not limit the present invention. Numerical values, conditions, and effects may vary depending on operating conditions, data acquisition source configuration, or implementation differences.

[0242] For example, when time-series data obtained from multiple data sources (102) are synchronized based on a single time criterion, the median of the synchronization error index (e.g., |Δt|) may be around a few milliseconds, and the quantiles (e.g., p99) may fall within a range of about 10 milliseconds. In this case, generating structured reporting data with time reference information can reduce the rework required for subsequent resynchronization.

[0243] For example, if low-quality sections (e.g., poorly exposed frames, sections with excessive data loss, etc.) are present in video or waveform data, excluding these sections based on input quality indicators and including the reason for exclusion in the structured report data can contribute to reducing false reports and improving the reliability of reports. Furthermore, the time required to generate summary document data or create initial findings may be shortened.

[0244] For example, calculating an index of agreement between stimulus sequences and response sequences related to extrinsic stimuli and including this index in structured reporting data can provide useful additional information for screening potential causes (affirmative / negative / unknown) and for post-hoc analysis.

[0245] For example, by setting multiple candidate intervals before and after the time of an abnormal event, selecting an appropriate interval based on input quality, stability, degree of loss, etc., and further aligning the start or end time with a frame boundary, coding boundary, or chunk boundary, the playback consistency and delivery efficiency of reference information (waveform / media, etc.) can be improved.

[0246] For example, if delivery fails due to a network outage or other reasons, encrypting and temporarily storing structured report data and resending it after the line is restored can improve the continuity of delivery. Furthermore, if an acknowledgment of receipt is not received, switching to an alternative delivery destination or route to continue delivery can contribute to reducing undelivered messages.

[0247] For example, by storing structured reporting data and generated logs in a chained hash structure and outputting record columns and verification results via an audit interface, it may be easier to detect modifications after storage and to respond to audits.

[0248] For example, if no response or an anomaly (e.g., timeout, NaN, etc.) is detected during the structured report data generation process, it will be notified as an unconfirmed anomaly, and a safe operation will be triggered, making it possible to transition to a safer state even when processing is stopped.

[0249] Next, we will describe implementation examples. The following implementation examples are illustrative for illustrative purposes only and do not limit the present invention.

[0250] The report generation system (100) can be implemented, for example, by an in-vehicle SoC, MCU, ECU, or server device. The data acquisition source (102) can be configured, for example, as a sensor or log acquisition unit related to motion, gaze, voice, operation, environment, etc. The data acquisition source (102) and the report generation unit (110) can be connected, for example, by Ethernet, TSN, wireless communication, or an in-vehicle bus.

[0251] Examples of data acquisition sources (102) include, but are not limited to, electrocardiograms, pulse waves, acceleration / angular velocity, cameras, microphones, steering, brakes, position, illuminance, temperature, etc.

[0252] Setting a single time reference can be achieved, for example, by a group of PTP masters and slaves, PPS wiring and PLLs, gPTP-compatible switches, etc. The report generation unit (110) can generate an analysis window, for example by interrupt or timer, store time-series data in a ring buffer, and perform processing by adding window identification information and time reference information.

[0253] The report generation unit (110) may consist of, for example, a time tagging module, a consistency module, an input quality index calculation module, a feature extraction module, an anomaly probability index calculation module, a match index calculation module, and a structured report data generation module. These may be implemented as a single program or as separate services or processes.

[0254] When using machine learning models to calculate anomaly-likeness indices or extract features, the inference execution environment may, but is not limited to, ONNX Runtime or TensorRT. When rule-based processing is used in conjunction, it can be implemented using a rule engine, a state machine, or conditional branching processing.

[0255] The output unit (120) can protect delivery by, for example, TLS 1.2 or higher encrypted communication, server certificate verification, and certificate pinning. In the event of delivery failure, Store & Forward can be implemented, where the data is temporarily held in an encrypted buffer (encryption queue) and resent after the connection is restored. Chunk splitting, retransmission control based on ACK / NAK, and fallback to an alternative endpoint can also be implemented.

[0256] The tamper-proof storage unit (150) may store structured reporting data and logs using an appendable storage medium (e.g., WORM) or a chained hash structure. The audit interface (160) may output the stored record sequence and verification results in JSON or similar format. Alternative or additional integrity protection may be used, such as TPM signing, remote notarization, or timestamp authority cooperation.

[0257] Each function of the report generation system (100) may be distributed between the edge and the cloud. For example, a minimal report may be generated on the edge, and coding, profile conformance, signature assignment, anonymization strength application, or audit storage may be performed on the cloud.

[0258] The embodiments, optional embodiments, and examples described above can be combined in any way without contradiction. Furthermore, the present invention is not limited to the above embodiments, and various modifications are possible without departing from its essence.

[0259] The present invention relates to a report generation system, a report generation method, and a program that, in response to a detection signal relating to an abnormal event, aligns time-series data obtained from multiple data sources based on a single time reference and generates structured report data including time reference information related to the alignment.

[0260] Therefore, the present invention can be widely applied to various monitoring, safety, recording, analysis, and reporting applications that handle time-series data from multiple sources. The application areas may include, but are not limited to, in-vehicle safety systems, driver condition monitoring, fleet management, industrial machinery condition monitoring, equipment maintenance, worker safety support, remote monitoring, home care monitoring, monitoring in medical and nursing care, disaster response, security monitoring, and accident investigation and auditing.

[0261] According to the present invention, since the output can be structured report data with time reference information attached, post-analysis, reproduction, auditing, and integration with other systems can be facilitated. Furthermore, since it may include exclusion of low-quality data based on input quality indicators, assignment of anomaly-like indicators and information on the reasons and origins, association with reference information (waveforms, media, etc.), response to operational requirements such as delivery, storage, auditing, anonymization, and policy switching, as well as fail-closed operation in case of unresponsiveness or anomalies, it can contribute to improving reliability and security in actual operation.

[0262] Furthermore, as one embodiment of structured reporting data, it can be configured in a form compliant with medical standard formats such as FHIR or DICOM-SR, making it applicable to uses such as linking with medical information systems, integration into electronic records, or diagnostic support. Therefore, the present invention is useful in industrial fields such as traffic safety, medical care and welfare, industrial safety, information and communication, and auditing and legal services. [Explanation of symbols]

[0263] 100 Report Generation System 101 Detection signal 102 Data sources (multiple) 110 Report generation section 120 Output section 130 Policy Profile Switching Section (Optional) 140 Anonymization section (optional) 150 Tamper-proof storage section (optional) 160 Audit Interface (Optional) 170 Monitoring Department (optional) 180 Notification section (optional) 190 Safety operation part (optional) 900 External device / recording medium / delivery destination (optional)

Claims

1. Equipped with a report generation unit, The report generation unit, in response to a detection signal related to an abnormal event, acquires time-series data from multiple data sources, aligns the time-series data based on a single time reference, and generates structured report data including time reference information related to the alignment. Based on the time-series data obtained from the multiple data sources, a stimulus series relating to an external stimulus and a response series relating to the response to the external stimulus are obtained or estimated, and a degree of agreement index representing the degree of agreement between the stimulus series and the response series is calculated. The structured reporting data includes the agreement index, The process for calculating the aforementioned degree of agreement index includes at least one of the following: a process based on cross-correlation in the time domain, a process based on coherence in the frequency domain, and a process based on correlation based on time-frequency analysis. Report generation system.

2. Equipped with a report generation unit, The report generation unit, in response to a detection signal related to an abnormal event, acquires time-series data from multiple data sources, aligns the time-series data based on a single time reference, and generates structured report data including time reference information related to the alignment. The report generation unit calculates an input quality index for the aligned time-series data, The aforementioned input quality index is a composite index based on at least two evaluation values ​​related to signal noise, loss, and gate fit. Report generation system.

3. Equipped with a report generation unit, The report generation unit, in response to a detection signal related to an abnormal event, acquires time-series data from multiple data sources, aligns the time-series data based on a single time reference, and generates structured report data including time reference information related to the alignment. The report generation unit calculates an input quality index for the aligned time-series data, Data that is judged to be of low quality based on the aforementioned input quality indicator will not be used in the generation of the structured report data. Report generation system.

4. Equipped with a report generation unit, The report generation unit, in response to a detection signal related to an abnormal event, acquires time-series data from multiple data sources, aligns the time-series data based on a single time reference, and generates structured report data including time reference information related to the alignment. The aforementioned multiple data sources include information belonging to at least two groups from among movement, behavior, consciousness, autonomous response, gaze, voice, operation, and environment. The report generation unit extracts features from the aligned time-series data, and the structured report data includes the features. The report generation unit calculates an anomaly probability index for the anomaly event based on the aligned time-series data or the feature quantities, and the structured report data includes the anomaly probability index. The aforementioned anomaly index is a value obtained by a calculation process that includes a monotonically non-decreasing aggregation process in the direction of increasing anomaly, and a monotonic transformation process on the result of the aggregation process, and belongs to the genus of anomaly indexes that treat anomaly indexes with different expressions due to differences in the monotonic transformation process as equivalent anomaly indexes. Report generation system.

5. Equipped with a report generation unit, The report generation unit, in response to a detection signal related to an abnormal event, acquires time-series data from multiple data sources, aligns the time-series data based on a single time reference, and generates structured report data including time reference information related to the alignment. The aforementioned multiple data sources include information belonging to at least two groups from among movement, behavior, consciousness, autonomous response, gaze, voice, operation, and environment. The report generation unit extracts features from the aligned time-series data, and the structured report data includes the features. The report generation unit calculates an anomaly probability index for the anomaly event based on the aligned time-series data or the feature quantities, and the structured report data includes the anomaly probability index. The structured reporting data includes reference information to at least one of waveform data or media data corresponding to the aligned time-series data. The aforementioned reference information is associated with at least one of the aforementioned time reference information or the aforementioned anomaly index, The aforementioned reference information includes multiple reference formats that can be selected depending on the recipient's capabilities or configuration policy. The aforementioned multiple reference forms include at least one of the reference forms corresponding to Media, DocumentReference, and Binary in FHIR. Report generation system.

6. A report generation system according to any one of claims 1 to 5, further comprising an output unit that outputs the structured report data generated by the report generation unit to an external device or records it on a recording medium, Report generation system.

7. In the report generation system according to any one of claims 1 to 5, the time reference information includes at least one of: start time information indicating the start of the time series data matching target interval; end time information indicating the end of the matching target interval; window identification information identifying the interval or analysis window used for matching; data acquisition source identification information identifying the data acquisition source; and time offset information for each data acquisition source. Report generation system.

8. In the report generation system according to claim 7, the time reference information further includes a consistency error index representing the consistency error between the plurality of data acquisition sources, Report generation system.

9. In the report generation system according to claim 7, the time reference information further includes synchronization source identification information that identifies the synchronization source that gives the single time reference, and at least one of the switching history of the synchronization source. Report generation system.

10. In the report generation system according to claim 7, the time reference information further includes time quality metadata representing at least one of jitter or drift. Report generation system.

11. In the report generation system according to claim 10, the time quality metadata includes distribution statistics relating to at least one of the time difference between the plurality of data sources, the jitter, or the drift, wherein the distribution statistics include at least one of the quantile, the maximum value, and the representative value. Report generation system.

12. In the report generation system according to claim 7, the time reference information further includes compensation information for compensating for time differences for each of the plurality of data acquisition sources. Report generation system.

13. In the report generation system according to any one of claims 1 to 5, the report generation unit divides the aligned time-series data into a plurality of analysis windows, assigns window identification information to identify each analysis window as time reference information, and the analysis window is at least one of a fixed-length window, a semi-fixed-length window, and an event-driven variable-length window based on the abnormal event. Report generation system.

14. In the report generation system according to any one of claims 1 to 5, the report generation unit assigns a time tag based on the single time standard when acquiring the time series data. Report generation system.

15. In the report generation system according to any one of claims 1, 4, or 5, the report generation unit calculates an input quality index for the aligned time-series data. Report generation system.

16. In the report generation system according to claim 3, the structured report data includes at least one of the input quality index and exclusion reason information corresponding to the data determined to be of low quality. Report generation system.

17. In the report generation system according to any one of claims 1 to 3, the plurality of data acquisition sources include information belonging to at least two groups from among movement, behavior, consciousness, autonomous response, gaze, voice, operation, and environment. Report generation system.

18. In the report generation system according to claim 17, the report generation unit extracts feature quantities from the aligned time-series data, and the structured report data includes the feature quantities. Report generation system.

19. In the report generation system according to claim 18, the report generation unit calculates an anomaly probability index of the anomaly event based on the aligned time-series data or the feature quantities, and the structured report data includes the anomaly probability index. Report generation system.

20. In the report generation system according to claim 4, the monotonic transformation process includes at least one of transformations corresponding to rescaling, probabilistic calibration, and reparameterization. Report generation system.

21. In the report generation system according to claim 19, the calculation of the anomaly index is performed by at least one of the following: calculation by a machine learning model, calculation by rule-based processing, and calculation by a combination thereof. Report generation system.

22. In the report generation system according to claim 19, the report generation unit calculates the anomaly index based on at least some of the available data from the plurality of data acquisition sources. Report generation system.

23. In the report generation system according to claim 19, the structured report data includes at least one of a reason code associated with the anomaly index, or a note relating to the anomaly index. Report generation system.

24. In the report generation system according to claim 19, The structured reporting data includes, as source information for calculating the anomaly probability index, at least one of the following: model identification information, rule set identification information, threshold or parameter version control information, learning date and time information, and applied profile identification information. Report generation system.

25. In the report generation system according to any one of claims 1 to 5, the report generation unit extracts the time-series data for a predetermined interval related to the time of the abnormal event associated with the detection signal, which spans before and after the time of the abnormal event, and includes it in the structured report data. Report generation system.

26. In the report generation system according to claim 25, the report generation unit sets a plurality of candidate intervals as the predetermined interval, and selects an adopted interval from the plurality of candidate intervals based on an evaluation based on at least one of the input quality, stability, or degree of deficiency for each candidate interval. Report generation system.

27. In the report generation system according to claim 25, the report generation unit aligns at least one of the start time and end time of the extracted interval with a delimiter boundary (including a frame boundary, encoding boundary, or data chunk boundary) of the time series data or media data related to the time series data. Report generation system.

28. In the report generation system according to any one of claims 1 to 5, The structured reporting data further includes readable summary document data generated based on the information contained in the structured reporting data, The aforementioned summary document data includes at least one format, such as PDF or HTML, and has language identification information including language tags, and the presentation language is switched according to the settings of the delivery destination or recipient. Report generation system.

29. In the report generation system according to claim 6, the output unit delivers the structured report data via a network, and the delivery is protected by encryption and authentication of the recipient. Report generation system.

30. In the report generation system according to claim 29, the output unit encrypts and temporarily stores the structured report data when delivery is impossible, and resends the structured report data when delivery is possible. Report generation system.

31. In the report generation system according to claim 29, the output unit performs retransmission control based on the receipt acknowledgment from the delivery destination, and if the receipt acknowledgment is not obtained, switches to an alternative delivery destination or an alternative delivery route and continues delivery. Report generation system.

32. A report generation system according to any one of claims 1 to 5, further comprising a tamper-resistant storage unit that stores at least one of the structured report data, the time reference information, or a log relating to the generation or delivery of the structured report data in a tamper-resistant form, wherein the tamper-resistant form includes at least one of storage on an appendable recording medium or storage using a chained hash structure. Report generation system.

33. The report generation system according to claim 32 further comprises an audit interface that outputs information or results of integrity verification that enable verification of the integrity of the data stored by the tamper-proof storage unit. Report generation system.

34. The report generation system according to claim 5 further comprises an anonymization unit that performs anonymization processing on data included in or referenced from the structured report data, wherein the anonymization processing includes at least one of anonymization of video data, anonymization of location information, and anonymization of audio information. Report generation system.

35. In the report generation system according to claim 34, the anonymization unit performs the anonymization process based on a profile defining the anonymization strength, records a log related to the anonymization process, or separates and manages key information that can be used to restore the anonymization process. Report generation system.

36. A report generation system according to any one of claims 1 to 5, further comprising: a monitoring unit that detects no response or abnormality in the processing related to the generation of the structured report data; and a notification unit that notifies the system of an undetermined abnormality in response to the detection of the no response or abnormality, Report generation system.

37. The report generation system according to claim 36 further comprises a safety operation unit that, in response to notification of an undetermined abnormality, activates a safety operation that transitions the device on which the report generation system is installed or the device controlled by the report generation system to the safety side. Report generation system.

38. In the report generation system according to any one of claims 1 to 5, the structured report data includes screening information for candidate causes related to the abnormal event, and the screening information includes a value indicating affirmative, negative, or unknown for each candidate cause. Report generation system.

39. A report generation system according to any one of claims 1 to 5, further comprising a policy profile switching unit that switches at least one of the data items to be included in the structured report data, the intensity of the anonymization process performed on the data included in or referenced from the structured report data, or the output destination of the structured report data, according to the policy of the jurisdiction or organization. Report generation system.

40. A process of acquiring time-series data from multiple data sources in response to a detection signal related to an abnormal event, A process of aligning the aforementioned time-series data based on a single time criterion, A step of acquiring or estimating, based on the time-series data obtained from the plurality of data sources, a stimulus series relating to an external stimulus and a response series relating to the response to the external stimulus, A step of calculating a degree of agreement index representing the degree of agreement between the stimulus series and the response series, the step of calculating the degree of agreement index comprising at least one of the following processes: a process based on cross-correlation in the time domain, a process based on coherence in the frequency domain, and a process based on correlation based on time-frequency analysis. A step of generating structured reporting data including time reference information related to the aforementioned consistency and the degree of agreement index, A computer-based method for generating reports, including [the specified method].

41. A process of acquiring time-series data from multiple data sources in response to a detection signal related to an abnormal event, A process of aligning the aforementioned time-series data based on a single time criterion, A step of calculating an input quality index for the aforementioned aligned time series data, The process of calculating the aforementioned input quality index as a composite index based on at least two evaluation values ​​related to signal noise, loss, and gate fit, A step of generating structured report data including time reference information related to the aforementioned consistency, A computer-based method for generating reports, including [the specified method].

42. A process of acquiring time-series data from multiple data sources in response to a detection signal related to an abnormal event, A process of aligning the aforementioned time-series data based on a single time criterion, A step of calculating an input quality index for the aforementioned aligned time series data, A step of generating structured report data including time reference information related to the consistency, without using data that has been determined to be of low quality based on the input quality indicators, A computer-based method for generating reports, including [the specified method].

43. In response to a detection signal related to an abnormal event, A process for acquiring time-series data obtained from multiple data sources, The aforementioned multiple data acquisition sources include a process that includes information belonging to at least two groups from among movement, behavior, consciousness, autonomous response, gaze, voice, operation, and environment. A process of aligning the aforementioned time-series data based on a single time criterion, The process of extracting features from the aforementioned aligned time-series data, A step of calculating an anomaly index for the anomaly event based on the consistent time series data or the feature quantities, The aforementioned anomaly index is a value obtained by a calculation process that includes a monotonically non-decreasing aggregation process in the direction of increasing anomaly, and a monotonic transformation process on the result of the aggregation process, and the process belongs to the anomaly index family, in which anomaly indexes with different expressions due to differences in the monotonic transformation process are treated as equivalent anomaly indexes, A step of generating structured reporting data including time reference information related to the aforementioned alignment, the feature quantities, and the anomaly probability index, A computer-based method for generating reports, including [the specified method].

44. In response to a detection signal related to an abnormal event, A process for acquiring time-series data obtained from multiple data sources, The aforementioned multiple data acquisition sources include a process that includes information belonging to at least two groups from among movement, behavior, consciousness, autonomous response, gaze, voice, operation, and environment. A process of aligning the aforementioned time-series data based on a single time criterion, The process of extracting features from the aforementioned aligned time-series data, A step of calculating an anomaly index for the anomaly event based on the consistent time series data or the feature quantities, A step of generating structured report data which includes time reference information relating to the matching, the feature quantities, the anomaly probability index, and reference information to at least one waveform data or media data corresponding to the matching time series data, The aforementioned reference information is associated with at least one of the aforementioned time reference information or the aforementioned anomaly index, The aforementioned reference information includes multiple reference formats that can be selected depending on the recipient's capabilities or configuration policy. The process includes including at least one of the reference forms corresponding to Media, DocumentReference, and Binary in FHIR, A computer-based method for generating reports, including [the specified method].

45. A program for causing a computer to execute the report generation method described in any one of claims 40 to 44.

Images