Communication method and system using multimodal domain knowledge fusion based on successive latent tokens
The communication system addresses the challenge of integrating heterogeneous data by generating structured JSON messages through modality-specific encoders and domain knowledge alignment, improving accuracy and compliance with predefined schemas.
Patent Information
- Application Number
- KR1020260076300
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2026-04-28
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2046-04-28
AI Technical Summary
Conventional communication systems struggle to integrate and process heterogeneous data from multiple modalities such as image, vehicle CAN, and audio data consistently, failing to reflect semantic and causal relationships, leading to misinterpretation and non-compliance with predefined communication message schemas.
A communication system that generates a structured communication message by converting input data into latent feature sequences using modality-specific encoders, integrating them with domain knowledge, and aligning them to a predefined schema, ensuring consistency and compliance.
The system effectively integrates multimodal data into a structured JSON message, enhancing accuracy and reliability of state recognition, ensuring compliance with message schemas, and improving interoperability and data processing efficiency.
Smart Images

Figure 112026051479334-PAT00001_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a communication system and method for generating a structured communication message based on input data received from a plurality of modalities.
[0002] More specifically, the invention relates to a technique for generating a sequence of latent features for each modality from heterogeneous multimodal inputs such as image data, vehicle CAN data, and audio data, generating a sequence of continuous latent tokens that integrates the sequences of latent features by reflecting domain knowledge, and then generating a structured JSON communication message that complies with a predefined schema of a communication message. Background Technology
[0003] Generally, in systems containing multiple sensors or input means, such as vehicles, industrial equipment, robots, surveillance systems, and various electronic devices, different forms of data—such as image data, time-series control data, and audio data—are often collected together to determine the state or perform control based on this data. For example, in a vehicle environment, video information from cameras, CAN (Controller Area Network) data transmitted through the vehicle's internal network, and audio data collected from various acoustic sensors can represent the same situation or event from different perspectives. While data obtained from multiple modalities can provide useful information individually, it is necessary to interpret these data comprehensively to more accurately determine the actual state or to generate communication messages to be transmitted to subsequent systems.
[0004] However, since data obtained from multiple modalities differ in representation format, dimensional structure, sampling period, and time alignment method, it is not easy to integrate and process them in a consistent manner. In particular, because image data is represented based on spatial features, CAN data has asynchronous multi-channel time-series characteristics, and audio data includes changes in frequency components over time, it is difficult to sufficiently reflect their interrelated meanings by simply combining them in parallel or analyzing them individually. Furthermore, when each modality provides different clues regarding a single domain situation, failing to reflect the semantic or causal relationships between them risks misinterpreting the actual state or missing important events.
[0005] Meanwhile, conventional communication systems faced limitations in consistently generating structured communication messages that subsequent devices or services could immediately utilize, even when necessary information was extracted from multiple input data. For example, even when it was necessary to generate messages containing device status, event information, numerical measurements, and alarm information, different processing results could be obtained depending on the type of input data, or the format and constraints of message fields might not be satisfied. In particular, when generating messages in a structured format such as JSON, schema requirements—including field names, data types, allowed value ranges, and the presence of required fields—must be satisfied; however, conventional systems often remained at the level of simple data conversion or text generation without sufficiently considering these structural requirements.
[0006] A related background technology is Korean Patent Publication No. 10-2016-0170055 (December 14, 2016). The problem to be solved
[0007] The objective of the present invention to solve the above-mentioned problems is to provide a communication method and a system using multimodal domain knowledge fusion based on continuous latent tokens. means of solving the problem
[0008] To solve the above problem, a structured communication message is generated based on input data received from a plurality of modalities, and a communication system mounted on each modality (hereinafter referred to as the "system according to embodiments") is disclosed.
[0009] A system according to embodiments comprises: an input interface unit that receives input data from each of a plurality of modalities; a multimodal encoder unit that generates a latent feature sequence for each modality based on the received input data; and an integrated sequence generation unit that generates a continuous latent token sequence by integrating the latent feature sequences of each of the plurality of modalities based on domain knowledge corresponding to the plurality of modalities; and when communication is performed between the plurality of modalities, a communication message is generated based on a predefined communication message schema and the continuous latent token sequence.
[0010] Meanwhile, the plurality of modalities according to the embodiments include image data, vehicle CAN (Controller Area Network) data, and audio data.
[0011] Furthermore, the multimodal encoder unit according to the embodiments includes: an image encoder that generates a first latent feature sequence corresponding to the image data; a CAN encoder that generates a second latent feature sequence corresponding to the vehicle CAN data; and an audio encoder that generates a third latent feature sequence corresponding to the audio data.
[0012] Furthermore, the integrated sequence generation unit according to the embodiments performs cross-attention, perceiver bottleneck, feature vector concatenation, and multi-layer perceptron-based operations to derive associations between latent feature sequences of each of the plurality of modalities.
[0013] Meanwhile, a continuous potential token sequence according to the embodiments is composed of a plurality of slots, and each of the plurality of slots represents semantic information corresponding to different fields or field groups of the communication message. Effects of the invention
[0014] According to the embodiments, the present invention converts heterogeneous input data received from a plurality of modalities into a latent feature sequence tailored to the characteristics of each modality, generates a continuous latent token sequence by integrating the data by reflecting domain knowledge, and then generates a structured communication message that complies with a predefined communication message schema. Accordingly, there is an effect of reliably providing data having different formats and semantic systems to an external device, remote server, or application service by organizing them into a message of a consistent structure.
[0015] In addition, the present invention according to the embodiments can form an integrated semantic representation by reflecting the relationships between different modalities, such as image data, vehicle CAN data, and audio data, even if they represent the same state or event in different ways, thereby improving the accuracy of state recognition and the reliability of message generation compared to using only a single modality. In particular, by integrating multiple latent feature sequences by reflecting domain knowledge, states, events, and causal relationships that are difficult to reflect through simple feature combination alone can be more effectively reflected in a structured message.
[0016] Furthermore, the present invention according to the embodiments forms a sequence of consecutive latent tokens into an intermediate semantic expression composed of a plurality of slots, thereby enabling more systematic separation and reflection of information corresponding to different fields or field groups of a communication message. Accordingly, the semantic correspondence of each field can be more clearly secured during the subsequent message generation process, and the consistency and interpretability of the message structure can be improved.
[0017] Furthermore, the present invention according to the embodiments can generate a structured JSON communication message by reflecting field names, data types, and value constraints included in the communication message schema, thereby ensuring that the generated message is not only formally complete but also satisfies the structural conformity required in actual application environments. Accordingly, problems such as missing required fields, data type mismatches, and exceeding allowed ranges can be reduced, and message usability can be enhanced in application processes such as status monitoring, event reporting, control command generation, and subsequent data processing.
[0018] Furthermore, the present invention according to the embodiments provides a generated structured JSON communication message to an external device, a remote server, or an application service, thereby enabling a subsequent system to perform necessary processing based on integrated structured information without having to individually reinterpret multiple modality data. Accordingly, this has the effect of improving interoperability between systems, increasing data processing efficiency, and allowing for flexible application to various application environments. Brief explanation of the drawing
[0019] FIG. 1 is a diagram showing the overall configuration of a communication system according to embodiments. FIG. 2 is a diagram illustrating an internal processing process in which a communication message is generated from multimodal input data in a communication system according to embodiments. FIG. 3 is a diagram showing the detailed configuration of a multimodal encoder unit according to embodiments. FIG. 4 is a diagram showing the specific internal configuration of an integrated sequence generation unit according to embodiments. FIG. 5 is a diagram illustrating the process of a communication message generation unit according to embodiments generating a structured JSON message using a communication message schema and a sequence of consecutive latent tokens. FIG. 6 is a diagram showing the learning structure of an artificial intelligence model unit according to embodiments. Specific details for implementing the invention
[0020] The present invention is susceptible to various modifications and may have various embodiments; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention. Similar reference numerals have been used for similar components in the description of each drawing.
[0021] Terms such as first, second, A, B, etc., may be used to describe various components, but the components should not be limited by these terms. The terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of multiple related described items or any of the multiple related described items.
[0022] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.
[0023] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0024] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.
[0025] FIG. 1 is a drawing showing the overall configuration of a communication system (100) according to embodiments.
[0026] Referring to FIG. 1, a communication system (100) according to embodiments may include a data source unit (10a), a data receiving unit (101), a multimodal encoder unit (102), an integrated sequence generating unit (103), a communication message generating unit (104), a communication message schema storage unit (103a), and an external device (10b). The communication system (100) according to embodiments may generate a structured communication message based on input data received from a plurality of modalities, and by providing or transmitting the generated communication message to an external device (10b), support the interpretation and utilization of semantic information between a plurality of modalities in a consistent format.
[0027] The data source unit (10a) may refer to a configuration that generates or holds raw data corresponding to multiple modalities. For example, the data source unit (10a) may include a camera or image sensor that generates image data, an in-vehicle electrical system or time-series sensor network that generates vehicle CAN data, and a microphone or acoustic sensor that generates audio data. However, the data source unit (10a) according to the embodiments is not necessarily limited to a single device and may be distributed among multiple devices provided at different locations. Additionally, the data provided from the data source unit (10a) may be real-time streaming data or batch data that is accumulated and stored over a certain period.
[0028] The data receiving unit (101) can receive input data corresponding to each of the plurality of modalities from the data source unit (10a). According to embodiments, the data receiving unit (101) can receive image data, vehicle CAN data, and audio data by arranging them into an input set representing a single event, state, or situation associated with each other. For example, in a vehicle environment, the data receiving unit (101) can treat camera images acquired before and after a specific point in time, CAN time series data representing the vehicle state at the same point in time, and audio data representing an acoustic event at the same point in time as an associated input set. Accordingly, the multimodal encoder unit (102) described below can process each data to reflect the state, event, or context expressed by the plurality of modalities within the same domain.
[0029] The multimodal encoder unit (102) can generate a latent feature sequence for each modality based on input data transmitted from the data receiver (101). According to embodiments, the multimodal encoder unit (102) may include an image encoder unit (102a), a CAN encoder unit (102b), and an audio encoder unit (102c). The image encoder unit (102a) can generate a first latent feature sequence that reflects visual features, object features, scene features, or event-related features included in the image data. The CAN encoder unit (102b) can generate a second latent feature sequence that reflects time-series state information, sensor value changes, control signal changes, or correlations between asynchronous channels included in the vehicle CAN data. At this time, the CAN encoder unit (102b) can generate a latent feature sequence by reflecting mask information, timestamp information, or an asynchronous multi-channel structure together. The audio encoder unit (102c) can generate a third latent feature sequence that reflects acoustic features, event sound patterns, frequency characteristics, or time-series speech / non-speech features included in the audio data.
[0030] That is, the multimodal encoder unit (102) can convert the raw data of each modality into a latent feature sequence that reflects the physical, temporal, and semantic characteristics expressed by each modality, rather than simply abbreviating the raw data of each modality. Accordingly, multiple modalities with different expression formats can be compared, aligned, and integrated within a common semantic space in the integrated sequence generation unit (103) described later.
[0031] The integrated sequence generation unit (103) receives potential feature sequences of each of the plurality of modalities, integrates them by reflecting domain knowledge corresponding to the plurality of modalities, and can generate a continuous potential token sequence. Here, the domain knowledge is not simple general knowledge, but may be information representing entities, states, events, causal relationships, or interrelationships within a domain that are commonly expressed by the plurality of modalities. For example, in a vehicle domain, specific image features, CAN state changes, and audio events may correspond to a vehicle state anomaly, a warning situation, or a control target event, and the integrated sequence generation unit (103) can integrate the potential feature sequences by reflecting these correspondences.
[0032] According to the embodiments, the integrated sequence generation unit (103) can derive the association between the potential feature sequences using at least one of cross-attention, a perceiver bottleneck, feature vector concatenation, and a multilayer perceptron-based operation, and convert them into a single continuous potential token sequence. Additionally, the integrated sequence generation unit (103) can perform at least one of projection operation and normalization operation to align the dimensions or distribution of the potential feature sequences. The continuous potential token sequence thus generated can be composed of multiple slots, and each slot can express semantic information corresponding to different fields or field groups of a communication message. In other words, the continuous potential token sequence can function as an intermediate semantic expression encompassing the entire multimodal input data, and simultaneously as a potential message expression that the communication message generation unit (104), described later, references to generate a structured communication message.
[0033] The communication message schema storage unit (103a) can store a predefined schema of a communication message. The schema may include at least one of a plurality of field names, a data type corresponding to each field, constraints on values allowed for each field, whether a field is mandatory, a hierarchical structure between fields, or logical rules between fields. For example, in the case of a structured message for reporting a specific situation, fields such as event type, status code, time of occurrence, related sensor value, risk level, or recommended action may be predefined. According to the embodiments, the integrated sequence generation unit (103) can form a slot structure of a continuous potential token sequence more coherently by referring to the schema information stored in the communication message schema storage unit (103a), and the communication message generation unit (104) can generate a final message that satisfies structural constraints by referring to the same schema information.
[0034] The communication message generation unit (104) can generate a communication message based on a predefined communication message schema and the continuous potential token sequence. More specifically, the communication message generation unit (104) can generate a communication message consisting of structured JSON data by referencing together the slot-specific semantic expression included in the continuous potential token sequence and the schema stored in the communication message schema storage unit (103a), determining a value suitable for each field, and arranging them in a structured form. At this time, the communication message generation unit (104) can ensure that the generated communication message satisfies the data type, value range, and required field conditions by correcting or restricting fields that may violate the schema.
[0035] The generated communication message may be provided to or transmitted to an external device (10b). The external device (10b) may be a remote server, a control device, a monitoring terminal, an application service server, or another modality processing device. The external device (10b) may perform status monitoring, event reporting, alarm generation, derivation of control commands, or subsequent data processing using the structured JSON data. Accordingly, the communication system (100) according to the embodiments organizes heterogeneous data obtained from a plurality of modalities into a single structured message form and transmits it, thereby enabling the subsequent system to receive and process information in a consistent format without interpreting each modality individually.
[0036] Meanwhile, the communication system (100) according to the embodiments does not necessarily have to be entirely implemented within a single physical device. For example, at least part of the data receiving unit (101) and the multimodal encoder unit (102) may be performed at the edge device or sensor device side corresponding to each modality, and at least part of the integrated sequence generation unit (103), the communication message schema storage unit (103a), and the communication message generation unit (104) may be performed in a server or cloud environment. Conversely, each component of the system (100) may be integrated and performed within a single device. Accordingly, the communication system (100) according to the embodiments can be implemented in various ways, such as a single device type, a distributed processing type, or an edge-cloud collaborative type structure.
[0037] Additionally, while three modalities—image data, vehicle CAN data, and audio data—are illustrated in FIG. 1 as examples, the embodiments are not limited thereto. According to other embodiments, the plurality of modalities may further include radar data, lidar data, text log data, biosignal data, or other time-series sensor data. In this case as well, the multimodal encoder unit (102) generates a latent feature sequence corresponding to each modality, the integrated sequence generation unit (103) generates a continuous latent token sequence that integrates them by reflecting domain knowledge, and the communication message generation unit (104) can generate a structured communication message based thereon.
[0038] FIG. 2 is a diagram illustrating a specific internal processing process in which a communication message is generated from multimodal input data in a communication system according to embodiments. More specifically, FIG. 2 is a detailed diagram illustrating the detailed configuration of the multimodal encoder unit (102), integrated sequence generation unit (103), communication message generation unit (104), and communication message schema storage unit (103a) of FIG. 1.
[0039] Referring to FIG. 2, a communication system according to embodiments may receive a plurality of modality inputs, such as an image input (110), a CAN input (120), and an audio input (130), and generate a structured communication message based thereon. Here, the image input (110) may include image data provided from a camera, an image sensor, a stored image frame, or an external system; the CAN input (120) may include time-series CAN data generated in an internal vehicle network or equivalent multi-channel sensor data; and the audio input (130) may include audio data obtained from a microphone, an acoustic sensor, or stored acoustic data. However, the embodiments are not necessarily limited to the three modalities, and other sensor data or signal data may be added.
[0040] The multimodal encoder unit (102) can generate a latent feature sequence corresponding to each of the plurality of modality inputs. According to embodiments, the multimodal encoder unit (102) may include an image encoder unit (102a), a CAN encoder unit (102b), and an audio encoder unit (102c). The image encoder unit (102a) can generate a latent feature sequence for image data by reflecting visual features related to objects, scenes, change patterns, positional relationships, or events included in the image input (110). For example, the image encoder unit (102a) can generate a plurality of visual feature vectors using a Vision Transformer (ViT), a Convolutional Neural Network (CNN), or a similar image processing model.
[0041] The CAN encoder unit (102b) can generate a latent feature sequence for CAN data by reflecting time series state values, control signals, sensor change amounts, or correlations between multiple channels included in the CAN input (120). In particular, the CAN encoder unit (102b) according to the embodiments can generate latent features by reflecting characteristics of vehicle CAN data such as missing values, differences in packet generation times, and channel-specific asynchronousness. To this end, the CAN encoder unit (102b) can apply a one-dimensional CNN, a transformer-based time series encoder, or a similar time series processing model, and can also reflect mask information and timestamp information as needed. Accordingly, the CAN encoder unit (102b) can generate a latent feature sequence that contains not just a simple numerical sequence, but state changes on the time axis and the context of event occurrence.
[0042] The audio encoder unit (102c) can generate a latent feature sequence for audio data by reflecting speech, noise, warning sounds, mechanical sounds, or other acoustic events included in the audio input (130). For example, the audio encoder unit (102c) can generate a latent feature sequence representing frequency characteristics, time series patterns, and acoustic event characteristics using a wav2vec family model, a spectrogram-based CNN, or an equivalent acoustic processing model.
[0043] In this way, the multimodal encoder unit (102) generates multiple latent feature sequences that reflect the physical characteristics, representation format, and semantic characteristics of each input modality. That is, the image input (110), CAN input (120), and audio input (130) are raw data of different formats, but by passing through the multimodal encoder unit (102), they can be converted into latent feature sequences that can be aligned and integrated in subsequent steps.
[0044] The integrated sequence generation unit (103) receives potential feature sequences generated from the multimodal encoder unit (102) and can generate a continuous potential token sequence by integrating them by reflecting domain knowledge corresponding to multiple modalities. According to embodiments, the integrated sequence generation unit (103) may include a multimodal fusion unit (103b), a potential slot generation unit (103c), and an alignment processing unit (103d).
[0045] The multimodal fusion unit (103b) can derive associations between latent feature sequences corresponding to each of a plurality of modalities and fuse them into a single integrated feature representation. Here, the associations may be based on states, events, signs, or causal relationships that each modality commonly exhibits within the same domain. For example, if a specific image feature, a specific CAN state change, and a specific audio event represent the same abnormal state or the same situation, the multimodal fusion unit (103b) can treat these features as mutually related information by reflecting the domain knowledge. According to embodiments, the multimodal fusion unit (103b) can perform at least one of cross-attention, a perceiver bottleneck, feature vector concatenation, and a multilayer perceptron-based operation. Through this, latent feature sequences generated from different modalities can be aligned and integrated in a common semantic space.
[0046] The potential slot generation unit (103c) can generate a sequence of consecutive potential tokens consisting of multiple slots from the feature representation integrated by the multimodal fusion unit (103b). According to embodiments, each of the multiple slots can represent semantic information corresponding to different fields or field groups of a communication message. For example, one slot may be associated with information indicating the type of event, another slot may be associated with a state value or numerical information, and yet another slot may correspond to time information, location information, cause information, or action information. Accordingly, instead of generating a single compression vector, the potential slot generation unit (103c) can generate a sequence of consecutive potential tokens in which information is separated into multiple semantic units suitable for generating a structured communication message.
[0047] The alignment processing unit (103d) can align the dimensions, distribution, or representation format of the continuous latent token sequence generated by the latent slot generation unit (103c) so that the communication message generation unit (104), described later, can reliably generate communication messages. For example, the alignment processing unit (103d) can perform projection operations, normalization operations, or similar alignment operations to align the number of dimensions of the vector per slot or adjust the value distribution to a certain range. Accordingly, the continuous latent token sequence can be utilized as a more consistent input in the message generation stage.
[0048] The communication message schema storage unit (103a) can store a predefined schema of a communication message. The schema may include a plurality of field names, data types corresponding to each field, constraints on values allowed for each field, whether a field is mandatory, or rules regarding relationships between fields. For example, a communication message to be provided to a specific application service may be defined to include an event identifier, a state classification, a numeric sensor value, a risk level, a time of occurrence, and recommended measures. According to embodiments, the communication message schema storage unit (103a) may be connected to both the integrated sequence generation unit (103) and the communication message generation unit (104). Accordingly, the integrated sequence generation unit (103) can form a slot configuration more coherently by considering the schema, and the communication message generation unit (104) can generate a final message that satisfies structural constraints by referencing the same schema.
[0049] The communication message generation unit (104) can generate a structured communication message by referring to a sequence of consecutive potential tokens provided by the alignment processing unit (103d) and a schema stored in the communication message schema storage unit (103a). According to embodiments, the communication message generation unit (104) may include a message formatting unit (104a), a data encoding unit (104b), and a packetization unit (104c).
[0050] The message formatting unit (104a) can determine the composition, order, data type, and logical structure of fields to be included in a communication message by referring to the schema stored in the communication message schema storage unit (103a). Additionally, the message formatting unit (104a) can form a message structure such that appropriate values are placed in each field by correlating the semantic information per slot included in a continuous sequence of potential tokens with the semantic information of each field defined in the schema. That is, the message formatting unit (104a) can perform the role of mapping potential semantic expressions to a schema-based structure.
[0051] The data encoding unit (104b) can encode values corresponding to each field into an actual data representation format based on the structure formed by the message formatting unit (104a). For example, the data encoding unit (104b) can convert numeric data, string data, categorical data, or boolean data into a format that fits the schema. According to the embodiments, the data encoding unit (104b) can generate a message in a structured JSON data format, and in this process, it can perform missing field correction, prevent type mismatch, and check value constraints. Therefore, the data encoding unit (104b) can generate a structured JSON communication message that complies with a predefined message schema, rather than a simple string output.
[0052] The packetization unit (104c) can packetize the structured JSON data generated by the data encoding unit (104b) into a form suitable for actual communication transmission. For example, the packetization unit (104c) can add header information, identifier information, length information, or transmission protocol-related information to convert the structured JSON data into packets or data units so that it can be received by an external system. The result processed by the packetization unit (104c) can be provided as a communication message output (150).
[0053] The communication message output (150) can provide or transmit a structured communication message packetized by the packetizing unit (104c) to an external device, a remote server, or an application service. At this time, the external system can perform status monitoring, event reporting, generation of control commands, or subsequent data processing based on the received structured JSON communication message. Accordingly, the configuration of FIG. 2 according to the embodiments represents a detailed processing process for generating a semantically consistent structured communication message from a plurality of modality inputs and outputting it in a form that can be actually communicated.
[0054] Additionally, according to the embodiments, each component illustrated in FIG. 2 may be performed entirely within a single device, or some may be performed on the edge device side and others on the server or cloud side. For example, at least a portion of the multimodal encoder unit (102) may be performed at a terminal adjacent to each sensor or field device, and at least a portion of the integrated sequence generator (103) and the communication message generator (104) may be performed at a remote server. However, this arrangement is merely one example according to the embodiments, and the embodiments are not limited to a specific processing location.
[0055] Furthermore, while image input (110), CAN input (120), and audio input (130) are illustrated as examples in FIG. 2, in other embodiments, text input, radar input, lidar input, biosignal input, or other time-series inputs may be further included among the plurality of modalities. In this case as well, the multimodal encoder unit (102) may operate according to the same basic principle of generating a latent feature sequence for each modality, the integration sequence generation unit (103) may integrate them by reflecting domain knowledge, and the communication message generation unit (104) may convert them into a structured communication message.
[0056] FIG. 3 is a drawing showing the detailed configuration of a multimodal encoder unit according to embodiments, illustrating the process in which inputs of different modalities, such as image data, vehicle CAN data, and audio data, are converted into latent feature sequences through each encoder unit. More specifically, FIG. 3 can be understood as a detailed drawing of the multimodal encoder unit (102) of FIG. 2.
[0057] Referring to FIG. 3, a multimodal encoder unit according to embodiments may receive image data (110), vehicle CAN data (120), and audio data (130) as inputs. The image data (110) may be a still image or video frame provided from a camera, image sensor, or external storage device, the vehicle CAN data (120) may include time-series data, control signals, sensor values, status information, or event information generated in a vehicle internal network, and the audio data (130) may include voice, warning sounds, mechanical sounds, or other acoustic event data obtained from a microphone or acoustic sensor. However, each input data according to embodiments is not limited to the above examples and may be replaced or expanded with other forms of heterogeneous data occurring in the same or similar domains.
[0058] Image data (110) can be transmitted to an image encoder unit (310). The image encoder unit (310) can generate a first latent feature sequence (311) by reflecting visual features, object information, spatial arrangement, shape change, boundary information, scene context, or event-related visual patterns included in the image data (110). According to embodiments, the image encoder unit (310) may include a Vision Transformer (ViT), a Convolutional Neural Network (CNN), or a similar structure, and after decomposing an input image or video frame into a plurality of patches or feature units, it can output latent vectors corresponding to each patch or feature unit in the form of a sequence. Accordingly, the first latent feature sequence (311) can function not as a simple image compression result, but as a semantic feature sequence capable of comparison and alignment with latent features of other modalities in a subsequent fusion step.
[0059] Vehicle CAN data (120) can be transmitted to a CAN encoder unit (320). Vehicle CAN data (120) can generally be generated asynchronously across multiple channels, and since certain signals may be missing or sampling periods may differ from each other at some points in time, it may include time-series non-uniformity and non-consistency between channels, unlike image data or audio data. The CAN encoder unit (320) according to the embodiments may include a mask information processing unit (321) and a timestamp information processing unit (322) to reflect these characteristics.
[0060] The mask information processing unit (321) can generate or reflect corresponding mask information when at least some of the multiple channels included in the vehicle CAN data (120) do not exist at a specific point in time, have low reliability, or are missing. For example, the mask information processing unit (321) can generate information indicating whether there is a missing value at a time when a specific sensor value or control value is not received, and reflect this in the subsequent processing of the CAN encoder unit (320), thereby preventing missing data from distorting the overall feature extraction result. Accordingly, the CAN encoder unit (320) can distinguish and process the actual observed signal and the missing section.
[0061] The timestamp information processing unit (322) can reflect time information assigned to each packet, each channel value, or each event of the vehicle CAN data (120). Since the vehicle CAN data (120) may not have all channels occurring simultaneously at the same time, the timestamp information processing unit (322) can normalize the relationship on the time axis or assign additional features by reflecting the order of occurrence, time difference, and interval information of each signal. Accordingly, the CAN encoder unit (320) can derive potential features that consider the temporal relationship of the signals, rather than simply processing a list of values.
[0062] The CAN encoder unit (320) can generate a second latent feature sequence (323) by reflecting the processing results of the mask information processing unit (321) and the timestamp information processing unit (322). According to embodiments, the CAN encoder unit (320) can generate sequence-type features that reflect state changes, trends in control signals, correlations between multi-channel sensors, and abnormal event patterns included in vehicle CAN data (120) using a one-dimensional CNN, a transformer-based time series encoder, or a similar structure. Accordingly, the second latent feature sequence (323) can function as a semantic time series representation representing vehicle state or domain events, rather than as a simple numerical sequence conversion result.
[0063] Audio data (130) can be transmitted to an audio encoder unit (330). The audio encoder unit (330) can generate a third latent feature sequence (331) by reflecting frequency components, energy changes on the time axis, speech or non-speech patterns, warning sounds, friction sounds, collision sounds, or other acoustic events included in the audio data (130). According to embodiments, the audio encoder unit (330) may use a wav2vec family model, a spectrogram-based CNN, or a similar acoustic processing model. For example, the audio encoder unit (330) may receive a raw waveform or spectrogram as input, generate feature vectors for multiple frames or acoustic segments, and output them in the form of a sequence. Accordingly, the third latent feature sequence (331) can become an acoustic semantic representation that can be fused with other modalities in a subsequent step.
[0064] As such, the multimodal encoder unit illustrated in FIG. 3 can generate a latent feature sequence corresponding to each modality by processing input data having different representation formats and physical characteristics according to the characteristics of each modality. In other words, image data (110), vehicle CAN data (120), and audio data (130) are not easy to directly compare or integrate with each other in their raw form, but can be converted into a sequence form that can be utilized in a common latent representation space by passing through the image encoder unit (310), CAN encoder unit (320), and audio encoder unit (330).
[0065] In addition, the first latent feature sequence (311), the second latent feature sequence (323), and the third latent feature sequence (331) according to the embodiments may be subject to mutual alignment and fusion in the integrated sequence generation unit described later. For example, for a single domain event, an image modality may provide external changes or scene information, a vehicle CAN modality may provide changes in state values and control signals, and an audio modality may provide warning sounds or mechanical abnormal sounds. The integrated sequence generation unit according to the embodiments may receive each of these latent feature sequences, interpret them as multifaceted representations of the same domain situation, and integrate them into a continuous latent token sequence.
[0066] Meanwhile, in FIG. 3, image data (110), vehicle CAN data (120), and audio data (130) are illustrated as examples, but the embodiments are not limited thereto. According to other embodiments, text data, radar data, lidar data, biosignal data, or other time-series sensor data may be additionally included as additional inputs. In this case as well, a separate encoder unit corresponding to each additional modality may be provided to generate a latent feature sequence that reflects the characteristics of the data, and the generated multiple latent feature sequences may be used for subsequent integrated processing according to the same principle.
[0067] Additionally, each encoder unit illustrated in FIG. 3 may be executed entirely in a single physical device, some may be executed at the sensor or edge device side, and some may be executed in a remote server or cloud environment. For example, at least some of the image encoder unit (310) and audio encoder unit (330) may be executed near a data acquisition device to generate a preprocessed feature sequence, and the CAN encoder unit (320) may be executed in an in-vehicle control unit or gateway device. However, this arrangement is merely one example of an embodiment, and the execution location of each encoder unit may vary depending on the system configuration.
[0068] FIG. 4 is a diagram showing the specific internal configuration of an integrated sequence generation unit (400) according to embodiments, illustrating a process in which potential feature sequences generated from a plurality of modalities are fused by reflecting domain knowledge, structured into a potential slot form, and then converted into a continuous potential token sequence usable for generating a subsequent message through alignment processing. More specifically, FIG. 4 can be understood as a detailed diagram of the integrated sequence generation unit (103) of FIG. 2.
[0069] Referring to FIG. 4, the integrated sequence generation unit (400) according to the embodiments may receive a first potential feature sequence (314), a second potential feature sequence (327), and a third potential feature sequence (334) as inputs. Here, the first potential feature sequence (314) may be a visual feature sequence extracted from image data, for example; the second potential feature sequence (327) may be a time-series state feature sequence extracted from vehicle CAN data; and the third potential feature sequence (334) may be an acoustic feature sequence extracted from audio data. However, the embodiments are not limited to the three sequences, and additional potential feature sequences corresponding to additional modalities may be input.
[0070] The integration sequence generation unit (400) may not simply merge the first potential feature sequence (314), the second potential feature sequence (327), and the third potential feature sequence (334), but may integrate them by reflecting the domain context that is common to multiple modalities. To this end, the integration sequence generation unit (400) may include a domain knowledge block (410), a multimodal fusion unit (420), a potential slot generation unit (430), and an alignment processing unit (450).
[0071] The domain knowledge block (410) may be a block that stores or provides domain knowledge corresponding to the plurality of modalities. The domain knowledge may be information representing entities, states, events, rules, interrelationships, or causal relationships in a specific domain to which information input from the plurality of modalities belongs. For example, in a vehicle domain, a specific image pattern, a specific CAN state change, and a specific acoustic event may correspond to the same vehicle state, abnormal state, or control situation, and the domain knowledge block (410) may include information expressing these relationships. According to embodiments, the domain knowledge block (410) may include at least one of a pre-established rule set, a learned knowledge representation, an ontology structure, a key-value table, or knowledge information linked to an external database.
[0072] The multimodal fusion unit (420) can integrate the first latent feature sequence (314), the second latent feature sequence (327), and the third latent feature sequence (334) while reflecting the domain knowledge provided from the domain knowledge block (410). According to embodiments, the multimodal fusion unit (420) can derive the association between the latent feature sequences, determine whether features expressed in different modalities refer to the same state or event, or generate an integration result by reflecting the importance that each modality has at a specific point in time or in a specific context.
[0073] For example, the multimodal fusion unit (420) can precisely reflect the correspondence between different modalities by using a cross-attention technique to utilize one modality sequence as a query and another modality sequence as a key and value. Additionally, the multimodal fusion unit (420) can generate an integrated representation centered on highly interrelated features while compressing multiple large-scale feature sequences into a limited number of latent arrays using a perceiver bottleneck structure. Furthermore, the multimodal fusion unit (420) can generate an integrated feature representation that reflects the complex relationship between different modality features by connecting feature vectors for each modality using combination and multilayer perceptron methods and then performing a non-linear transformation. Figure 4 illustrates that these operations can be utilized selectively or combinatorially.
[0074] The potential slot generation unit (430) can form a potential expression structure composed of multiple slots from the integrated feature expression generated by the multimodal fusion unit (420). According to embodiments, the potential slot generation unit (430) can decompose or reconstruct the integrated feature expression into multiple slot units so that they correspond to different fields or field groups in the subsequent message generation step, rather than simply compressing the integrated feature expression into a single vector. For example, one slot may more strongly reflect a meaning related to the event type, another slot may have a greater association with a numerical state value, and yet another slot may have a greater association with visual information or time information. Accordingly, the potential slot generation unit (430) can form an intermediate semantic expression that facilitates the generation of a structured communication message.
[0075] Potential slots generated by the potential slot generation unit (430) can be transmitted to a subsequent alignment processing unit (450). The alignment processing unit (450) can align the dimensions, distribution, or representation format of the potential slots so that a subsequent communication message generation unit can utilize them more reliably. According to embodiments, the alignment processing unit (450) may include a dimension alignment unit (451) and a distribution normalization unit (452).
[0076] The dimension alignment unit (451) can adjust the number of dimensions of each slot or each latent vector, or perform projection operations to resolve dimensional differences between feature representations derived from multiple modalities. For example, the dimension alignment unit (451) can ensure consistency in subsequent processing by mapping latent feature sequences generated by different encoder units into a common dimensional space when they have different dimensional structures. Additionally, it may perform rearrangement or linear transformation on the slot vectors formed by the latent slot generation unit (430) to fit the representation space required in the subsequent message generation step.
[0077] The distribution normalization unit (452) can adjust the value distribution of the potential slots or dimensionally aligned feature representations. According to embodiments, the distribution normalization unit (452) can suppress the phenomenon in which information is excessively concentrated in some slots or a specific dimension value has an excessively dominant influence during the subsequent decoding process by adjusting the range of each feature value to a certain level or reducing the distribution deviation between specific slots. Through this, the continuous potential token sequence can be referenced more stably in subsequent steps.
[0078] In this way, the integrated sequence generation unit (400) according to the embodiments can convert a plurality of latent feature sequences into a continuous latent token sequence through fusion, slot structuring, and alignment processing that reflect domain knowledge, rather than simply maintaining them in parallel. Here, the continuous latent token sequence can be used as an intermediate semantic expression for generating a structured JSON communication message in a subsequent communication message generation unit. In other words, the continuous latent token sequence can be described as an integrated expression organized to have a structure that corresponds to each field of a communication message while maintaining individual features for each modality.
[0079] Meanwhile, according to the embodiments, the domain knowledge block (410) may not only affect the multimodal fusion unit (420), but may also be indirectly reflected in the potential slot generation unit (430) or the alignment processing unit (450). For example, in a specific domain, there may be cases where a specific field must be included, or where a specific state and a specific numerical value must be expressed together, and such domain rules may affect the slot configuration method or the alignment priority. Therefore, the domain knowledge block (410) can function as a higher-level information source that determines the processing direction of the entire integrated sequence generation unit (400).
[0080] Additionally, FIG. 4 illustrates an exemplary form in which the first latent feature sequence (314), the second latent feature sequence (327), and the third latent feature sequence (334) are directly input into the multimodal fusion unit (420), but the embodiments are not limited thereto. According to other embodiments, some latent feature sequences may be input after undergoing selective filtering or importance evaluation, and if a specific modality is absent, fusion may be performed using only the remaining modalities. In this case as well, the fusion sequence generation unit (400) may generate a continuous latent token sequence by reflecting available latent feature sequences and domain knowledge.
[0081] Additionally, the multimodal fusion unit (420), potential slot generation unit (430), and alignment processing unit (450) illustrated in FIG. 4 may all be performed on a single physical device, some may be performed on an edge device or field device, and others may be performed on a remote server or in a cloud environment. For example, the multimodal fusion unit (420) may be performed on the server side as it is responsible for fusion processing that involves a relatively large amount of computation, and the alignment processing unit (450) may be performed in a location close to the subsequent message generation unit. However, this is merely one example according to the embodiment, and the actual placement and execution location of each component may vary depending on the system implementation environment.
[0082] FIG. 5 is a diagram showing the detailed configuration of a communication message generation unit (600) according to embodiments, illustrating the process of deriving a structured JSON message (700) generated using a communication message schema (500) and a continuous latent token sequence (460). More specifically, FIG. 5 can be understood as a detailed diagram of the communication message generation unit (104) of FIG. 2.
[0083] Referring to FIG. 5, a communication message generation process according to embodiments may include a communication message schema (500), a sequence of potential tokens (460), a communication message generation unit (600), and a generated structured JSON message (700). The communication message generation unit (600) receives a sequence of potential tokens (460) output from an integrated sequence generation unit and can generate a structured communication message by referring to the communication message schema (500). According to embodiments, the structured communication message may be generated in JSON format and subsequently provided to or transmitted to an external device, a remote server, or an application service.
[0084] A communication message schema (500) may include information defining the structural format of a communication message and the attributes of each field. According to embodiments, the communication message schema (500) may include field name information (510), data type information (520), and value constraint information (530). The field name information (510) may indicate the names of multiple fields included in the communication message. For example, in the case of a status report message, field names corresponding to time information, device identifier, speed information, temperature information, event identification information, or alarm-related information may be defined. However, these field names are merely examples, and the actual types and number of fields may vary depending on the domain and application purpose.
[0085] Data type information (520) can define the format of data corresponding to each field. For example, a specific field can be defined as a String, Integer, Floating-point number, Boolean, Array, or Object. Accordingly, the communication message generation unit (600) can determine what type the value corresponding to a specific field should be represented in. Value constraint information (530) can define the range of values allowed for each field, whether input is required, format type, set of allowed strings, or logical constraints. For example, a specific numeric field may allow only values within a predetermined range, a specific identification field may allow only strings of a defined format, and a specific field may be defined as a mandatory field that must exist. Therefore, the communication message schema (500) can provide not only a list of fields but also standard information that ensures the generated message described below is structurally valid and semantically consistent.
[0086] A continuous latent token sequence (460) may be a latent semantic expression composed of multiple slots. As described in FIG. 4, the continuous latent token sequence (460) may be the result generated after latent feature sequences corresponding to multiple modalities are fused with domain knowledge and then undergo latent slot structure and alignment processing. According to embodiments, each slot included in the continuous latent token sequence (460) may express semantic information corresponding to different fields or field groups of a communication message. For example, one slot may more strongly reflect state information, another slot may reflect event information, and yet another slot may more strongly reflect numerical measurement values or time-related information. However, the continuous latent token sequence (460) according to embodiments is not limited to a specific slot and a specific field necessarily having a one-to-one correspondence; a single slot may contribute to the creation of multiple fields, or multiple slots may jointly contribute to the creation of a single field.
[0087] The communication message generation unit (600) may be a functional block that derives a structured JSON message (700) generated based on the communication message schema (500) and a sequence of potential tokens (460). According to embodiments, the communication message generation unit (600) may include a schema verification unit (610), a token-field mapping unit (620), and a JSON decoder unit (630).
[0088] The schema verification unit (610) can verify the structural requirements that the target message must satisfy by referring to the field name information (510), data type information (520), and value constraint information (530) included in the communication message schema (500). For example, the schema verification unit (610) can verify the existence of required fields, the suitability of the data type of each field, whether the allowable range of numerical values is satisfied, the suitability of the string format, or the relationship conditions between multiple fields. According to the embodiments, the schema verification unit (610) may not only perform verification after final creation but may also function as a reference standard in the token-field mapping and decoding process described later. That is, the schema verification unit (610) can be understood as a control block for ensuring structural suitability throughout the communication message creation process.
[0089] The token-field mapping unit (620) can determine which field or field group of the communication message schema (500) corresponds to each slot or each potential token included in the continuous potential token sequence (460). According to embodiments, the token-field mapping unit (620) can select a token or set of tokens suitable for each field by comparing the semantic expression included in the continuous potential token sequence (460) with the meaning of the field defined in the schema. For example, a slot that strongly reflects the state-related meaning may be prioritized for a field representing a state, and a slot that reflects numerical information may be corresponded to a field corresponding to a numerical measurement value. Additionally, in the case of a field having a complex meaning, multiple slots may be referenced together. Accordingly, the token-field mapping unit (620) can perform a key role in connecting the continuous potential token sequence (460), which is a potential semantic expression, to the structural message field.
[0090] The JSON decoder unit (630) can generate values corresponding to each field in the form of actual messages based on the token-field mapping results. According to embodiments, the JSON decoder unit (630) can generate a structured message by determining the representation format of the value according to the type of each field and arranging it according to the JSON structure. For example, numeric fields can be encoded as numeric values, string fields as string values, and logical fields as true or false, and nested objects or array structures can also be generated as needed. Additionally, the JSON decoder unit (630) can refer to the verification results of the schema verification unit (610) to suppress the generation of values with unsuitable types, correct missing fields, or adjust so that unacceptable values are not output. Accordingly, the JSON decoder unit (630) can perform the role of a structured message generator that complies with the schema, rather than a simple text generator.
[0091] The generated structured JSON message (700) may be the output result of the communication message generation unit (600). According to embodiments, the generated structured JSON message (700) may be a structured JSON object containing a plurality of fields and field values. This object may include time information, identification information, status information, numerical information, event information, or alarm information. However, the JSON form illustrated in FIG. 5 is merely an example for convenience of explanation, and the actual types, number, nesting levels, and array structures of the fields may be changed according to the domain and application purpose. The generated structured JSON message (700) may subsequently be provided to or transmitted to an external device, remote server, application service, or other module, and may be utilized for status monitoring, event reporting, generation of control commands, or subsequent data processing.
[0092] Meanwhile, according to the embodiments, the communication message generation unit (600) may be extended to other structured message formats other than JSON format. For example, in a specific application environment, XML, Protocol Buffers, YAML, or other machine-readable structured data formats may be required, and in this case, a structured communication message can be generated based on the same principle using the communication message schema (500) and a continuous latent token sequence (460). However, for convenience of explanation, the embodiments are described with a focus on the JSON format.
[0093] Additionally, according to the embodiments, the communication message schema (500) may be stored in a fixed form or may be dynamically updated depending on the application environment. For example, if the message structure required by a specific external device or a specific application service changes, part of the field name information (510), data type information (520), or value constraint information (530) may be modified, and the communication message generation unit (600) may generate a structured JSON message (700) of a new format by referring to the updated schema. Accordingly, the communication message generation structure according to the embodiments can flexibly respond to various communication protocols or application scenarios.
[0094] Additionally, the configuration of the communication message generation unit (600) illustrated in FIG. 5 may be entirely performed within a single physical device, or some may be performed at an edge device, and some at a remote server or cloud environment. For example, the token-field mapping unit (620) may be performed at a location close to the system that generated the continuous potential token sequence (460), and the JSON decoder unit (630) and schema verification unit (610) may be performed on the application service server side. However, this configuration is merely one example of an embodiment, and the actual execution location may vary depending on the system design.
[0095] FIG. 6 is a diagram showing the learning structure of an artificial intelligence model unit (400) according to embodiments, illustrating a process of training the artificial intelligence model unit (400) using an input data set corresponding to a plurality of modalities and a correct answer message, and optimizing the model by calculating a loss based on the difference between the generated communication message and the correct answer message. More specifically, FIG. 6 can be understood as a diagram for explaining the operation of the model learning unit described in claims 10 to 14, and represents the encoding, fusion, and structured message generation processes described in FIG. 3 to 5 from a learning perspective.
[0096] Referring to FIG. 6, the learning structure according to the embodiments may include learning data (800), an artificial intelligence model unit (400), a generated communication message (900), and a loss function block (1000). According to the embodiments, the learning data (800) may include an input data set (810) and a correct answer message (820). The input data set (810) may include image data (811), CAN data (812), and audio data (813), and these may be a cross-related data set representing the same domain situation, the same event, the same state, or the same context. For example, image data capturing a scene at a specific point in time, CAN data representing the vehicle state before and after the same point in time, and acoustic data acquired in the same situation may be composed of a single learning input sample. However, the input data set (810) according to the embodiments is not limited to the three modalities mentioned above and may further include other sensor data, text data, or time series data.
[0097] The correct answer message (820) may be a target structured message corresponding to the input data set (810). According to embodiments, the correct answer message (820) may be a structured JSON message that complies with a predefined communication message schema and may include a state, event, numeric value, warning information, control-related information, or other domain information represented by the input data set (810). That is, the correct answer message (820) may be a reference message that the artificial intelligence model unit (400) must ultimately generate from the input data set (810).
[0098] The artificial intelligence model unit (400) can receive an input data set (810) and output a generated communication message (900). According to embodiments, the artificial intelligence model unit (400) may include an encoder (410), a fusion unit (420), and a decoder (430). The encoder (410) can convert image data (811), CAN data (812), and audio data (813) into latent feature sequences according to their respective characteristics. The fusion unit (420) can integrate the latent feature sequences by reflecting domain knowledge and derive a continuous latent token sequence suitable for generating a subsequent message or an intermediate semantic expression equivalent thereto. The decoder (430) can generate a communication message based on the intermediate semantic expression and, if necessary, output it in a structured message format by referring to a communication message schema. Accordingly, the generated communication message (900) can be derived as a prediction result for the input data set (810).
[0099] The generated communication message (900) can be provided as a loss function block (1000). The loss function block (1000) can quantitatively calculate how much the output of the artificial intelligence model unit (400) differs from the target message by comparing the generated communication message (900) with the correct answer message (820). According to embodiments, the loss function block (1000) may include a first loss item (1010), a second loss item (1020), and a third loss item (1030).
[0100] The first loss item (1010) may be a loss item that reflects a difference in token units. For example, the first loss item (1010) may reflect how much each token or each value included in the generated communication message (900) differs from the corresponding token or corresponding value of the correct answer message (820). Accordingly, the first loss item (1010) may function as a base loss that evaluates the sequence generation accuracy of the entire message. According to embodiments, the first loss item (1010) may be implemented as a cross-entropy loss, a sequence prediction loss, or an equivalent loss function.
[0101] The second loss item (1020) may be a loss item that reflects a schema violation penalty. According to the embodiments, if the generated communication message (900) violates a predefined communication message schema, for example, if a required field is missing, the data type does not match, the value constraint is exceeded, or an unallowed structure is formed, the second loss item (1020) may impose a corresponding penalty. Accordingly, the artificial intelligence model unit (400) can be trained to generate a message that satisfies structural constraints, going beyond simply outputting a string similar to the correct answer message.
[0102] The third loss item (1030) may be a loss item that reflects field-unit accuracy. According to embodiments, the third loss item (1030) may reflect the accuracy of values for each field, errors in numeric fields, whether categorical fields match, whether specific key fields are accurately filled, or consistency at the field group level. For example, for fields such as time information, status values, event identification information, or numeric measurement values, it may evaluate whether semantically accurate values have been generated as well as simple token matching. Thus, the third loss item (1030) may provide precise accuracy criteria to enhance the practical usability of structured messages.
[0103] According to embodiments, the loss function block (1000) can calculate the learning loss of the artificial intelligence model unit (400) using at least one of the first loss item (1010), the second loss item (1020), and the third loss item (1030). Additionally, according to other embodiments, the loss function block (1000) may further incorporate normalization items in addition to the loss items. For example, normalization items such as KL divergence, L2 regularization, token diversity penalty, or low-rank constraint may be added to control the information density of a continuous latent token sequence, to suppress the phenomenon of information being excessively concentrated in a specific slot, or to adjust the complexity of the model. In this case, the loss function block (1000) can calculate the final learning loss by reflecting the basic loss items and the normalization items together.
[0104] The loss calculated by the loss function block (1000) can be used to update the parameters of the artificial intelligence model unit (400). That is, the model learning unit according to the embodiments can adjust the weights included in the encoder (410), fusion (420), and decoder (430) of the artificial intelligence model unit (400) by applying backpropagation, gradient descent-based optimization, or equivalent learning techniques based on the loss. As a result, when the same or similar input data set (810) is subsequently input, the artificial intelligence model unit (400) can generate a communication message that is more accurate, has higher schema conformance, and has better field-unit consistency.
[0105] Additionally, according to the embodiments, the input data set (810) and the correct answer message (820) used in the learning process may be obtained from a learning database consisting of multiple samples. In this case, the learning data may be configured to encompass various situations, various states, various noise conditions, and various domain events. For example, image data may include lighting, viewpoint, background changes, etc., CAN data may include various patterns according to normal state, warning state, and abnormal state, and audio data may include different environmental noise or event sounds. Accordingly, the artificial intelligence model unit (400) may be trained to generate structured communication messages corresponding to complex situations that may occur in an actual operating environment.
[0106] Meanwhile, according to the embodiments, the learning structure of FIG. 6 is not limited to offline learning. For example, additional learning or fine-tuning may be performed using communication messages generated during system operation, actual subsequent processing results, or user correction results. In this case, the system can secure more accurate structured communication message generation performance while adapting to new data accumulated in the operating environment. In addition, if a change in the message format required by a specific external device, a specific remote server, or a specific application service occurs, relearning or additional learning may be performed based on the updated correct answer message or the updated schema.
[0107] In addition, the learning structure described in FIG. 6 can be directly linked to the utilization structure of FIG. 7. That is, the artificial intelligence model unit (400) learned in FIG. 6 can be optimized so that the subsequently generated structured communication message can be used for state monitoring, event reporting, control command generation, and subsequent data processing in an external device, remote server, or application service. In other words, the learning process of FIG. 6 can be described as a foundational process that not only improves the accuracy of message generation but also ensures that the generated communication message can be used structurally appropriately and semantically validly in the actual utilization stage. For example, when used for state monitoring, the accuracy of state-related fields may be important; when used for control command generation, the accuracy of command targets and numerical control parameters may be important; and when used for event reporting, the accuracy of event identification information and the time of occurrence may be important. The loss function block (1000) according to the embodiments may adjust the weight of each loss item or reflect different importance for specific fields to suit these utilization purposes.
[0108] Additionally, each component of the artificial intelligence model unit (400) and the loss function block (1000) illustrated in FIG. 6 may be performed entirely on a single physical device, or some may be performed on an edge device, and some on a remote server or in a cloud environment. For example, training data management and loss calculation may be performed on the server side, and fine-tuning of some encoder parameters may be performed on a field device or terminal side. However, this arrangement is merely an example, and the actual execution location and processing method may vary depending on the implementation purpose, computational resources, and system environment.
[0109] Accordingly, FIG. 6 can be described as a diagram showing a structure in which an artificial intelligence model unit (400) according to embodiments is trained using an input data set (810) and a correct answer message (820), and is optimized based on a loss reflecting the difference between the generated communication message (900) and the correct answer message (820), schema suitability, field unit accuracy, etc. The model trained in this way can generate structured communication messages during actual operation and be effectively utilized for status monitoring, event reporting, control command generation, and subsequent data processing in external devices, remote servers, and application services.
[0110] The embodiments of the present invention disclosed in this specification and drawings are provided merely as specific examples to facilitate the explanation of the technical content of the present invention and to aid in understanding the present invention, and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art that other variations based on the technical concept of the present invention are possible in addition to the embodiments disclosed herein.
[0111] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims.
Claims
Claim 1 A communication system mounted on each modality, which generates a structured communication message based on input data received from a plurality of modalities, comprising: an input interface unit that receives input data from each of the plurality of modalities; a multimodal encoder unit that generates a latent feature sequence for each modality based on the received input data; and an integrated sequence generation unit that generates a continuous latent token sequence by integrating the latent feature sequences of each of the plurality of modalities, reflecting domain knowledge corresponding to the plurality of modalities. A communication system comprising, wherein the continuous latent token sequence is composed of a plurality of slots, each of the plurality of slots includes semantic information corresponding to different fields of the communication message, and the integrated sequence generation unit derives the association between the latent feature sequences by reflecting the domain knowledge, determines whether the features expressed in each modality represent the same state, forms the slot structure of the continuous latent token sequence by referring to a predefined schema of the communication message, and when performing communication between the plurality of modalities, the communication message is generated by selecting a slot or latent token corresponding to each field or field group defined in the schema, and structuring a field value based on the selected slot or latent token to fit the data type and value constraints of the schema. Claim 2 A communication system according to claim 1, wherein the plurality of modalities include image data, vehicle CAN (Controller Area Network) data and audio data. Claim 3 In paragraph 2, the multimodal encoder unit comprises: an image encoder that generates a first latent feature sequence corresponding to the image data; a CAN encoder that generates a second latent feature sequence corresponding to the vehicle CAN data; and an audio encoder that generates a third latent feature sequence corresponding to the audio data; a communication system. Claim 4 A communication system according to claim 1, wherein the integrated sequence generation unit performs cross-attention, perceiver bottleneck, feature vector concatenation, and multi-layer perceptron-based operations to derive association relationships between latent feature sequences of each of the plurality of modalities. Claim 5 delete
Citation Information
Patent Citations
Heterogeneous data fusion device and method for AI large model pre-training and medium
CN120974435A
Integrated compressing apparatus and integrated decompressing apparatus for multimodal data including legacy system
KR1020250173859A
