SYSTEM AND METHOD FOR PROCESSING ASYNCHRONOUS MULTIMEDIA USER DATA ACCORDING TO PERSON-SPECIFIC REFERENCES, WITH REGARD TO QUALITY AND CURRENCY VALUES.

Abstract

The invention is derived from multiple samples obtained at different acquisition times or at different sampling frequencies. It relates to a system and method for processing modular user data. The system; User device receiving user data (100), preprocessing performing technical integrity check module (140), feature module (160) which generates modality-specific feature data, person-specific database storing reference data (150), trust which creates quality and currency values module (170) common evaluation of feature data with different timestamps Combining module (180) that combines asynchronously for time and user status It includes the status module (190) which generates the data. Common feature data Its contribution to the evaluation is determined according to quality and timeliness values; low quality Alternatively, the effect of the missing modality can be reduced, or the relevant modality can be excluded. In this way, user data from different age and technical skill levels are processed according to individual references. An architecture is provided that is conducive to their collaborative processing.

Description

1 TARIFF QUALITY AND CURRENCY OF ASYNCHRONOUS MULTIMEDIA USER DATA. A SYSTEM FOR PROCESSING INDIVIDUALS BASED ON THEIR VALUES AND PERSONAL REFERENCES. METHOD 5 Technical Area The invention relates to computer-applied data processing, signal processing, multimodal data integration, and time. It relates to series analysis and person-specific reference modeling areas, and at different times 10 or a common group of heterogeneous user data obtained at different sampling frequencies a system that ensures reliable processing in terms of evaluation time and It relates to the method. The invention specifically applies to audio, visual, cognitive measurement, contextual user data, and usage data. 15 derived from modalities with different acquisition and processing characteristics, such as Feature data includes technical quality, data age, currency, and personalized historical reference information. combining them asynchronously, using low reliability or outdated methods limiting the impact of lost data on the combined result and over time It is related to the creation of machine-processable user state data that changes. (Word 20) The modalities in question are heart rate, heart rate variability (HRV), sleep duration and stages. activity, movement, respiration, skin conductivity, temperature, or similar biophysiological or It can also include digital biomarker data from wearable devices. State of the Art 25 Human-computer interaction, sentiment analysis, user behavior analysis, and multimodal data. in processing areas, audio, video, text, biophysiological measurements or Several systems are known for the combined evaluation of behavioral data. These The use of different data sources together in systems, 30 based on a single data type According to the assessments, this allows for the acquisition of more technical information; However, data from different sources should not have the same technical quality and the same temporal range. the assumption that it is valid is not always true under real-world conditions of use It is not available. 35 2 While short audio data points received from a user can be generated at frequent intervals, image data... It is preferred that this be done only at certain checkpoints or upon user approval. Cognitive measurements, brief user responses, and usage logs can also be used, along with voice and audio. Image data can be generated at different times. In such a structure, the same Data acquisition times, data ages, and technical quality to be used in the evaluation 5 Their levels and inference confidences differ. These differences are taken into account. Direct joining processes carried out without taking into account old or low-quality materials. This can cause the combined effect of the modality to be unduly significant. In patent document number WO2016195474A1, a user's face, speech and 10 Monitoring body language characteristics through various sensors, machine learning. classification using and resulting in fusion processes of the obtained information. A method for creating a comprehensive user profile is described. The document also includes profile information obtained from various sources. 15 Creating a multidimensional dynamic profile state by combining them It is anticipated that the aforementioned solution will not work at different sampling frequencies and at different times. a common evaluation period for user data obtained during acquisition times Determining the age of the data, deriving the currency value from the age of the data in question, assessment together with the modality-specific technical quality value and the individual's background Data integration based on asynchronous processing relative to reference data 20 Its architecture does not reveal this. Patent document number CN112686048A mentions sound, semantic text, and face. the generation of emotion scores from these statements through separate learning models and words The subject is combining scores after smoothing and weighting processes. 25 This approach is explained by the evaluation obtained from different modalities. This shows the combined use of scores obtained at different times. Calculating the currency of data relative to an evaluation time is a low-tech process. Reducing or excluding the contribution of a modality with quality is accepted. Creating personalized references from historical data and controlling the reference process 30 It does not integrate the update processes into the same technical processing chain. Within the known techniques, multimodal classification, data weighting, personalization, and While time series processing concepts are used separately in different applications; Heterogeneous user data have different acquisition times and technical quality 35 3 outdated or unreliable data that directly incorporates these levels into the data processing process limiting the user's contribution to shared representation and their own accepted historical data. There is still a need for an integrated processing structure that references data. In addition, continuous acquisition of image data increases the number of camera activations. the volume of raw data to be processed, the amount of data transfer, and device resource usage 5 It can increase [the system's capacity]. However, in a structure where image data is acquired less frequently, the system is lacking. It can continue working with the modality by combining the old image data with new audio data. sharing data that is not evaluated in terms of timeliness and does not meet technical quality requirements. It should be possible to exclude it from the evaluation. Therefore, data acquisition and quality... control, updating, personal reference management, and asynchronous merging 10 A new technical solution is needed where these processes work together. Purpose of the Invention The primary purpose of the invention is to obtain 15 at different acquisition times or at different sampling frequencies. a common evaluation time for the multimodal user data obtained a system that can process data, taking into account differences in data quality and data age. The goal is to develop systems and methods. One of the aims of the invention is to define quality 20, which refers to the technical usability for each modality. the value and the timeliness value which expresses the data age according to the evaluation time the creation and contribution of these values ​​to the unification of the modality outdated or low-quality data can be used in determining user status. The goal is to limit its disproportionate impact on the data. Another purpose of the invention is to compare user data to a general population average or a fixed rate. Instead of being processed according to a general threshold value, it meets the user's technical acceptance criteria. associating the created personal reference data with historical characteristic data and thus a machine-processable representation of intra-temporal change to provide. 30 Another objective of the invention is the absence of data belonging to a modality, requiring completeness. failure to meet the condition or the quality value falling below the specified threshold level to prevent the termination of the entire processing chain in this case; the contribution of the relevant modality 4 to reduce, exclude the modality in question, or create a new data acquisition request The aim is to ensure that the process continues with the remaining available modalities. Another aim of the invention is to eliminate the need for continuous acquisition of image data, at specified checkpoints or data uncertainty, age of previous image, change 5 depending on technical conditions such as size, user choice, or device resource availability The goal is to create a structure that makes it possible to acquire it. Another aim of the invention is to produce only a single class or final score as a result of combination. Instead of producing; direction of change according to reference, confidence information, uncertainty information and temporal 10 by creating user status data that may include one or more trend data points a data structure that is traceable and extensible for subsequent computer-implemented processes to provide. Another objective of the invention is to process user status data into a controlled output library of 15 to allow matching and restrictions on eligibility, exclusion, order, duration and repetition. By enabling the selection of outputs below, the user is guided through the technical evaluation chain. The goal is to separate the communication layers offered. The invention is intended to achieve the above purposes; it was acquired at different times or in different 20 Multimode user data obtained at sampling frequencies, at least one of which is sensor-based. a system for processing data, with at least two users from different modalities enabling the system to import data with timestamps showing the acquisition times. a user device, that user data, modality-specific technical integrity and A pre-processing module that operates in terms of quality conditions, 25 by the pre-processing module A feature module that generates modality-specific feature data from processed user data. Personalized reference created from the user's accepted historical data. a database that stores data, quality which refers to the technical usability of feature data. the value and the timeliness value, which expresses the age of the data according to an evaluation time. a trust module that processes feature data based on personalized reference data 30 and feature data with different timestamps at the time of joint evaluation by determining its contribution based on at least the quality and up-to-dateness values ​​of the feature in question. a merge module that combines data asynchronously, merge module machine-processable and time-varying feature data combined by It includes a status module that generates user status data. 35 Explanation of the Figures Figure 1 shows the main components of the system in question and the data between these components. It is a representative block diagram showing the flow. Figure 2 shows the requirement for simultaneous collection of user data from different modalities. that is not the case, as the data in question were acquired at different times and with different sampling. It shows the sample data acquisition pattern in which the frequencies were obtained. Figure 3 shows the quality and timeliness of modality-specific feature data. evaluation, correlation with personalized reference data, and shared user This is a representative data processing flow showing the generation of state data. 10 Figure 4 shows the user status data being associated with the protocol module and the logging module. transferring it to the user interface and user interaction of the security module This is a representative flowchart showing its implementation. Description of Part References 15 100. User device 110. Microphone Camera 120 130. Data interface 20 140. Preprocessing module 150. Database 160. Feature module 170. Trust module 180. Merging module 25 190. Status module Protocol Module 200 210. Registration module 220. User interface 230. Security module 30 Detailed Description of the Invention The invention involves the acquisition of multimodal user data simultaneously or with the same frequency. Without the obligation to do so, the common 35 of the data in question, which have different acquisition times 6 It is ensured that they are processed together in terms of an evaluation time. The invention is a In its implementation, the system uses: user device (100), microphone (110), camera (120), data interface (130), preprocessing module (140), database (150), feature module (160), trust module (170), merging module (180), status module (190), protocol module (200), It includes the registration module (210), user interface (220) and security module (230). (Speech 5) All of the subject elements can be run on a single user device (100), some of the functions will be on the user device (100) and some on a remote server It can also be implemented on a client-server or split-process architecture. User device (100), mobile device containing at least one processor, memory and communication unit, 10 This can be done via tablet computer, personal computer, or web client. The user device (100), microphone (110) and camera (120) as built-in hardware. It can include or communicate with an external microphone (110) and camera (120). Data The interface (130) includes cognitive measurement data, short user response, usage log, contextual It enables the entry of data or other authorized user data into the system. 15 Within the scope of this specification, modality, acquisition source or processing characteristic It refers to a type of data that differs from other user data in terms of its characteristics. Voice data is a type of data. one modality, image data another modality, and cognitive measurement data yet another modality. It can be considered as such. In terms of invention, asynchronous processing involves at least two 20 It is not mandatory for the modality data to be generated at the same acquisition time, and different These data, which have timestamps, will then be jointly evaluated. This means that they can be brought together in terms of time. The sound data obtained from the microphone (110) is 25 at the first sampling frequency. Each audio data record includes a timestamp indicating the time the data was acquired. and is associated with a modality identifier that identifies the source of the data. Audio data. The duration can vary depending on the application needs, but it is a short and repeatable acquisition. The use of windows reduces the processing load of the user device (100) and user interaction It provides a structure that is suitable for limiting its duration. 30 The camera (120) must be activated at the same frequency or continuously as the microphone (110). is not. In one implementation, the camera (120) is less than the acquisition frequency of the audio data. It operates at a certain sampling frequency and processes image data for baseline, intermediate comparison, and 7 The final comparison is obtained at the control points. Limiting factors regarding product application. In an example that does not exist, the checkpoints in question are the eighth, fourteenth, and twenty-first. These can be specified in days. This timing is merely an example of an application, and The scope of protection for the invention is not limited to specific calendar days. When new image data is to be received from the camera (120), it is connected to a fixed program. It is not necessary. User status generated by status module (190) the uncertainty level of the data, the time elapsed since the acquisition of the previous image data, the last magnitude of change in assessments, number of available modalities, user choice or taking into account the processing, energy or communication resources of the user device (100) 10 The next image acquisition time can be determined. The user device (100), this using at least one of the parameters the camera (120) activation time It can determine and, if necessary, change the time in question. In this case Instead of keeping the camera (120) constantly on, it is technically necessary to wait for new image data. It can be activated at a checkpoint once it is detected. 15 Cognitive or contextual user data received via the data interface (130), microphone (110) and data obtained via camera (120) are not subject to the same sampling frequency. Response time, error pattern, task completion time, short text response, usage. time, application completion information, or contextual data entered by the user 20 They can be created at different times. The data records in question are also created at different times depending on the acquisition time and... It is associated with the modality ID and passed on to the next processing steps. Raw user data received by user device (100) preprocessing module (140) It is subjected to modality-specific technical integrity and quality controls. Sound 25 Data includes recording time, interrupt rate, signal-to-noise ratio, sampling integrity, and amplitude. saturation and the application of speech-to-text conversion in question The confidence level regarding the conversion can be evaluated. Lighting for image data. level, blur, face visibility, exposure angle, image integrity, motion detection Distortion and square continuity can be checked. 30 in terms of cognitive measurement data. Completion status, device latency, and outlier response times are technical control parameters. It can be used as such. 8 Data processed by the preprocessing module (140) is sent to the feature module (160) is transferred. Feature module (160), machine-processable feature data for each modality. It produces speech rate, pause distribution, energy change, and pitch from audio data. Data on variability, speech continuity, and linguistic representation can be generated. Image The data includes facial movement values, eye opening, eyebrow movement, and facial expression 5. Geometry, expression change, and temporal motion data can be obtained. Cognitive Response time, error pattern, attention indicator, and completion indicator from measurement data. It can be derived. The feature data generated by the feature module (160) is a single numerical value out of 10 It does not have to be. A feature containing one or more dimensions for each modality. The vector can be generated; along with the feature vector, the modality ID and acquisition information are also included. Time, quality sub-metrics, data version, and feature inference confidence (if any) are all in the same data record. It can be stored within. Thanks to this structure, all of the raw audio or video data is preserved. Derived technical data that does not necessarily have to be transferred to subsequent processing stages 15 Transactions can be made through this channel. The trust module (170) is created by the feature module (160) for each modality. It specifies at least one quality value and one recency value for the feature data. Quality 20 is a quantitative value that expresses the technical integrity and suitability for processing of the relevant user data. It can be created as a value and multiple quality sub-metrics can be normalized. or can be calculated by weighting. In an application, the quality value is between 0 and 1. normalization occurs between them; however, the use of different scales is also acceptable. It is possible. The recency value is the difference between the time a user's data was created and the time the data is currently being updated. It takes into account the difference between the time of evaluation and when it will be evaluated. For evaluation time t* and acquisition time t_m of a modality, data age Δt_m = It can be defined as t*-t_m. As the data age increases, the modality becomes less likely to be included in the joint evaluation. To minimize its contribution, the currency value is a decreasing function of data age. 30 It can be calculated as follows. In a non-restrictive implementation, the current value This is obtained using the following relationship: r_m = exp(-λ_m × Δt_m) 9 Here, r_m represents the current value for the relevant modality, and λ_m represents the modality-specific aging value. Δt_m represents the coefficient and Δt_m represents the data age. The exponential decay function... Its use is not mandatory. The current value shows a piecemeal linear decrease, incremental. decrease, fixed time window, logistic decrease, or another modality-specific monotonic 5 It can also be determined via the function. For audio, image and other modalities. Different aging coefficients or time windows can be defined. The trust module (170) extracts features in addition to quality and currency values. an inference expressing the reliability of the model or data processing chain that performed the operation It can also establish trust value. Thus, a modality can be jointly evaluated. 10 Its contribution depends not only on the technical quality of the data, but also on the condition of the data at the time of evaluation. It can also be limited based on its currency and the reliability of the feature extraction process. In an implementation, the quality value is q_m, the timeliness value is r_m, and the inference confidence is c_m. The modality weight w_m is calculated using a non-restrictive sample of 15. The calculation can be applied as follows: w_m = normalize(q_m × r_m × c_m) The weighting process is not limited to the multiplication relationship above. Quality, timeliness, and 20 Inference confidence values: linear combination, weighted sum, threshold function, rule This can also be done through a table, a probabilistic model, or a learned weighting function. can be combined. The confidence module (170) combines these values ​​on a modality basis. can create; the merging module (180) allows the relevant modality to be jointly evaluated. Its contribution is 25, with a modality weight created using at least two of these values. It can limit it. The essential point regarding the invention is that it has different acquisition times. The contribution of modality data to common user status data must be at least of technical quality and This is determined by taking current information into consideration. Pre-processing module (140) and confidence module (170) together form a two-stage quality gate 30 It can work in such a way as to create. The preprocessing module (140) is mandatory in the first stage. It enforces integrity requirements; for example, minimum recording integrity of audio data. failure to provide, absence of a usable facial region in the image data, or lack of data If the record does not contain the required timestamp, merge the relevant modality. It can exclude. The trust module (170) requires mandatory integrity in the second stage. a product that meets the condition but whose quality value is below the defined threshold level It can reduce the weighting of data aggregation. Excluding data belonging to a modality necessarily disrupts the system's operation. It does not terminate. The merging module (180) includes the remaining accepted modalities. It enables the creation of user status data by processing it. Available When the number of modalities or the total confidence value falls below the defined lower limit, the system can create a new data acquisition request, status module (190) low confidence sign 10 It can generate user status data or evaluate it for subsequent data acquisition. It can postpone the process to a later time. Thus, a single corrupted or incomplete data source can be used to restore the entire system. It is possible to prevent the data processing chain from being interrupted. The database (150) contains past features that meet the user's technical acceptance conditions. It stores personalized reference data created from the data. Personalized reference Data can be stored as a separate data structure for each modality, or multiple modality at once. It can also be stored as a common structure containing the reference components of the modality. The purpose of the reference data is to present the user in a fixed manner according to a general population distribution. not to classify, but to analyze temporal change according to the user's own accepted history. The aim is to provide a technical comparison basis that allows for calculations. In an implementation, the initial reference is a certain number of initial steps that meet the quality requirements. It is generated from feature data. Reference data includes the center and spread of features. It can include the following values: The central value is the arithmetic mean or the healthy median; 25 The dispersion value can be determined by standard deviation or median absolute deviation. Alternative implementations include sliding windows, exponential moving averages, and probabilistic distributions. or personalized referrals using learned user representation. can be created. New feature data does not necessarily have to directly change the person's unique reference. New feature data is first evaluated in terms of quality, timeliness, and outlier criteria. is being evaluated. Those that do not meet the quality requirement, have insufficient technical integrity, or Data identified as an outlier is not included in the reference update. Accepted 11 The impact of the newly acquired data on the reference is determined by a defined update rate with an upper limit. It can be restricted. The database (150) contains personal, central and spread values. It stores specific reference data and uses that reference data only under these conditions. It allows for controlled updates using new feature data that meets the requirements. This structure provides a unique, instantaneous reference to a single corrupted or unusual piece of data. It is suitable for preventing it from dragging in that way. If one modality is introduced later than others, the term... The subject of this modality will be discussed in a separate initial reference at a later time. It can be created. For example, in the first period, sound was only produced through the microphone (110). By retrieving the data, a reference for the audio modality can be created; the first suitable image data is then... then when obtained via camera (120) the reference to the image modality is also can be initiated. Thus, the system is available from the moment all modalities begin. It does not require that. The merging module (180) combines feature data with different timestamps. It operates in terms of evaluation time. Each feature data is in the database (150) It can be normalized or standardized according to the specific reference data of the relevant individual. This The direction of change of the relevant feature as a result of the process, according to the user's own reference, and Difference data can be generated to express the magnitude of the change. 20 In an application, a feature value is f_m, a reference center value is μ_m, and a reference spread is... The value σ_m can be standardized as follows: z_m = (f_m - μ_m) / σ_m 25 If robust statistics are used, the median should be used instead of μ_m and the absolute median instead of σ_m. The diffusion value derived from the deviation can be used. This standardization is in question. This is merely an example implementation; individual comparisons based on minimum and maximum values ​​may be required. normalization, probabilistic distance, similarity function, or learned comparison 30 It can also be done using the model. 12 The merging module (180) combines feature data processed according to the individual reference. its contribution during the evaluation time determined by the confidence module (170) It is limited by weights. The merging operation is weighted sum, feature level. combination, state-space model, time series model, attention-based model, rule-based This can be achieved using fusion or a combination thereof. Fusion 5 Thanks to the fact that the process is carried out in terms of a common evaluation time, different ages This prevents the data from being considered equally up-to-date. The status module (190) uses the combined data generated by the merge module (180). It generates machine-processable user status data. The user status data is a single 10 It doesn't have to be a class or a single score. The data in question, characteristic dimensions, personal Directions of change, confidence values, uncertainty values, and time according to the specific reference. It may include one or more trend data points. Thus The evaluation results can then be used in subsequent computer-based processes and It can be stored as a structured data representation that can be updated over time. 15 The application of the consistency requirement established among the accepted modalities is also included. It is possible. The merging module (180) allows a modality to be combined according to the person's specific reference. direction of change with direction of change derived from other accepted modalities can compare and if a discrepancy is found above the specified tolerance, a 20 It can generate a contradiction signal. If a contradiction signal occurs, the merging module... (180) can reduce the weight of the relevant modality or request a new data acquisition. It can generate; the status module (190) provides low confidence information instead of precise classification. It can generate user status data. The registration module (210) determines which technical inputs and which processing the user status data A machine-readable transaction record was generated showing that it was created under the given conditions. It brings about the modality IDs used in the transaction log, and the excluded modality IDs. Exclusion reasons, quality values, timeliness values, inference confidences, integration weights, personalized reference version, evaluation time, contradiction indicator, and 30 one or more fields such as the ID of the user status data created It can be held. 13 The technical transaction log created by the registration module (210) is displayed in the user interface (220) This can be kept separate from the displayed natural language description. This allows the user to see a simpler explanation. When a communication is presented, it is important to know which data is used within the system and which data is not. which version was excluded and which reference version was used was then determined by the machine. It can be protected in a verifiable manner. 5 where the protocol module (200) is used. rule ID selected in implementations, eligibility condition applied and exclusion This condition can also be added to the transaction record. The protocol module (200) is implemented by the status module (190) in an optional implementation. It matches the generated user status data with a controlled output library. Output 10 In its library, the eligibility criteria, exclusion criteria, and sequence information are listed for each output or output series. Time limit, repetition limit, and intensity parameters can be maintained. Protocol module (200), by evaluating user status data and the conditions in question together, only It makes a selection from among the allowed outputs. Protocol module (200), decision table, state machine, rule graph, constrained optimization This can be implemented as an engine or a combination of these. Free text generator If a software component is used, the protocol selection for that component is made directly. It is not mandatory to change it; the component in question is changed by the protocol module (200) It can be used to explain the selected controlled output to the user. Thus, 20 The technical selection mechanism and the layer of expression seen by the user are distinct from each other. They are leaving. The user interface (220) outputs the selected output by the protocol module (200) as text or voice. redirection, card interface, chat interface or avatar to the user 25 It can provide. The user interface (220) also shows whether the output is complete or not, feedback from the user on whether they found the output acceptable or the direction of the perceived effect. They can receive notifications. This feedback will only be used in the next output selection. It can be used within predefined variant or intensity limits. Technical aspects in presenting user status data to the user are handled by the user. They can be displayed with understandable labels. In an application, these labels might include Stress Tone, etc. It can be used for Mental Clarity, Life Rhythm, and Emotional Tone. The aforementioned These terms are merely designations relating to the presentation layer in the user interface, and 14 the content of the technical data structure maintained by the status module (190) or of the invention It does not limit the scope of protection. Security module (230), camera (120) or other sensitive data sources are designated It can be activated after a user action or user approval. 5 Security module (230), separate storage time, access authorization and deletion for different modalities. This allows us to apply the rule and thus have different options for raw data and derived feature data. It is possible to define data lifecycles. After the feature data is created. In implementations where the deletion of raw audio or video data is foreseen, the relevant deletion The implementation of the rule can be controlled by the security module (230). 10 In an implementation, at least one of the preprocessing module (140) or feature module (160) Part of the data is run on the user device (100). Raw audio or video data, After the feature data is generated by the feature module (160), it is sent to the remote server. It can be deleted without being transferred. In this case, the remote processing infrastructure only takes 15 minutes. branding, modality ID, feature data, quality information, and inference confidence, if any. It can be transferred. In alternative implementations, all processing is done on the user device (100), all processing on the remote server or some of the functions on both sides It can be implemented. An external calendar, wearable device, or other data provider requires a mandatory 20-day period. It does not require integration. However, the data interface (130) allows the user to open additional data provided by the action or obtained from an authorized integration over time It can be included in the system with its stamp and modality ID. Thus, a new data type, The main asynchronous processing chain remains unchanged as a new modality in the system. It can be added. Heart data from an authorized wearable device or data provider. 25 heart rate, heart rate variability (HRV), sleep, activity, movement, respiration, skin conductivity, and At least one of the temperature data points will be added as an additional modality with a timestamp and modality ID. It can be processed. It is mandatory to have new data for all modalities within a specific evaluation period. 30 That's not it. At the time of that evaluation, only new data for one modality was available. when this happens, previously accepted feature data belonging to other modalities are outdated. It can be used subject to limitations based on its values; the condition of currency or quality Modalities that do not meet the requirements are excluded, and processing continues with the remaining available data. It is possible. In a sample application, a short audio data is received from the user device (100) and The acquisition time is assigned to the data in question; preprocessing module (140) recording time, interruption 5 controlling the ratio and signal-to-noise ratio; feature module (160) speech rhythm, It generates audio feature data including pause distribution and energy change. Trust module (170) determines the quality and up-to-date values ​​of the audio feature data and In the implementation where inference confidence is used, the confidence value in question is also the modality. It is included in the calculation of its weight. 10 At a later checkpoint, the camera (120) is activated upon user action or approval. It receives image data. Image data includes lighting or face visibility. If the image modality does not meet the essential integrity requirement in terms of excluded; merging module (180), existing audio data and data interface if any 15 (130) Processing continues through other accepted modalities. Registration module (210) indicates that the image modality is excluded and the reason for exclusion is recorded in the transaction log. It can store it. According to the (150) person-specific reference data in the accepted feature data database, 20 is being processed and joint evaluation time by the merging module (180) They are weighted and brought together. The state module (190) emerges By relating the resulting user status data to previous user status data, time It allows tracking the direction of change within it. The quality and nature of the new feature data. If it meets the exceptional sample conditions, the (150) person-specific reference in the database 25 The data can be updated with a defined update rate. In another application, image acquisition time is not tied to a fixed control point. When the uncertainty level of the user status data exceeds the defined limit and the previous image When the age of the data exceeds the specified upper limit, user 30 will be required to provide new image data. The user can request this action. If the user's action does not occur, the camera will be disabled. (120) is not activated and the system is working through the remaining available modalities. It continues. 16 In an implementation using the protocol module (200), user status data is controlled. It is matched with the output library, eligibility and exclusion conditions are applied, and permission is granted. One of the given outputs is selected. The user interface (220) displays the selected output to the user. They can transfer information and receive feedback after the process. This feedback... In subsequent elections, it is only used within predetermined limits; registration 5 module (210) uses the user status data, the selected rule ID and the selection It is saving time.

Claims

17 REQUESTS 1. Obtained at different acquisition times or different sampling frequencies and at least a system for processing multimodal user data, one of which is sensor-derived and its feature is; 5  Showing the acquisition times of at least two user data points from different modalities a user device (100) that enables the system to be accessed with timestamps,  the technical integrity and quality of the user data in question, specific to the modality. a preprocessing module that operates in terms of conditions (140),  User data processed by the preprocessing module (140) into modality 10 a feature module that generates specific feature data (160),  Personalized results created from the user's accepted historical data. a database that stores reference data (150),  a quality value that expresses the technical usability for feature data and a The recency value, which indicates the age of the data according to the evaluation time, is 15. a trust module (170),  Processing feature data based on individual reference data and at different times feature data with their stamps at the time of joint evaluation by determining its contribution based on at least the values ​​of quality and up-to-dateness. a merging module (180) that combines feature data asynchronously, 20  from feature data combined by the merging module (180) machine a state that generates processable and time-varying user state data module (190), It includes.

2. The system is according to claim 1 and its feature is; the user device (100), first user data. a microphone (110) that receives the audio data and the second user data, the image data. the area includes a camera (120) and the camera (120) receives through the microphone (110) It has a lower sampling frequency compared to audio data.

3. The system is according to claim 2 and its feature is; the user device (100), the camera (120) selected from initial, intermediate comparison, or final comparison checkpoints enabling it at certain times and setting the image acquisition time based on user status data. uncertainty, age of previous image data, magnitude of change, user choice or It is determined according to at least one of the source statuses of the user device (100). 35 18 4. A system that meets any of the requirements 1-3, and whose characteristic is; cognitive measurement data, at least one of the following: contextual user data, usage data, or short user response. It includes the field data interface (130) as an additional modality.

5. The system is defined according to any of the requirements 1-4, and its characteristic is; quality value q_m and In addition to the recency value r_m, the value c_m represents the feature extraction confidence. the contribution of the related modality to the joint assessment with the confidence module (170) A modality created using at least two of the values ​​q_m, r_m, and c_m. It includes a coupling module (180) that limits its weight. 10 6. A system that meets any of the requirements 1-5, and whose characteristic is: mandatory integrity. Preprocessing that excludes user data that does not meet the condition from being combined. module (140), threshold that meets the integrity condition but the quality value is determined Confidence level 15 reduces the aggregate weight of user data below that level. module (170) and merging module (180) which handles the remaining accepted modalities It includes.

7. The system is according to any of the requirements 1-6 and its feature is; the database (150), center and spread values ​​generated from accepted historical feature data 20 storing the personalized reference data containing that reference data only new feature data that meets the quality condition and the outlier condition It can be updated at a defined update rate using a specified upper limit. It is holding on.

8. The system is defined according to any of the requirements 1-7, and its characteristic is; accepted. above the defined tolerance between the directions of change derived from the modalities If an inconsistency is found, it constitutes a sign of contradiction, and that contradiction... according to the signal, reducing the weight of the relevant modality or requesting re-acquisition of data. The combination module (180) that creates the low confidence 30 according to the contradiction signal in question. It includes a status module (190) that generates user status data containing information.

9. The system is defined according to any of the requirements 1-8, and its characteristic is the modality used. identities, excluded modality identities, quality values, timeliness values, integration 19 weights, personalized reference version and at least one of the assessment times It includes a logging module (210) that creates a machine-readable transaction log.

10. The system is according to any of the requirements 1-9 and its feature is; status module (190) User status data generated by eligibility, exclusion, order, duration and 5 By matching it with a controlled output library that includes repetition conditions, the maximum of the allowed outputs can be selected. The protocol module (200) selects one of the outputs and presents the selected output to the user. It includes a user interface that receives feedback (220).

11. The system is according to any of the requirements numbered 1-10 and its feature is; camera (120) or 10 other sensitive data sources after specified user action activation and raw audio or video after feature data is generated. A security module that ensures data is deleted according to a defined retention rule. (230) must include at least one of the preprocessing module (140) or feature module (160). part of it is located on the user device (100). 15 12. Obtained at different acquisition times or different sampling frequencies and at least one of them is multimodal user data originating from sensors, processed by computer applications. It is a method for processing, and its characteristic feature is;  User device (100) 20 with at least two user data belonging to different modalities Obtained via timestamps indicating the relevant acquisition times  the user data in question is processed by the preprocessing module (140) into the modality processing in terms of specific technical integrity and quality conditions,  Modality-specific feature module (160) from processed user data Generating feature data, 25  Personalized results created from the user's accepted historical data. Obtaining reference data from the database (150),  Technical availability expressed by the confidence module (170) for feature data the quality value and the data age according to an evaluation time Determining the current value, 30  Processing characteristic data based on individual reference data and at different times feature data with their stamps at the time of joint evaluation by combining the contributions based on at least the quality and up-to-date values. asynchronous combination by module (180),  The combined feature data is machined by the status module (190) Generating processable and time-varying user status data, It includes the steps of the process.

13. The method according to Request 12, and its feature is; the first user data is microphone (110) 5 voice data received via the camera, second user data received via the camera (120) the fact that it is image data and image data has a lower sampling rate compared to audio data It involves taking the medication at a specific frequency.

14. Method according to claim 12 or 13, its feature is cognitive 10 via data interface (130). from measurement data, contextual user data, usage data, or brief user response at least one of them must be obtained and that data must be used as an additional modality asynchronously. The inclusion of the merge process involves a specific step.

15. This method is based on any of the requirements 12-14 and its characteristic is; trust module 15 (170) by quality value q_m, currentness value r_m and feature extraction confidence The determination of c_m values ​​relates to the timeliness value to the data acquisition time. deriving the modality weight from the difference between the evaluation times q_m and r_m The steps involved in creating it using at least two of the c_m values. It includes. 20 16. A method according to any of the requirements 12-15, characterized by its mandatory integrity. Pre-processing module (140) of modality data that does not meet the condition Exclusion from integration, meeting the integrity requirement but exceeding the quality value threshold Reducing the merging weight of modality data below the level and 25 the operation with the remaining accepted modalities by the merging module (180) It involves the steps involved in maintaining the process.

17. This method is based on any of the requirements numbered 12-16 and its characteristic is that it is specific to the individual. 30 accepted historical feature data on the reference data database (150) Storing the generated central and dispersion values ​​and the quality of the new feature data. If the reference data in question meets the condition and the outlier condition, then... the steps for updating at a defined update rate It includes. 35 21 18. Method according to any of the requests numbered 12-17, its feature is; camera (120) start, intermediate comparison, or end of image data acquisition time selection from comparison checkpoints or user status data uncertainty, age of previous image data, magnitude of change, user choice, or The user device must be adapted to at least one of the (100) source states and 5 If image data cannot be obtained, processing will be done using the remaining accepted modalities. The process involves continuing with the necessary steps.

19. The method is based on any of the requests numbered 12-18, and its characteristic is that it is accepted. The tolerance defined between the directions of change derived from the modalities is 10 If a discrepancy is found, the merging module (180) will send a conflict signal. creation and weighting of the relevant modality according to the said contradiction signal. reduction, creation of a new data acquisition request or status module (190) the process of generating user status data containing low confidence information It must include at least one of the steps. 15 20. This method is based on any of the requests numbered 12-19 and its characteristic is; registration module. Modality IDs used or excluded by (210), quality and timeliness values, combined weights, personalized reference version and evaluation Creating a machine-readable transaction log that includes at least one of the times, 20 user status data by protocol module (200) with controlled output library By matching and allowing, at least one of the outputs is selected and the selected output is displayed to the user. The process includes presenting the steps to the user via the interface (220).

21. The system is defined according to any of the requirements 1-11, and its characteristic is that the data interface is 25. (130), heart data from an authorized wearable device or data provider rate, heart rate variability, sleep, activity, movement, respiration, skin conductivity or Add at least one of the temperature data points to the additional modality with a timestamp and modality ID. It is the process of adding it to the system.

22. The method is based on any of the requests numbered 12-20, and its characteristic is that it is authorized. heart rate received from a wearable device or data provider, heart rate variability of sleep, activity, movement, respiration, skin conductivity, or temperature. at least one of the data points as an additional modality with a timestamp and modality ID. This involves the process of acquiring and including it in asynchronous combining. 35

Images