A hardware optimization method of a low-power-consumption multi-modal emotion analysis terminal

Abstract

This invention relates to the field of intelligent hardware and multimodal emotion computing, specifically a hardware optimization method for a low-power multimodal emotion analysis terminal, comprising: controlling the main computing core to sleep; extracting feature data from video, audio, and physiological sensor data on the direct memory access bus side and calculating feature residuals between two consecutive frames; when the residual is lower than a basic residual threshold, performing cache discarding and preventing writing to dynamic random access memory; when the residual reaches the threshold, generating an event token, storing it in static random access memory, and accumulating the multimodal joint activity; when the joint activity exceeds the activity threshold, sending a hardware interrupt to wake up the main computing core; the main computing core reads the event token, performs sparse attention mechanism calculation, outputs the emotion state evaluation result, and adjusts the basic residual threshold in reverse; this invention can reduce invalid memory access and periodic calculations, maintain continuous sensor sampling, and not miss key events.

Description

Technical Field

[0001] This invention relates to the field of smart hardware and multimodal emotion computing, specifically to a hardware optimization method for a low-power multimodal emotion analysis terminal. Background Technology

[0002] With the rapid development of medical monitoring terminals and intelligent sensing devices, multimodal emotion analysis terminals for continuous patient monitoring are gradually being applied in scenarios such as psychiatric wards, sleep observation, and care for high-risk groups. These terminals typically require long-term continuous collection and analysis of video, audio, and physiological sensor data. Under the premise that the terminal needs to work continuously, how to reduce storage access overhead and main computing core power consumption while ensuring timely response to abnormal emotional changes has become a technical problem that needs to be solved. Traditional multimodal emotion analysis terminals currently rely mainly on the following methods: the main computing core is woken up at fixed intervals to continuously transfer and store the raw data from each modality sensor, and then performs fusion analysis based on the complete video, audio and physiological data in the dynamic random access memory; or anomaly detection is completed by using a single modality priority triggering and fixed threshold filtering method. However, fixed-period wake-up, continuous writing of raw data to memory, and single-modal triggering or fixed-threshold filtering all have the following technical limitations. For example, fixed-period wake-up causes the main computing core to frequently exit sleep mode, resulting in additional power consumption; continuous writing of raw multimodal data increases the memory bus occupancy rate of dynamic random access memory, causing stable data without state changes to occupy bandwidth and storage space; single-modal triggering is easily affected by environmental noise, changes in lighting, or fluctuations in sensor wearing status, while fixed thresholds are difficult to adapt to the dynamic changes of different patient states and monitoring scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide a hardware optimization method for a low-power multimodal emotion analysis terminal, which solves the following technical problems: reducing invalid writes to dynamic random access memory and maintaining low power consumption of the main computing core for a long time, while improving the reliability of emotion state assessment without missing key events.

[0004] The objective of this invention can be achieved through the following technical solutions: A hardware optimization method for a low-power multimodal emotion analysis terminal, the terminal comprising a main computing core, a direct memory access bus, a hardware feature evaluation circuit, static random access memory, dynamic random access memory, an interrupt controller, a cross-modal activity accumulation register, and various modal sensors, including: S0. Control the main computing core to enter a sleep state; S1. Acquire multimodal sensor data collected by each modal sensor, extract feature data through hardware feature evaluation circuit on the direct memory access bus side, and calculate feature residuals of the feature data for two consecutive frames; S2. If the feature residual is less than a preset basic residual threshold, perform a cache discard operation to prevent the multimodal sensor data from being written to the dynamic random access memory; S3. If the feature residual is not less than the basic residual threshold, generate an event token containing the feature data and store it in the static random access memory. According to the positive correlation residual-activity mapping rule, convert the feature residual into a feature activity value and accumulate it to the cross-modal activity accumulation register to generate multimodal joint activity. S4. If the multimodal joint activity is greater than the preset activity threshold, a hardware interrupt signal is sent to the interrupt controller to wake up the main computing core and enter the running state; otherwise, it remains in the dormant state. S5. The event token in the static random access memory is read through the main computing core, feature data is extracted and sparse attention mechanism is executed to output a multimodal classification result containing confidence parameters and the contribution of each modality, and the basic residual threshold is adjusted in reverse based on the multimodal classification result.

[0005] Preferably, before controlling the main computing core to enter a sleep state, S0 further includes: Each modal sensor is allocated an independent block of static random access memory. Initialize the basic residual threshold corresponding to each modal sensor.

[0006] Preferably, the step of calculating the feature residual of two consecutive frames of feature data in S1 includes: The degree of difference between the two consecutive frames of feature data is calculated by an arithmetic logic unit deployed on the direct memory access bus side and located within the hardware feature evaluation circuit. The degree of difference is defined as the characteristic residual.

[0007] Preferably, the step of generating multimodal joint activity in S3 includes: Obtain the preset activity weights corresponding to each modal sensor that are pre-configured; Based on the preset activity weights, the feature activity values ​​corresponding to the feature residuals output by each modal sensor are weighted and summed to generate the multimodal joint activity.

[0008] Preferably, before reading the event token from the static random access memory via the main computing core in step S5, the method further includes: In response to the hardware interrupt signal, the main computing core is switched from the hibernation state to the running state.

[0009] Preferably, the step in S5 of adjusting the basic residual threshold in reverse based on the multimodal classification result includes: Extract the confidence parameter and the contribution of each modality from the multimodal classification results; Determine the relationship between the confidence parameter and the preset confidence threshold; If the confidence parameter is greater than the confidence threshold, then the modal sensor that contributes the most to the multimodal classification result is determined as the target modal sensor, and the basic residual threshold corresponding to the target modal sensor is reduced. If the confidence parameter is less than or equal to the confidence threshold, then the basic residual threshold remains unchanged.

[0010] Preferably, the step of calculating the difference between the feature data of two consecutive frames includes: The feature data of the two consecutive frames are processed using a pre-defined Manhattan distance algorithm or an XOR gate logic operation to generate the difference.

[0011] Preferably, each modal sensor includes a video sensor, an audio sensor, and a physiological sensor, and the multimodal sensor data includes video data, audio data, and physiological sensing data collected by the above sensors respectively; The multimodal classification results are the emotional state assessment results for the preset target users.

[0012] The beneficial effects of this invention are: 1. This invention extracts feature data and calculates residuals on the direct memory access bus side; if the residual is below a threshold, the cache is discarded to prevent data from being written to dynamic memory; only when the residual meets the threshold is it stored in static memory and the joint activity is accumulated, and the main computing core is woken up when the threshold is exceeded; this event-driven mechanism effectively intercepts invalid memory accesses and avoids the extra power consumption generated by fixed-cycle calculations. 2. This invention obtains the preset activity weights corresponding to each modal sensor, and generates multimodal joint activity by weighted summation of the feature activity values; this mechanism realizes cross-modal evidence reliability adjustment at the hardware bus level; effectively overcomes the defect of false wake-up caused by single modality being susceptible to environmental noise, occlusion and other influences, while maintaining accurate response to real emotional changes; 3. This invention extracts the confidence parameter of the multimodal classification results, and determines the target modality with the highest contribution when the confidence is higher than the threshold and reduces its basic residual threshold. This feedback mechanism enables the terminal to automatically increase the sensitivity of specific modalities when the risk increases, maintain stability when the classification is unstable, dynamically adapt to changes in the patient's state, and take into account both low power consumption and high reliability. 4. This invention deploys an arithmetic logic unit on the direct memory access bus side and uses the Manhattan distance algorithm or XOR gate logic to process feature data to generate a difference degree. This mechanism moves the computation forward to a low-power path, avoids complex operations or deep model inference, significantly reduces the power consumption and area of ​​the evaluation circuit, and ensures the real-time performance of event filtering. 5. Before the main computing core goes into sleep mode, the present invention allocates independent static random access memory blocks for each modal sensor and initializes the basic residual threshold. This mechanism effectively avoids the problem of a large number of writes to a single modality causing it to overwrite and crowd out event tokens of other modalities in high-noise scenarios, thus ensuring the integrity of multimodal event data in space and reducing the false wake-up rate. Attached Figure Description

[0013] The invention will now be further described with reference to the accompanying drawings.

[0014] Figure 1 This is a flowchart illustrating a hardware optimization method for a low-power multimodal emotion analysis terminal provided in this application embodiment. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figure 1 A hardware optimization method for a low-power multimodal emotion analysis terminal, the terminal including a main computing core, a direct memory access bus, a hardware feature evaluation circuit, static random access memory, dynamic random access memory, an interrupt controller, a cross-modal activity accumulation register, and various modal sensors, including: S0. Controls the main computing core to enter hibernation mode; S1. Acquire multimodal sensor data collected by each modal sensor, extract feature data through hardware feature evaluation circuit on the direct memory access bus side, and calculate feature residuals of two consecutive frames of feature data; S2. If the feature residual is less than the preset basic residual threshold, perform a cache discard operation to prevent multimodal sensor data from being written to dynamic random access memory; S3. If the feature residual is not less than the basic residual threshold, generate an event token containing feature data and store it in static random access memory. According to the positive correlation residual-activity mapping rule, convert the feature residual into feature activity value and accumulate it to the cross-modal activity accumulation register to generate multimodal joint activity. S4. If the multimodal joint activity level is greater than the preset activity threshold, send a hardware interrupt signal to the interrupt controller to wake up the main computing core and enter the running state; otherwise, remain in the hibernation state. S5. The event tokens in the static random access memory are read through the main computing core, feature data is extracted and sparse attention mechanism is executed to output a multimodal classification result containing confidence parameters and the contribution of each modality. The basic residual threshold is adjusted in reverse based on the multimodal classification result.

[0017] This embodiment provides a hardware optimization mechanism for a low-power multimodal emotion analysis terminal for medical monitoring scenarios. Specifically, the terminal is deployed in a bedside monitoring system for high-risk patients in a closed psychiatric ward to continuously assess patients' emotional stability, increased anxiety, and agitation prodromes without reducing the sampling capabilities of video, audio, and physiological sensors. The terminal includes a main computing core, a direct memory access bus, a hardware feature evaluation circuit, static random access memory, dynamic random access memory, an interrupt controller, a cross-modal activity accumulation register, and modal sensors such as video, audio, and physiological sensors. The main computing core can be a neural network processor, a digital signal processor, or a low-power central processing core, mainly responsible for sparse attention fusion and classification output. The direct memory access bus is responsible for data transfer from the sensor side. Static random access memory is used to store event tokens; dynamic random access memory is used to store large blocks of raw data or intermediate model data; cross-modal activity accumulator register is used to store the joint activity level of multiple modalities formed on the hardware side; The cross-modal activity accumulator register is mounted on the direct memory access bus in the hardware topology. Its data input is connected to the write control logic of the static random access memory to synchronously acquire the characteristic activity value, and its output is directly connected to the dedicated trigger pin of the interrupt controller. Internally, it contains a group register circuit for latching the sub-activity of each modality and a parallel adder tree for joint summation calculation. In step S0, when the patient is in the routine observation stage, the terminal controls the main computing core to enter a sleep state. This sleep state does not turn off sensor sampling, but turns off the matrix operation array, on-chip cache refresh unit or high-frequency clock domain of the main computing core, and only retains the sensor interface, hardware feature evaluation circuit on the direct memory access bus side, low-power retention circuit of static random access memory and interrupt response circuit. In this way, the system can still continuously receive data from the bedside camera, directional microphone, wristband heart rate or skin conductance sensor, but will not generate continuous power consumption because every frame of data enters the main computing core. In step S1, the real-time data generated by each modal sensor reaches the direct memory access bus side instead of being directly written to the dynamic random access memory. The direct memory access bus side has a pre-set hardware feature evaluation circuit, which can be composed of a lightweight register set, comparator, adder, XOR gate array and local state cache; The hardware feature evaluation circuit adopts a bus monitoring or data bypass branching mechanism on the data path. During the period when the data of each modal sensor is transmitted to the dynamic random access memory through the direct memory access bus, the data stream is extracted synchronously and loaded into a small-scale state buffer for real-time feature extraction, without blocking the main transmission pipeline of the original direct memory access bus. For video data, this hardware feature evaluation circuit can extract feature data such as facial contour changes, eyelid opening and closing changes, glabellar texture changes, or head posture changes; for audio data, it can extract short-time energy, zero-crossing changes, abrupt changes in sound intensity, or the start and end features of speech segments. For physiological sensor data, feature data such as heart rate fluctuations, skin conductance rising edges, or respiratory rhythm changes can be extracted; the circuit compares the feature data of two consecutive frames or two consecutive sampling windows to obtain feature residuals; in hardware terms, the feature residuals represent whether the current segment has information changes sufficient to change the emotional judgment relative to the previous segment, rather than performing complex neural network inference on the complete original data. To facilitate the explanation of data flow, assume that facial muscle region change marker vector A and facial muscle region change marker vector B are extracted from two consecutive windows of the video modality, with values ​​of [0,0,1,0] and [0,1,1,0] respectively. The hardware feature evaluation circuit only needs to determine the difference between the two at the corresponding positions to determine that a local change has occurred in the window. If two consecutive windows of the audio modality are both marked as having no speech and low energy, then its feature residual remains low. The above example is only used to illustrate the feature transfer method on the hardware side. The actual circuit can set a longer register field according to the sensor bit width and sampling format. In step S2, the terminal judges the feature residual based on the preset basic residual threshold. When the feature residual of a certain mode is less than the corresponding basic residual threshold, it means that the current data of the mode has not formed a valid event compared with the previous moment. For example, if the patient's face is still for a long time, the background noise of the ward is stable, and the heart rate curve is near the baseline, writing the complete video frame or audio waveform into the dynamic random access memory at this time will cause a large number of invalid memory accesses. Therefore, the cache discard operation performed on the direct memory access bus side can specifically manifest as canceling the current dynamic random access memory write request, releasing the direct memory access descriptor, overwriting the sensor-side temporary buffer, or only retaining the timestamp digest, thereby preventing the original multimodal sensor data from entering the dynamic random access memory. In step S3, if the feature residual is greater than or equal to the basic residual threshold, it indicates that the modality has generated a valid event that may be related to the change in emotion. At this time, the terminal does not need to wake up the main computing core immediately, but instead generates an event token and stores it in static random access memory. Event tokens must include at least a modality identifier, a timestamp, feature data, residual level, and sensor quality flags; The event token uses a fixed-width structured data packet format at the underlying hardware level. The modality identifier, timestamp, feature data, residual level, and sensor quality flag are hard-wired and mapped to consecutive bit fields of a preset number of bits to ensure that the static random access memory can perform high-speed, low-power addressing read and write operations according to a fixed address step. For example, a video event token can record video modality, bed area, eyebrow and eye area changes, timestamp, and valid frame number; an audio event token can record audio modality, speech energy spikes, short vocalizations, and timestamp. Simultaneously, the terminal converts the feature residuals into feature activity values ​​according to the preset residual-activity mapping rule and writes them into the cross-modal activity accumulation register. The residual-activity mapping rule is a mapping rule in which the value of the feature residual is positively correlated with the feature activity value. The residual-activity mapping rule is implemented in the hardware structure through hardware lookup table or shift register logic deployed on the direct memory access bus side or static random access memory controller side. The hardware circuit directly outputs the corresponding feature activity value by performing table lookup or shift operations on the input feature residual, thereby avoiding calling the floating-point arithmetic unit. This register does not store the complete original waveform or image, but rather stores hardware-level cumulative evidence of how each modal event affects the current emotional state. Furthermore, to avoid the erroneous superposition of joint activity across time spans exceeding a preset time limit due to asynchronous sampling of different modalities, event tokens can also be accompanied by their respective observation window identifiers when written to static random access memory. The cross-modal activity accumulation register prioritizes the accumulation of event tokens within the same observation window or adjacent preset time ranges; old tokens exceeding this time range are only retained for subsequent tracing and will not continue to increase the current joint activity. In this way, the joint activity corresponds to multimodal common evidence in the same emotional change process, rather than an unrelated superposition of multiple time segments with time intervals exceeding a preset interval threshold. Furthermore, to ensure that the input object is clear after the main computing core is woken up, the feature data in the event token is preferably a structured field quantized by the hardware feature evaluation circuit, rather than continuous raw video frames, complete audio waveforms or uncompressed physiological sequences. In step S5, the main computing core reads these structured fields and their temporal relationships, quality indicators and residual levels, and performs sparse attention mechanism calculations based on these. Therefore, the aforementioned cache discarding operation does not conflict with subsequent classification, and the data path always remains the original data after being filtered by the bus side, retaining only event-level features for subsequent inference. In step S4, after the cross-modal activity accumulation register forms the multimodal joint activity, it is compared with a preset activity threshold. When the multimodal joint activity is greater than the activity threshold, it indicates that different modalities jointly show evidence of emotional changes within the same observation time range. The cross-modal activity accumulation register sends a hardware interrupt signal to the interrupt controller, which wakes up the main computing core. The activity threshold is provided by a read-only threshold register pre-configured in the system control address space. The hardware comparator integrated inside the cross-modal activity accumulator register performs a parallel hardware-level comparison of the current multimodal joint activity with the value in the read-only threshold register during the processing cycle of each event token. When the multimodal joint activity is less than or equal to the activity threshold, it indicates that the current events are not enough to support a complete classification. The main computing core continues to remain dormant, and the event token can be retained in static random access memory until the preset timeout threshold is reached, waiting for subsequent modal events to supplement it. In step S5, after the main computing core is woken up, it reads the event token from the static random access memory instead of reading the continuous historical video frames, complete audio waveforms and all physiological sampling sequences from the dynamic random access memory; the main computing core extracts the feature data from the event token and performs sparse attention mechanism calculation; This sparse attention mechanism establishes associations only between event tokens, such as fusing facial feature deformation, audio energy surges, and heart rate rising edges as candidate evidence in the same temporal neighborhood, without performing intensive computation on stable background frames. The output multimodal classification results can be in states such as emotional stability, increased anxiety, agitation precursors, or requiring manual review; the terminal adjusts the basic residual threshold in reverse according to the classification results, so that the hardware feature evaluation circuit has the sensitivity to match the patient's state in subsequent monitoring; Furthermore, the reverse adjustment of the basic residual threshold is applied to subsequent unprocessed sampling windows, without retrospectively changing event tokens already written to static random access memory or joint activity determinations already completed; this avoids the timing conflict of the same batch of events being generated according to the old threshold and reinterpreted according to the new threshold, keeping the threshold adjustment and event screening separate in time. As an anomaly handling mechanism, if a sensor fails briefly, data frame verification fails, or timestamp is abnormal, the hardware feature evaluation circuit will not write the abnormal data as a valid event token into the static random access memory, but will write it into a low-priority status flag. If the event token storage space of the static random access memory reaches the preset limit, the old tokens will be cleared according to the time order or low activity priority strategy, and the most recent high activity tokens will be retained. If the duration for which the cross-modal activity accumulation register does not exceed the critical value is greater than the preset time threshold, it can be periodically cleared or decayed to prevent weak events in time segments with time intervals greater than the preset interval threshold from being erroneously accumulated as a strong event. During the nighttime monitoring phase in the psychiatric ward, when the patient is in a quiet bed resting state, the bedside camera still captures images at a predetermined frequency, the microphone still captures ambient sound, and the wristband still captures heart rate and skin conductance signals. Since facial features, audio energy, and physiological fluctuations are close to the baseline, the direct memory access bus side continuously executes cache discarding, and the dynamic random access memory does not experience continuous large block writes. If a patient exhibits frowning, short vocalizations, and increased skin conductance, event tokens are generated from video, audio, and physiological modalities respectively. After the cross-modal activity accumulation register accumulates and reaches a critical condition, the interrupt controller wakes up the main computing core. The main computing core performs sparse fusion on these event tokens only, outputs the emotional state assessment results indicating arousal precursors and triggering nursing review. In the context of medical monitoring, the multimodal classification results correspond to the emotional state assessment results of the target user. The emotional state assessment results, emotional risk judgments, or emotional change judgments mentioned in this article are all specific expressions of the classification results in the monitoring context and do not change their meaning as the same output object. In this embodiment, the target user is specifically the patient receiving monitoring. Furthermore, throughout the text, event tokens refer to structured event records that are filtered by the direct memory access bus side and written to static random access memory, and do not refer to original video frames, original audio waveforms, or original physiological sampling sequences. The purpose of this step is to change the triggering condition of multimodal sentiment analysis from fixed-period intensive computation to hardware-side event-driven computation, thereby achieving continuous sensor data collection, no omission of key events, reduction of invalid writes to dynamic random access memory, and long-term low-power operation of the main computing core.

[0018] In a preferred embodiment of the present invention, before controlling the main computing core to enter a sleep state, S0 further includes: allocating independent static random access memory blocks for each modal sensor; and initializing the basic residual threshold corresponding to each modal sensor.

[0019] This embodiment provides a storage isolation and threshold initialization mechanism before hibernation; specifically, before the main computing core enters hibernation, the terminal first allocates independent static random access memory blocks for video, audio and physiological sensors, and initializes their respective basic residual thresholds. If multiple modalities share the same static random access memory area, in high-noise scenarios, a large number of event tokens from one modality may be written, overwriting event tokens from other modalities. For example, if a nursing device prompt tone occurs in the ward within the first preset time period, the audio modality may generate more event tokens than the preset threshold. If there is no independent storage block, it may overwrite the valid event records of the physiological modality. Therefore, in this embodiment, before hibernation, the static random access memory is divided into video storage blocks, audio storage blocks, and physiological storage blocks. Each storage block is configured with a write pointer, a read pointer, and an overflow flag. After the main computing core hibernates, the direct memory access bus side can still write event tokens to the corresponding storage blocks according to the modality identifier. When initializing the baseline residual threshold, the terminal can record sensor feature changes in a relatively stable state within the baseline acquisition window after the patient wears the device and completes bedside terminal binding; the baseline residual threshold of the video modality is used to distinguish between normal body position micro-movements and obvious changes in facial expressions; the baseline residual threshold of the audio modality is used to distinguish between ward background noise and patient voices; the baseline residual threshold of the physiological modality is used to distinguish between normal heart rate and respiratory fluctuations and emotion-related mutations. To illustrate with a simplified example, if the energy level of the audio modality in a steady state is mostly level 1 or 2 in a continuous window, then its basic residual threshold can be set higher than the level of background fluctuation; if the video modality causes the image noise to be greater than the preset noise threshold due to nighttime illumination being lower than the preset illuminance threshold, then the video basic residual threshold can be set more conservatively to avoid sensor noise generating a large number of invalid tokens. As an anomaly handling mechanism, if the signal quality of a certain modality is unqualified during the initialization phase, such as when the camera is blocked, the microphone is not connected, or the wristband is loose, the terminal will not use the real-time data of that modality to establish the basic residual threshold. Instead, it will use the factory safety threshold or the default threshold issued by the ward equipment management system. If the available capacity of the static random access memory is insufficient, priority will be given to ensuring the token space of the physiological modality and the audio modality, and the maximum number of event tokens for the video modality will be limited to maintain the security in the medical monitoring scenario. During the device binding phase after the same patient is admitted to the hospital, the nursing staff will pair the wristband with the bedside terminal, and the terminal will complete the baseline acquisition within the first preset time period when the patient is at rest and cooperates. The system writes video event tokens to the first static storage block, audio event tokens to the second static storage block, and heart rate and skin conductance event tokens to the third static storage block, and respectively forms basic residual thresholds that are adapted to ward lighting, background sound and the patient's basic physiological state. The purpose of this mechanism is to complete the spatial isolation of event tokens and the setting of basic judgment boundaries before the main computing core goes into sleep mode, so as to avoid multimodal tokens from overwriting each other and reduce the probability of false wake-up caused by environmental noise.

[0020] In a preferred embodiment of the present invention, the step of calculating the feature residual of two consecutive frames of feature data in S1 includes: calculating the degree of difference between two consecutive frames of feature data by means of an arithmetic logic unit deployed on the direct memory access bus side and located in the hardware feature evaluation circuit; and determining the degree of difference as the feature residual.

[0021] This embodiment provides a difference calculation mechanism on the direct memory access bus side; specifically, considering that the aforementioned scheme needs to continue to determine whether an event has occurred while the main computing core is in sleep mode, if the residual calculation is handed over to the main computing core for execution, the sleep effect will be destroyed. If the complete data is written to the dynamic random access memory before calculation, the power consumption of memory access cannot be reduced; therefore, in this embodiment, an arithmetic logic unit is deployed on the direct memory access bus side to calculate the difference between two consecutive frames of feature data and to determine the difference as the feature residual. The arithmetic logic unit is located between the sensor interface and the dynamic random access memory write path, and can quickly compare feature data before the direct memory access descriptor is submitted; this arithmetic logic unit does not undertake complete neural network inference, but only performs low-complexity difference operations; For example, for video modalities, feature data can be simplified to several facial region change markers; for audio modalities, feature data can be simplified to the energy level of the current window, speech start and end markers, and abrupt change markers; for physiological modalities, feature data can be simplified to heart rate rise, skin conductance rise, or respiratory rhythm change markers. To illustrate with a simplified example, if the previous frame feature V1 extracted from a certain video window contains states such as stable eyes, stable eyebrows, and stable mouth, and the current frame feature V2 contains states such as stable eyes, changing eyebrows, and stable mouth, the arithmetic logic unit only needs to compare which of the three fields have changed to obtain the degree of difference. If the previous frame feature P1 of the physiological sensing window contains stable heart rate and stable skin conductance, and the current frame feature P2 contains increased heart rate and increased skin conductance, then the difference reflects the synchronous change in physiological mode. The above comparison can be completed in parallel on the bus side by latching the previous frame feature through the register and comparing it when the current frame arrives, without calling the main computing core. As an exception handling mechanism, if the previous frame of two consecutive frames is missing, such as when the device has just started up or the sensor has just reconnected, the arithmetic logic unit writes the current frame as the reference frame into the previous frame register, but does not immediately generate a high activity event; if the current frame verification fails, the previous frame register is not updated to avoid erroneous data contaminating subsequent difference judgments; if there are no valid frames in multiple consecutive windows, the modality enters a low confidence state, and the cross-modal activity accumulation register can reduce its impact when generating joint activity. During nighttime monitoring, the main computing core of the bedside terminal is in hibernation; when the patient makes a positional adjustment with an amplitude less than the preset displacement threshold, the head position changes in the video features, but there is no significant change in the eyebrow and mouth emotional areas, and the difference obtained by the arithmetic logic unit is insufficient to form an emotional event; when facial tension and rapid breathing occur, the difference between the video and the continuous window of physiological features increases significantly, and event token candidates can be generated directly by accessing the memory bus side. The purpose of this step is to move the calculation of consecutive frame differences forward to a low-power hardware path, thereby reducing the number of times the main computing core participates in basic filtering and completing the interception of invalid data before the data enters the dynamic random access memory. In a preferred embodiment of the present invention, the step of generating multimodal joint activity in S3 includes: obtaining the preset activity weights corresponding to each modal sensor that are pre-configured; and performing weighted summation on the feature activity values ​​corresponding to the feature residuals output by each modal sensor based on the preset activity weights to generate multimodal joint activity.

[0022] This embodiment provides a cross-modal activity weighted accumulation mechanism; specifically, the aforementioned scheme can identify valid events of a single modality, but in the medical monitoring field, the reliability of different modalities is not constant; If all modal events are simply treated as equally important, false wake-ups or missed wake-ups may occur in the presence of strong background noise, occlusion, or loose sensors. Therefore, this embodiment sets a preset activity weight for each modal sensor and performs a weighted summation of the feature activity values ​​based on this weight to generate a multimodal joint activity. The preset activity weight can be determined by the equipment configuration, ward environment, individual patient condition, or sensor quality indicators; the video modality can have a weight value greater than the preset first weight when the lighting is sufficient and the face is not obstructed; the weight can be reduced when the infrared image noise at night is greater than the preset noise threshold or when it is obstructed by bedding. The audio modality is more sensitive to anxiety and agitation precursors when the ward is quiet, but background noise may reduce its reliability when there are frequent nursing procedures in the corridor; the physiological modality usually has a continuity advantage when the wristband is worn stably, but its weight should be reduced if the wristband has poor contact. Furthermore, the preset activity weight is preferably understood as a set of baseline weights pre-stored in the hardware, rather than being freely calculated by the main computing core on a temporary basis each time an event occurs; Specifically, the baseline weights of video, audio and physiological modalities can be preset in the register table or lookup table, and several discrete adjustment coefficients can be preset for common quality states, such as normal, low confidence, occlusion, noise greater than the preset noise threshold, poor contact and other states corresponding to different adjustment coefficients; After reading the quality flag in the event token, the cross-modal activity accumulation register directly obtains the currently effective weight from the corresponding entry, thus satisfying both the preset weight value and adapting to changes in on-site quality. To illustrate with a simplified data flow example, suppose that within a certain observation window, the video modality generates an event of facial expression change, the audio modality generates an event of short vocalization, and the physiological modality generates an event of increased heart rate. The cross-modal activity accumulation register does not store the original video, audio, and physiological data. Instead, it reads the characteristic activity values ​​corresponding to the three events and writes them into the accumulation field according to their respective weights. If the current ward is in a dark state and the video quality indicator is low, the video event can still be recorded, but its contribution to the joint activity is reduced. If the wristband contact is stable and the skin conductance increases significantly, the contribution of the physiological event to the joint activity is retained. Furthermore, to avoid confusion between preset weights and real-time quality changes, the effective weight of the current modality can be limited to the baseline weight multiplied by the preset adjustment coefficient corresponding to the quality state. For example, the baseline weight of the video modality is written to a register when the device is deployed. If the quality flag in the event token shows low illumination, the hardware reads the adjustment coefficient corresponding to low illumination from the lookup table and reduces the video modality contribution accordingly. If the quality flag returns to normal, the adjustment coefficient corresponding to the normal state is reused. The above processing still belongs to hardware weighting within the preset weight framework, rather than unbounded, non-predefined weighting at runtime. Furthermore, the meanings determined by the above factors, including equipment configuration, ward environment, individual patient condition, or sensor quality indicators, are preferably discrete configuration items that are pre-written into register tables, configuration tables, or lookup tables, rather than new free weight values ​​being regenerated by the main computing core after an event occurs. In other words, when generating multimodal joint activity, the cross-modal activity accumulation register preferably executes only a deterministic hardware process of reading preset entries, obtaining effective weights, and weighted summation of feature activity values. Among them, the ward environment can select the corresponding weight level through preset scene identifiers, the patient's individual situation can select the corresponding benchmark weight set through preset monitoring file number, and the sensor quality indicator can correct the current contribution through preset adjustment coefficient, thereby ensuring that the preset activity weight benchmark weight and effective weight in the whole text always belong to the same pre-configured weight system. In abnormal or boundary situations, if the weight of a certain modality is reduced to a low confidence range, the modality event token can still be stored in static random access memory for subsequent manual review or model-assisted judgment, but will not trigger the main computing core to wake up independently. If the quality of all modalities is below the usable level, the terminal can send a device maintenance prompt and switch to a periodic low-frequency wake-up strategy to avoid completely stopping the emotional state assessment when the sensor fails. If a certain modality continuously generates high activity but other modalities are completely unresponsive, the system can first keep the tokens accumulating and wait for cross-modal verification within the first preset time window. After the waiting time is exceeded, it will be reported as a single-modal high-priority anomaly. During nighttime rounds in the ward, nursing equipment in the corridor emitted short-term alerts, and the audio modality showed high changes. However, the bedside camera did not detect any changes in the patient's facial or limb tension, and the wristband physiological data remained stable. Because the environmental noise weight of the audio modality was low during this period, the cross-modal joint activity did not exceed the threshold, and the main computing core remained dormant. Conversely, when the patient made low, rapid vocalizations in a quiet environment, and at the same time, skin conductance and heart rate increased, the joint activity was rapidly accumulated, thereby triggering subsequent classification. The purpose of this mechanism is to achieve reliability tuning of multimodal evidence at the hardware bus level, thereby reducing false wake-ups caused by noise from a single sensor and maintaining responsiveness to real emotional change events.

[0023] In a preferred embodiment of the present invention, before reading the event token in the static random access memory through the main computing core in step S5, the method further includes: switching the main computing core from a sleep state to a running state in response to a hardware interrupt signal.

[0024] This embodiment provides a hardware interrupt wake-up mechanism; specifically, the aforementioned scheme determines whether sparse fusion calculation is needed by using a cross-modal activity accumulation register, but if the main computing core cannot be reliably and timely switched from sleep state to running state, event tokens may accumulate in static random access memory, affecting the monitoring timeliness; Therefore, in this embodiment, before reading the event token, the interrupt controller responds to the hardware interrupt signal and switches the main computing core from sleep state to running state; When the combined activity of the multimodal modes exceeds the activity threshold, the cross-modal activity accumulator register outputs a hardware interrupt signal to the interrupt controller. The interrupt controller determines whether the signal belongs to the emotion risk monitoring interrupt according to the preset priority. If so, it restores the clock of the main computing core, de-gated the corresponding power domain, restores the on-chip cache access permission, and points the program counter entry to the event token reading service program. This process does not rely on the operating system's periodic polling and does not require the main computing core to switch to the running state periodically to check the dynamic random access memory, thus maintaining the event-driven characteristics. To simplify the process, three event tokens already exist in the static random access memory: video token T1, audio token T2, and physiological token T3; after the cross-modal activity accumulator register confirms that the joint activity corresponding to these tokens has reached the critical condition, it sends an interrupt signal. The interrupt controller wakes up the main computing core; the main computing core reads T1, T2, and T3, and clears the corresponding read pointers or marks the token as processed after reading is complete; in this way, the data path remains that the event token triggers the wake-up, instead of the main computing core periodically scanning all sensor data; As an exception handling mechanism, if the main computing core fails to wake up within a preset time, the interrupt controller can repeatedly send interrupts or switch to the backup low-power processing core to perform minimum classification. If the main computing core is processing other high-priority medical alarms, the event token is kept in static random access memory and a wait flag is set to prevent it from being overwritten. If the interrupt signal jitters or occurs repeatedly within the first preset duration, the interrupt controller can merge adjacent interrupts and only respond to the next batch after the current batch of event tokens has been processed, so as to avoid frequent wake-ups that lead to increased power consumption. When a patient experiences continuous short vocalizations and an increase in heart rate at night, the cross-modal activity accumulation register sends an emotional risk interrupt to the interrupt controller. The interrupt controller switches the main computing core from deep sleep to running state. The main computing core reads the audio and physiological event tokens in the static random access memory, and then supplements the reading with adjacent video event tokens based on the timestamp, completing a fast sparse fusion judgment. The purpose of this mechanism is to establish a deterministic wake-up path from the joint activity level of the hardware side to the operating status of the main computing core, thereby avoiding the power waste caused by periodic polling and ensuring the timeliness of handling high-risk emotional events. In a preferred embodiment of the present invention, the step of adjusting the basic residual threshold based on the multimodal classification result in S5 includes: extracting the confidence parameter and the contribution of each modality from the multimodal classification result; Determine the relationship between the confidence parameter and the preset confidence threshold; If the confidence parameter is greater than the confidence threshold, the modal sensor that contributes the most to the multimodal classification result is determined as the target modal sensor, and the basic residual threshold corresponding to the target modal sensor is reduced; if the confidence parameter is less than or equal to the confidence threshold, the basic residual threshold is kept unchanged.

[0025] This embodiment provides a threshold feedback mechanism based on classification confidence. Specifically, the aforementioned scheme can trigger sparse classification through event tokens, but the fixed baseline residual threshold may not be sensitive enough when the patient's emotional state transitions from a stable state to a risky state. Conversely, if the classification results themselves are unstable, unconditionally lowering the threshold will cause a large number of invalid events to wake up the main computing core. Therefore, this embodiment extracts the confidence parameter from the multimodal classification results and determines whether to adjust the basic residual threshold based on the relationship between the confidence parameter and the preset confidence threshold. After the main computing core completes the sparse attention mechanism calculation, it outputs the multimodal classification results and their confidence parameters. The confidence parameters are used to represent the stability of the current event token combination in judging a certain emotional state. For example, if video events, audio events, and physiological events corroborate each other in time, the model outputs a high confidence level for arousal precursors. If there is only a single audio event and the environmental noise is strong, the classification confidence level is low. The system compares this confidence parameter with a preset confidence threshold. If the confidence parameter is greater than the threshold, it indicates that the current classification result has high credibility. The terminal further determines the target modal sensor that contributes the most to the classification result and lowers the basic residual threshold corresponding to the target modal sensor, making it easier for subsequent similar early changes to form event tokens. If the confidence parameter is less than or equal to the threshold, it indicates that the classification result is insufficient to support the hardware threshold change, and the basic residual thresholds remain unchanged. Furthermore, the highest contribution is determined based on the modal-level contribution markers, attention weight summaries, or event token matching scores already generated by the main computing core in this sparse fusion, without the need to introduce an additional large-scale analysis process. In other words, the terminal can simultaneously output target modality identifiers such as high video contribution, high audio contribution, or high physiological contribution at the end of this classification calculation. The threshold control logic will then only perform a reduction operation on the basic residual threshold of the target modality. This can explain the source of the threshold feedback without compromising the low-power design. To illustrate with a simplified data flow example, in a certain sparse fusion, the event tokens include physiological modality tokens for increased skin conductance, increased heart rate, and low-pitched rapid vocalization. The model outputs increased anxiety with confidence scores higher than the preset boundary. Meanwhile, contribution analysis shows that physiological modality tokens play a major role in classification. At this point, the terminal lowers the physiological modality baseline residual threshold so that subsequent increases in skin conductance or changes in heart rate with amplitudes less than the preset threshold can also generate event tokens; if another classification is mainly triggered by background noise and the confidence level does not reach the preset boundary, the audio threshold is not lowered to avoid frequent wake-ups caused by ward environmental noise. Furthermore, to avoid repeated decreases in the threshold within a preset time period causing system sensitivity oscillations, the action of reducing the basic residual threshold can be limited to taking effect on subsequent preset observation periods, and the magnitude of a single reduction is limited to a preset step value. If no high-confidence classification result of the same type is obtained again within the observation period, the basic residual threshold can remain at its current value or be gradually restored to the initial level in subsequent periods. In this way, the threshold feedback can reflect the increased sensitivity when the risk increases, and will not be in a state where the trigger condition is lower than the preset baseline for a long time due to a single incident. In abnormal or boundary situations, the threshold reduction can set a lower limit to prevent the basic residual threshold from being too low, which would cause random noise to continuously generate event tokens; if the same mode continuously generates low confidence classifications after the threshold is reduced, the system can restore its threshold to the initial level or enter the sensor quality check. If the contribution of multiple modes is close, the terminal can reduce the threshold of only the mode with the highest quality indicator, or use a preset first step amplitude synchronous adjustment to avoid the system sensitivity from suddenly becoming too high. During the same nighttime monitoring cycle in the psychiatric ward, the patient experienced a simultaneous increase in heart rate and skin conductance accompanied by low-pitched rapid vocalizations for the first time. Sparse fusion results showed a high confidence level of increased anxiety, with the physiological modality contributing the most. The terminal lowers the baseline residual threshold of the physiological modality, causing the system to pay more attention to early changes in skin conductance and heart rate in the following period; if the classification confidence is low only due to the audio token generated by corridor noise, the audio modality threshold remains unchanged. The purpose of this mechanism is to configure the basic residual threshold on the hardware side in reverse through the classification results, so that the terminal can increase the sensitivity of the relevant modalities when the risk increases and remain stable when the evidence is insufficient, thereby taking into account both low power consumption and monitoring reliability. In a preferred embodiment of the present invention, the step of calculating the difference between two consecutive frames of feature data includes: processing the two consecutive frames of feature data using a pre-set Manhattan distance algorithm or an XOR gate logic operation to generate the difference.

[0026] This embodiment provides a lightweight difference generation mechanism. Specifically, the aforementioned arithmetic logic unit needs to run continuously on the direct memory access bus side. If complex floating-point operations or deep model inference are used, the hardware area and power consumption will increase significantly, which is not conducive to the long-term operation of bedside terminals and wearable terminals. Therefore, this embodiment uses the Manhattan distance algorithm or XOR gate logic operation to process two consecutive frames of feature data to generate difference. For feature data with level values ​​or count values, the Manhattan distance algorithm can be used; in hardware implementation, this algorithm mainly involves the calculation and accumulation of absolute differences, and is suitable for processing features such as audio energy level, heart rate change level, and skin conductance change level. For example, the physiological characteristics of the previous window can be quantified as heart rate level 2, skin conductance level 1, and respiratory level 2, and the physiological characteristics of the current window can be quantified as heart rate level 3, skin conductance level 3, and respiratory level 2. The arithmetic logic unit compares the changes in the levels of each field and accumulates them to obtain the difference. This process does not require a complex multiplication array and is suitable for low-power register-level implementation. For binary flags or bitmap-type feature data, XOR gate logic operations can be used; XOR gates are suitable for determining whether there is a change, such as facial region change flags, voice occurrence flags, wristband contact status flags, etc. If the feature flag of the previous frame is 0010 and the feature flag of the current frame is 0110, the XOR result is 0100, indicating that only one preset area has changed; if the feature flags of the previous frame and the current frame are the same, the XOR result is 0000, indicating that no change has been detected; the XOR gate array can process multiple bits in parallel and directly output the number of changed bits or the change flag. In abnormal or boundary situations, when the feature data is in a mixed format, the terminal can use Manhattan distance for the rank field, XOR gate logic for the binary field, and combine the two into a difference degree through a preset mapping. When the sensor output has saturation, overflow, or missing fields, the corresponding fields are not used in the difference generation, but are written into the quality flag; when the time interval between consecutive frames is abnormally long, the system does not directly use the difference to trigger the event, but first re-establishes the reference frame to avoid misjudgment caused by excessive time interval. In the bedside terminal in the ward, whether the eyebrow, eye, and mouth regions of the video modality have changed can be represented by binary flags, so XOR gate logic is used for fast judgment; changes in heart rate and skin conductance collected by the wristband can be converted into a rank field, so Manhattan distance is used for processing; both types of results are generated on the direct memory access bus side, without the need for the main computing core to participate; The purpose of this mechanism is to provide a low-complexity difference calculation method suitable for hardware implementation, thereby reducing the power consumption and area of ​​the bus-side evaluation circuit and ensuring that event filtering has sufficient real-time performance.

[0027] In a preferred embodiment of the present invention, each modal sensor includes a video sensor, an audio sensor, and a physiological sensor, and the multimodal sensor data includes video data, audio data, and physiological sensing data collected by the above sensors respectively. The multimodal classification results are the emotional state assessment results for the preset target users.

[0028] This embodiment provides a mechanism for collaboratively generating emotional state assessment results from video, audio, and physiological sensor data; specifically, the aforementioned scheme can be applied to a variety of sparse asynchronous signals, but in the monitoring of high-risk patients in closed psychiatric wards, a single modality is often difficult to reliably reflect emotional state. Using only video data may be affected by occlusion and lighting, using only audio data may be affected by noise in the ward environment, and using only physiological sensor data may be affected by wearing status and underlying diseases. Therefore, this embodiment limits the multimodal sensor data to video data, audio data, and physiological sensor data, and outputs the multimodal classification results as the emotional state assessment results of the target user. Video data is used to capture overt behavioral changes such as facial expressions, eye tension, head posture, and body movements; audio data is used to capture acoustic changes such as speech energy, short phonations, abrupt changes in tone, or sudden phonation after silence. Physiological sensor data is used to capture changes in autonomic nervous activity such as heart rate, skin conductance, and respiratory rhythm. The three types of data have different temporal characteristics in the formation of emotional states: physiological changes may precede obvious behaviors, audio changes may precede communication or agitation, and video changes can reflect facial and motor performance. After filtering by hardware-side event tokens, the main computing core only performs sparse fusion on these feature data that have change significance, and outputs the emotional state assessment results of the target user. To simplify the state mapping, if the event token mainly manifests as stable video, stable audio, and stable physiological state, the system outputs stable emotion; if it manifests as increased physiological state, short, abrupt vocalizations with audio energy exceeding a preset energy threshold, or tense facial expressions in the video, the system can output increased anxiety. If the symptoms include increased range of motion, sudden increase in volume, or significant physiological elevation, the system may output a warning sign of agitation or require nursing review. The above status assessment results can be sent to the nursing station monitoring platform or generated locally on the bedside terminal to form a graded prompt. As an anomaly handling mechanism, if video data is unavailable due to occlusion, the system can still generate a low-confidence assessment based on audio and physiological sensor data, along with a video missing flag; if audio data is temporarily disabled due to privacy policies, the system can retain only the audio energy event without saving the voice content. If the physiological sensor falls off, the system will reduce the weight of the physiological modality and prompt the caregiver to check the wearing status; for any single modality abnormality, the system does not directly equate it with the target user's emotional abnormality, but combines the remaining modalities and confidence mechanism to form an assessment result; Furthermore, to ensure that the terminology in this embodiment is consistent with that in the full text, the target user in this scenario is specifically the monitored patient; The emotional state assessment results, as the output objects defined in the embodiment, can be specifically represented as classification labels such as stable mood, increased anxiety, agitation, or need for manual review in the context of psychiatric care. Therefore, the emotional state assessment results or emotional risk assessment results mentioned in this article refer to the same multimodal classification output, rather than adding different result types. Accordingly, the video modality, audio modality, and physiological modality mentioned above correspond to the video data, audio data, and physiological sensor data in this embodiment, respectively, without changing the defined range of the three types of sensor data; In the same psychiatric ward monitoring link, the bedside camera captures images of the patient's face and upper body, the directional microphone captures acoustic events near the bed, and the wristband captures heart rate and skin conductance signals. The terminal discards unchanged cache during most stable periods and generates event tokens only when there are valid changes in video, audio or physiology. When the main computing core is woken up, these tokens are sparsely fused and the emotional state assessment results, such as emotional stability, increased anxiety, arousal, or need for manual review, are output. The purpose of this mechanism is to improve the reliability of emotional state assessment by leveraging the complementarity of video, audio, and physiological sensor data, while avoiding high-power memory access caused by continuous raw data through hardware-level event filtering.

[0029] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A hardware optimization method for a low-power multi-modal sentiment analysis terminal, the terminal comprising a main computing core, a direct memory access bus, a hardware feature evaluation circuit, a static random access memory, a dynamic random access memory, an interrupt controller, a cross-modal activity accumulation register, and a plurality of modal sensors, the method comprising: include: ​ S0. Control the main computing core to enter a sleep state; S1. Acquire multimodal sensor data collected by each modal sensor, extract feature data through hardware feature evaluation circuit on the direct memory access bus side, and calculate feature residuals of the feature data for two consecutive frames; S2. If the feature residual is less than a preset basic residual threshold, perform a cache discard operation to prevent the multimodal sensor data from being written to the dynamic random access memory; S3. If the feature residual is not less than the basic residual threshold, generate an event token containing the feature data and store it in the static random access memory. According to the positive correlation residual-activity mapping rule, convert the feature residual into a feature activity value and accumulate it to the cross-modal activity accumulation register to generate multimodal joint activity. S4. If the multimodal joint activity is greater than the preset activity threshold, a hardware interrupt signal is sent to the interrupt controller to wake up the main computing core and enter the running state; otherwise, it remains in the dormant state. S5. The event token in the static random access memory is read through the main computing core, feature data is extracted and sparse attention mechanism is executed to output a multimodal classification result containing confidence parameters and the contribution of each modality, and the basic residual threshold is adjusted in reverse based on the multimodal classification result.

2. The hardware optimization method for the low-power multimodal emotion analysis terminal according to claim 1, characterized in that, Before controlling the main computing core to enter a hibernation state, S0 also includes: Each modal sensor is allocated an independent block of the static random access memory; Initialize the basic residual threshold corresponding to each modal sensor.

3. The hardware optimization method for the low-power multimodal emotion analysis terminal according to claim 1, characterized in that, The step of calculating the feature residual of two consecutive frames of feature data in S1 includes: The degree of difference between the two consecutive frames of feature data is calculated by an arithmetic logic unit deployed on the direct memory access bus side and located within the hardware feature evaluation circuit. The degree of difference is defined as the characteristic residual.

4. The hardware optimization method for the low-power multimodal emotion analysis terminal according to claim 1, characterized in that, The step of generating multimodal joint activity in S3 includes: Obtain the preset activity weights corresponding to each modal sensor that are pre-configured; Based on the preset activity weights, the feature activity values ​​corresponding to the feature residuals output by each modal sensor are weighted and summed to generate the multimodal joint activity.

5. The hardware optimization method for the low-power multimodal emotion analysis terminal according to claim 1, characterized in that, Before reading the event token from the static random access memory via the main computing core in step S5, the process further includes: In response to the hardware interrupt signal, the main computing core is switched from the hibernation state to the running state.

6. The hardware optimization method for the low-power multimodal emotion analysis terminal according to claim 1, characterized in that, The step in S5 of adjusting the base residual threshold based on the multimodal classification result includes: Extract the confidence parameter and the contribution of each modality from the multimodal classification results; Determine the relationship between the confidence parameter and the preset confidence threshold; If the confidence parameter is greater than the confidence threshold, then the modal sensor that contributes the most to the multimodal classification result is determined as the target modal sensor, and the basic residual threshold corresponding to the target modal sensor is reduced. If the confidence parameter is less than or equal to the confidence threshold, then the basic residual threshold remains unchanged.

7. The hardware optimization method for the low-power multimodal emotion analysis terminal according to claim 3, characterized in that, The step of calculating the difference between the feature data of two consecutive frames includes: The feature data of the two consecutive frames are processed using a pre-defined Manhattan distance algorithm or an XOR gate logic operation to generate the difference.

8. The hardware optimization method for the low-power multimodal emotion analysis terminal according to claim 1, characterized in that, The modal sensors include video sensors, audio sensors, and physiological sensors, and the multimodal sensor data includes video data, audio data, and physiological sensing data collected by the above sensors respectively. The multimodal classification results are the emotional state assessment results for the preset target users.

Images