Multimedia data acquisition method and system

Abstract

The invention relates to the related technical field of data acquisition and processing, in particular to a multimedia data acquisition method and system, which comprises the following steps: connecting a multimedia monitoring device, acquiring audio and image data, drawing up individual and environment modal parameters, executing multi-modal aggregation model fitting, and configuring a risk item evaluation network layer. According to the health monitoring system, the technical problems that the health monitoring equipment can only obtain limited physiological data, neglects the influence of non-physiological factors and cannot make effective response in time aiming at health risk events are solved, and the health monitoring efficiency is improved. By integrating various types of data such as audio monitoring data, image monitoring data and environmental noise, a monitoring reminding unit and an emergency response mechanism are introduced, and the technical effects of timely and effectively responding to health risk events and improving the response speed and timeliness are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field related to data acquisition and processing, and particularly to a multimedia data acquisition method and system. Background Art

[0002] With the rapid development of information technology, technologies such as the Internet of Things, artificial intelligence, and big data analysis have gradually penetrated into the field of medical and health, especially in aspects such as intelligent health monitoring, telemedicine, and personalized medicine. Commonly, health monitoring devices have been able to monitor some basic physiological data (such as electrocardiogram, blood pressure, respiration, etc.) in real time. However, the application scenarios of these devices are mostly limited to static monitoring, and usually can only collect single physiological data, lacking the ability of multi-modal data integration, and unable to provide users with personalized and real-time responsive health services. Especially in emergency situations, it is unable to quickly trigger an emergency response mechanism and make rapid processing.

[0003] In summary, there are technical problems in the prior art that health monitoring devices can only obtain limited physiological data, ignore the influence of non-physiological factors, and cannot effectively respond to health risk events in a timely manner. Summary of the Invention

[0004] The present application provides a multimedia data acquisition system, aiming to solve the technical problems in the prior art that health monitoring devices can only obtain limited physiological data, ignore the influence of non-physiological factors, and cannot effectively respond to health risk events in a timely manner.

[0005] In view of the above problems, the technical solution of the present application is as follows: On the one hand, the present application provides a multimedia data acquisition method, wherein the method includes: connecting a multimedia monitoring device to obtain audio monitoring data and image monitoring data, the audio monitoring data including respiration sound parameters and environmental noise parameters, and the image monitoring data including facial expression parameters and environmental light parameters; formulating individual modal parameters through the respiration sound parameters in the audio monitoring data and the facial expression parameters in the image monitoring data; formulating environmental modal parameters through the environmental noise parameters in the audio monitoring data and the environmental light parameters in the image monitoring data; performing fitting of a multi-modal aggregation model based on the individual modal parameters and the environmental modal parameters, the multi-modal aggregation model being used to recursively integrate different modal data; configuring a risk event assessment network layer, the risk event assessment network layer being quantitatively evaluated according to risk assessment criteria; performing feature registration on a monitoring reminder unit built in the multimedia monitoring device based on the multi-modal aggregation model and the risk event assessment network layer to obtain a monitoring reminder instruction; and uploading the monitoring reminder instruction to a monitoring terminal device for real-time display and alarm.

[0006] On the other hand, the present application provides a multimedia data acquisition system, wherein the system includes: a data monitoring module for connecting to a multimedia monitoring device to obtain audio monitoring data and image monitoring data, the audio monitoring data including breathing sound parameters and environmental noise parameters, and the image monitoring data including facial expression parameters and environmental light parameters; a first parameter determination module for determining individual modality parameters through the breathing sound parameters in the audio monitoring data and the facial expression parameters in the image monitoring data; a second parameter determination module for determining environmental modality parameters through the environmental noise parameters in the audio monitoring data and the environmental light parameters in the image monitoring data; a recursive integration module for performing fitting of a multi-modal aggregation model based on the individual modality parameters and the environmental modality parameters, the multi-modal aggregation model being used for recursively integrating different modality data; a quantization evaluation module for configuring a risk event evaluation network layer, the risk event evaluation network layer performing quantization evaluation according to a risk evaluation standard; a feature registration module for performing feature registration on a built-in monitoring reminder unit of the multimedia monitoring device based on the multi-modal aggregation model and the risk event evaluation network layer to obtain a monitoring reminder instruction; and an instruction uploading module for uploading the monitoring reminder instruction to a monitoring terminal device for real-time display and alarm.

[0007] In summary, one or more technical solutions provided in the present application solve the technical problem that health monitoring devices can only obtain limited physiological data, ignore the influence of non-physiological factors, and cannot make effective responses to health risk events in a timely manner, and achieve the technical effect of integrating various types of data such as audio monitoring data, image monitoring data, and environmental noise, introducing a monitoring reminder unit and an emergency response mechanism, making effective responses to health risk events in a timely manner, and improving the response speed and timeliness. Description of the Drawings

[0008] Figure 1 is a schematic flow chart of a multimedia data acquisition method provided by the present application; Figure 2 is a schematic structural diagram of a multimedia data acquisition system provided by the present application.

[0009] Description of the Reference Numerals: data monitoring module M100, first parameter determination module M200, second parameter determination module M300, recursive integration module M400, quantization evaluation module M500, feature registration module M600, instruction uploading module M700. Detailed Embodiments

[0010] Embodiment 1 The present application will be specifically described below with reference to the drawings. As Figure 1 shown, the present application provides a multimedia data acquisition method, wherein the method includes: S1: Connect the multimedia monitoring device to obtain audio monitoring data and image monitoring data. The audio monitoring data includes breathing sound parameters and ambient noise parameters, and the image monitoring data includes facial expression parameters and ambient light parameters; S2: Determine individual modal parameters based on the breathing sound parameters in the audio monitoring data and the facial expression parameters in the image monitoring data; S3: Determine environmental modal parameters based on the ambient noise parameters in the audio monitoring data and the ambient light parameters in the image monitoring data.

[0011] Specifically, audio monitoring data refers to various types of data related to sound collected by audio sensors. The audio monitoring data includes: breathing sound parameters (sound signals related to an individual's breathing collected by an audio device, analyzing their frequency, intensity, interval, etc. characteristics for evaluating the individual's breathing health status), ambient noise parameters (data on the ambient background noise level monitored by an audio device, reflecting the surrounding sound conditions and having an impact on the individual's health).

[0012] Image monitoring data refers to visual data collected by an image acquisition device (such as a camera), usually used to analyze an individual's facial expressions and the lighting changes in the environment. The image monitoring data includes: facial expression parameters (parameters of facial muscle changes related to an individual's emotions, stress, fatigue, etc. extracted through facial recognition technology, helping to evaluate the individual's mental and physical states), ambient light parameters (data on the ambient light intensity collected by an image device (such as a camera or a light sensor), reflecting the changes in environmental light and affecting the individual's sleep quality, emotional state, etc.).

[0013] Individual modal parameters refer to characteristic data describing an individual's health, emotions, behaviors, etc. extracted based on audio and image monitoring data, mainly involving the combination of breathing sounds and facial expressions, used to comprehensively evaluate the individual's physiological and psychological states; environmental modal parameters refer to data extracted from audio and image monitoring data, describing the impact of environmental factors (such as noise and light) on an individual's health and comfort, used to analyze the potential impact of environmental changes on an individual's health.

[0014] Execution steps: By connecting multimedia monitoring devices (such as audio sensors and image acquisition devices), real-time audio and image monitoring data are obtained, including microphones, environmental noise sensors, cameras, infrared sensors, etc.; Audio data refers to the breathing sounds collected from the microphone and the noise level of the surrounding environment. For example, the breathing sound parameters can be the frequency, intensity, and regularity of breathing, and the environmental noise parameters are the magnitude and fluctuation of the background noise; Image data refers to the facial expression data and environmental light data obtained through the camera. The facial expression data can analyze a person's emotional state, whether they are tired or stressed through computer vision algorithms (such as facial recognition technology), and the environmental light data is collected through sensors or image analysis techniques to obtain the light intensity of the current environment, reflecting the degree of influence of light on the individual.

[0015] Individual modal parameters are mainly based on breathing sound parameters and facial expression parameters. Specifically, the breathing frequency and breathing pattern (such as shallow breathing or deep breathing) are closely related to an individual's physiological state, and facial expressions are closely related to the psychological state. For example, if the breathing becomes rapid or irregular, it indicates that the individual is anxious or has respiratory problems; while the analysis of facial expressions (such as frowning, downturned corners of the mouth, etc.) indicates that the individual is stressed or in a low mood; By combining these data, individual modal parameters can be generated, such as health indicators like an individual's anxiety level and stress level. The corresponding tools include speech analysis algorithms (for analyzing breathing patterns) and facial recognition algorithms (for facial expression analysis).

[0016] Environmental modal parameters are mainly based on environmental noise parameters and environmental light parameters, reflecting the potential impact of the surrounding environment on the individual. The environmental noise parameter refers to the noise level in the environment. If the noise is too loud, it affects the individual's sleep quality or causes mood swings. For example, when the environmental noise exceeds a certain threshold, it means the environment is too noisy and the individual's health status is affected as a result; The environmental light parameter refers to the brightness level in the environment. In an environment with insufficient light, it causes the individual's biological clock to be disrupted or affects the mood. Therefore, the environmental light parameter is crucial for evaluating an individual's health and comfort; By analyzing these environmental data, environmental modal parameters can be generated, such as the degree of interference of environmental noise on the individual and the impact of insufficient light on health.

[0017] After obtaining the individual modal parameters and environmental modal parameters, these data will be fused to form a multi-dimensional health status model, which means considering not only the individual's physiological and psychological states but also the influence of environmental factors; Through a multi-modal aggregation model, multiple factors such as breathing sounds, facial expressions, environmental noise, and light are combined to conduct health risk assessment and status prediction. For example, when the environmental noise is large and the individual's facial expression shows stress, a health warning is issued.

[0018] S4: Based on the individual modality parameters and the environmental modality parameters, perform the fitting of the multi-modal aggregation model, where the multi-modal aggregation model is used to recursively integrate different modality data; S5: Configure the risk event assessment network layer, where the risk event assessment network layer conducts quantitative assessment based on risk assessment criteria; S6: Based on the multi-modal aggregation model and the risk event assessment network layer, perform feature registration on the built-in guardianship reminder unit of the multimedia monitoring device to obtain a guardianship reminder instruction; S7: Upload the guardianship reminder instruction to the guardianship terminal device for real-time display and alarm.

[0019] Specifically, the individual modality parameters refer to the health status indicator variables extracted from the audio and image monitoring data related to the individual (such as breathing sound parameters and facial expression parameters), which reflect the physical and mental states of the individual, such as emotions, stress, fatigue, etc.; the environmental modality parameters refer to the environmental characteristics extracted from the audio and image monitoring data related to the environment (such as environmental noise and environmental light parameters), which reflect the potential impact of the environment on the individual, such as the impact of factors like insufficient light on health.

[0020] The multi-modal aggregation model is a computational model used to integrate information from different monitoring modalities (such as audio and image data). Through a recursive algorithm, it aggregates and analyzes multi-modal data to generate a comprehensive health status assessment result; the risk event assessment network layer is a network layer based on risk assessment criteria, and its function is to perform quantitative risk assessment on the individual health status. By deeply analyzing the modality data, it assesses potential health risks and generates an assessment result.

[0021] The guardianship reminder unit is a functional module built into the multimedia monitoring device, and its task is to generate a guardianship reminder instruction for health risks based on the assessment result, which is used to remind the guardian to pay attention to health risks; the guardianship reminder instruction is an instruction generated by the guardianship reminder unit, which is used to trigger health risk alarms, prompts or other response measures. The instruction is uploaded to the guardianship terminal device for real-time display and alarm; the guardianship terminal device refers to the terminal device that receives the guardianship reminder instruction and performs real-time display and alarm, which can be a smart phone, a computer, a medical monitoring terminal, etc., to display the guardianship information of the system and remind users or health guardians.

[0022] Execution steps: Collect monitoring data from audio and image devices to obtain individual modality parameters (such as breathing sounds and facial expressions) and environmental modality parameters (such as environmental noise and light data); use these modality data to enter a multimodal aggregation model for fitting. The goal of the multimodal aggregation model is to integrate multimodal information from different data sources and continuously update and optimize the model through a recursive algorithm to more accurately describe the health status of individuals and the environment. For example, a Recurrent Neural Network (RNN) or a Long Short-Term Memory (LSTM) network can be applied to this task, and these networks can process time-series data and learn the interdependencies between data; generate a comprehensive health assessment output through the multimodal aggregation model, and this output can reflect the health status of an individual under current environmental conditions. For example, by combining an individual's breathing condition and emotional expression, as well as the noise and light data of the surrounding environment, the model can evaluate whether an individual is in a health risk state.

[0023] Based on the health assessment model, configure a risk item assessment network layer. The risk item assessment network layer evaluates health risks through a quantitative method based on preset risk assessment criteria. For example, a set of thresholds can be set. If an individual's health parameters (such as breathing rate, stress index of facial expression) exceed a certain safe range, it indicates that the individual has potential health risks. At this time, the network layer will perform a quantitative assessment on the multimodal data to generate specific risk values.

[0024] Based on the output results of the generated multimodal aggregation model and the risk item assessment network layer, perform feature registration on the built-in monitoring reminder unit, that is, determine the specific monitoring measures or reminders to be taken according to the results of the risk assessment. For example, if the risk assessment shows that an individual is in a highly anxious state, it will remind the individual to take a deep breath or relax. At this time, the monitoring reminder unit will generate a monitoring reminder instruction, and the instruction includes specific health warning information, such as "Rapid breathing, it is recommended to relax" or "Excessive environmental noise, adjust the environment".

[0025] After the monitoring reminder unit generates reminder instructions, these instructions will be uploaded to the monitoring terminal device through the network. The terminal device includes a mobile phone, a tablet, or a dedicated medical monitoring device; the uploaded reminder instructions will be displayed in real time on the terminal device and provided to the user or the caregiver. For example, if an individual has abnormal physiological signals or environmental impacts during the monitoring process, the terminal device will display an alarm and remind the user to take corresponding countermeasures.

[0026] After the guardianship reminder instruction is uploaded to the guardianship terminal device, the terminal will display the health status information and trigger corresponding alarms. The alarms can remind the user through means such as sound, vibration, or visual warnings to ensure that corresponding measures are taken in a timely manner. Exemplarily, if it is detected that an individual is in a high-risk state, the terminal device issues a warning, prompting the individual to take deep breaths or relaxation activities to help reduce the risk. In this way, the entire process not only provides comprehensive health monitoring but also ensures that when a health risk occurs, the individual can obtain timely reminders and take corresponding measures, achieving the effects of health monitoring and early warning.

[0027] Furthermore, uploading the guardianship reminder instruction to the guardianship terminal device for real-time display and alarm, the method of the present application includes: Based on the guardianship terminal device, in comparison with the guardianship reminder instruction, connect to the peripheral intelligent devices. The peripheral intelligent devices include a window lifting control unit and a curtain opening and closing control unit; upload the device status information of the window lifting control unit and the curtain opening and closing control unit; based on the device status information, synchronize the execution progress of the operation instruction issued by the guardianship terminal device.

[0028] Specifically, the guardianship reminder instruction is an instruction generated by the guardianship reminder unit built in the multimedia monitoring device according to the risk assessment result, usually including warning or reminder information about the user's health status. This instruction will be uploaded to the guardianship terminal device to prompt the user or the guardian to take actions; the guardianship terminal device refers to the terminal device that receives and displays the guardianship reminder instruction, usually including smartphones, tablets, medical monitoring devices, etc. These devices are used to display health monitoring data and alarm information in real time and provide real-time feedback to the user or the guardian.

[0029] The peripheral intelligent device refers to the external intelligent hardware connected to the guardianship terminal device, which is used to automatically control the environmental settings in response to the guardianship reminder instruction. The peripheral intelligent device can include a window lifting control unit, a curtain opening and closing control unit, etc., which are used to automatically adjust environmental factors (such as air circulation, light adjustment, etc.); the window lifting control unit is a device that can automatically control the opening or closing of the window, which is used to adjust environmental factors such as indoor air circulation and temperature; the curtain opening and closing control unit is a device that can automatically control the opening and closing of the curtain, which is used to adjust indoor light and privacy protection.

[0030] The device status information refers to the current working status data of the peripheral intelligent device, including whether the device is running normally, the current executed actions, the on / off status of the device, etc. The device status information is used to report the operation progress and status to the guardianship terminal device; the execution progress synchronization refers to comparing and synchronizing the execution status of the operation instruction issued by the guardianship terminal device with the feedback of the peripheral intelligent device to ensure that the device executes tasks according to the predetermined plan and enables the guardianship terminal device to update the execution progress in real time.

[0031] Execution steps: Upload the guardianship reminder instructions to the guardianship terminal device. These instructions are based on the health assessment results and remind the user or the guardian to take measures. For example, if it is detected that an individual has anxiety or excessive stress, the guardianship reminder instructions include content such as suggesting to adjust the environment (such as opening the window for ventilation, adjusting the light, etc.) or performing relaxation exercises; after receiving these instructions, the guardianship terminal device displays the reminder information in real time, usually prompting the user or the guardian through a warning pop-up window, sound or vibration.

[0032] Based on the reminder instructions on the guardianship terminal device, connect to peripheral intelligent devices (such as window lifting control units, curtain opening and closing control units) to automatically adjust environmental factors, indicating that the window is opened, the curtain is closed, or the light is adjusted. The specific instructions depend on the health assessment results. For example, if the health assessment results show that the environmental noise is too high or the air is not circulating, the guardianship reminder instructions prompt the user "Please open the window for ventilation" or "Please adjust the curtain to improve the indoor lighting".

[0033] During the process of the peripheral intelligent device performing the operation, upload the device status information to the guardianship terminal device. The device status information includes the current status of the device (such as whether the window has been opened, whether the curtain has been adjusted), and the progress of the operation. For example, the window lifting control unit will upload the current position of the window (such as "10% opened", "closed"), while the curtain opening and closing control unit will upload the opening and closing degree of the curtain (such as "fully closed").

[0034] Based on the device status information, the guardianship terminal device synchronizes with the execution progress of the peripheral intelligent device. For example, if the window lifting control unit receives an instruction and starts to perform the operation of "opening the window", the guardianship terminal device will display the opening progress of the window in real time. The user can see whether the window has been fully opened. At the same time, the guardianship terminal device will update the operation progress in real time to ensure that the operation instruction is successfully completed and ensure that the user or the guardian can see the execution status of the task. For example, after the window and curtain control units complete the task, the guardianship terminal device will display "The environmental adjustment is completed, and the health guardianship task has been executed".

[0035] With the intelligent upload and real-time execution of the guardianship reminder instructions, and the operation progress synchronization through the feedback information of the peripheral intelligent devices, the guardianship terminal device can display the guardianship status and the device execution progress in real time, ensuring that each operation can obtain timely and accurate feedback. This not only enhances the interactivity and intelligence of the system, but also ensures that individuals can quickly and effectively adjust the environment when facing health risks to help restore the healthy state.

[0036] Furthermore, based on the device status information, perform the execution progress synchronization of the operation instructions issued by the guardianship terminal device. The method of the present application includes: Parse the operation instructions sent by the monitoring terminal device, decompose them into multiple subtasks, and each subtask corresponds to an execution action of a device; send the multiple subtasks to the corresponding peripheral intelligent devices respectively; after the peripheral intelligent devices receive them, return an operation confirmation signal, and the operation confirmation signal is used to indicate the start and end of each subtask.

[0037] Specifically, the monitoring terminal device refers to a device that receives and executes monitoring reminder instructions, such as a smartphone, a tablet computer, etc. After this device receives the user's health risk assessment, it will send operation instructions to peripheral intelligent devices (such as a curtain control unit, a window lifting control unit, etc.); the operation instruction is a specific execution command sent by the monitoring terminal device, usually to change the user's environment (such as adjusting lighting, air circulation, etc.) or execute health-related tasks (such as reminding to rest, performing relaxation exercises, etc.), and the operation instruction usually mainly controls other peripheral devices; the subtask is the decomposition result of an operation instruction, which means that a large task (such as "adjusting environmental lighting") is decomposed into multiple specific small tasks, and each small task corresponds to an execution action of a device.

[0038] The peripheral intelligent device refers to a hardware device that is connected to the monitoring terminal device and responds to operation instructions, such as a window lifting control unit, a curtain control unit, an air conditioner control unit, etc. These devices adjust the environmental state according to the instructions; the operation confirmation signal refers to the signal returned from the peripheral intelligent device to the monitoring terminal device, which is used to confirm whether the device has started to execute a certain task and indicate whether the task is completed. For example, after the curtain control unit receives the window opening instruction, it will send a confirmation signal to indicate whether the curtain has been fully opened; the execution progress synchronization means that the monitoring terminal device updates the execution status in real time according to the operation confirmation signal received from the peripheral intelligent device, which can help ensure that the progress of each task can be synchronously displayed, facilitating the user or the caregiver to understand the execution situation of the current device.

[0039] Execution steps: When the monitoring terminal device receives an operation instruction from a health assessment or monitoring system, it needs to parse the instruction; the parsing process can be completed through natural language processing or a preset rule system to ensure that each operation instruction is correctly split and mapped to specific device actions; decompose the large task into multiple subtasks, and each subtask corresponds to a specific device action. For example: Subtask 1: Adjust the curtain (perform the opening and closing action of the curtain); Subtask 2: Adjust the lighting (perform the adjustment of the light brightness or color temperature). Each subtask has clear device and action requirements, and through decomposition, it can be ensured that each device executes the task correctly according to the instruction.

[0040] The monitoring terminal device sends the parsed multiple subtasks to the corresponding peripheral intelligent devices respectively. For example, subtask 1 (adjusting the curtain) will be sent to the curtain control unit, and subtask 2 (adjusting the light) will be sent to the light control unit. The communication can be carried out through smart home protocols such as Wi-Fi, Bluetooth, ZigBee, etc. Each device will receive its own task and be ready to execute it.

[0041] After receiving the instruction, the peripheral intelligent device starts to execute the subtask. For example, the curtain control unit starts to move the curtain, and the light control unit starts to adjust the brightness. During the execution process, the device will return an operation confirmation signal to the monitoring terminal device in real time, indicating the progress of the task. For example, the curtain control unit returns the signal: "The curtain has been opened 50%", and the light control unit returns: "The brightness has been adjusted to medium level". The return of the operation confirmation signal not only indicates whether the task has started to be executed, but also informs the monitoring terminal device of the execution status of the task (completed or in progress).

[0042] Based on the operation confirmation signal returned from the peripheral intelligent device, the monitoring terminal device will synchronize the execution progress of the task in real time. For example, if the curtain control unit returns that the curtain has been opened 30%, the monitoring terminal device will update the display status: "Curtain opening progress: 30%", and display the next execution plan. The synchronization mechanism ensures that the monitoring terminal device can provide timely feedback during the task execution process and display the progress in real time through the interface update, helping users or guardians to understand the device execution situation.

[0043] Exemplarily, Scenario 1: When the user is performing relaxation training, it is recommended to adjust the ambient light to create a relaxing atmosphere. The monitoring terminal device issues an operation instruction: "Adjust the curtain and the light". This command is parsed into two subtasks: subtask 1 (adjust the curtain), subtask 2 (adjust the light brightness). Subtask 1 is sent to the curtain control unit to request opening the curtain, and the curtain control unit starts to execute and returns the signal: "The curtain has been opened 30%". Subtask 2 is sent to the light control unit to request adjusting the light brightness to 60%, and the light control unit returns the signal: "The brightness has been adjusted to 60%".

[0044] Scenario 2: After the indoor air quality is detected, the monitoring terminal device recommends opening the window and starting the air purifier, and issues the instructions: "Open the window" and "Start the air purifier". Subtask 1 (open the window) is sent to the window lifting control unit, and the returned signal is: "The window has been opened 20%". Subtask 2 (start the air purifier) is sent to the air purifier control unit, and the returned signal is: "The air purifier has been started, filtration mode: high efficiency".

[0045] Through the above steps, the monitoring terminal device can parse and disassemble the operation instructions into multiple specific subtasks, ensuring that each subtask corresponds to a device action and is executed. This not only improves the flexibility and accuracy of task execution but also ensures that the execution progress of each task is synchronized in real time, thus realizing the efficient cooperation between devices and the intelligentization of health management.

[0046] Furthermore, a risk matter assessment network layer is configured. The risk matter assessment network layer conducts quantitative assessment based on risk assessment criteria. The method of this application includes: Integrate an emergency response mechanism to determine whether to trigger an emergency response program; at the same time, record the emergency response events and extract effective features; based on the effective features and in combination with the risk assessment criteria, establish a feedback loop.

[0047] Specifically, the risk matter assessment network layer is used to evaluate and monitor potential risk matters in real time. The risk assessment conducts quantitative analysis on the user's health status, environmental conditions, or device operation conditions based on preset criteria. This network layer can combine multiple data sources (such as user physical sign data, environmental data, etc.) to determine the risk level, thereby guiding subsequent processing steps; the emergency response mechanism can quickly take predetermined emergency measures, such as notifying the user, activating the alarm, starting the device, etc., to minimize the impact of the risk on the user.

[0048] An emergency response event refers to a specific event or state that occurs when the emergency response mechanism is activated. For example, when an abnormal heart rate of a certain user is detected and the emergency response mechanism is triggered, the event recorded at this time is an "emergency response event". These events will be recorded in detail for post-event analysis and feedback; effective features refer to the important information related to risks extracted by screening and analyzing data in the emergency response event. For example, a certain waveform change in the electrocardiogram data, a drastic temperature fluctuation in the environmental data, etc. are effective features of the emergency response. These features will assist in subsequent risk assessments; the feedback loop is to establish a feedback loop based on the effective features extracted from the emergency response event through decision-making responses according to the risk assessment criteria. The purpose of the feedback loop is to adjust subsequent decisions based on the previous risk assessment results, so as to optimize the ability to handle risks in continuous practice.

[0049] Execution steps: Real-time monitor the user's status according to the set risk assessment criteria (such as electrocardiogram data, body temperature, environmental humidity, etc.). When an abnormal situation is found and the abnormality meets the trigger conditions for the emergency response (such as too high heart rate or too high environmental temperature), the emergency response mechanism will be activated. The core purpose of the emergency response mechanism is to take necessary measures as soon as possible when there are major risks to the user's health or safety, such as automatically sending an alarm signal or notifying relevant personnel for intervention.

[0050] Once the emergency response mechanism is triggered, detailed information related to the event is recorded. For example, the time when the event occurred, specific abnormal data (such as the user's electrocardiogram waveform, abnormal body temperature value, etc.), and changes in the external environment (such as air quality, temperature and humidity, etc.); the data related to the emergency response event will be recorded for subsequent analysis and processing. For example, when recording an emergency response event of a too-fast heart rate, the specific heart rate data, electrocardiogram waveform, and environmental data (such as noise, light, etc.) that triggered the event will be saved.

[0051] After an emergency response event occurs, the relevant data is analyzed to screen out the information that is most critical for judging risks and optimizing emergency measures. For example, abnormal fluctuations in heart rate, sharp changes in body temperature, abnormal noise in the environment, etc. can all be regarded as effective features. For example, if during a certain emergency, the user's body temperature continuously rises above a certain threshold, then the change in body temperature is an effective feature. In addition, changes in the temperature or air quality in the environment affect the health status, so they are also regarded as effective features.

[0052] The extracted effective features will be transmitted to the risk assessment network layer, which re-evaluates the current risk status based on these features. Through the analysis of these data, the current assessment model is adjusted or a new emergency operation is triggered. For example, if a certain user has a too-high body temperature and is accompanied by a too-fast heart rate, based on these effective features combined with the preset risk assessment criteria, the assessment result is automatically updated to determine that the user is in a high-risk state; the response strategy is continuously optimized through a feedback loop. For example, if a previous assessment fails to successfully identify a certain risk (such as the combined effect of a too-fast heart rate and a too-high body temperature), this risk pattern is added to the subsequent risk assessment criteria to enhance the accuracy and response ability; the feedback loop adjusts its risk assessment model according to the continuously updated data, thereby improving the accuracy of risk prediction.

[0053] By integrating the emergency response mechanism, recording emergency response events, extracting effective features, and establishing a feedback loop in combination with risk assessment criteria, it is possible to continuously monitor the user's health status and optimize the response strategy. When the user's health shows abnormalities, a rapid response is made, and the assessment criteria and emergency strategies are adjusted according to real-time feedback, so as to provide more accurate and efficient health monitoring services.

[0054] Furthermore, by using the breathing sound parameters in the audio monitoring data and the facial expression parameters in the image monitoring data, individual modal parameters are formulated. The method of the present application includes: Based on the individual modal parameters, a monitoring database with user markings is established; through the monitoring database, it is docked with the health database to set up a targeted monitoring service; through the targeted monitoring service, group monitoring reminders are carried out.

[0055] Specifically, individual modal parameters refer to the quantitative indicators of the physiological and emotional states of individual users obtained through the breathing sound parameters in audio monitoring data and the facial expression parameters in image monitoring data. These parameters reflect the physical condition, emotional response, etc. of an individual at a specific moment and can help analyze the health and psychological state of the user; the monitoring database is a database used to store individual modal data of users, containing the monitoring data of all users at specific times and in specific environments, such as breathing sounds, facial expressions, physical sign data, etc. Each data item will be associated with a specific user tag, so as to distinguish and track the health and state changes of different users.

[0056] The health database is a data storage unit containing a large amount of health information, usually related to data such as individual health history, disease records, diagnostic results, etc. The role of the health database is to provide background data for individual health management to help with analysis and decision-making; the targeted guardianship service refers to providing personalized and customized health guardianship services based on the health data and status of specific users. These services can be health reminders, guardianship reports, risk warnings, etc. The goal is to carry out health interventions according to the specific needs of users; the group guardianship reminder refers to the centralized monitoring and reminder service for a group or category of users. Group guardianship not only focuses on individual health but also on the health risks of the overall user group, and conducts timely health warnings and interventions.

[0057] Execution steps: Obtain the physiological and emotional data of the user through audio monitoring and image monitoring devices. For example, obtain individual modal parameters by capturing the user's breathing sound (such as whether it is stable, whether there is wheezing, frequency, etc.) through an audio device and capturing the user's facial expression (such as whether they are anxious, in pain, or calm) through an image monitoring device; based on these data, establish a monitoring database, and the relevant data of each user will be marked as the health record of a specific user. For example, if the breathing sound of user A is stable and the facial expression shows relaxation, then this information will be recorded in the database entry of this user.

[0058] After the individual modal parameters of the user and other health data (such as body temperature, blood pressure, etc.) are stored in the monitoring database, connect these data with the health database. The health database usually contains a wide range of health information. Based on the individual modal parameters of the user, compare with the existing health patterns in the health database, so as to provide personalized health advice and warnings for this user. For example, if the facial expression of user A shows anxiety, and the health database indicates that anxiety is related to a certain disease risk, set personalized targeted guardianship services based on these data. The targeted guardianship services can include regular health reminders, risk warnings, appointment for health checks, etc.

[0059] When health risks are detected in individuals within certain groups, or when there are changes in the group's health status, group guardianship reminders are initiated. Such reminders are usually targeted at multiple users and can focus on a specific group or scenario. For example, providing guardianship reminder services for a certain type of high-risk population; Group guardianship reminder services can be issued in various forms, such as reminding caregivers or relevant departments, pushing health warning notifications, or providing users with risk warning information about group health. For example, by monitoring the physiological data of multiple elderly groups to detect whether there are abnormal patterns, such as the risk of multiple people having low blood pressure, so as to carry out group-level early warnings and interventions.

[0060] By establishing a monitoring database based on individual modal parameters and docking it with a health database, personalized and customized targeted guardianship services can be provided for each user. These services help users timely understand and respond to potential health risks. In addition, when group health risks are discovered, guardianship reminders and interventions at the group level are provided. This method not only improves the accuracy of individual health monitoring but also enhances the overall management ability of group health.

[0061] Furthermore, based on the effective features and combined with the risk assessment criteria, a feedback loop is established. The method of the present application includes: ; where is the updated risk assessment value, is the current risk assessment value, n is the number of effective features, is the weight of the i-th effective feature, is the updated i-th effective feature, is an adjustment factor between 0 and 1, t is the current time point, is the timestamp corresponding to the update of the i-th effective feature, is the time decay factor.

[0062] Specifically, effective features refer to the feature data with practical significance after being screened and processed during the risk assessment process. For example, environmental factors (such as noise, temperature), etc. These feature data are considered to have a direct impact on risk assessment. Effective features are derived from multi-modal data (audio, images, etc.) and are helpful for judging the health status or risk level of an individual; Risk assessment represents a comprehensive assessment value of the current health status of a specific user or group, usually a numerical index used to reflect the level of health risk. The higher the risk assessment value, the greater the health risk.

[0063] Adjustment factor Used to adjust the parameters in the risk assessment model, usually a value between 0 and 1, indicating the degree of dynamic adjustment of the model parameters. The adjustment factor is affected by external changes (such as time, environment, health status changes) or internal factors (such as changes in the user's immediate health data); Time decay factor When processing health data, the time factor causes the weights of some data to gradually decrease. The time decay factor is used to control these changes, that is, during the passage of time, the importance of specific data gradually decreases over time. Usually, the time decay factor is used to strengthen the impact of real-time data and weaken the impact of historical data.

[0064] Execution steps: Extract effective features through multimodal data collection (such as audio, images, and sensor data). These features can include an individual's physiological parameters (such as heart rate, respiratory rate), environmental factors (such as temperature, noise), etc. For example, assume the respiratory rate in audio data, the anxiety index in facial expression analysis, etc. Combine the effective features and calculate a preliminary risk assessment value according to the preset risk assessment criteria (such as medical models, statistical analysis, etc.). This value represents the current health risk level. For example, if a user has a high respiratory rate and a facial expression showing anxiety, the risk assessment value is high, indicating that the user has a high health risk.

[0065] When conducting risk assessment, the adjustment factor and the time decay factor will dynamically adjust the risk assessment result according to the changes in the user's immediate health data. For example, if a user maintains relatively stable health data (such as steady breathing sounds and facial expressions) for a long time, gradually reduce the weight of this data, so that the impact of this feature on the risk assessment gradually decreases; The application of the time decay factor means that when the user's health status changes rapidly (for example, the health condition suddenly deteriorates), the weight of recent data will increase, and earlier data will be weakened over time. This decay factor is usually applied in time series analysis to simulate the actual situation of health data changing over time.

[0066] The core of the feedback loop is to dynamically adjust the risk assessment value by continuously updating the user's health data and corresponding effective features. After each data update, calculate the risk assessment value again according to the new data and features, and correct the output of the model through the adjustment factor. For example, assume that the weight of a certain effective feature (such as respiratory rate) is high, and this feature changes within a certain time period. Use this change to update the risk assessment value. If the respiratory rate increases and the facial expression shows uneasiness, automatically adjust the risk assessment value according to the set assessment criteria and issue a corresponding health warning.

[0067] After each feedback loop, the updated risk assessment value is used to generate health warning messages, which are output to the monitoring terminal or subsequent health intervention measures. This assessment value is the result of multiple rounds of updates and adjustments, which can accurately reflect the user's current health risks and provide data support for subsequent monitoring measures.

[0068] Through the feedback loop of continuously updating effective features and adjustment factors to adapt to the changes in the user's health status, each time through the adjustment of the newly collected data and the time decay factor, the risk assessment model can adaptively optimize its output to ensure that the health intervention measures are more timely and accurate. Through this feedback loop, the intelligent level of health monitoring is improved, and more personalized and timely health management services are provided.

[0069] Furthermore, the method of the present application further includes: Tracking the individual modal parameters and environmental modal parameters to obtain the change trend of the individual's health status; according to the updated risk assessment value, comparing with the change trend of the individual's health status, adaptively adjusting the response threshold of the risk matter assessment network layer; introducing the individual's health needs to verify the risk assessment result of the risk matter assessment network layer, and after passing the verification, performing feature registration on the monitoring reminder unit again with the adaptive adjustment result.

[0070] Specifically, the individual modal parameters refer to the characteristic data reflecting the individual's health status collected through various monitoring methods such as audio and images. These parameters can include physiological data (such as heart rate, respiratory rate), facial expressions, postures, etc., which can reflect the user's immediate health status. Through the individual modal parameters, the user's physical health or emotional changes can be monitored in real time; the environmental modal parameters refer to the characteristic data reflecting the environmental status through data collection indicators such as audio and images, such as environmental temperature, humidity, noise, light, etc. These environmental parameters are closely related to the individual's health status, especially in different scenarios such as sleep and work, the environmental modal parameters have a significant impact on health.

[0071] The change trend of the health status refers to the change pattern and law of the health data of an individual over a period of time. By tracking the changes in the individual's health data (such as body temperature, heart rate, respiratory rate, etc.), it can be evaluated whether the health status is tending to improve, stable or deteriorate, which is helpful for predicting and timely intervening in the individual's health; the risk assessment value is a comprehensive index calculated based on the individual modal parameters and environmental modal parameters, which is used to reflect the individual's health risk level. The risk assessment value is usually obtained through a specific assessment model and expressed in numerical form. For example, factors such as too high heart rate and rapid breathing result in a higher risk assessment value, indicating that preventive or intervention measures need to be taken.

[0072] In a risk assessment network, the response threshold is the critical value that triggers a health intervention or an alarm. If the health assessment value exceeds this threshold, corresponding response measures will be triggered (such as sending health reminders, automatically adjusting devices, etc.). The response threshold will be automatically adjusted according to the changes in the individual's health status; individual health needs refer to the specific health needs of each user or patient, which are formulated based on factors such as their personal health records, lifestyle habits, and disease history; the monitoring reminder unit is a component integrated in the monitoring device, which is used to send reminders or alarms based on health data and risk assessment values. Its main function is to timely remind users or medical staff to take corresponding intervention measures according to the monitored health parameters and risk assessment results.

[0073] Execution steps: Real-time collect individual and environmental data from various monitoring devices. Individual data includes physiological data (such as respiratory rate, heart rate, body temperature, etc.) and emotional state data (such as facial expressions, postures, etc.). Environmental data includes noise, light, temperature and humidity, etc.; By comparing this data with the health database, generate the trend of changes in the individual's health status. For example, if an individual's respiratory rate gradually increases over a certain period of time, or the body temperature changes under different environmental conditions (such as too high noise), these can all reflect the dynamic changes in the health status.

[0074] Through the combination of multi-modal data (individual and environmental modal parameters) and risk assessment criteria, calculate the updated risk assessment value, so as to accurately reflect the user's health risk. For example, if the respiratory rate increases and the facial expression shows anxiety, a higher risk assessment value will be given; As the health status changes, the response threshold will be adjusted according to the updated risk assessment value. For example, when the health risk increases, the response threshold will be lowered, which means that the alarm or intervention measures will be triggered earlier. On the contrary, if the health status is stable, the response threshold will be increased to avoid over-intervention.

[0075] In the risk assessment process, not only rely on real-time monitoring data, but also dynamically adjust in combination with individual health needs. For example, if a certain user has a known history of hypertension and is particularly sensitive to changes in heart rate, give priority to monitoring the situation where the heart rate fluctuates greatly; After assessing the risk, verify the risk assessment result according to the individual's health needs. For example, if the risk assessment value of this user is assessed to be high, check his health record to judge whether this health condition meets his health needs. If it meets, it means that the risk assessment result is valid, otherwise the assessment model needs to be adjusted again.

[0076] Once the verified risk assessment results match the health needs, adjust the parameters of the guardianship reminder unit according to the updated assessment results to ensure that it can accurately respond according to the health risks. For example, the reminder frequency, reminder content, etc. of the guardianship reminder unit are dynamically adjusted according to the changes in health risks; after the adaptive adjustment, the guardianship reminder unit will more effectively reflect the changes in the individual's health status. For example, if the user's health risk is high, increase the reminder frequency and urgency; on the contrary, choose to reduce the reminder frequency to avoid excessive interference.

[0077] By tracking the multi-modal data of the individual and the environment, the health status of the individual is evaluated in real time, and dynamic adjustment is carried out in combination with the user's health needs. By adaptively adjusting the response threshold and verifying the risk assessment results, the user's health risks are accurately judged and the most appropriate intervention measures are taken. At the same time, the guardianship reminder unit ensures the accuracy of health intervention through feature registration, avoiding excessive or inappropriate intervention and improving the intelligent level of health management.

[0078] In summary, the beneficial effects of the embodiments of the present application are as follows: 1. Through the recursive integration of different modal data by the multi-modal aggregation model, more accurate health risk assessment can be achieved. Combining the individual's health status with the changes in the environment, potential health risks can be warned, so as to detect and avoid the occurrence of health risks at an early stage.

[0079] 2. By introducing the guardianship reminder unit and the emergency response mechanism, when potential health risks or environmental problems are detected, through the linkage between the peripheral intelligent devices (such as window lifting control, curtain opening and closing control, etc.) and the health monitoring unit, automated operation instructions can be executed, and task synchronization and confirmation during the execution process can be ensured.

[0080] 3. Based on the docking of individual health data and the group health database, group guardianship services can be implemented. By monitoring the individual modal parameters and dynamically adjusting the evaluation criteria, personalized health services can be provided for different users, and the health monitoring standards and thresholds can be adaptively adjusted to ensure the accuracy and pertinence of the guardianship effect.

[0081] 4. By tracking individual modal parameters and environmental modal parameters, the changing trend of the individual's health status is obtained; according to the updated risk assessment value, the response threshold of the risk event assessment network layer is adaptively adjusted in contrast to the changing trend of the individual's health status; the individual's health needs are introduced to verify the risk assessment result of the risk event assessment network layer. After the verification passes, the feature registration of the guardianship reminder unit is performed again with the adaptively adjusted result. By tracking the multi-modal data of the individual and the environment, the health status of the individual is evaluated in real time, and dynamic adjustment is performed in combination with the user's health needs. By adaptively adjusting the response threshold and verifying the risk assessment result, the user's health risk is accurately judged and the most appropriate intervention measures are taken. At the same time, the guardianship reminder unit ensures the accuracy of health intervention through feature registration, avoids excessive or inappropriate intervention, and improves the intelligent level of health management.

[0082] Embodiment 2 Based on the same inventive concept as a multimedia data acquisition method in the foregoing embodiment, as Figure 2 shown, the embodiment of the present application provides a multimedia data acquisition system, wherein the system includes: A data monitoring module M100, configured to connect to a multimedia monitoring device to obtain audio monitoring data and image monitoring data, where the audio monitoring data includes breathing sound parameters and environmental noise parameters, and the image monitoring data includes facial expression parameters and environmental light parameters; A first parameter determination module M200, configured to determine individual modal parameters through the breathing sound parameters in the audio monitoring data and the facial expression parameters in the image monitoring data; A second parameter determination module M300, configured to determine environmental modal parameters through the environmental noise parameters in the audio monitoring data and the environmental light parameters in the image monitoring data; A recursive integration module M400, configured to perform fitting of a multi-modal aggregation model based on the individual modal parameters and the environmental modal parameters, where the multi-modal aggregation model is used to recursively integrate different modal data; A quantization evaluation module M500, configured to configure a risk event assessment network layer, where the risk event assessment network layer performs quantization evaluation according to a risk assessment standard; A feature registration module M600, configured to perform feature registration on a guardianship reminder unit built in the multimedia monitoring device based on the multi-modal aggregation model and the risk event assessment network layer to obtain a guardianship reminder instruction; An instruction uploading module M700, configured to upload the guardianship reminder instruction to a guardianship terminal device for real-time display and alarm.

[0083] Further, the instruction uploading module M700 is configured to execute the following method: Based on the monitoring terminal device, connect to the peripheral intelligent devices according to the monitoring reminder instructions. The peripheral intelligent devices include a window lifting control unit and a curtain opening / closing control unit. Upload the device status information of the window lifting control unit and the curtain opening / closing control unit. Based on the device status information, synchronize the execution progress of the operation instructions issued by the monitoring terminal device.

[0084] Furthermore, the instruction uploading module M700 is also used to execute the following method: Analyze the operation instructions issued by the monitoring terminal device, and decompose them into multiple subtasks. Each subtask corresponds to a device execution action. Send the multiple subtasks to the corresponding peripheral intelligent devices respectively. After the peripheral intelligent devices receive them, return an operation confirmation signal, which is used to indicate the start and end of each subtask.

[0085] Furthermore, the quantitative evaluation module M500 is used to execute the following method: Integrate an emergency response mechanism to determine whether to trigger an emergency response program. Meanwhile, record the emergency response events and extract effective features. Based on the effective features, establish a feedback loop in combination with the risk assessment criteria.

[0086] Furthermore, the first parameter formulation module M200 is used to execute the following method: Based on the individual modal parameters, establish a monitoring database with user markings. Through the monitoring database, interface with the health database and set up a targeted monitoring service. Through the targeted monitoring service, conduct group monitoring reminders.

[0087] Furthermore, the quantitative evaluation module M500 is also used to execute the following method: ; Wherein, is the updated risk assessment value, is the current risk assessment value, n is the number of effective features, is the weight of the i-th effective feature, is the updated i-th effective feature, is an adjustment factor between 0 and 1, t is the current time point, is the time stamp corresponding to the updated i-th effective feature, is the time decay factor.

[0088] Furthermore, the described multimedia data acquisition system is also used to execute the following method: Track the individual modal parameters and environmental modal parameters to obtain the change trend of the individual's health status; According to the updated risk assessment value, and in contrast to the change trend of the individual's health status, adaptively adjust the response threshold of the risk matter assessment network layer; Introduce the individual's health needs, verify the risk assessment results of the risk matter assessment network layer, and after passing the verification, perform feature registration on the guardianship reminder unit again with the adaptive adjustment result.

[0089] In summary, any step can be stored as computer instructions or programs in an unrestricted computer memory and can be called and recognized by an unrestricted computer processor, without further limitation here.

[0090] Furthermore, the above technical solution only reflects the preferred technical solution of the technical solution of the embodiments of the present application. Some changes that those skilled in the art may make to some parts thereof all reflect the principles of the new type of the embodiments of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application.

Claims

1. A method for multimedia data acquisition, characterized in that, The method includes: Connect a multimedia monitoring device to obtain audio monitoring data and image monitoring data. The audio monitoring data includes breathing sound parameters and environmental noise parameters, and the image monitoring data includes facial expression parameters and environmental light parameters; Formulate individual modal parameters through the breathing sound parameters in the audio monitoring data and the facial expression parameters in the image monitoring data; Formulate environmental modal parameters through the environmental noise parameters in the audio monitoring data and the environmental light parameters in the image monitoring data; Based on the individual modal parameters and the environmental modal parameters, perform fitting of a multi-modal aggregation model, which is used to recursively integrate different modal data; Configure a risk event assessment network layer, which is quantitatively evaluated according to risk assessment criteria; Based on the multi-modal aggregation model and the risk event assessment network layer, perform feature registration on the built-in guardianship reminder unit of the multimedia monitoring device to obtain a guardianship reminder instruction; Upload the guardianship reminder instruction to a guardianship terminal device for real-time display and alarm.

2. The multimedia data acquisition method according to claim 1, wherein Upload the guardianship reminder instruction to a guardianship terminal device for real-time display and alarm. The method includes: Based on the guardianship terminal device, connect to peripheral intelligent devices according to the guardianship reminder instruction. The peripheral intelligent devices include a window lifting control unit and a curtain opening / closing control unit; Upload the device status information of the window lifting control unit and the curtain opening / closing control unit; Based on the device status information, synchronize the execution progress of the operation instructions issued by the guardianship terminal device.

3. The multimedia data acquisition method according to claim 2, wherein Based on the device status information, synchronize the execution progress of the operation instructions issued by the guardianship terminal device. The method includes: Analyze the operation instructions issued by the guardianship terminal device and decompose them into multiple subtasks, and each subtask corresponds to a device execution action; Send the multiple subtasks to the corresponding peripheral intelligent devices respectively; After the peripheral intelligent devices receive them, return an operation confirmation signal, which is used to indicate the start and end of each subtask.

4. The multimedia data acquisition method according to claim 3, characterized in that, Configure a risk event assessment network layer, which is quantitatively evaluated according to risk assessment criteria. The method includes: Integrate an emergency response mechanism to determine whether to trigger an emergency response program; At the same time, record emergency response events and extract effective features; Based on the effective features and in combination with the risk assessment criteria, establish a feedback loop.

5. The multimedia data acquisition method according to claim 4, wherein Formulate individual modal parameters through the breathing sound parameters in the audio monitoring data and the facial expression parameters in the image monitoring data. The method includes: Based on the individual modal parameters, establish a monitoring database with user marks; Through the monitoring database, dock with a health database to set up a targeted guardianship service; Through the targeted guardianship service, perform group guardianship reminders.

6. The multimedia data acquisition method according to claim 5, wherein Based on the effective features and in combination with the risk assessment criteria, establish a feedback loop as follows: ； Among them, is the updated risk assessment value, is the current risk assessment value, n is the number of effective features, is the weight of the i-th effective feature, is the updated i-th effective feature, is an adjustment factor between 0 and 1, t is the current time point, is the timestamp corresponding to the updated i-th effective feature, is the time decay factor.

7. The multimedia data acquisition method according to claim 6, wherein The method further includes: Track the individual modal parameters and environmental modal parameters to obtain the change trend of the individual's health status; According to the updated risk assessment value, adaptively adjust the response threshold of the risk item assessment network layer in contrast to the change trend of the individual's health status; Introduce the individual's health needs, verify the risk assessment results of the risk item assessment network layer, and after passing the verification, perform feature registration on the guardianship reminder unit again with the adaptive adjustment result.

8. A multimedia data acquisition system, characterized in that, For implementing a multimedia data acquisition method according to any one of claims 1-7, the system includes: A data monitoring module, configured to connect to a multimedia monitoring device and obtain audio monitoring data and image monitoring data. The audio monitoring data includes breathing sound parameters and environmental noise parameters, and the image monitoring data includes facial expression parameters and environmental light parameters; A first parameter determination module, configured to determine individual modality parameters through the breathing sound parameters in the audio monitoring data and the facial expression parameters in the image monitoring data; A second parameter determination module, configured to determine environmental modality parameters through the environmental noise parameters in the audio monitoring data and the environmental light parameters in the image monitoring data; A recursive integration module, configured to perform fitting of a multi-modal aggregation model based on the individual modality parameters and the environmental modality parameters. The multi-modal aggregation model is used to recursively integrate different modality data; A quantization evaluation module, configured to configure a risk item assessment network layer, and the risk item assessment network layer performs quantization evaluation according to a risk assessment standard; A feature registration module, configured to perform feature registration on a guardianship reminder unit built in the multimedia monitoring device based on the multi-modal aggregation model and the risk item assessment network layer to obtain a guardianship reminder instruction; An instruction upload module, configured to upload the guardianship reminder instruction to a guardianship terminal device for real-time display and alarm.

Images