End side large model-based multi-dimensional sensing system, device and method

Abstract

The invention discloses a multi-dimensional sensing system, device and method based on an end-side large model, and the method comprises the steps: obtaining multi-modal sensor data in real time, including environment data, audio data and visual motion data; based on the signal variation, the historical standard deviation and the preset environment weight of each sensor, an environment anomaly index is calculated in combination with the time period factor; calculating a user intention confidence coefficient based on a cross-correlation value of the audio data and the visual motion data, a visual motion information entropy and a personnel existence probability; and when the environment anomaly index is greater than a preset safety threshold value or the user intention confidence coefficient is greater than a preset intention threshold value, triggering an end-side large model to analyze multi-modal sensor data and generate a control instruction. The invention relates to the technical field of multi-dimensional perception, and solves the technical problems of overhigh equipment energy consumption and potential safety hazard early warning lag caused by large-model real-time reasoning in a smart home scene.

Description

Technical Field

[0001] The present invention belongs to the field of multidimensional perception and involves big data processing technology, specifically a multidimensional perception system, device and method based on a large end-side model. Background Art

[0002] Traditional smart home systems' environmental perception capabilities rely heavily on explicit user commands or simple rules-based control logic, lacking the ability to proactively understand the home environment and anticipate potential hazards. For example, these systems typically only control devices in response to user voice commands or manual operations, failing to proactively identify potential risks through multi-dimensional sensory data. This makes it difficult to detect and intervene in safety hazards in a timely manner.

[0003] In recent years, with the introduction of large-scale model technology, some systems have attempted to enhance intelligent perception through real-time environmental data analysis. However, devices equipped with large models must continuously run complex inference tasks to maintain real-time response, resulting in long-term high load on the end-user computing units. Especially when processing multimodal data, the parallel computing requirements of large models conflict with the limited end-user hardware resources. This not only significantly increases device energy consumption but also can cause performance throttling due to chip overheating, further weakening system reliability. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a multidimensional perception system, device and method based on a large end-side model to solve the technical problems of excessive energy consumption of equipment, low resource utilization and delayed safety hazard warning caused by real-time reasoning of large models in smart home scenarios.

[0005] To achieve the above objectives, a first aspect of the present invention provides a multi-dimensional perception method based on a large device-side model, comprising:

[0006] Real-time acquisition of multimodal sensor data, including environmental data, audio data, and visual motion data;

[0007] The environmental anomaly index is calculated based on the signal change, historical standard deviation and preset environmental weight of each sensor, combined with the time period factor;

[0008] The confidence level of user intention is calculated based on the cross-correlation value between audio data and visual motion data, visual motion information entropy, and the probability of person presence.

[0009] When the environmental anomaly index is greater than a preset safety threshold, or the user intention confidence is greater than a preset intention threshold, the end-side big model is triggered to analyze the multimodal sensor data and generate control instructions; wherein, the end-side big model includes a first end-side big model and a second end-side big model.

[0010] Furthermore, the multimodal sensor data includes:

[0011] Environmental data composed of temperature and humidity sensors, smoke sensors, gas sensors, and light intensity sensors;

[0012] Audio data collected by a microphone array;

[0013] Visual motion data calculated by thermal infrared sensors or cameras;

[0014] Status signal data obtained by door and window magnetic sensors or pressure sensors.

[0015] Furthermore, the calculation formula of the environmental anomaly index is:

[0016]

[0017] Where, ΔS i (t) represents the data change of sensor i in the time window Δt, represents the standard deviation of sensor i in the same historical period, W env，i (t) represents the preset environmental weight of sensor i at time t, k1 and k2 represent the preset time period factors, and k1>k2>0, δ night (t) represents the night mode indicator function, n represents the number of sensors, and t represents time.

[0018] The Environmental Anomaly Index (EAI) is the system's core indicator for proactive environmental protection. By monitoring multimodal sensor signal changes in real time and combining historical data standard deviations with dynamic weight distribution, it determines whether the current environment requires initiating a large end-to-end model for further analysis.

[0019] Furthermore, the night mode indication function is determined based on the night mode time range, including:

[0020] Collect user historical behavior data and generate activity frequency matrix;

[0021] Perform cluster analysis on the activity frequency matrix based on the clustering algorithm to obtain the user's work and rest cycle;

[0022] Calculate the night mode time range [T night-start ,T night-end ], the calculation formula is: Among them, T night-start Indicates the start time of night mode, T night-end Indicates the end time of night mode, μ sleep-time and σ sleep-time They represent the mean and standard deviation of the sleep time determined by the user's sleep cycle, μ wakeup-timeand σ wakeup-time They represent the mean and standard deviation of the wake-up time determined by the user's sleep and rest cycle, and α represents the adjustment factor.

[0023] When the current time is within the night mode time range, δ night (t) = 1; otherwise δ night (t)=0.

[0024] Furthermore, the calculation formula of the preset environment weight is: Among them, W base，i represents the preset basic weight of the i-th sensor, and γ represents the night mode adjustment coefficient.

[0025] In terms of abnormal event warning, when sensor data suddenly deviates from historical patterns, EAI quickly amplifies abnormal signals through weighted calculations, triggers hidden danger prediction models, and identifies high-risk events such as gas leaks and fire risks in advance; in terms of day and night sensitivity adaptation, a time period factor is introduced to enhance detection sensitivity at night when user activity is less, avoiding missed reports due to environmental interference; in terms of dynamic priority allocation, high-risk sensor signals are responded to first through preset environmental weights to ensure that resources are tilted towards key hidden dangers.

[0026] Furthermore, the calculation process of the user intention confidence includes:

[0027] Get the audio data A of the jth audio channel of the microphone array at time t j (t);

[0028] The optical flow estimation algorithm is used to calculate the pixel displacement of adjacent frame images obtained by the visual sensor to obtain the optical flow displacement vector v(x, y);

[0029] The visual motion data V(t) at time t is calculated according to the optical flow energy formula: Where N represents the total number of pixels of the frame image acquired by the visual sensor, and |||| represents the symbol for modulus calculation;

[0030] Discretize the direction of the optical flow displacement vector into n intervals and count the proportion of moving pixels in each direction p i , the visual motion information entropy at time t is calculated according to the formula: The motion pixel refers to a pixel whose modulus of the optical flow displacement vector exceeds a preset modulus threshold;

[0031] Get the temperature matrix output by the thermal infrared sensor and calculate the local variance of the temperature matrix The probability of personnel presence at time t is calculated according to the formula P human (t), the formula is: Among them, β represents the slope parameter and θ represents the judgment threshold, both of which are determined through experimental calibration;

[0032] The user intention confidence UIC(t) at time t is calculated according to the formula: Where m represents the number of audio channels of the microphone array, CrossCorr(A j (t), V(t)) represents the cross-correlation value between the audio data and the visual motion data calculated using the calculation formula of the Pearson correlation coefficient.

[0033] User intention confidence (UIC) is the core criterion for achieving natural interaction and precise control. It verifies the effectiveness of user commands through multimodal data fusion. Its specific functions are as follows: in multimodal intent verification, it uses the cross-correlation characteristics of audio and visual signals to distinguish real user commands from environmental noise, reducing the probability of false triggering; in scenario-based intent reasoning, it quantifies the complexity of the environment through visual motion information entropy, and dynamically adjusts the response strategy based on the probability of human presence; in adaptive threshold management, it suppresses unnecessary interactions according to different time periods or human presence status, while ensuring reliable responses to key commands.

[0034] Furthermore, the user intention confidence determination rule also includes:

[0035] When the current time is within the night mode time range, the preset intention threshold is reduced to a preset percentage;

[0036] When the probability of the person existing is less than the preset probability threshold, the user intention confidence UIC is directly determined to be 0.

[0037] Furthermore, the generating of the control instruction includes:

[0038] When the environmental anomaly index is greater than a preset safety threshold, the first end-side large model is triggered to predict hidden danger events based on the multimodal sensor data, and generate a control instruction for an alarm notification based on the prediction result of the hidden danger event;

[0039] When the user intention confidence is greater than a preset intention threshold, the second end-side large model is triggered to analyze the user voice data and provide an interactive answer;

[0040] The first large model on the end is built based on an artificial intelligence algorithm, with environmental data and audio data as input and hidden danger prediction types as output, including gas leak and fire risks, to determine whether hidden dangers exist.

[0041] The second end-side large model is built based on an artificial intelligence algorithm, and its input is environmental data and audio data, and its output is an interactive answer, which is used to confirm with the user and generate control instructions for the device.

[0042] A second aspect of the present invention provides a multi-dimensional perception system based on a large device-side model, including:

[0043] Data acquisition module, used to acquire multimodal sensor data in real time, including environmental data, audio data, and visual motion data;

[0044] The edge computing module includes an environmental anomaly analysis unit and a user intention inference unit; wherein the environmental anomaly analysis unit calculates the environmental anomaly index based on the signal change amount, historical standard deviation and preset environmental weight of each sensor, combined with the time period factor; the user intention inference unit calculates the user intention confidence based on the cross-correlation value of audio data and visual motion data, visual motion information entropy and the probability of human presence;

[0045] The model analysis module is used to trigger the first end-side large model to predict hidden danger events and generate alarm instructions when the environmental anomaly index is greater than a preset safety threshold; and to trigger the second end-side large model to parse user voice data and generate control instructions when the user intention confidence is greater than a preset intention threshold; the control instructions include at least one of temperature adjustment instructions, lighting adjustment instructions, and humidity adjustment instructions.

[0046] A third aspect of the present invention provides a multi-dimensional perception device based on a large end-to-end model, comprising:

[0047] A processing unit for:

[0048] Processing multimodal sensor data acquired in real time, the multimodal sensor data including environmental data, audio data, and visual motion data;

[0049] Calculate the environmental anomaly index based on the signal change of each sensor, historical standard deviation, preset environmental weight and time period factor;

[0050] The confidence level of user intention is calculated based on the cross-correlation value between audio data and visual motion data, visual motion information entropy, and the probability of person presence.

[0051] When the environmental anomaly index is greater than a preset safety threshold, the first end-side large model is triggered to predict hidden danger events on the multimodal sensor data and generate an alarm instruction; when the user intention confidence is greater than the preset intention threshold, the second end-side large model is triggered to parse the user voice data and generate a control instruction; the control instruction includes at least one of an alarm notification instruction, a temperature adjustment instruction, a lighting adjustment instruction, and a humidity adjustment instruction.

[0052] Communication unit for:

[0053] Establishing a communication connection with an external multimodal sensor to receive the multimodal sensor data in real time;

[0054] Establish a communication connection with the controlled device, push the control instruction to the target device, and implement the execution of the instruction.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] In the smart home application scenario, the present invention realizes the precise scheduling of real-time analysis of the end-side large model through the threshold trigger mechanism of dual-core indicators and the dynamic resource allocation strategy, thereby effectively improving the battery life and life of the device. Specifically, the present invention determines whether to activate the end-side large model based on the environmental anomaly index EAI and the user intention confidence UIC: only when the sensor data deviates from the historical rules or the user instruction passes the multi-modal verification, the model in-depth analysis is triggered, avoiding the continuous scanning of the daily stable environment and invalid interactive response, and reducing the unnecessary computing load of the device. At the same time, the time period factor and the personnel presence detection mechanism are introduced to automatically increase the weight of high-risk sensors and reduce the analysis frequency of non-critical modules at night, turn off the high-power interactive function in the unmanned scene, and realize the optimized distribution of energy consumption in different time periods. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 A schematic diagram of the framework of the multi-dimensional perception system based on the large device-side model provided by the present invention;

[0059] Figure 2 A schematic diagram of the process of the multi-dimensional perception method based on the large device-side model provided by the present invention;

[0060] Figure 3 A schematic structural diagram of a multi-dimensional perception device based on a large end-side model is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0062] The multi-dimensional perception method based on the large device model provided in the embodiment of the present application can be applied to Figure 1 In the multi-dimensional perception system shown in Figure 1As shown, the system includes: a data acquisition module, an edge computing module and a model analysis module.

[0063] Among them, the data acquisition module is used to obtain multimodal sensor data in real time, including environmental data, audio data and visual motion data.

[0064] The edge computing module includes an environmental anomaly analysis unit and a user intention reasoning unit: the former calculates the environmental anomaly index EAI based on the change in sensor signals, historical standard deviation, preset environmental weights and time period factors; the latter calculates the user intention confidence UIC based on the cross-correlation value of audio and visual motion data, visual motion information entropy and the probability of human presence.

[0065] The model analysis module is used to trigger the first end-side large model to predict hidden danger events from multimodal data and generate alarm instructions when the EAI is greater than the preset safety threshold; and to trigger the second end-side large model to parse user voice data and generate control instructions when the UIC is greater than the preset intention threshold.

[0066] It should be noted that the data acquisition module, edge computing module and model analysis module of the multi-dimensional perception system of this embodiment are communicatively connected.

[0067] In order to solve the technical problems in the prior art of excessive energy consumption, waste of hardware resources and shortened device life caused by continuous real-time inference of large models on the end side, the embodiment of the present application provides a multi-dimensional perception method based on large models on the end side. This method uses a non-continuous trigger mechanism to activate the large model on the end side only when the environment is abnormal or the user's intention is clear, avoiding its indiscriminate real-time operation, thereby achieving a dynamic balance between performance requirements and energy consumption control, extending device life and reducing hardware loss. Figure 2 As shown, the embodiment of the present application provides a multi-dimensional perception method based on a large device-side model, including:

[0068] S1. Acquire multimodal sensor data in real time, including environmental data, audio data, and visual motion data.

[0069] Among them, multimodal sensor data is collected by multiple types of sensors and is used to comprehensively perceive the environment and user status.

[0070] In some implementations, the multimodal sensor may include a temperature and humidity sensor for collecting environmental data, a microphone array for collecting audio data, and a thermal infrared sensor or camera for calculating visual motion data.

[0071] It should be pointed out that the process of obtaining data is continuous and real-time to ensure the timeliness and effectiveness of the data.

[0072] For example, in a smart home scenario, user behaviors such as cooking and sleeping can be identified through these sensor data.

[0073] S2. Calculate the environmental anomaly index based on the signal change of each sensor, historical standard deviation, and preset environmental weight, combined with the time period factor.

[0074] Among them, the environmental anomaly index is the core indicator for the system to achieve active protection of environmental security, and is used to determine whether the current environment requires the activation of the end-side large model for further analysis.

[0075] In some implementations, the signal delta represents the change in sensor data within a time window, the historical standard deviation reflects the fluctuation of sensor data within the same historical time period, the preset environmental weight can be set based on the importance of the sensor's monitoring content, and the time period factor is used to reflect differences between different time periods. When sensor data suddenly deviates from historical patterns, the environmental anomaly index can quickly amplify the abnormal signal.

[0076] S3. Calculate the user intention confidence based on the cross-correlation value between audio data and visual motion data, visual motion information entropy, and the probability of person presence.

[0077] Among them, user intention confidence is the core criterion for achieving natural interaction and precise control, and is used to verify the effectiveness of user instructions.

[0078] In some implementations, the cross-correlation value is used to reflect the degree of correlation between audio and visual signals, the visual motion information entropy can quantify the complexity of the environment, and the probability of human presence is used to determine whether a person is present.

[0079] It should be noted that by calculating the confidence level of user intent through multimodal data fusion, it is possible to distinguish between real user commands and environmental noise. For example, when a user issues a voice command, visual motion data can be combined to determine whether the command is the user's true intention.

[0080] S4. When the environmental anomaly index is greater than a preset safety threshold, or the user intention confidence is greater than a preset intention threshold, the end-side large model is triggered to analyze the multimodal sensor data and generate control instructions, wherein the end-side large model includes a first end-side large model and a second end-side large model.

[0081] Among them, triggering the end-side large model analysis is to respond to abnormal environments or user intentions and generate corresponding control instructions to achieve protection or interaction.

[0082] In some implementations, the first end-side large model can be built based on an artificial intelligence algorithm to analyze environmental data and audio data to predict potential hazards; the second end-side large model is also built based on an artificial intelligence algorithm to analyze user voice data, etc. to provide interactive answers.

[0083] It's important to note that the preset safety and intent thresholds can be set based on actual application scenarios. When the environmental anomaly index exceeds the threshold, the first-end large model predicts whether there are hidden dangers such as gas leaks and sends an alarm notification. When the user intent confidence exceeds the threshold, the second-end large model confirms the control command with the user.

[0084] Based on the above technical solution, in the multi-dimensional perception method based on the end-side large model provided by this application, the end-side large model is triggered by collecting and analyzing multimodal sensor data in combination with the environmental anomaly index and user intention confidence. It can not only achieve active protection of environmental safety, timely discover and deal with hidden dangers, but also achieve natural interaction and precise control, while balancing the energy consumption and performance of the equipment and reducing resource consumption in unnecessary situations.

[0085] In a possible implementation of the embodiment of the present application, the above step S1 can be specifically implemented by the following S101, S102, S103 and S104, which are specifically described below:

[0086] S101: Collect environmental data. Environmental data is collected in real time by temperature and humidity sensors, smoke sensors, gas sensors, and light intensity sensors to monitor environmental safety and physical parameters.

[0087] In some implementations, the sensor collects data at a period of seconds (eg, once per second) and removes outliers through a filtering algorithm.

[0088] It should be noted that smoke / gas sensor data has the highest priority, and its sudden changes (such as a sudden increase in gas concentration) will trigger emergency data marking.

[0089] S102: Collect audio data. The audio data is collected synchronously through multiple channels (eg, 8 channels) of a microphone array to capture user voice commands or ambient sounds.

[0090] In some implementations, beamforming technology is used to enhance the signal strength in the direction of the target sound source, and short-time Fourier transform (STFT) is used to convert the time domain signal into frequency domain features.

[0091] It's important to note that audio data requires real-time noise reduction processing (such as Wiener filtering) to remove background noise from air conditioners, fans, and other sources. Furthermore, when high-frequency human voices (800-4000Hz) are detected for more than 300ms, the audio cache module is automatically triggered to identify the human voice data in real time, facilitating subsequent in-depth analysis of user intent.

[0092] S103: Collect visual motion data. The visual motion data is obtained through a thermal infrared sensor or a camera: the thermal infrared sensor outputs a temperature matrix, and the camera calculates the displacement of pixels in adjacent frames using an optical flow estimation algorithm.

[0093] In some implementations, the modulus of the optical flow displacement vector is calculated as Valid motion pixels are screened by using a preset modulus threshold, such as 2 pixels / frame; where u and v are the speeds of the pixels in the x / y directions.

[0094] It should be pointed out that thermal infrared sensors can still identify human movement through temperature differences even in the absence of light at night, and the local variance of the temperature matrix can reflect the intensity of human activity.

[0095] For example, the camera detects 5 consecutive frames of optical flow energy More than 10 -4 When , it can be determined that the personnel are in a continuous activity state.

[0096] S104: Collect status signal data. The status signal data is obtained by the door and window magnetic sensor or pressure sensor and is used to monitor the device switch status or physical contact events.

[0097] In some implementations, the door and window magnetic sensors can detect the switch status through the reed switch, and the pressure sensor can identify the weight of an object (such as a person sitting down) through the change in resistance.

[0098] It should be noted that the status signal data adopts an event trigger mechanism and reports data only when the status changes to reduce power consumption.

[0099] In one possible implementation, combining Figure 2 , the above S2 can be specifically implemented through the following S201, S202, S203 and S204:

[0100] S201, calculate the signal change and historical standard deviation. The signal change of sensor i in the time window Δt is ΔS i (t)=|S i (t)-S i (t-Δt)|, historical standard deviation It is the standard deviation of the data in the same time period in the past 7 days or the preset period.

[0101] In some implementations, Δt can be dynamically adjusted according to the sensor type, such as 1 minute for a smoke / gas sensor and 10 minutes for a temperature and humidity sensor.

[0102] S202: Determine the night mode time range.

[0103] First, the user's historical rest data is analyzed by clustering algorithm to generate an activity frequency matrix, and then the mean sleep time μ is calculated. sleep-time and standard deviation σ sleep-time , mean time of waking up μ wakeup-time and standard deviation σ wakeup-time , according to these data, the night mode time range [T night-start ,T night-end ]for: Here, α represents an adjustment factor, which is usually set to 1 to 2. For example, if the user's historical bedtime is concentrated between 22:00 and 24:00 (mean 23:00, standard deviation 0.5 hours), the night mode start time is set to 22:30 (set α = 1).

[0104] It should be noted that the night mode indicator function δ night (t) takes the value 1 during the night mode time range and 0 otherwise. It is used to adjust the system detection sensitivity.

[0105] S203, calculate the preset environment weight. The weight calculation formula of smoke / gas sensors is W env，i (t) = W base，i ·(1+γ·δ night (t)), the weight calculation formula of other sensors is W env，i (t) = W base，i ·(1-γ·δ night (t)); where W base，i represents the preset basic weight of the i-th sensor, and γ represents the night mode adjustment coefficient, which is usually set to 0.2-0.5.

[0106] In some implementations, it is assumed that the gas sensor base weight W base，i =0.4, γ=0.3, then the weight is increased to 0.4×(1+0.3)=0.52 in night mode.

[0107] It should be pointed out that the dynamic weighting mechanism enables high-risk sensors to obtain higher priority at night, avoiding the omission of hidden dangers due to environmental interference.

[0108] S204, calculate the environmental anomaly index EAI(t). Substitute the above calculation result into the formula:

[0109]

[0110] ; Wherein k1 and k2 represent preset time cycle factors, and k1>k2>0, for example, k1=2, k2=1 enhances the night detection sensitivity through the time cycle factor.

[0111] For example, suppose a smoke sensor in night mode The weight of the sensor is calculated to be 0.6, and the time factor is Then the contribution of this sensor to EAI is 1.8×0.6×0.88=0.95.

[0112] Based on the above technical solution, EAI achieves accurate quantification of environmental anomalies through dynamic weighting and time factors, providing core criteria for triggering large models on the end side.

[0113] In one possible implementation, combining Figure 2 The user intention confidence in the above step S3 can be specifically achieved through the following S301, S302, S303 and S304:

[0114] S301: Calculate the cross-correlation value between audio and visual data.

[0115] First, obtain the audio data A of the jth audio channel of the microphone array at time t j (t);

[0116] Then, the optical flow estimation algorithm is used to calculate the pixel displacement of adjacent frame images obtained by the visual sensor to obtain the optical flow displacement vector v(x, y);

[0117] The visual motion data V(t) at time t is calculated according to the optical flow energy formula: Where N represents the total number of pixels of the frame image acquired by the visual sensor, and |||| represents the symbol for modulus calculation;

[0118] Calculate the j-th audio channel A by Pearson correlation coefficient j The cross-correlation value of (t) and visual motion data V(t): Reflects the synchronization between audio signals and visual motion; Indicates the standard deviation of the audio channel j signal in the same historical time period, reflecting the degree of fluctuation of the audio data, σ V It represents the standard deviation of the visual motion data V(t) in the same historical time period, reflecting the dynamic fluctuation degree of visual motion.

[0119] S302: Calculate visual motion information entropy.

[0120] Discretize the optical flow displacement direction v(x, y) into n intervals (e.g. 8 directions), and count the proportion of moving pixels in each direction p i , the information entropy is calculated according to the formula: Among them, the higher the entropy value of the motion information entropy, the more complex the environment. For example, the motion information entropy value in a scene with multiple people moving in multiple directions will be higher than the entropy value of a single person walking.

[0121] S303: Calculate the probability of personnel presence.

[0122] Local variance of the temperature matrix by the thermal infrared sensor Calculate the probability of personnel presence, the formula is: Where β represents the slope parameter and θ represents the judgment threshold, both of which are determined through experimental calibration. For example, β = 5, θ = 10. At this time, when the local variance of the temperature matrix exceeds 20, P human When (t) approaches 1, it is judged to be in a occupied state.

[0123] S304: Calculate user intention confidence UIC(t).

[0124] The user intention confidence UIC(t) at time t is calculated according to the formula: Where m is the number of audio channels, and the calculation result of UIC ranges from 0 to 1.

[0125] In some implementations, when the current time is within the night mode time range, the preset intention threshold is reduced to a preset percentage. For example, the preset intention threshold in night mode is reduced from 0.7 percent to 80%, that is, the intention threshold in night mode is 0.56, to adapt to command recognition under low background noise.

[0126] For example, if the sum of the cross-correlation values ​​of the three audio channels is 1.8, the information entropy is 2.0, and the probability of a person being present is 0.9, then UIC=1.8 / (1+2.0)×0.9=0.54.

[0127] Based on the above technical solution, UIC verifies the effectiveness of user commands through multimodal data fusion, which can reduce the probability of false triggering and improve interaction accuracy.

[0128] In one possible implementation, combining Figure 2 In the above step S4, triggering the end-side large model to analyze the multimodal sensor data and generate control instructions may specifically include:

[0129] S401: Trigger a first end-side large model based on an environmental abnormality response value.

[0130] When EAI(t) exceeds the preset safety threshold (e.g., 1.2), the first end-side large model is activated, environmental data and audio data are input, the hidden danger prediction type (e.g., gas leak, fire risk) is output, and an alarm notification is pushed to the user terminal; the alarm notification includes the abnormal sensor type, real-time data value, and historical trend comparison chart.

[0131] It should be pointed out that the first end-side large model is trained based on the LSTM or Transformer algorithm and can predict the probability of hidden dangers occurring within the next 30 minutes.

[0132] For example, if the EAI reaches 1.5 and the gas sensor data is abnormal, the model predicts the hidden danger type as "gas leak" and triggers an audible and visual alarm and SMS notification.

[0133] S402: Trigger a second end-side large model based on the user intention response value.

[0134] Among them, when UIC(t)>preset intention threshold (such as 0.6), the second end-side large model is activated to parse the user voice data and generate control instructions (such as temperature adjustment, lighting control), or confirm the validity of the instructions with the user.

[0135] In some implementations, control commands must be verified through two factors (such as voice + gesture) to avoid misoperation.

[0136] It should be noted that the second-end large model supports multiple rounds of dialogue and can automatically adjust the response strategy (such as lowering the brightness of lights at night) based on environmental data (such as light intensity).

[0137] For example, when the user says "adjust to 24°C" and UIC=0.8, the model directly sends a temperature adjustment instruction to the air conditioner and replies "The temperature has been set to 24°C for you."

[0138] Based on the above technical solution, the dual-threshold trigger mechanism can realize "on-demand wake-up" of the large model on the edge, significantly reducing the continuous operation energy consumption of the large model on the edge and extending the battery life of the device.

[0139] In one possible implementation, the first end-side large model of the present application may use an LSTM network structure to process time series data of the environment and audio. The input of the first end-side large model is the normalized environmental data of a preset time period, such as the temperature and humidity, smoke, gas concentration, and the mean of the audio Mel spectrum after noise reduction within the last 30 minutes; the output is the hazard type of "gas leak", "fire hazard", and "normal". Labels are generated by manually annotating historical hazard events. When training the model, a cross-entropy loss function can be used, and the model is required to achieve an accuracy of more than 95% on the validation set after iterative training, or an F1-score value of more than 0.9, which can be output as a pre-trained first end-side large model and finally deployed on the edge computing device.

[0140] In one possible implementation, the second-end-side large model of the present application may adopt a Transformer network structure to process voice data in the environment and audio data. The input data of the second-end-side large model are the voice text features converted by automatic speech recognition technology, the normalized environmental sensor data, and the Mel spectrum features corresponding to the voice data. The model captures the semantic associations in the voice text and the contextual dependencies between the environmental data and the voice commands through the Transformer's multi-head attention mechanism. The Mel spectrum features are extracted by short-time Fourier transform and converted into the Mel spectrum mean to reflect the frequency domain distribution of the voice signal.

[0141] During the training phase, a multimodal dataset consisting of speech text, environmental data, and control commands is constructed. This involves collecting daily household commands from users, such as "Set the air conditioner to 26°C" and "Turn off the kitchen gas valve," while simultaneously recording the environmental data at the time the commands are triggered, such as the current temperature of 28°C and the kitchen gas concentration of 0 ppm. The corresponding control command labels, such as "AC temperature adjustment" and "Gas valve closing," are also annotated. The model uses a cross-entropy loss function to optimize the intent classification task (i.e., determining command types such as temperature adjustment and device on / off). The negative log-likelihood loss (NLLLLoss) is also used to optimize the control command generation task. End-to-end training is performed using the AdamW optimizer, with the goal of achieving a command type classification accuracy of ≥92% on the validation set.

[0142] During deployment, the pre-trained second-end large model is quantized and compressed into int8 format before being embedded in the smart home edge computing device. When a user's voice command triggers the microphone array to collect audio data, the system first determines whether to activate the large model using the user intent confidence (UIC) calculation module. If the UIC exceeds a preset threshold, the model is fed with the automatic speech recognition text, environmental data sequence, and mel-spectrogram features. The model extracts multimodal features using a Transformer encoder. The classification head first outputs the intent category, such as "temperature adjustment" with a probability of 0.95. The generation head then generates a control command, such as "Set the living room air conditioning temperature to 26°C," and confirms the user's voice with a voice prompt, such as "I've adjusted it to 26°C. Do you need confirmation?" If the environmental data contains outliers, such as a gas concentration greater than 50 ppm, the model prioritizes safety warning logic, suspending non-urgent control commands to ensure the interaction process is aligned with environmental safety policies.

[0143] Based on the above technical solution, the activation of the model is determined by the dual thresholds of the environmental anomaly index and the user intention confidence. The end-side large model analysis is triggered only when the environment is abnormal or the user intention is clear, avoiding its indiscriminate real-time operation, achieving a dynamic balance between performance requirements and energy consumption control, extending device life and reducing hardware loss; the first end-side large model predicts hidden danger events based on environmental data and audio data, realizing active protection of environmental safety; the second end-side large model analyzes user voice based on environmental data and audio data and gives interactive answers, realizing natural interaction and precise control.

[0144] The above mainly introduces the scheme of the embodiment of the present application from the perspective of device implementation. It can be understood that each device, for example, a multi-dimensional perception device based on a large end-side model, includes at least one of the hardware structures and software modules corresponding to the execution of each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] The embodiment of the present application can divide the multi-dimensional sensing device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0146] In the case of an integrated unit, Figure 3 A possible structural diagram of the multi-dimensional sensing device involved in the above embodiment (denoted as the multi-dimensional sensing device 30) is shown. The multi-dimensional sensing device 30 includes a processing unit 301 and a communication unit 302, and may also include a storage unit 303. Figure 3 The structural schematic diagram shown can be used to illustrate the structure of the multi-dimensional sensing device involved in the above embodiments.

[0147] when Figure 3 The structural schematic diagram shown is used to illustrate the structure of the multi-dimensional perception device involved in the above embodiments. The processing unit 301 is used to control and manage the actions of the multi-dimensional perception device, the communication unit 302 is used for the multi-dimensional perception device to communicate with other devices, and the storage unit 303 is used to store the program code and data of the multi-dimensional perception device.

[0148] For example, the communication unit 302 is used to establish a communication connection with an external multimodal sensor to receive multimodal sensor data such as environmental data, audio data, and visual motion data in real time; establish a communication connection with a controlled device (such as an air conditioner, light, gas valve, etc.) to push alarm instructions or control instructions to the target device;

[0149] The processing unit 301 is used to preprocess the multimodal sensor data acquired in real time (such as noise reduction and normalization); calculate the environmental anomaly index EAI based on the signal change, historical standard deviation, preset environmental weight and time period factor of each sensor; calculate the user intention confidence UIC based on the cross-correlation value of audio data and visual motion data, visual motion information entropy and the probability of human presence; when EAI is greater than the preset safety threshold or UIC is greater than the preset intention threshold, the first end-side large model or the second end-side large model is triggered to analyze the multimodal data and generate an alarm notification or control instruction.

[0150] In a possible implementation, the processing unit 301 is further configured to generate an activity frequency matrix based on the user's historical behavior data, determine the user's work and rest cycle through a clustering algorithm, and dynamically adjust the night mode time range and sensor weight parameters.

[0151] In one possible implementation, the communication unit 302 is further configured to receive configuration instructions from the user terminal, such as instructions for adjusting preset security thresholds and intention thresholds, and the processing unit 301 is further configured to update model parameters or threshold policies according to the configuration instructions.

[0152] Among them, the processing unit 301 can be a processor or controller (such as an ARM processor, FPGA chip, etc.), and the communication unit 302 can be a communication interface, a transceiver, a transceiver circuit (such as a Wi-Fi module, a Bluetooth module, a Zigbee module, etc.). Among them, the communication interface is a general term and can include one or more interfaces (such as a GPIO interface for sensor access, an Ethernet interface for network communication). The storage unit 303 can be a memory (such as a Flash memory, a DDR memory, etc.) for storing sensor historical data, model parameters, user configuration information, etc. When the multi-dimensional perception device 30 is a chip, the processing unit 301 can be a processor or controller (such as an integrated CPU core or NPU core), and the communication unit 302 can be an input interface and / or output interface, pin or circuit (such as an SPI interface, an I2C interface pin). The storage unit 303 can be a storage unit within the chip (such as a register, a cache), or a storage unit located outside the chip (such as an external ROM, RAM).

[0153] Among them, the communication unit can also be called a transceiver unit. The antenna and control circuit (such as a wireless communication module) with transceiver functions in the multi-dimensional sensing device 30 can be regarded as a communication unit 302, and the processor with a processing function can be regarded as a processing unit 301. Optionally, the device for realizing the receiving function (such as a radio frequency receiving circuit) in the communication unit 302 can be regarded as a receiving unit, which is used to perform the step of receiving sensor data in the embodiment of the present application; the device for realizing the sending function (such as a radio frequency sending circuit) in the communication unit 302 can be regarded as a sending unit, which is used to perform the step of sending alarm / control instructions in the embodiment of the present application.

[0154] Some of the data in the above formula are calculated by removing the dimensions and taking their numerical values. The formula is a formula that is closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0155] Working principle of the present invention:

[0156] First, environmental data, audio data, and visual motion data are acquired in real time through multimodal sensors;

[0157] Next, the edge computing module calculates the environmental anomaly index based on the sensor signal change, historical standard deviation, preset environmental weight, and time period factor. It also calculates the user intention confidence based on the cross-correlation value of audio and visual motion data, visual motion information entropy, and the probability of human presence.

[0158] When the environmental anomaly index exceeds a preset safety threshold, the first end-side large model is triggered to analyze multimodal data to predict potential hazards and push an alarm notification. When the user intent confidence exceeds a preset intent threshold, the second end-side large model is triggered to analyze user voice data and generate control instructions or interactive responses.

[0159] Among them, the night mode can dynamically adjust the sensor weight and detection threshold according to the user's daily routine. The presence status of people is used to assist in judging the effectiveness of user intentions. Through the threshold trigger mechanism, the on-demand wake-up of the large model on the terminal side is realized. While ensuring active protection of environmental safety and natural interaction with users, it reduces the energy consumption of continuous operation of the equipment, extends battery life and improves the accuracy of interaction.

[0160] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A multi-dimensional perception method based on a large device-side model, characterized by: include: Real-time acquisition of multimodal sensor data, including environmental data, audio data, and visual motion data; The environmental anomaly index is calculated based on the signal change, historical standard deviation and preset environmental weight of each sensor, combined with the time period factor; Calculating the user intention confidence based on the cross-correlation value between the audio data and the visual motion data, the visual motion information entropy, and the probability of the presence of a person; the visual motion information entropy and the probability of the presence of a person are calculated based on the visual motion data; When the environmental anomaly index is greater than a preset safety threshold, or the user intention confidence is greater than a preset intention threshold, the end-side big model is triggered to analyze the multimodal sensor data and generate control instructions; wherein, the end-side big model includes a first end-side big model and a second end-side big model.

2. The multi-dimensional perception method based on a large device-side model according to claim 1 is characterized in that: The multimodal sensor data includes: Environmental data composed of temperature and humidity sensors, smoke sensors, gas sensors, and light intensity sensors; Audio data collected by a microphone array; Visual motion data calculated by thermal infrared sensors or cameras; Status signal data obtained by door and window magnetic sensors or pressure sensors.

3. The multi-dimensional perception method based on a large device-side model according to claim 1, characterized in that: The calculation formula of the environmental anomaly index is: Where, ΔS i (t) represents the data change of sensor i in the time window Δt, represents the standard deviation of sensor i in the same historical period, W env，i (t) represents the preset environmental weight of sensor i at time t, k1 and k2 represent the preset time period factors, and k1>k2>0, δ night (t) represents the night mode indicator function, n represents the number of sensors, and t represents time.

4. The multi-dimensional perception method based on a large end-to-end model according to claim 3, characterized in that: The night mode indication function is determined based on the night mode time range, and includes: Collect user historical behavior data and generate activity frequency matrix; Perform cluster analysis on the activity frequency matrix based on the clustering algorithm to obtain the user's work and rest cycle; Calculate the night mode time range [T night-start ,T night-end ], the calculation formula is: Among them, T night-start Indicates the start time of night mode, T night-end Indicates the end time of night mode, μ sleep-time and σ sleep-time They represent the mean and standard deviation of the sleep time determined by the user's sleep cycle, μ wakeup-time and σ wakeup-time They represent the mean and standard deviation of the wake-up time determined by the user's sleep and rest cycle, and α represents the adjustment factor. When the current time is within the night mode time range, δ night (t) = 1; otherwise δ night (t)=0.

5. The multi-dimensional perception method based on a large end-to-end model according to claim 3 is characterized in that: The calculation formula of the preset environment weight is: Among them, W base，i represents the preset basic weight of the i-th sensor, and γ represents the night mode adjustment coefficient.

6. The multi-dimensional perception method based on a large end-to-end model according to claim 4, characterized in that: The calculation process of the user intention confidence includes: Get the audio data A of the jth audio channel of the microphone array at time t j (t); The optical flow estimation algorithm is used to calculate the pixel displacement of adjacent frame images obtained by the visual sensor to obtain the optical flow displacement vector v(x, y); The visual motion data V(t) at time t is calculated according to the optical flow energy formula: Where N represents the total number of pixels of the frame image acquired by the visual sensor, and |||| represents the symbol for modulus calculation; Discretize the direction of the optical flow displacement vector into n intervals and count the proportion of moving pixels in each direction p i , the visual motion information entropy at time t is calculated according to the formula: The motion pixel refers to a pixel whose modulus of the optical flow displacement vector exceeds a preset modulus threshold; Get the temperature matrix output by the thermal infrared sensor and calculate the local variance of the temperature matrix The probability of personnel presence at time t is calculated according to the formula P human (t), the formula is: Among them, β represents the slope parameter and θ represents the judgment threshold, both of which are determined through experimental calibration; The user intention confidence UIC(t) at time t is calculated according to the formula: Where m represents the number of audio channels of the microphone array, CrossCorr(A j (t), V(t)) represents the cross-correlation value between the audio data and the visual motion data calculated using the calculation formula of the Pearson correlation coefficient.

7. The multi-dimensional perception method based on a large device-side model according to claim 6, characterized in that: The user intention confidence determination rule also includes: When the current time is within the night mode time range, the preset intention threshold is reduced to a preset percentage; When the probability of the person existing is less than the preset probability threshold, the user intention confidence UIC is directly determined to be 0.

8. The multi-dimensional perception method based on a large device-side model according to claim 1, characterized in that: The generating control instruction comprises: When the environmental anomaly index is greater than a preset safety threshold, the first end-side large model is triggered to predict hidden danger events based on the multimodal sensor data, and generate a control instruction for an alarm notification based on the prediction result of the hidden danger event; When the user intention confidence is greater than a preset intention threshold, the second end-side large model is triggered to analyze the user voice data and provide an interactive answer; The first large model on the end is built based on an artificial intelligence algorithm, with environmental data and audio data as input and hidden danger prediction types as output, including gas leak and fire risks, to determine whether hidden dangers exist. The second end-side large model is built based on an artificial intelligence algorithm, and its input is environmental data and audio data, and its output is an interactive answer, which is used to confirm with the user and generate control instructions for the device.

9. A multi-dimensional perception system based on a large model on the device side, applied to the multi-dimensional perception method based on a large model on the device side according to any one of claims 1 to 8, characterized in that: include: Data acquisition module, used to acquire multimodal sensor data in real time, including environmental data, audio data, and visual motion data; The edge computing module includes an environmental anomaly analysis unit and a user intention inference unit; wherein the environmental anomaly analysis unit calculates the environmental anomaly index based on the signal change amount, historical standard deviation and preset environmental weight of each sensor, combined with the time period factor; the user intention inference unit calculates the user intention confidence based on the cross-correlation value of audio data and visual motion data, visual motion information entropy and the probability of human presence; The model analysis module is used to trigger the first end-side large model to predict hidden danger events and generate alarm instructions when the environmental anomaly index is greater than a preset safety threshold; and to trigger the second end-side large model to parse user voice data and generate control instructions when the user intention confidence is greater than a preset intention threshold; the control instructions include at least one of alarm notification instructions, temperature adjustment instructions, lighting adjustment instructions, and humidity adjustment instructions.

10. A multi-dimensional perception device based on a large model on the device side, applied to the multi-dimensional perception method based on a large model on the device side according to any one of claims 1 to 8, characterized in that: include: A processing unit for: Processing multimodal sensor data acquired in real time, the multimodal sensor data including environmental data, audio data, and visual motion data; Calculate the environmental anomaly index based on the signal change of each sensor, historical standard deviation, preset environmental weight and time period factor; The confidence level of user intention is calculated based on the cross-correlation value between audio data and visual motion data, visual motion information entropy, and the probability of person presence. When the environmental anomaly index is greater than a preset safety threshold, the first end-side large model is triggered to predict hidden danger events on the multimodal sensor data and generate an alarm instruction; when the user intention confidence is greater than the preset intention threshold, the second end-side large model is triggered to parse the user voice data and generate a control instruction; the control instruction includes at least one of an alarm notification instruction, a temperature adjustment instruction, a lighting adjustment instruction, and a humidity adjustment instruction. Communication unit for: Establishing a communication connection with an external multimodal sensor to receive the multimodal sensor data in real time; Establish a communication connection with the controlled device, push the control instruction to the target device, and implement the execution of the instruction.

Images