A mobile terminal rich context-aware and personalized agent decision-making method

Abstract

The application relates to the technical field of personalized intelligent decision-making, in particular to a mobile terminal rich-situation perception and personalized intelligent agent decision-making method, which comprises the following steps: constructing a three-state monitoring mechanism based on a minimum perceptible energy threshold, wherein the three-state monitoring mechanism comprises a dormant state, a short-time monitoring state and a long-time monitoring state; in the short-time monitoring state, a light-weight Gaussian mixture model is used to perform coarse-granularity classification on a sampled audio frame, a binary label representing human voice or environmental sound is outputted, and a continuous audio stream is converted into a structured acoustic event; when the structured acoustic event is a non-silence event, a multi-sensor context is collected and inputted into a multi-modal context model to obtain a scene label; and the scene label, the structured acoustic event and an audio segment are jointly inputted into a personalized intelligent agent decision-making model to perform situation-level reasoning and generate a concise and executable situational guide. The method realizes a closed-loop processing of low-power continuous monitoring, deep situation understanding and personalized guidance.

Description

Technical Field

[0001] The embodiments of this application relate to the field of personalized intelligent decision-making technology, and in particular to a mobile-based context-rich and personalized intelligent agent decision-making method. Background Technology

[0002] In recent years, with the widespread integration of various sensors such as microphones, accelerometers, gyroscopes, and positioning modules in mobile terminals such as smartphones and wearable devices, as well as the development of large-scale pre-trained models, mobile devices have significantly improved their ability to perceive and understand surrounding acoustic scenes and physical environments. Applications based on acoustic sensors have gradually expanded from traditional keyword detection and voice assistants to environmental sound event detection, soundscape classification, audio description generation, and multimodal large-model-driven human-computer interaction. Compared to earlier systems that could only output "sound category," the new generation of multimodal models can simultaneously access contextual information such as sound, location, and motion state, enabling higher-level semantic reasoning such as "what does this sound mean in the current context" and "whether immediate attention or action is needed." Especially in scenarios where users are in noisy environments, are moving, or have limited attention, how to integrate acoustic cues and physical context in real time on mobile terminals to achieve rich contextual understanding and contextualized interpretation of complex acoustic scenes has become an important research direction in the fields of mobile perception and human-computer collaboration.

[0003] However, current sound-based mobile systems still have significant shortcomings in terms of contextualization and personalization capabilities. On the one hand, due to limitations in energy consumption and computing power, mobile devices typically employ strategies such as duty-line sampling and lightweight models, which can easily miss short-term, low signal-to-noise ratio, or key acoustic events superimposed on background noise. Simultaneously, many solutions compress continuous acoustic streams into discrete labels or simple prompts, making it difficult to fully encode contextual elements such as "location, activity, and distance / direction of the sound source," resulting in systems providing only vague or overly general feedback. On the other hand, although large multimodal models possess strong reasoning capabilities, existing solutions often rely on users manually supplementing fine-grained information such as location, intent, and risk preferences during interaction. This increases the interaction burden and is prone to unstable reasoning results when input is incomplete. Furthermore, different users vary greatly in terms of risk sensitivity, tolerance for disturbance, and focus. Simple rule configuration or direct fine-tuning of large models on a user-by-user basis is not only difficult to learn a reliable individual preference structure under conditions of sparse data and skewed context distribution, but may also introduce uncontrolled bias-variance trade-offs, causing the guidance strategy to drift or even reduce security.

[0004] Therefore, there is an urgent need in this field for a technical solution that can seamlessly integrate "perception, reasoning, and action" on mobile terminals, ensure high recall of key acoustic events under energy-constrained conditions, and achieve rich contextual understanding, contextualized decision-making, and personalized guidance for users and scenarios through structured contextual modeling and controlled personalized intelligent agent decision-making. Summary of the Invention

[0005] In view of this, embodiments of this application propose a mobile context-aware and personalized intelligent agent decision-making method, which can achieve closed-loop processing of low-power continuous monitoring, deep context understanding and personalized guidance while ensuring real-time performance and privacy. It is applicable to scenarios such as smartphones and in-vehicle terminals, and is superior to traditional technologies in terms of energy consumption control and decision-making effectiveness.

[0006] To achieve the above objectives, embodiments of this application propose a mobile-terminal context-aware and personalized intelligent agent decision-making method. The method includes: constructing a three-state listening mechanism based on a minimum perceptible energy threshold, comprising a sleep state, a short-term listening state, and a long-term listening state; wherein, in the short-term listening state, the microphone samples periodically with a low duty cycle, in the long-term listening state, the microphone samples continuously, and in the sleep state, the microphone is turned off; in the short-term listening state, a lightweight Gaussian mixture model is used to perform coarse-grained classification of the sampled audio frames, outputting binary labels representing human voices or ambient sounds, transforming the continuous audio stream into structured acoustic events; when the acoustic event is a non-silent event, multiple sensors are activated to collect multi-sensor context, which is then input into a multimodal context model to infer the current scene and obtain a scene label; the scene label, acoustic event, and audio segment are jointly input into a personalized intelligent agent decision-making model to perform context-level reasoning, generating concise and executable contextualized guidance.

[0007] To achieve the above objectives, embodiments of this application also propose an electronic device, including: a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a mobile context-aware and personalized intelligent agent decision-making method as described above.

[0008] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a mobile context-aware and personalized intelligent agent decision-making method as described above.

[0009] Optionally, a three-state monitoring mechanism is constructed based on the minimum perceptible energy threshold, including: Based on the energy density of the acoustic signal in the time-frequency domain, the following formula is used to calculate the duration of the sound signal. The sensing bandwidth is Cumulative energy estimation of the sound: ; in, For the first Frequency within a time window Time-frequency representation at a location, for The corresponding energy density, The length of the time integration window. Indicates approximation. This represents the estimated cumulative energy; Let the minimum perceptible energy threshold be ; when When the microphone is switched off, it enters sleep mode and remains off. when When the time is reached, it enters short-time listening mode, utilizes the temporal sparsity of acoustic events to perform 10-second duty cycle sampling, and adaptively adjusts the short-time listening duration according to the remaining battery power. when Furthermore, when the current acoustic event is a non-silent event, it enters a long-term listening state to capture the complete context of the current acoustic event. After the current acoustic event ends, it switches back to a short-term listening state or a sleep state.

[0010] Optionally, a lightweight Gaussian mixture model is used to perform coarse-grained classification of the sampled audio frames, outputting binary labels representing human voices or ambient sounds, including: A lightweight Gaussian mixture model is used to model the stable energy structure of audio frames in the low / mid frequency band. The speech consistency likelihood of the feature vector extracted from the audio frame is expressed as: ; in, Indicates the speech consistency of audio frames. The number of Gaussian components. For parameters of the hybrid model, , , The first The weights, mean, and variance of each Gaussian component. The microphone uses a multivariate Gaussian distribution with a sampling rate of 32kHz to cover the sensing frequency band from 0kHz to 16kHz. The sensing frequency band from 0kHz to 4kHz is given high weight as the speech-dominant frequency band, and the sensing frequency band from 4kHz to 16kHz is given low weight as the non-speech-dominant frequency band. Let the speech consistency threshold be... ,Will and The audio frames are labeled as human voice frames. and The audio frames are labeled as ambient sound frames.

[0011] Optionally, after calculation Then, the speech consistency score is obtained by performing a logarithmic transformation on the likelihood proportion of each Gaussian component using the following formula. : .

[0012] Optionally, multi-sensor context is acquired and input into a multimodal context model to infer the current scene and obtain scene labels, including: Step frequency / motion cues reflecting mobility are collected by accelerometers, typical indoor location cues are based on the number of Wi-Fi access points and SSID semantics, and outdoor openness cues are based on the number of satellites visible and the average signal-to-noise ratio. All cues are encoded into multimodal context vectors. Input the multimodal context vector into the multimodal context model. Perform fusion inference on the current scene to obtain Output scene labels; where scene labels are used to represent indoor / outdoor, stationary / moving, station / airport / shopping mall / vehicle interior.

[0013] Optionally, scene labels, acoustic events, and audio clips are input into a personalized agent decision-making model to perform context-level reasoning, generating concise and actionable contextualized guidance, including: Scene labels, acoustic events, and audio clips are constructed into structured inference inputs, which are then fed into the personalized intelligent agent decision-making model. Perform context-level reasoning to generate concise and actionable contextualized instructions; To ensure consistent and learnable inference inputs, a phased supervised fine-tuning approach is adopted. First, learn contextual inference and reasoning input comprehension; then, learn to generate concise, prioritized instructions in different scenarios. The supervised fine-tuning goal is expressed by the formula: ; in, To monitor and fine-tune the targets, For inference input, Output the target sequence. To generate the step index, for In parameters The conditional probability of a token.

[0014] Optionally, based on supervised fine-tuning, direct preference optimization is used for personalized agent decision-making, targeting... Constructing Paired Responses , To provide a better preferred response that better aligns with user preferences, This is a non-preferred response; The direct preference optimization objective is expressed by the formula: ; in, To optimize the objective of direct preference, For the Sigmoid function, For reference strategy, This is the temperature coefficient.

[0015] Optionally, a lightweight adapter that only operates on the output mapping can be introduced on the mobile device, freezing... The core parameters enable local updates of human-machine interaction in the loop, defining personalized strategies. for: ; in, For lightweight adapters, For personalized strength coefficients; To prevent overfitting due to sparse feedback and maintain the consistency of group preferences, a regularization objective with KL constraints is adopted. Conduct training; ; in, The preference data is formed by users' feedback. The regularization coefficient, KL constraint, restricts personalized updates to only a small, controllable offset near the group alignment strategy, in order to reduce the risk of overfitting and safety drift caused by sparse feedback.

[0016] This application proposes a mobile-based rich context-aware and personalized intelligent agent decision-making method that can achieve a closed-loop integration of "perception, reasoning, and action" on mobile devices. It ensures high recall of key acoustic events under energy-constrained conditions and achieves rich context-aware understanding, contextualized decision-making, and personalized guidance for users and scenarios through structured context modeling and controlled personalized intelligent agent decision-making. First, a three-state listening mechanism is constructed based on a minimum perceptible energy threshold. Higher-intensity feature extraction is activated only when the acoustic energy crosses the threshold. A lightweight Gaussian mixture model is used to achieve coarse-grained annotation of human voice / ambient sound for silence suppression, transforming continuous audio streams into low-entropy structured acoustic events. Then, without requiring users to write prompts in real time, the coarse-grained annotations are fused with low-power multi-sensor contextual information. A compact multimodal context model infers the current scene, forming a contextualized semantic representation. Finally, a personalized intelligent agent decision-making model is used to perform context-level reasoning, generating concise and executable contextualized guidance. This application utilizes the above principles to achieve closed-loop processing of low-power continuous monitoring, deep contextual understanding, and personalized guidance while ensuring real-time performance and privacy. It is applicable to scenarios such as smartphones and in-vehicle terminals, and outperforms traditional technologies in terms of energy consumption control and decision-making effectiveness. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0018] Figure 1 This is a flowchart of a mobile context-rich and personalized intelligent agent decision-making method provided in one embodiment of this application; Figure 2 This is a detailed schematic diagram of a mobile context-rich and personalized intelligent agent decision-making method provided in one embodiment of this application; Figure 3 This is a schematic diagram of a three-state monitoring mechanism provided in one embodiment of this application; Figure 4 This is a schematic diagram of the weight allocation of speech and non-speech segments provided in one embodiment of this application; Figure 5 This is a schematic diagram of staged scenario reasoning provided in another embodiment of this application; Figure 6 This is a schematic diagram of the personalized intelligent agent decision-making model update provided in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0020] This application primarily utilizes two principles. Firstly, based on the characteristics of human hearing, ambient sound possesses a minimum perceptible energy threshold. Acoustic events below this threshold have limited impact on user decisions, and low-power continuous perception can be achieved through threshold gating. Secondly, the semantic meaning, risk level, and expected response to the same type of sound vary significantly across different contexts. Simply outputting static sound categories or event labels is insufficient to meet the guidance requirements of being both "executable" and "understandable."

[0021] Utilizing the above two principles, this application takes "low-power continuous perception with perception threshold gating" as the entry point, transforms continuous audio streams into low-entropy coarse-grained acoustic tags, and adopts a closed-loop scheme of "event-triggered multi-sensor context acquisition + two-stage edge-side contextual reasoning + contextual conditional personalized intelligent agent decision-making + local lightweight personalization" to achieve rich contextual understanding of the user's scene and generate contextualized guidance. Under strict constraints on mobile power consumption and latency, it outputs results that meet individual preferences and security requirements.

[0022] Based on this, one embodiment of this application proposes a mobile context-aware and personalized intelligent agent decision-making method. The implementation details of the mobile context-aware and personalized intelligent agent decision-making method proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding only and are not necessary for implementing this solution.

[0023] The specific process of the mobile context-rich awareness and personalized intelligent agent decision-making method proposed in this embodiment can be as follows: Figure 1 As shown, its details are as follows Figure 2 As shown, the method includes: Step 101: Construct a three-state listening mechanism based on the minimum perceptible energy threshold. The three-state listening mechanism includes a sleep state, a short-time listening state, and a long-time listening state. In the short-time listening state, the microphone samples periodically with a low duty cycle. In the long-time listening state, the microphone samples continuously. In the sleep state, the microphone is turned off.

[0024] In practical implementation, the first step in achieving context-aware and personalized intelligent agent decision-making on mobile devices is to construct a three-state listening mechanism based on the minimum perceptible energy threshold. The three-state listening mechanism includes a sleep state, a short-term listening state, and a long-term listening state. In the short-term listening state, the microphone samples periodically with a low duty cycle; in the long-term listening state, the microphone samples continuously; and in the sleep state, the microphone is turned off.

[0025] It is understandable that perception has a lower limit and is less sensitive to instantaneous amplitude but more affected by accumulated energy over time. This embodiment uses the following formula, based on the energy density of the sound signal in the time-frequency domain, to calculate the energy density over a duration of... The sensing bandwidth is Cumulative energy estimation of the sound: ; in, For the first Frequency within a time window Time-frequency representation at a location, for The corresponding energy density, The length of the time integration window. Indicates approximation. This represents the estimated cumulative energy.

[0026] Let the minimum perceptible energy threshold be (Generally speaking) Pick .when When [the microphone is in sleep mode], it enters sleep mode and keeps the microphone off. When the time limit is reached, a short-time listening state is entered, utilizing the temporal sparsity of acoustic events to perform 10-second duty cycle sampling, and the short-time listening duration is adaptively adjusted according to the remaining battery power. Furthermore, when the current acoustic event is a non-silent event, it enters a long-term listening state to capture the complete context of the current acoustic event. After the current acoustic event ends, it switches back to a short-term listening state or a sleep state.

[0027] The short-time listening state samples periodically with a low duty cycle to quickly detect perceptible events and perform early silence suppression. If non-silence is detected and the energy reaches a threshold, it is necessary to switch to a long-time listening state to capture the full context of the acoustic event. After the acoustic event ends, it returns to the short-time listening or sleep state. This is achieved by setting multiple detection points within a period of approximately 10 seconds (e.g., Figure 3 As shown in the figure, by jointly adjusting the frame length and listening duration, a balance can be achieved between recall, latency and energy consumption.

[0028] To ensure stable operation of the gating strategy on mobile devices, this embodiment maps perceptual constraints to sampling parameters. A 32kHz sampling rate is used to cover the perceptual frequency band from 0kHz to 16kHz. A 20ms frame length is used as the energy integration window to balance stability and responsiveness. The "temporal sparsity" characteristic of sound events is utilized for 10s duty cycle sampling (sampling at 0s, 2.5s, 5s, and 7.5s). The short-term listening duration is adaptively adjusted based on battery level (longer upper limit for higher battery levels), thereby reducing constant-on power consumption without significantly sacrificing perceptible event recall.

[0029] Step 102: In the short-time listening state, a lightweight Gaussian mixture model is used to perform coarse-grained classification of the sampled audio frames, and output binary labels representing human voices or ambient sounds, thus transforming the continuous audio stream into structured acoustic events.

[0030] In the specific implementation, during short-term listening, we need to use a lightweight Gaussian mixture model to perform coarse-grained classification of the sampled audio frames, outputting binary labels representing human voices or ambient sounds, and transforming the continuous audio stream into structured acoustic events.

[0031] In the short-term listening state, to further reduce invalid inference, this embodiment does not perform fine-grained classification, but instead generates "coarse-grained perceptual tags" that facilitate subsequent inference, used to filter silent segments as early as possible and distinguish between "human voices / ambient sounds". To adapt to continuous operation on mobile devices, this embodiment uses lightweight statistical modeling instead of computationally intensive neural detectors, that is, it uses a lightweight Gaussian mixture model (GMM) to model the stable energy structure of audio frames in the low / mid frequency bands.

[0032] set up The speech consistency likelihood of the feature vector (sub-band SNR feature, such as composed of multiple sub-band energies or signal-to-noise ratios) extracted from the audio frame is expressed as: ; in, Indicates the speech consistency of audio frames. The number of Gaussian components. For parameters of the hybrid model, , , The first The weights, mean, and variance of each Gaussian component. The frequency band is a multivariate Gaussian distribution. To improve sensitivity to speech, this embodiment assigns high weight to the 0kHz to 4kHz sensing band as the speech-dominant band and low weight to the 4kHz to 16kHz sensing band as the non-speech-dominant band (e.g., ...). Figure 4 As shown in the figure, this improves the robustness of coarse-grained detection in noisy environments.

[0033] Let the speech consistency threshold be... ,Will and The audio frames are labeled as human voice frames. and The audio frames are labeled as ambient sound frames, thus completing coarse-grained classification. The detector has a small parameter size and can operate at the edge with low overhead.

[0034] In practical implementation, frame-level energy consistency or component log-likelihood differences can be further calculated to distinguish between silence, speech, and typical ambient sound. For example, a logarithmic transformation can be performed on the likelihood proportions of each Gaussian component to obtain a speech consistency score. , .

[0035] Through the coarse-grained detection described above, this embodiment completes the event screening in the continuous listening phase with a very small model size and computational overhead, and compresses the continuous audio stream into a low-entropy structured trigger signal (the existence of the event and its "human voice / ambient sound" attributes) so that subsequent high-order inference can be started when the event occurs.

[0036] Step 103: When the acoustic event is a non-silent event, activate the multi-sensor, collect the multi-sensor context, and input it into the multimodal context model to infer the current scene and obtain the scene label.

[0037] In the specific implementation, when an acoustic event is detected to be a non-silent event, multiple sensors can be activated to collect the multi-sensor context and input into the multimodal context model to infer the current scene and obtain the scene label.

[0038] User interaction bandwidth is limited while on the move. Requiring users to describe the context and write prompts in real time for each event would significantly reduce usability. Therefore, this embodiment automatically collects low-power sensor features related to scene inference after detecting a non-silent event.

[0039] The multi-sensor context includes at least: step frequency / motion cues collected by accelerometers to reflect mobility, indoor typical location cues based on the number of Wi-Fi access points and SSID semantics, and outdoor openness cues based on the number of satellite-visible points and average signal-to-noise ratio.

[0040] All cues are encoded into multimodal context vectors, and these multimodal context vectors are then input into a multimodal context model. Perform fusion inference on the current scene to obtain The output scene labels form interpretable, low-dimensional, and stable intermediate representations, reducing the interference of uncertainties in the original sensor stream on subsequent inference. Scene labels are used to represent indoor / outdoor, stationary / moving, and station / airport / shopping mall / vehicle interior.

[0041] Step 104: Input scene labels, acoustic events and audio clips into the personalized intelligent agent decision model to perform context-level reasoning and generate concise and executable contextualized guidance.

[0042] In the specific implementation, after obtaining the scene label, the scene label, acoustic event, and audio clip are input together into the personalized intelligent agent decision model to perform context-level reasoning and generate concise and executable contextualized guidance.

[0043] This embodiment adopts a two-stage approach (such as...) Figure 5 The mobile-side inference method (shown) replaces the approach of "directly fusing all modalities from a single large model". The first stage outputs structured scene labels from a multimodal context model. The second stage uses a personalized intelligent agent decision model to perform context-level inference by combining "audio clips + coarse-grained acoustic labels + scene labels", outputting user-oriented notifications and actionable guidance. The above inputs are automatically generated by the system and form structured prompts, which are used to stably trigger multi-step inference without relying on users to write prompts in real time. Furthermore, by abstracting the high-dimensional raw sensor stream into semantic scene labels, inference noise is reduced and output stability and usability are improved.

[0044] To ensure consistent and learnable inference inputs, a phased supervised fine-tuning approach is adopted. First, learn contextual inference and reasoning input comprehension; then, learn to generate concise, prioritized instructions in different scenarios. The supervised fine-tuning goal is expressed by the formula: ; in, To monitor and fine-tune the targets, For inference input, Output the target sequence. To generate the step index, for In parameters The conditional probability of a token.

[0045] To ensure that the guidance better aligns with users' preferences regarding "scene-dependent priorities (e.g., prioritizing outdoor traffic-related information and indoor voice cues)," "accuracy-latency tradeoffs varying with risk," and "safety-critical voices not being weakened," this embodiment employs a training strategy of "supervised fine-tuning followed by personalized agent decision-making." Supervised fine-tuning first solidifies the basic capabilities of both the context model and the guidance model. Then, direct preference optimization is used to widen the likelihood interval between preferred and non-preferred outputs under the same scene input, thereby achieving context-conditional personalized agent decision-making.

[0046] Based on supervised fine-tuning, direct preference optimization is used for personalized agent decision-making, targeting... Constructing Paired Responses , To provide a better preferred response that better aligns with user preferences, This is a non-preferred response.

[0047] The direct preference optimization objective is expressed by the formula: ; in, To optimize the objective of direct preference, For the Sigmoid function, For reference strategy, This is the temperature coefficient.

[0048] By making personalized intelligent agent decisions in different scenarios, the model can learn to prioritize alerts for security-related events and reasonably downgrade low-priority background noise, outputting simpler and more scenario-appropriate guidance.

[0049] It is important to note that while personalized agent decision-making at the group level can improve overall usability, individual users' daily environments and risk preferences may still differ and change over time. Directly personalizing the entire model can easily lead to overfitting and drift, especially in security-related scenarios, potentially compromising semantic and inference stability. Therefore, this embodiment introduces a lightweight adapter on the mobile device that only operates on the output mapping, freezing... The core parameters enable local updates of human-machine interaction in the loop (such as...). Figure 6 (As shown).

[0050] Define personalization strategy for: ; in, For lightweight adapters, This is a personalized strength coefficient.

[0051] To prevent overfitting due to sparse feedback and to maintain the consistency of group preferences, this embodiment employs a regularization objective with KL constraints. Conduct training: ; in, The preference data is formed by users' feedback. The regularization coefficient, KL constraint, restricts personalized updates to only a small, controllable offset near the group alignment strategy, in order to reduce the risk of overfitting and safety drift caused by sparse feedback.

[0052] In terms of implementation, A multilayer perceptron (MLP) employing a two-bottom bottleneck structure projects the backbone output vector down to a low-dimensional bottleneck (e.g., 128 dimensions) and then up, thereby compressing the number of trainable parameters to approximately 100,000 to 500,000. We only update... It is trained with half precision to meet the resource constraints of mobile devices.

[0053] Furthermore, this embodiment supports both default and customized modes. Default mode provides direct scenario-based guidance. In customized mode, users can correct identified events or guidance content. Correction information is recorded locally and triggers incremental updates to the adapter, allowing the system to gradually adapt to personal habits in repetitive scenarios.

[0054] This embodiment proposes a mobile-based rich context-aware and personalized intelligent agent decision-making method that can seamlessly integrate "perception, reasoning, and action" on mobile devices. It ensures high recall of key acoustic events under energy-constrained conditions and achieves rich context-aware understanding, contextualized decision-making, and personalized guidance for users and scenarios through structured context modeling and controlled personalized intelligent agent decision-making. First, a three-state listening mechanism is constructed based on a minimum perceptible energy threshold. Higher-intensity feature extraction is activated only when the acoustic energy crosses the threshold. A lightweight Gaussian mixture model is used to achieve coarse-grained annotation of silence suppression of human voices / ambient sounds, transforming continuous audio streams into low-entropy structured acoustic events. Then, without requiring users to write prompts in real time, the coarse-grained annotations are fused with low-power multi-sensor contextual information. A compact multimodal context model infers the current scene, forming a contextualized semantic representation. Finally, a personalized intelligent agent decision-making model is used to perform context-level reasoning, generating concise and executable contextualized guidance. This embodiment utilizes the above principles to achieve closed-loop processing of low-power continuous monitoring, deep contextual understanding, and personalized guidance while ensuring real-time performance and privacy. It is applicable to scenarios such as smartphones and in-vehicle terminals, and outperforms traditional technologies in terms of energy consumption control and decision-making effectiveness.

[0055] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.

[0056] Another embodiment of this application provides an electronic device, such as Figure 7 As shown, it includes a processor 201 and a memory 202. The memory 202 stores instructions that the processor 201 can execute. When the processor 201 is configured to execute the instructions, the electronic device can realize a mobile terminal rich context awareness and personalized intelligent agent decision-making method as described in the above method embodiment.

[0057] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges. The bus can connect various circuits of one or more processors and memories, as well as other circuits such as peripherals, voltage regulators, and power management circuits—all well-known in the art and therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which also receives and transmits data to the processor.

[0058] The processor manages the bus and handles general processing, providing various functions, including but not limited to timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory, on the other hand, is used to store data used by the processor during operation.

[0059] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a mobile context-aware and personalized intelligent agent decision-making method as described in the above method embodiments.

[0060] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0061] It will be understood by those skilled in the art that the above embodiments are specific implementations of this application, and various changes in form and detail can be made in practical applications without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A mobile-based context-aware and personalized intelligent agent decision-making method, characterized in that, include: A three-state listening mechanism is constructed based on the minimum perceptible energy threshold. The three-state listening mechanism includes a sleep state, a short-time listening state, and a long-time listening state. In the short-time listening state, the microphone samples periodically with a low duty cycle. In the long-time listening state, the microphone samples continuously. In the sleep state, the microphone is turned off. In short-term listening mode, a lightweight Gaussian mixture model is used to perform coarse-grained classification of sampled audio frames and output binary labels representing human voices or ambient sounds, thus transforming continuous audio streams into structured acoustic events. When the acoustic event is a non-silent event, multiple sensors are activated to collect the multi-sensor context and input it into the multimodal context model to infer the current scene and obtain the scene label; By inputting scene labels, acoustic events, and audio clips into a personalized intelligent agent decision-making model, context-level reasoning is performed to generate concise and actionable contextualized guidance.

2. The mobile context-rich and personalized intelligent agent decision-making method as described in claim 1, characterized in that, A three-state monitoring mechanism is constructed based on the minimum perceptible energy threshold, including: Based on the energy density of the acoustic signal in the time-frequency domain, the following formula is used to calculate the duration of the sound signal. The sensing bandwidth is Cumulative energy estimation of the sound: ； in, For the first Frequency within a time window Time-frequency representation at a location, for The corresponding energy density, The length of the time integration window. Indicates approximation. This represents the estimated cumulative energy; Let the minimum perceptible energy threshold be ; when When the microphone is switched off, it enters sleep mode and remains off. when When the time is reached, it enters short-time listening mode, utilizes the temporal sparsity of acoustic events to perform 10-second duty cycle sampling, and adaptively adjusts the short-time listening duration according to the remaining battery power. when Furthermore, when the current acoustic event is a non-silent event, it enters a long-term listening state to capture the complete context of the current acoustic event. After the current acoustic event ends, it switches back to a short-term listening state or a sleep state.

3. The mobile context-rich and personalized intelligent agent decision-making method as described in claim 2, characterized in that, A lightweight Gaussian mixture model is used to perform coarse-grained classification of sampled audio frames, outputting binary labels representing human voices or ambient sounds, including: A lightweight Gaussian mixture model is used to model the stable energy structure of audio frames in the low / mid frequency band. The speech consistency likelihood of the feature vector extracted from the audio frame is expressed as: ； in, Indicates the speech consistency of audio frames. The number of Gaussian components. For parameters of the hybrid model, , , The first The weights, mean, and variance of each Gaussian component. The microphone uses a multivariate Gaussian distribution with a sampling rate of 32kHz to cover the sensing frequency band from 0kHz to 16kHz. The sensing frequency band from 0kHz to 4kHz is given high weight as the speech-dominant frequency band, and the sensing frequency band from 4kHz to 16kHz is given low weight as the non-speech-dominant frequency band. Let the speech consistency threshold be... ,Will and The audio frames are labeled as human voice frames. and The audio frames are labeled as ambient sound frames.

4. The mobile context-rich and personalized intelligent agent decision-making method as described in claim 3, characterized in that, In the calculation Then, the speech consistency score is obtained by performing a logarithmic transformation on the likelihood proportion of each Gaussian component using the following formula. : 。 5. The mobile context-rich and personalized intelligent agent decision-making method as described in claim 1, characterized in that, Collect multi-sensor context and input it into a multimodal context model to infer the current scene and obtain scene labels, including: Step frequency / motion cues reflecting mobility are collected by accelerometers, typical indoor location cues are inferred based on the number of Wi-Fi access points and SSID semantics, outdoor openness cues are inferred based on the number of satellites visible and the average signal-to-noise ratio, and all cues are encoded into multimodal context vectors. Input the multimodal context vector into the multimodal context model. Perform fusion inference on the current scene to obtain Output scene labels; where scene labels are used to represent indoor / outdoor, stationary / moving, station / airport / shopping mall / vehicle interior.

6. The mobile context-rich and personalized intelligent agent decision-making method as described in claim 5, characterized in that, Scene labels, acoustic events, and audio clips are input into a personalized intelligent agent decision-making model to perform context-level reasoning, generating concise and actionable contextualized guidance, including: Scene labels, acoustic events, and audio clips are constructed into structured inference inputs, which are then fed into the personalized intelligent agent decision-making model. Perform context-level reasoning to generate concise and actionable contextualized instructions; To ensure consistent and learnable inference inputs, a phased supervised fine-tuning approach is adopted. First, learn contextual inference and reasoning input comprehension; then, learn to generate concise, prioritized instructions in different scenarios. The supervised fine-tuning goal is expressed by the formula: ； in, To monitor and fine-tune the targets, For inference input, Output the target sequence. To generate the step index, for In parameters The conditional probability of a token.

7. The mobile context-rich and personalized intelligent agent decision-making method as described in claim 6, characterized in that, Based on supervised fine-tuning, direct preference optimization is used for personalized agent decision-making, targeting... Constructing Paired Responses , To provide a better preferred response that better aligns with user preferences, This is a non-preferred response; The direct preference optimization objective is expressed by the formula: ； in, To optimize the objective of direct preference, For the Sigmoid function, For reference strategy, This is the temperature coefficient.

8. The mobile context-rich and personalized intelligent agent decision-making method as described in claim 7, characterized in that, Introduce a lightweight adapter on mobile devices that only applies to output mapping and freeze... The core parameters enable local updates of human-machine interaction in the loop, defining personalized strategies. for: ； in, For lightweight adapters, For personalized strength coefficients; To prevent overfitting due to sparse feedback and maintain the consistency of group preferences, a regularization objective with KL constraints is adopted. Conduct training; ； in, The preference data is formed by users' feedback. The regularization coefficient, KL constraint, restricts personalized updates to only a small, controllable offset near the group alignment strategy, in order to reduce the risk of overfitting and safety drift caused by sparse feedback.

9. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores instructions that the processor can execute, and the processor is configured to, when executing the instructions, enable the electronic device to implement a mobile context-aware and personalized intelligent agent decision-making method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a mobile-terminal context-aware and personalized intelligent agent decision-making method as described in any one of claims 1 to 8.

Images