Mobile terminal and cloud interaction system and communication method

Abstract

The invention provides a mobile terminal and cloud interaction system and a communication method, and belongs to the technical field of intelligent wearable equipment. Comprising an intelligent bracelet, a mobile terminal and a cloud AI service platform, a loudspeaker, a microphone and a wireless communication module are arranged in the intelligent bracelet, a wide area network communication module and a near field communication module are arranged in the mobile terminal, the intelligent bracelet is in wireless communication with the mobile terminal, and the mobile terminal is in wireless communication with the cloud AI service platform. According to the invention, health management service is provided for the elderly through the smart bracelet, early warning is given out in time when abnormity is found, the elderly and family members are reminded to take corresponding measures, the health of the elderly is protected, and the smart bracelet carries out AI interaction with the cloud AI service platform through the mobile terminal, so that diversified accompanying modes can be provided for the elderly, and the health of the elderly is improved. The intelligent bracelet can accompany the elderly to chat and answer questions of the elderly in real time, the loneliness of the elderly is effectively relieved, and the emotional requirements of the elderly are met.

Description

Technical Field

[0001] This invention belongs to the field of smart wearable device technology, specifically relating to a mobile terminal and cloud interaction system and communication method. Background Technology

[0002] With the rapid development of technology, the smart wearable device market is booming. Smart bracelets, as an important component of this market, have gained popularity among consumers due to their convenience, multifunctionality, and relatively affordable price. Smart bracelets primarily focus on activity monitoring, helping users understand their exercise status and develop reasonable exercise plans. They can also monitor physiological indicators such as heart rate, blood pressure, and blood oxygen saturation in real time, providing crucial data support for users' health management. In addition to activity and health monitoring functions, smart bracelets also feature intelligent reminder functions, such as call reminders, SMS reminders, and schedule reminders, enabling users to receive important information promptly and avoid missing important tasks.

[0003] With the increasing number of elderly people in my country and their children moving further away, the number of lonely elderly people in society is growing. As society enters the era of artificial intelligence, smart bracelets, commonly worn by the elderly, have become essential daily necessities. Many elderly people lack daily companionship and communication due to their children's busy work schedules and fast-paced lifestyles. This not only easily leads to feelings of loneliness and psychological problems but may also negatively impact their physical health. Summary of the Invention

[0004] (1) Technical problems to be solved

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a mobile terminal and cloud interaction system and communication method to solve the problem that existing smart bracelets cannot perform AI interaction and cannot provide diversified companionship for the elderly.

[0006] (2) Technical solution

[0007] To address the aforementioned technical problems, this invention provides a mobile terminal and cloud interaction system, comprising a smart bracelet, a mobile terminal, and a cloud AI service platform.

[0008] The smart bracelet has a built-in speaker, microphone, and wireless communication module, while the mobile terminal has a built-in wide area network communication module and short-range communication module. The smart bracelet communicates wirelessly with the mobile terminal, and the mobile terminal communicates wirelessly with the cloud AI service platform.

[0009] Preferably, a method for communication between a mobile terminal and the cloud includes the following steps:

[0010] Step 1: When the smart bracelet's microphone detects the wake word, it begins to listen to the user's spoken voice. After the user finishes listening to the spoken voice, the voice data is sent to the mobile terminal.

[0011] Step 2: The mobile terminal uploads data to the cloud AI service platform using AES-256 encryption.

[0012] Step 3: After receiving the interaction request from the mobile terminal, the cloud AI service platform verifies the integrity of the data, sorts the correct data by sequence number, returns the failed data, and requests a retransmission.

[0013] Step 4: The cloud-based AI service platform decrypts the successful encrypted data packets frame by frame, generates a response result, and sends it to the mobile terminal. Finally, the mobile terminal sends the response result to the wristband.

[0014] Preferably, the mobile terminal has accessibility services installed and communicates with the APP installed on the mobile terminal in real time. When the system detects that the APP is silenced, the accessibility services on the mobile terminal enable Bluetooth or Wi-Fi scanning and connect to the smart bracelet via Bluetooth or Wi-Fi. After the mobile terminal and the smart bracelet establish accessibility service connections, all voice requests from the smart bracelet are uploaded to the cloud AI service platform by the accessibility services.

[0015] Preferably, the cloud-based AI service platform verifies the MAC address of the smart bracelet when receiving data from the mobile terminal.

[0016] Preferably, when the mobile terminal uploads data using AES-256 encryption, it dynamically slices the audio data every 300ms and then encrypts each slice independently.

[0017] Preferably, the slicing formula is: Frame(i) = Audio[i×4800:(i+1)×4800]; where: sampling rate = 16kHz, bit depth = 16bit, 300ms = 4800 sample points;

[0018] Encryption process:

[0019] Key k = HKDF(master key, device ID);

[0020] IV(i) = HMAC(k, timestamp||i)[0:16];

[0021] Ciphertext C(i)=AES-256-GCM(k,IV(i),Frame(i));

[0022] Transmission packet = Base64(i||timestamp||C(i)||GCM-Tag).

[0023] Preferably, frame-by-frame decryption includes:

[0024] A1. First, align the sample points, then dynamically adjust the 50-200ms buffer window to eliminate jitter in the overlapping area. ;

[0025] A2. Resampling is used for linear prediction to compensate for clock drift, where the clock drift rate is: ;

[0026] A3. Compensated sampling point mapping: ;

[0027] in .

[0028] Preferably, in step four, the decrypted speech data is completely assembled and sent to the ASR speech-to-text module for speech processing. Then, using a dynamic load balancing architecture and a hybrid recognition model, the text is semantically processed through an NLP text semantic understanding model. Finally, based on the acoustic model of the Generative Adversarial Network (GAN) and the corresponding speech prosody predictor, TTS speech and specific execution instructions are generated.

[0029] Preferably, the speech processing includes:

[0030] B1. Feature extraction of multimodal recognition architecture: Mel-Frequency Cepstral Coefficients;

[0031] B2, ASR multi-layer speech recognition

[0032] ;

[0033] B3. Zero-shot dialect adaptation based on meta-learning

[0034] .

[0035] Preferably, the communication method between the mobile terminal's APP and the accessibility service is: AIDL file sharing; the communication method between the mobile terminal's APP and the accessibility service and the smart bracelet is: Bluetooth or Wi-Fi communication; the communication method between the mobile terminal's APP and the accessibility service and the cloud AI service platform is: HTTPS request to the cloud API.

[0036] (3) Beneficial effects

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] The above solution provides health management services for the elderly through smart bracelets. When abnormalities are detected, timely warnings are issued to remind the elderly and their families to take appropriate measures to safeguard their health. In addition, the smart bracelets interact with the cloud AI service platform through mobile terminals, providing the elderly with diverse ways to accompany them. The smart bracelets can chat with the elderly in real time, answer their questions, effectively alleviate their loneliness, and meet their emotional needs.

[0039] The above solution, by combining localized learning of user dialects with cloud-based collaborative optimization, enables smart bracelets to better adapt to the language habits of the elderly, improve the accuracy and fluency of voice interaction, increase response speed, and allow the elderly to receive more timely responses. Furthermore, it can fully utilize the powerful computing capabilities and rich language resources of the cloud to improve recognition accuracy, thereby accurately understanding and recognizing the elderly's dialects and enhancing the quality of companionship. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the structure of the mobile terminal and cloud interaction system and communication method. Detailed Implementation

[0041] Embodiments of the present invention provide a mobile terminal and cloud interaction system, such as... Figure 1 As shown, it includes smart bracelets, mobile terminals, and a cloud AI service platform. The cloud AI service platform is a general-purpose large-scale cloud AI model service platform.

[0042] The smart bracelet has a built-in speaker, microphone, and low-power wireless communication module. It has a heart rate monitoring linkage mechanism that automatically enhances the microphone sensitivity when an abnormal heart rate is detected. The mobile terminal has a built-in wide area network communication module and a near-field communication module. The smart bracelet communicates wirelessly with the mobile terminal, and the mobile terminal communicates wirelessly with the cloud AI service platform.

[0043] The communication method includes the following steps:

[0044] Step 1: When the smart bracelet's microphone detects a wake word, such as a wake word starting with "xiao ya huan" spoken by the user, it begins to listen to the user's voice. After monitoring the user's voice ends, it sends the voice data to the mobile terminal's APP or accessibility service via Bluetooth or Wi-Fi.

[0045] Step 2: The mobile terminal's APP or accessibility service uploads data to the cloud AI service platform using AES-256 encryption.

[0046] Step 3: After receiving the interaction request from the mobile terminal, the cloud AI service platform verifies the integrity of the data, sorts the correct data by sequence number, returns the failed data, and requests a retransmission.

[0047] Step 4: The cloud-based AI service platform decrypts the successful encrypted data packets frame by frame, then generates a response and sends it to the mobile terminal's APP or accessibility service. After receiving the cloud response, the mobile terminal sends it to the wristband via Bluetooth or Wi-Fi to execute specific tasks and provide voice broadcasts.

[0048] In this embodiment, an accessibility service is installed on the mobile terminal. This accessibility service is an accessibility assistance backend service that communicates with the APP installed on the mobile terminal in real time. When the system detects that the APP is silenced, the accessibility service on the mobile terminal enables Bluetooth or Wi-Fi scanning and connects to the smart bracelet via Bluetooth or Wi-Fi. After the mobile terminal and the smart bracelet establish an accessibility service connection, all voice requests from the smart bracelet are uploaded to the cloud AI service platform by the accessibility service to enable accessibility communication.

[0049] In this embodiment, the cloud-based AI service platform verifies the MAC address of the smart bracelet when receiving data from the mobile terminal, making data transmission more secure.

[0050] In this embodiment, when the mobile terminal uploads data using AES-256 encryption, it dynamically slices the audio data every 300ms and then encrypts each slice independently.

[0051] In this embodiment, the slicing formula is: Frame(i) = Audio[i×4800:(i+1)×4800]; where: sampling rate = 16kHz, bit depth = 16bit, 300ms = 4800 sample points;

[0052] The encryption process is as follows:

[0053] Key k = HKDF(master key, device ID);

[0054] IV(i) = HMAC(k, timestamp||i)[0:16];

[0055] Ciphertext C(i)=AES-256-GCM(k,IV(i),Frame(i));

[0056] Transmission packet = Base64(i||timestamp||C(i)||GCM-Tag).

[0057] In this embodiment, frame-by-frame decryption includes:

[0058] A1. First, align the sample points, then dynamically adjust the 50-200ms buffer window to eliminate jitter in the overlapping area. Buffer the jitter;

[0059] A2. Resampling is used for linear prediction to compensate for clock drift, where the clock drift rate is: Ensure clock synchronization;

[0060] definition:

[0061] tk: Theoretical arrival time of the k-th slice

[0062] tk′: The actual arrival time of the k-th slice

[0063] fs: Nominal sampling rate (16kHz)

[0064] fs′: Actual sampling rate (including drift)

[0065] M: Slice length (4800 sample points)

[0066] A3. Compensated sampling point mapping: ;

[0067] in .

[0068] In this embodiment, in step four, the decrypted voice data is completely assembled and sent to the ASR speech-to-text module for speech processing to convert the speech into text. Then, using a dynamic load balancing architecture and a hybrid recognition model, the text is semantically processed through an NLP text semantic understanding model. Next, based on the acoustic model of a generative adversarial network (GAN) and the corresponding speech prosody predictor, TTS speech and specific execution instructions are generated. Finally, the information is returned to the mobile terminal's APP or accessibility service. After receiving the response from the cloud, the mobile terminal sends it to the wristband via Bluetooth or Wi-Fi to execute specific tasks and voice broadcasts.

[0069] In this embodiment, voice processing includes:

[0070] B1. Multimodal Recognition Architecture

[0071] Feature extraction: Mel-Frequency Cepstral Coefficients (MFCC);

[0072] Deep learning models (for solving speech confusion matrix compensation algorithms (e.g., dialect pronunciation problems such as confusion between "m" and "n")):

[0073] class DialectASR(nn.Module);

[0074] def __init__(self);

[0075] super().__init__();

[0076] self.cnn = ResNet34() # Dialect feature extraction;

[0077] self.gru = BiGRU(units=128) # Temporal modeling;

[0078] self.ctc = CTCLoss() # Connection time series classification;

[0079] B2. Multilayer Speech Recognition (ASR)

[0080]

[0081] in:

[0082] X: Speech feature sequence number

[0083] W: Candidate word sequence

[0084] D: Dialect category identifier

[0085] α, β: Weighting coefficients

[0086] B3. Zero-sample dialect adaptation

[0087] Dialect Adaptation Based on Meta-Learning

[0088]

[0089] in:

[0090] θdialect: Model parameters adapted to the new dialect

[0091] θmeta: Initial parameters of the meta-model (obtained through joint training with multiple dialects or using a third-party model).

[0092] α: Learning rate (adaptation step size), the magnitude of the control parameter update.

[0093] ∇θ‎: Gradient operator (means taking the partial derivative with respect to the parameter θ)

[0094] LL: Loss function (measures the model's prediction error)

[0095] Dnew: Limited data on the new dialect.

[0096] Specifically, the system uses an ASR (Automatic Speech Recognition) speech-to-text module. After speech-text processing, it calls a NLP (Natural Language Processing) semantic understanding model to identify the domain intent. Then, it performs TTS (Text-to-Speech) synthesis on the speech portion of the response result, stores the generated audio in a cache, and returns a response (response data includes execution instructions and audio data). After receiving the response from the cloud, the APP or accessibility service sends it to the smart bracelet via Bluetooth or Wi-Fi. The bracelet parses the returned response data, executes specific tasks (such as temperature measurement), and broadcasts the execution results via voice.

[0097] Below is the firmware code for the smart bracelet:

[0098] enable_noise_suppression(DB_LEVEL=-25); / / Noise reduction threshold;

[0099] set_sample_rate(16000Hz); / / Complies with GB / T 15279-2018 standard; start_adpcm_compression(bitrate=16kbps); / / Compression algorithm.

[0100] All technical features in this embodiment can be freely combined according to actual needs.

[0101] The above embodiments are preferred implementations of the present invention. In addition, the present invention can be implemented in other ways. Any obvious substitutions without departing from the concept of the present technical solution are within the protection scope of the present invention.

Claims

1. A mobile terminal and cloud interaction system, characterized in that, This includes smart bracelets, mobile terminals, and cloud-based AI service platforms; The smart bracelet has a built-in speaker, microphone, and wireless communication module. The mobile terminal has a built-in wide area network communication module and a short-range communication module. The smart bracelet communicates wirelessly with the mobile terminal, and the mobile terminal communicates wirelessly with the cloud AI service platform.

2. A mobile terminal-cloud communication method as described in claim 1, characterized in that, The communication method includes the following steps: Step 1: When the smart bracelet's microphone detects the wake word, it begins to listen to the user's spoken voice. After the user finishes listening to the spoken voice, the voice data is sent to the mobile terminal. Step 2: The mobile terminal uploads data to the cloud AI service platform using AES-256 encryption. Step 3: After receiving the interaction request from the mobile terminal, the cloud AI service platform verifies the integrity of the data, sorts the correct data by sequence number, returns the failed data, and requests a retransmission. Step 4: The cloud-based AI service platform decrypts the successful encrypted data packets frame by frame, generates a response result, and sends it to the mobile terminal. Finally, the mobile terminal sends the response result to the wristband.

3. The mobile terminal and cloud communication method according to claim 2, wherein the mobile terminal is equipped with an accessibility service and communicates with the APP installed on the mobile terminal in real time. When the APP is detected to be silenced by the system, the accessibility service of the mobile terminal enables Bluetooth or Wi-Fi scanning and connects to the smart bracelet via Bluetooth or Wi-Fi. After the mobile terminal and the smart bracelet establish an accessibility service connection, all voice requests of the smart bracelet are uploaded to the cloud AI service platform by the accessibility service.

4. The mobile terminal-cloud communication method according to claim 2, characterized in that, The cloud-based AI service platform verifies the MAC address of the smart bracelet when receiving data from the mobile terminal.

5. The mobile terminal-cloud communication method according to claim 4, characterized in that, When the mobile terminal uploads data using AES-256 encryption, it dynamically slices the audio data every 300ms and then encrypts each slice independently.

6. The mobile terminal and cloud communication method according to claim 5, characterized in that, The slicing formula is: Frame(i)=Audio[i×4800:(i+1)×4800]; where: sampling rate=16kHz, bit depth=16bit, 300ms=4800 sample points; Encryption process: Key k = HKDF(master key, device ID); IV(i) = HMAC(k, timestamp||i)[0:16]; Ciphertext C(i)=AES-256-GCM(k,IV(i),Frame(i)); Transmission packet = Base64(i||timestamp||C(i)||GCM-Tag).

7. The mobile terminal and cloud communication method according to claim 2, characterized in that, The frame-by-frame decryption includes: A1. First, align the sample points, then dynamically adjust the 50-200ms buffer window to eliminate jitter in the overlapping area. ; A2. Resampling is used for linear prediction to compensate for clock drift, where the clock drift rate is: ; A3. Compensated sampling point mapping: ; in .

8. The mobile terminal-cloud communication method according to claim 7, characterized in that, In step four, the decrypted speech data is assembled and sent to the ASR speech-to-text module for speech processing. Then, a dynamic load balancing architecture and a hybrid recognition model are used to perform semantic processing on the text through an NLP text semantic understanding model. Finally, based on the acoustic model of the Generative Adversarial Network (GAN) and the corresponding speech prosody predictor, TTS speech and specific execution instructions are generated.

9. The mobile terminal-cloud communication method according to claim 8, characterized in that, The speech processing includes: B1. Feature extraction of multimodal recognition architecture: Mel-Frequency Cepstral Coefficients; B2, ASR multi-layer speech recognition ； B3. Zero-shot dialect adaptation based on meta-learning 。 10. The mobile terminal and cloud communication method according to claim 2, characterized in that, The communication method between the mobile terminal's APP and the accessibility service is: AIDL file sharing; the communication method between the mobile terminal's APP and the accessibility service and the smart bracelet is: Bluetooth or Wi-Fi communication; the communication method between the mobile terminal's APP and the accessibility service and the cloud AI service platform is: HTTPS request to the cloud API.

Images