Data processing method and device, electronic equipment and computer readable storage medium

By combining voice and facial image data with smart glasses to perform fine-grained emotion prediction and generate subtitles, and using eye-tracking data to adjust the position, the problem of smart glasses subtitles lacking fine-grained emotion and positional adaptation is solved, thus improving communication efficiency and reading experience.

CN121262437BActive Publication Date: 2026-11-10WUHAN BOBODONG TECHNOLOGY CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202511276403.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-09-08
Publication Date
2026-11-10
Estimated Expiration
2045-09-08

AI Technical Summary

Technical Problem

The subtitles generated by smart glasses lack fine-grained emotion discrimination and cannot adaptively adjust the position of the subtitles, which affects communication efficiency and reading experience.

Method used

By acquiring voice data and facial images, and combining them with graph attention networks to predict emotion categories, fine-grained emotion captions are generated, and the caption positions are adaptively adjusted using eye-tracking data and location information.

Benefits of technology

It achieves accurate presentation of fine-grained emotional subtitles and adaptive adjustment of subtitle position, reduces eye movement load, and improves the subtitle reading experience in multi-person dialogue scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121262437B_ABST
    Figure CN121262437B_ABST
Patent Text Reader

Abstract

The application provides a data processing method and device, electronic equipment and a computer readable storage medium. The method comprises: obtaining voice data and a face image corresponding to a first object; when it is determined that an emotion prediction condition is met based on the voice data and the face image, performing emotion category prediction based on at least one of the voice data and the face image to obtain an emotion prediction result; generating text data based on the voice data, and generating subtitle data based on the emotion prediction result and the text data; obtaining eye movement data corresponding to a second object and position information of the first object relative to the second object; determining a target position of the subtitle data based on the position information and the eye movement data, and displaying the subtitle data at the target position. Through the application, a fine-grained emotion corresponding to the subtitle can be presented, and the position of the subtitle can be adaptively adjusted.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In everyday conversations, business meetings, and educational training, users may need to obtain real-time textual information about the conversation. Especially in noisy environments, or when users wish to avoid disturbing others, smart glasses can generate captions, providing a silent way to access information. For users engaging in cross-border communication or learning a foreign language, smart glasses can translate speech into captions in the target language in real time, helping them overcome language barriers and improve communication efficiency. Hearing-impaired individuals often face obstacles in daily communication; smart glasses can generate captions to provide real-time textual information, enhancing communication skills and engagement.

[0003] In related technologies, subtitles generated by smart glasses lack fine-grained emotional expression and are displayed in a fixed position on the screen, making it impossible to achieve adaptive adjustment of the subtitle position. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium that can present fine-grained emotions corresponding to subtitles and adaptively adjust the position of subtitles.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a data processing method, the method comprising:

[0007] Obtain the voice data and facial image corresponding to the first object;

[0008] When it is determined that the emotion prediction conditions are met based on the voice data and the facial image, the emotion category is predicted based on at least one of the voice data and the facial image to obtain the emotion prediction result.

[0009] Text data is generated based on the speech data, and subtitle data is generated based on the emotion prediction result and the text data;

[0010] Obtain eye-tracking data corresponding to the second object, and position information of the first object relative to the second object;

[0011] Based on the location information and the eye-tracking data, the target location of the subtitle data is determined, and the subtitle data is displayed at the target location.

[0012] This application provides a data processing apparatus, including:

[0013] The first acquisition module is used to acquire the voice data and facial image corresponding to the first object;

[0014] An emotion prediction module is used to predict the emotion category based on at least one of the voice data and the facial image when it is determined that the emotion prediction conditions are met, and to obtain an emotion prediction result.

[0015] The data generation module is used to generate text data based on the voice data, and to generate subtitle data based on the emotion prediction result and the text data;

[0016] The second acquisition module is used to acquire eye-tracking data corresponding to the second object, and position information of the first object relative to the second object;

[0017] The position determination module is used to determine the target position of the subtitle data based on the position information and the eye-tracking data, and to display the subtitle data at the target position.

[0018] This application provides an electronic device, the electronic device comprising:

[0019] Memory is used to store executable instructions or computer programs.

[0020] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method provided in the embodiments of this application.

[0021] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which are executed by a processor to implement the data processing method provided in this application.

[0022] This application provides a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the data processing method provided in this application.

[0023] The embodiments of this application have the following beneficial effects:

[0024] By applying the embodiments of this application, voice data and facial images corresponding to a first object are acquired. When it is determined that the emotion prediction conditions are met based on the voice data and facial images, emotion category prediction is performed based on at least one of the voice data and facial images to obtain an emotion prediction result. This allows for the prediction of the emotion category of subtitles by combining voice data and facial images, achieving fine-grained emotion discrimination of subtitles and improving the accuracy of emotion prediction results. Then, text data is generated based on the voice data, and subtitle data is generated based on the emotion prediction result and text data, enabling subtitles to present fine-grained emotions. Next, eye-tracking data corresponding to a second object and the positional information of the first object relative to the second object are acquired. Then, based on the positional information and eye-tracking data, the target position of the subtitle data is determined, and the subtitle data is displayed at the target position. In this way, by combining eye-tracking information and speaker positioning information, adaptive adjustment of the subtitle position is achieved, reducing the eye movement load and thus improving the subtitle reading experience in multi-person dialogue scenarios. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating the application mode of the data processing method provided in the embodiments of this application;

[0026] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0027] Figure 3A This is a first flowchart illustrating the data processing method provided in an embodiment of this application;

[0028] Figure 3B This is a second flowchart illustrating the data processing method provided in the embodiments of this application;

[0029] Figure 3C This is a schematic diagram of the third process of the data processing method provided in the embodiments of this application.

[0030] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0034] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0035] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0036] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0037] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0038] 1) Graph Attention Network (GAT): This is a graph neural network model based on the attention mechanism. It efficiently processes graph structure data by dynamically learning the importance weights between nodes.

[0039] 2) Differential Privacy Federated Learning: This is a technique that combines differential privacy and federated learning, aiming to protect data privacy while achieving efficient distributed model training.

[0040] This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium that can present fine-grained emotions corresponding to subtitles and adaptively adjust the position of subtitles.

[0041] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of terminals, such as smart wearable devices, laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, in-vehicle terminals, augmented reality (AR) glasses, and smart glasses. The following will describe an exemplary application when the electronic device is implemented as smart glasses.

[0042] See Figure 1 , Figure 1 This is a schematic diagram illustrating the application mode of the data processing method provided in the embodiments of this application, for example. Figure 1 The process involves a first object 200, a second object 300, and a terminal 400. Figure 1 The terminal 400 is shown as an example of smart glasses, and the smart glasses are worn on the second object.

[0043] During the subtitle generation process, terminal 400 acquires the voice data and facial image corresponding to the first object 200. When the emotion prediction conditions are met based on the voice data and facial image, emotion category prediction is performed based on at least one of the voice data and facial image to obtain the emotion prediction result. Text data is generated based on the voice data, and subtitle data is generated based on the emotion prediction result and the text data. Terminal 400 acquires the eye-tracking data corresponding to the second object 300, as well as the position information of the first object 200 relative to the second object 300. Based on the position information and eye-tracking data, the target position of the subtitle data is determined, and the subtitle data is displayed at the target position. Terminal 400 can be a smartphone, augmented reality (AR) glasses, etc., used in scenarios such as online meetings, game live streaming, online education, court recording, real-time dialogue, and accessible cinemas. For example, in an online meeting scenario, terminal 400 moves the subtitle data to the corresponding target position in the video window of each participant based on the determined subtitle data and the target position of the subtitle data. Alternatively, in an accessible cinema setting, the terminal 400 moves the subtitle data to the corresponding target position in the film / television character area based on the determined subtitle data and its target position.

[0044] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be a terminal or a server. Figure 2The illustrated electronic device includes at least one processor 410, a memory 450, and at least one network interface 420. The various components of the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0045] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0046] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0047] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0048] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0049] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0050] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, WiFi, and Universal Serial Bus (USB).

[0051] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A data processing device 455 stored in memory 450 is shown. It may be software in the form of programs and plug-ins, including the following software modules: a first acquisition module 4551, a sentiment prediction module 4552, a data generation module 4553, a second acquisition module 4554, and a location determination module 4555. These modules are logically related and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0052] The data processing method provided in this application will be described in conjunction with exemplary applications and implementations of the terminal provided in the embodiments of this application.

[0053] The following describes the data processing method provided in the embodiments of this application. For example, in order to facilitate understanding of the data processing method provided in the embodiments of this application, the example is described using the application in the scenario of emotion-adaptive real-time caption generation and layout.

[0054] As mentioned above, the electronic device implementing the data processing method of the embodiments of this application can be a terminal. See also Figure 3A , Figure 3A This is a first flowchart illustrating the data processing method provided in the embodiments of this application, which will be combined with... Figure 3A The steps shown are explained.

[0055] In step 301, the voice data and facial image corresponding to the first object are obtained.

[0056] Here, the first object is the object that initiates the real-time dialogue. The terminal can collect the voice data corresponding to the first object through the built-in microphone array at a frequency of 16kHz and a frame rate of 20 milliseconds, and can collect the facial image corresponding to the first object through the front-facing camera at a rate of 30 frames per second.

[0057] In step 302, when it is determined that the emotion prediction conditions are met based on the speech data and facial image, the emotion category is predicted based on at least one of the speech data and facial image to obtain the emotion prediction result.

[0058] Here, the signal-to-noise ratio of the speech data and the occlusion rate of the facial image are used to determine whether the emotion prediction conditions are met. When the emotion prediction conditions are met based on the signal-to-noise ratio of the speech data and the occlusion rate of the facial image, the emotion category is predicted based on at least one of the speech data and the facial image to obtain the emotion prediction result. The emotion categories include 12 categories such as joy, calmness, surprise, sadness, anger, disgust, fear, doubt, sarcasm, embarrassment, fatigue, and excitement.

[0059] In some embodiments, determining that the emotion prediction condition is met can be achieved through the following steps: determining the signal-to-noise ratio of the speech data and the occlusion rate of the facial image; when the signal-to-noise ratio of the speech data is greater than or equal to a first preset threshold, or the occlusion rate of the facial image is less than or equal to a second preset threshold, it is determined that the emotion prediction condition is met.

[0060] Here, the signal-to-noise ratio (SNR) refers to the ratio of valid speech signal to noise signal in speech data, reflecting the clarity and intelligibility of the speech data. Speech data with a high SNR is easier to recognize and understand, contributing to improved speech recognition accuracy. Speech data with a low SNR may lead to inaccurate speech recognition results, affecting the reliability of subsequent speech data processing. The effective speech signal is extracted from the speech data using speech activity detection technology. The noise signal is separated from the speech data using frequency domain analysis methods. The power of the effective speech signal and the power of the noise signal are calculated; the power is usually expressed as the square of the signal amplitude. The SNR of the speech data is obtained by performing an inverse logarithmic operation on the effective speech signal power and the noise signal power.

[0061] Occlusion rate refers to the proportion of facial regions in a facial image that are obscured or invisible by occlusions, such as hair, hands, glasses, hats, etc. A high occlusion rate reduces the detectability of facial images and affects the accuracy of expression recognition. Facial images with low occlusion rates help improve the accuracy of feature extraction and enhance facial image processing. Facial detection algorithms (such as deep learning models like multi-task convolutional neural networks) are used to identify facial regions in the image. Object detection algorithms are used to detect occlusions (such as hands, hair, etc.) in the image, determining the total area of ​​the facial region and the occlusion area of ​​the occluded objects within the facial region. The ratio of the occluded area to the total area of ​​the facial region is defined as the occlusion rate of the facial image.

[0062] The first preset threshold is a preset signal-to-noise ratio (SNR) threshold, and the second preset threshold is a preset occlusion rate threshold. When the SNR of the speech data is greater than or equal to the first preset threshold, or the occlusion rate of the facial image is less than or equal to the second preset threshold, the emotion prediction condition is determined to be met. For example, if the first preset threshold is 10 dB and the second preset threshold is 40%, then when the SNR of the speech data is greater than or equal to 10 dB, or the occlusion rate of the facial image is less than or equal to 40%, the emotion prediction condition is determined to be met.

[0063] In some embodiments, see Figure 3B , Figure 3B This is a second flowchart illustrating the data processing method provided in the embodiments of this application. Figure 3A Step 302 shown can be achieved through... Figure 3B Steps 3021 to 3023 are implemented, and will be explained in detail below.

[0064] In step 3021, when the signal-to-noise ratio of the speech data is greater than or equal to the first preset threshold and the occlusion rate of the facial image is less than or equal to the second preset threshold, emotion category prediction is performed based on the speech data and the facial image to obtain the emotion prediction result.

[0065] Here, when the signal-to-noise ratio of the speech data is greater than or equal to the first preset threshold and the occlusion rate of the facial image is less than or equal to the second preset threshold, the speech data and the facial image simultaneously meet the emotion prediction conditions. Emotion category prediction can be performed simultaneously based on the speech data and the facial image to obtain the emotion prediction result.

[0066] In some embodiments, emotion category prediction based on speech data and facial images can be achieved through the following steps: extracting features from the speech data and facial images respectively to obtain audio features and facial features; fusing the audio features and facial features to obtain fused features; using a graph attention network to predict the emotion category on the fused features to obtain an emotion category probability distribution; and mapping the emotion intensity based on the emotion category probability distribution to obtain the emotion prediction result.

[0067] Here, a convolutional neural network is used to extract features from the speech data to obtain audio features, including fundamental frequency, energy, zero-crossing rate, speech rate, spectral features, and prosodic features such as 512-dimensional acoustic embedding. The lightweight face recognition model MobileFace2 is used to extract features from the facial images to obtain facial features, including intensity features of 17 Action Units (AUs).

[0068] The timestamps of the speech data and facial images are aligned using interpolation or synchronization techniques, with each time window lasting 200 milliseconds. Within multiple consecutive time windows, audio features are identified as prosodic nodes, and facial features are identified as facial nodes. Within the same time window, the mutual information between prosodic nodes and facial nodes, i.e., the correlation between them, is determined. Prosodic nodes and facial nodes with mutual information exceeding a threshold are connected to achieve the fusion of audio and facial features, resulting in a fused feature.

[0069] The fused features are input into the fully connected layer of a Graph Attention Network (GAT) to predict the emotion category, resulting in an emotion category probability distribution for 12 emotion categories. Based on this distribution, the emotion category with the highest probability is determined as the target emotion category. Each emotion category corresponds to a preset emotion intensity, which includes five levels: extremely negative, negative, neutral, positive, and extremely positive. Based on the preset emotion intensity corresponding to the target emotion category, an emotion intensity mapping is performed on the target emotion category to obtain the emotion prediction result.

[0070] In this embodiment, emotion category prediction is performed based on voice data and facial images to obtain emotion prediction results. Voice data can reflect the voice information of the speaker, and facial images can provide visual emotional information. Combining voice data and facial images to perform multimodal emotion category prediction improves the accuracy of emotion prediction results.

[0071] Continue to refer to Figure 3B In step 3022, when the signal-to-noise ratio of the speech data is less than the first preset threshold and the occlusion rate of the facial image is less than or equal to the second preset threshold, emotion category prediction is performed based on the facial image to obtain the emotion prediction result.

[0072] Here, when the signal-to-noise ratio of the speech data is less than the first preset threshold and the occlusion rate of the facial image is less than or equal to the second preset threshold, that is, when the facial image meets the emotion prediction conditions, emotion category prediction can be performed based on the facial image to obtain the emotion prediction result.

[0073] In some embodiments, facial regions in a facial image are located using a facial landmark detection algorithm. Features are extracted from the facial regions using the feature extraction layer in the lightweight face recognition model MobileFace2, resulting in facial features, which include 17 Action Unit (AU) intensity features. The facial features are then mapped to emotion categories using the fully connected layer in the lightweight face recognition model to obtain an emotion prediction result.

[0074] Continue to refer to Figure 3B In step 3023, when the signal-to-noise ratio of the speech data is greater than or equal to the first preset threshold and the occlusion rate of the facial image is greater than the second preset threshold, emotion category prediction is performed based on the speech data to obtain the emotion prediction result.

[0075] Here, when the signal-to-noise ratio of the speech data is greater than or equal to the first preset threshold and the occlusion rate of the facial image is greater than the second preset threshold, that is, when the speech data meets the emotion prediction conditions, emotion category prediction can be performed based on the speech data to obtain the emotion prediction result.

[0076] In some embodiments, the speech data is segmented into multiple time windows. Within each time window, a convolutional neural network in the emotion recognition model is used to extract features from the speech data, resulting in audio features. These audio features include fundamental frequency, energy, zero-crossing rate, speech rate, spectral features, and prosodic features such as 512-dimensional acoustic embedding. The audio features are then mapped to emotion categories through a fully connected layer in the emotion recognition model to obtain emotion prediction results.

[0077] In this embodiment, the signal-to-noise ratio of the speech data and the occlusion rate of the facial image are used to determine whether the emotion prediction conditions are met. This avoids the impact of low-quality speech data and facial images on the accuracy of emotion prediction results. When the signal-to-noise ratio of the speech data or the occlusion rate of the facial image does not meet the preset threshold, emotion category prediction is performed based on either the speech data or the facial image. This automatically switches to unimodal emotion prediction, improving the robustness of emotion prediction in noisy environments and scenarios with missing facial information.

[0078] Continue to refer to Figure 3A In step 303, text data is generated based on the speech data, and subtitle data is generated based on the emotion prediction results and the text data.

[0079] Here, a lightweight 4-bit quantized speech recognition model, Whisper-tiny, can be loaded. This model is then used to transcribe the speech data into text. The sentiment prediction results are then combined with the text data to obtain caption data.

[0080] In some embodiments, generating subtitle data based on emotion prediction results and text data can be achieved through the following steps: determining the target display format and target subtitle color of the subtitle border corresponding to the emotion prediction result based on the emotion category corresponding to the emotion prediction result, wherein each emotion category corresponds to a preset display format and preset subtitle color of the subtitle border; rendering the text data using the target display format and target subtitle color of the subtitle border to obtain the subtitle data.

[0081] Here, the subtitle data can be displayed in the form of semi-transparent bubbles. Each emotion category corresponds to a preset display format and preset subtitle color for the subtitle border. The display format of the subtitle border includes the subtitle border color, the subtitle border line, and the flashing effect of the subtitle border. The subtitle color is the font color in the subtitle data.

[0082] Based on the emotion category corresponding to the emotion prediction result, the display format and color of the subtitle border corresponding to that emotion prediction result are determined, i.e., the target display format and target subtitle color of the subtitle border. The emotion prediction result, text data, and timestamp are combined to obtain triplet data. In response to the data rendering command, the triplet data is rendered using the target display format and target subtitle color of the subtitle border to obtain the subtitle data.

[0083] For example, if the emotion prediction result corresponds to the emotion category "anger", the corresponding caption data will be displayed as follows: the caption border color is red, the caption border line is thick, the caption border blinking effect is slightly blinking, and the caption color is red.

[0084] For example, the emotion category corresponding to the emotion prediction result is "question". The corresponding caption data is displayed as follows: the caption border color is blue, the caption border line is a dashed line, the blinking effect of the caption border is not blinking, and the caption color is blue.

[0085] In this embodiment, the target display format and target subtitle color of the subtitle border corresponding to the emotion prediction result are determined, and the text data is rendered using the target display format and target subtitle color of the subtitle border to obtain subtitle data. This can supplement the emotional information of the subtitles and realize the simultaneous presentation of the content and fine-grained emotions of the subtitles. This not only improves the richness of the subtitle display, but also provides a multi-dimensional understanding channel for hearing-impaired assistance and cross-language communication, thereby improving communication efficiency.

[0086] In some embodiments, the following steps may also be performed: when the vibration mode is detected to be in the active state, the target vibration frequency corresponding to the terminal is determined based on the emotion category corresponding to the subtitle data, and vibration is performed based on the target vibration frequency, wherein each emotion category corresponds to a preset vibration frequency; when the voice reading mode is detected to be in the active state, the subtitle voice corresponding to the subtitle data is output.

[0087] Here, each emotion category corresponds to a preset vibration frequency, which is a pre-set frequency at which the terminal vibrates. When the vibration mode is detected as enabled, the target vibration frequency is determined based on the emotion category corresponding to the subtitle data, and vibration is then performed based on the target vibration frequency. For example, if the emotion category corresponding to the subtitle data is "anger," and the preset vibration frequency for this emotion category is intense vibration, then the target vibration frequency for the terminal is determined to be intense vibration, and intense vibration is performed based on the target vibration frequency.

[0088] When the voice reading mode is detected to be enabled, the text corresponding to the subtitle data is converted into subtitle speech and then output.

[0089] In this embodiment, based on the emotion category corresponding to the subtitle data, the target vibration frequency corresponding to the terminal is determined, and vibration is performed based on the target vibration frequency. The subtitle voice corresponding to the subtitle data is also output. The subtitle content can be displayed in a visual way through vibration mode and voice reading mode, making it easier for visually impaired or colorblind users to understand the content and tone of the subtitle data.

[0090] Continue to refer to Figure 3A In step 304, eye-tracking data corresponding to the second object and position information of the first object relative to the second object are obtained.

[0091] Here, the second object can be someone who engages in dialogue or communication with the first object, or someone who listens to the first object speak. Eye movement data corresponding to the second object is collected using a built-in eye-tracking sensor. This data includes the eye's center position, eye rotation angle, and fixation time. The direction of the second object is measured using a built-in inertial sensor to obtain the position information of the first object relative to the second object.

[0092] Continue to refer to Figure 3A In step 305, the target position of the subtitle data is determined based on the location information and eye-tracking data, and the subtitle data is displayed at the target position.

[0093] Here, based on location information and eye movement data, the target position of the subtitle data is determined according to the principle of minimum eye saccades, so that the subtitle data can be displayed at the target position. The principle of minimum eye saccades is to minimize the eye movement load of the second object when viewing the subtitle data.

[0094] In some embodiments, eye-tracking data includes the center position of the eyeball, see [link to relevant documentation]. Figure 3C , Figure 3C This is a schematic diagram of the third process of the data processing method provided in the embodiments of this application. Figure 3A In step 305 shown, "determine the target position of the caption data based on location information and eye-tracking data," which can be achieved through... Figure 3C Steps 3051 to 3054 are implemented, and will be explained in detail below.

[0095] In step 3051, the unit vector pointing from the center of the eyeball to the preset gaze point is determined as the gaze vector.

[0096] Here, the preset fixation point is a pre-set point used for gazing at the second object. Vector operations are performed based on the coordinates of the eye center and the coordinates of the preset fixation point to obtain the vector operation result. The vector operation result is then normalized to obtain the unit vector pointing from the eye center to the preset fixation point, which is the gaze vector.

[0097] In step 3052, at least one preset position for displaying subtitle data is obtained, and for each preset position, a first distance between the gaze vector and the preset position is determined, and a second distance between the position information and the preset position is determined.

[0098] Here, at least one preset position for displaying subtitle data can be determined based on viewing preferences. The preset position can be the top, bottom, center, left, or right side of the screen. For each preset position, the difference between the gaze vector and the preset position is defined as a first distance. The difference between the position information and the preset position is defined as a second distance. For example, the first distance can be represented as |GP|, and the second distance can be represented as |SP|, where G represents the gaze vector, P represents the preset position, and S represents the position information.

[0099] In step 3053, the first distance and the second distance are weighted and summed using the first preset weight and the second preset weight to obtain the summation result.

[0100] Here, when the angle between the gaze vector and the position information is less than an angle threshold, the first preset weight is set to a first value; when the angle between the gaze vector and the position information is greater than or equal to the angle threshold, the first preset weight is set to a second value. The sum of the first preset weight and the second preset weight is 1. The first distance and the second distance are weighted and summed to obtain the summation result.

[0101] For example, with an angle threshold of 15°, when the angle between the gaze vector and the position information is less than 15°, the first preset weight is set to a first value of 0.7, so the first preset weight is 0.7 and the second preset weight is 0.3. When the angle between the gaze vector and the position information is greater than or equal to 15°, the first preset weight is set to a second value of 0.9, so the first preset weight is 0.9 and the second preset weight is 0.1. The summation result can be expressed as w1|GP|+w2|SP|, where G represents the gaze vector, P represents the preset position, S represents the position information, w1 represents the first preset weight, and w2 represents the second preset weight.

[0102] In step 3054, the preset position corresponding to the minimum summation result is determined as the target position of the subtitle data.

[0103] Here, a preset position is determined that minimizes the summation result, and this position is set as the target position for the subtitle data. For example, the target position for the subtitle data can be determined using formula (1):

[0104] P*=argmin(w1|GP|+w2|SP|) (1)

[0105] Where P* represents the target position of the subtitle data, G represents the gaze vector, P represents the preset position, S represents the position information, w1 represents the first preset weight, and w2 represents the second preset weight.

[0106] In this embodiment, the target position of the subtitle data is determined by combining the gaze vector, the first distance between the preset position used to display subtitle data, the position information of the first object relative to the second object, and the second distance between the preset positions. This enables adaptive adjustment of the subtitle position, reduces the load of eye movement, and improves the subtitle reading experience in multi-person dialogue scenarios.

[0107] In some embodiments, when the position information of the first object relative to the second object changes, the target position of the subtitle data is moved, which can be achieved through the following steps: when the position information of the first object relative to the second object changes, update the position information; based on the update position information and eye-tracking data, determine the update target position of the subtitle data; when the deviation angle between the target position of the subtitle data and the update target position of the subtitle data is greater than a preset angle, determine the third distance between the target position of the subtitle data and the update target position of the subtitle data; based on the third distance and a preset adjustment coefficient, move the target position of the subtitle data to the update target position of the subtitle data.

[0108] Here, when the position information of the first object relative to the second object changes, the direction of the second object is remeasured using the built-in inertial sensor to obtain the updated position information of the first object relative to the second object.

[0109] The unit vector pointing from the eye center position in the eye-tracking data to a preset gaze point is defined as the gaze vector. For each preset position used to display subtitle data, a first distance between the gaze vector and the preset position is determined, and a third distance between the updated position information and the preset position is determined. The first and third distances are weighted and summed using the first and third preset weights to obtain the sum. The preset position corresponding to the minimum sum is determined as the target position for updating the subtitle data.

[0110] When the deviation angle between the target position and the updated target position of the subtitle data is greater than a preset angle, the difference between the target position and the updated target position of the subtitle data is determined as the third distance. The third distance is multiplied by a preset adjustment coefficient to obtain the position adjustment amount, and this position adjustment amount is summed with the target position of the subtitle data to obtain the subtitle migration position. Using this migration position, the target position of the subtitle data is gradually moved to the updated target position. For example, the migration position of the subtitle can be determined using formula (2):

[0111] P_t=P_old+α(P_new-P_old) (2)

[0112] Where P_t represents the migration position of the subtitle, P_old represents the target position of the subtitle data, P_new represents the update target position of the subtitle data, and α represents the preset adjustment coefficient.

[0113] In this embodiment, based on the target position of the subtitle data, the distance between the target position of the subtitle data update and the preset adjustment coefficient, the target position of the subtitle data is moved to the target position of the subtitle data update, so as to realize the gradual migration of the subtitle position and avoid the subtitle jumping instantaneously, which would affect the subtitle reading experience.

[0114] In some embodiments, the following steps may also be performed: in response to an adjustment instruction for the subtitle font, the font size of the subtitle data is adjusted to obtain the adjusted subtitle data; or in response to a lock instruction for the subtitle position, the target position of the subtitle data is fixed and the subtitle data is displayed at the target position.

[0115] Here, in response to an adjustment command for the subtitle font (which could be a two-finger zoom touch operation applied by the second object on the screen), the font size in the subtitle data is adjusted to obtain the adjusted subtitle data. In response to a lock command for the subtitle position (which could be a "clench and unclench" gesture applied by the second object in front of the screen), the target position of the subtitle data is fixed, ensuring that the target position remains unchanged and the subtitle data is displayed at the fixed target position.

[0116] In this embodiment, in response to an adjustment command for the subtitle font, the font size of the subtitle data is adjusted; in response to a lock command for the subtitle position, the target position of the subtitle data is fixed. This allows for real-time control of the font and position of the subtitles, significantly improving the readability, aesthetics, and reading experience of the subtitles.

[0117] In some embodiments, during the training of the emotion recognition model, local speech data and facial images are used to train the model. Low-Rank Adaptation (LoRA) is then used locally to fine-tune the model, calculating its gradient. The gradient with added Gaussian noise is uploaded to the cloud, where it undergoes noise trimming and aggregation to obtain an optimized emotion recognition model through differential privacy federated learning. This optimized model is then distributed back to the local machine. Differential privacy is a mathematical framework used to ensure that sensitive individual information is not leaked during data analysis; its core idea is to add an appropriate amount of noise to the data. Federated learning is a distributed machine learning technique, and differential privacy federated learning combines differential privacy and federated learning to achieve efficient distributed model training while protecting data privacy.

[0118] In some embodiments, the data processing method provided in this application can be applied to the field of game live streaming. Voice data and facial images corresponding to a first object are acquired on a game live streaming platform. When it is determined that the emotion prediction conditions are met based on the voice data and facial images, emotion category prediction is performed based on at least one of the voice data and facial images to obtain an emotion prediction result. This allows for the prediction of the emotion category of subtitles by combining voice data and facial images, achieving fine-grained emotion discrimination of subtitles and improving the accuracy of emotion prediction results. Then, text data is generated based on the voice data, and subtitle data is generated based on the emotion prediction result and text data, enabling subtitles to present fine-grained emotions. Next, eye-tracking data corresponding to a second object and the positional information of the first object relative to the second object are acquired. Then, based on the positional information and eye-tracking data, the target position of the subtitle data is determined, and the subtitle data is displayed at the target position. Thus, by combining eye-tracking information and the speaker's location information, adaptive adjustment of the subtitle position is achieved, reducing the eye movement load and improving the subtitle reading experience in multiplayer game scenarios.

[0119] The following will describe an exemplary application of the data processing method provided in the embodiments of this application in the scenario of emotion-adaptive real-time caption generation and layout.

[0120] In related technologies, subtitles can only indicate "laughter" and "applause" with emoticons, failing to distinguish fine-grained emotions such as "anger," "sarcasm," and "hesitation" within the subtitles. This lack of fine-grained emotion discrimination affects the understanding of subtitles for hearing-impaired or foreign language learners. Displaying subtitles in fixed positions on the screen, such as at the bottom or center, prevents adaptive adjustment of the subtitle position. This forces users to constantly swivel between the speaker and the subtitle during multi-person conversations or when looking down at the screen, easily leading to information loss and visual fatigue. Relying solely on the speaker's voice features or facial information to identify the emotion category of subtitles significantly increases the error rate of emotion discrimination in noisy environments, at long distances, and in scenarios where facial information is lacking, and lacks alternative paths for emotion inference.

[0121] Traditional Augmented Reality (AR) captioning glasses upload audio to the cloud for automatic transcription, resulting in an average round-trip latency of 400–800 milliseconds. Furthermore, the call content and facial images are publicly displayed on the server side, raising data compliance risks. Additionally, misalignment between the captions and the speaker makes it difficult to identify the correct speaker, reducing the captions' usability.

[0122] This application proposes a data processing method to address the problems existing in related technologies, which includes the following improvements compared to related technologies:

[0123] In mobile terminals, cross-modal fusion based on speech prosody and facial expressions is used to accurately identify the emotion category of subtitles. In noisy or occluded environments, it automatically degenerates into unimodal emotion inference to supplement the emotional information of the subtitles, and can simultaneously present the content and tone of the subtitles. By combining eye tracking and the localization of the speaker (the first object in the above embodiment), the subtitles are accurately bound to the position of the speaker, reducing eye movement and improving the subtitle reading experience in multi-person dialogue scenarios. Differential privacy federated learning is used to optimize the emotion recognition model, taking into account data compliance and real-time performance, and realizing the synergy of technical links such as multimodal emotion recognition, eye-driven dynamic subtitle layout, and low-latency inference on the mobile terminal side.

[0124] At the product level, based on terminals such as smartphones, tablets, computer-based video conferencing, and AR subtitle glasses equipped with forward-facing cameras and eye-tracking sensors, subtitle bubbles are superimposed in real time near the user's (the second object in the above embodiment) line of sight, and the speaker's emotions are presented synchronously through multiple channels such as color, icons, and vibration. The subtitle position adaptively adjusts according to the user's gaze point and the speaker's location, significantly reducing the eye movement load for hearing-impaired individuals and in noisy environments.

[0125] At the algorithm level, the mobile terminal simultaneously extracts speech prosodic features (fundamental frequency, energy, and speech rate) and facial expression action units, and uses a graph attention network to perform cross-modal fusion of the two, outputting emotion labels and inserting subtitle streams. When the speech signal-to-noise ratio is low or the facial occlusion rate is high, it automatically degenerates into single-modal emotion inference to ensure the robustness of emotion recognition. The speech transcription uses a lightweight speech recognition model Whisper-tiny with 4-bit quantization, and the emotion recognition model uses low-rank adaptation (LoRA) fine-tuning. The average latency on the mobile terminal is less than or equal to 150 milliseconds, and the power consumption increase is less than 1 watt.

[0126] In this embodiment, the emotion recognition model is fine-tuned locally using differential privacy federated learning. The model is trained using local data, and its gradient is calculated. The gradient with added Gaussian noise is uploaded to the cloud. The cloud aggregates the noisy gradients to optimize the emotion recognition model and then distributes the optimized model, thus balancing dialect adaptation and data compliance.

[0127] In this embodiment, the emotion tagging of the subtitles adopts a "text-emotion-rendering instruction" triple data structure. This data structure is backward compatible with the Web Video Text Tracks (WebVTT) standard and the Accessible Rich Internet Applications (ARIA) standard, making it easy to integrate into scenarios such as conferencing software, game streaming, and accessible cinemas. The system also provides a Software Development Kit (SDK) for third-party applications to call the application programming interface (API) of real-time subtitles and emotions, achieving a consistent cross-platform experience.

[0128] The following section uses a smartphone and AR glasses as examples to explain the human-computer interaction process for real-time caption generation and layout.

[0129] When a user launches an application (App) on their smartphone for the first time, a two-step guided interface appears in the App: "Microphone and Camera Authorization" and "Gaze / Speaker Calibration." In the "Microphone and Camera Authorization" guided interface, the system prompts that audio and video data for caption generation and emotion recognition needs to be collected and processed locally. The user can choose the "On-device Processing Only" or "Allow Federated Learning" option. In the "Gaze / Speaker Calibration" guided interface, three random points appear on the screen for the user to gaze at, establishing the user's gaze vector. The user is then prompted to turn their head and read two short phrases to establish a speaker spatial model. The entire process is completed within 30 seconds.

[0130] The "Real-time Captions" switch is displayed at the bottom center of the app interface. A short press of the "Real-time Captions" switch will start listening to the speaker's voice. The pickup level is displayed in the status bar at the top of the app interface. If the system detects pure media audio (such as songs played by a music player or audio played by a short video app, i.e., the speaker's voice not captured by the microphone), it will not upload audio features or gradients of the emotion recognition model, and will not use non-conversational audio to train the emotion recognition model. A long press of the "Real-time Captions" switch will take you to the multilingual speech-to-text settings, supporting automatic detection or manual selection of bilingual captions. Pulling down the menu on the app interface allows you to switch to the "Emotion Marking Only" lightweight mode (in this mode, speech-to-text is still performed, but the transcribed text is not rendered; only emotion mark metadata is retained, using color / small icons to mark the emotion at the speaker's location, and the transcribed text is immediately discarded), to adapt to low battery scenarios.

[0131] Once a speaker is detected, subtitles appear as semi-transparent bubbles on the speaker's shoulder. The color and border style of the subtitle bubble depend on the emotion tag. For example, the subtitle bubble corresponding to the emotion tag "anger" displays a thick red border with a slight flicker, the subtitle bubble corresponding to the emotion tag "question" displays a blue dashed border, and the subtitle bubble corresponding to the emotion tag "neutral" displays a gray-white border. If the user's gaze deviates from the speaker's position by more than a preset angle, the system smoothly moves the subtitle bubble at an interpolation speed of 200 milliseconds to ensure that the subtitle is always within the user's 3-5 degree field of vision. If multiple speakers are speaking simultaneously, the subtitle bubbles are placed on top in chronological order, and a list of subtitle bubbles appears in the lower left corner of the app interface for the user to manually lock.

[0132] Users can swipe down or double-tap a subtitle on the app interface to expand the full text history (a full log of subtitles since the start of the current conversation, including text, emotion tags, and timestamps), making it easy to review missed subtitle content. Double-tapping the subtitle again collapses the full text history. Users can adjust the font size of the subtitles by pinching and zooming with two fingers on the app interface, and swiping left quickly mutes the speaker for 3 minutes. In AR glasses applications, users can lock the current subtitle position using a "clench and release fist" gesture to prevent subtitles from jumping in noisy environments. For users with poor vision or color blindness, the subtitles include optional vibration prompts; for example, the higher the emotion level, the faster the vibration frequency. In call mode, a reverse voice output (reading the subtitle content aloud) can be enabled to help users with both visual and hearing impairments understand the subtitle content.

[0133] The app settings interface offers four toggle options: "Local Mode / Federated Learning Mode," "Emotion Granularity," "Subtitle Layout Strategy," and "Data Cleanup." The "Local Mode / Federated Learning Mode" option defaults to Local Mode. When Federated Learning Mode is enabled, the system only uploads gradient information from the emotion recognition model when connected to the network and charging. The "Emotion Granularity" setting offers options for either 5 basic categories or 12 extended categories; higher granularity results in longer computation time. The "Subtitle Layout Strategy" setting offers options for Automatic Strategy, Fixed Bottom Bar Strategy, and Fixed Center Strategy. Under the Automatic Strategy, the system calculates subtitle positions in real-time using the minimum saccade formula. "Data Cleanup" clears locally cached audio data and intermediate emotion tensors with a single click.

[0134] When the system detects a low signal-to-noise ratio in the speech data or a high rate of facial occlusion in the facial image, a yellow "Low Quality" marker appears in the app's status bar, and it automatically switches to single-modal emotion recognition. If no effective emotion is output for 5 consecutive seconds, only plain text subtitles are retained. Users can manually switch to "Plain Text Mode" to save energy; the switch button is located in the upper right corner of the app's main interface.

[0135] This application's embodiments can be applied to scenarios such as online meetings, street conversations, and accessible cinemas. In online meeting scenarios, through a video conferencing plugin, subtitle bubbles are bound to the lower left corner of each participant's video window; hovering the mouse over a subtitle bubble displays the full subtitle content. In real-time conversation scenarios, by wearing AR glasses, the subtitles on the AR glasses move with the speaker's position; if the user looks down, the subtitles automatically move to the lower right corner of the glasses and shrink to a single scrolling line. In accessible cinema scenarios, by placing a smartphone horizontally on the armrest of a seat, the subtitles move left and right with the character's position, and vibration alerts are triggered when emotional intensity is high.

[0136] The overall technical architecture provided in the embodiments of this application will be described below. The system adds a real-time subtitle service layer to the mobile terminal. The real-time subtitle service layer includes seven functional layers connected in series. The seven functional layers are a multi-source acquisition layer, a single-modal feature extraction layer, a cross-modal fusion and emotion reasoning layer, a speech transcription layer, a subtitle synthesis layer, a dynamic layout engine layer, and a rendering and interaction layer.

[0137] In the multi-source acquisition layer, audio is acquired at a frequency of 16kHz and a frame rate of 20 milliseconds via a microphone array; facial images are captured at a rate of 30 frames per second via a front-facing camera; the gaze vector is updated every 8 milliseconds via an eye-tracking sensor; and inertial measurement unit (IMU) data is used to locate the speaker's position. The gaze vector can be determined in several ways. For AR glasses or smartphone front-facing cameras with auxiliary lighting, the vector from the camera's optical center to the pupil center is calculated. For single-camera devices, a convolutional neural network or regression model can be used to estimate the gaze vector based on head pose (IMU measurement) and the user's historical interaction positions.

[0138] In the single-modal feature extraction layer, prosodic features such as fundamental frequency, energy, zero-crossing rate, speech rate, spectral features, and 512-dimensional acoustic embedding of the audio data are extracted (audio features in the above embodiments). Using the lightweight face recognition model MobileFace2, the intensity features of 17 facial action units (AUs) of the video data (facial features in the above embodiments) are extracted, and then the timestamps of the two types of features are unified.

[0139] In the cross-modal fusion and emotion inference layer, the system aggregates the intensity features of 17 facial expression units and the 6-dimensional prosody feature in 200-millisecond sliding windows, forming 23-dimensional feature nodes within each window. A cross-modal emotion graph is constructed using temporal adjacency and cross-modal mutual information. This graph is then input into a two-layer graph attention network (GAT) to obtain a 12-dimensional emotion vector. The 12-dimensional emotion vector is normalized using the normalized exponential function softmax, yielding probability distributions for 12 emotion categories: joy, calm, surprise, sadness, anger, disgust, fear, confusion, sarcasm, embarrassment, fatigue, and excitement. The emotion category with the highest probability is used as the final emotion label, and based on the quantile of the highest probability value, the emotion category is adaptively mapped to five emotion intensity levels: extremely negative, negative, neutral, positive, and extremely positive.

[0140] In the speech transcription layer, incremental text is continuously generated using the lightweight speech recognition model Whisper-tiny with 4-bit quantization.

[0141] In the subtitle compositing layer, text, emotion, and timestamp are encapsulated into triples, and rendering instructions such as color, border, and vibration are added.

[0142] In the dynamic layout engine layer, the subtitle anchor point is calculated by combining the user's gaze vector and the direction of the speaking object (the position information of the first object relative to the second object in the above embodiment) according to the principle of minimum eye saccades. The subtitle anchor point is the floating position coordinate of the subtitle displayed on the device screen or in the field of view of the glasses. The principle of minimum eye saccades is to reduce the user's eye movement load and dynamically adjust the subtitle position, with the goal of minimizing the field of view angle between the user's gaze vector and the subtitle anchor point, while keeping the subtitle anchor point as close as possible to the direction of the speaking object. For example, the target position of the subtitle can be determined by formula (1):

[0143] P*=argmin(w1|GP|+w2|SP|) (1)

[0144] Where P* represents the target position of the subtitle, G represents the gaze vector, P represents the preset position of the subtitle, S represents the direction of the speaking object, w1 represents the first preset weight, and w2 represents the second preset weight.

[0145] The system obtains the current user gaze vector and the direction of the speaking object every 50 milliseconds. When the angle between the user gaze vector and the direction of the speaking object is less than 15°, the first preset weight w1 is set to 0.7, prioritizing proximity to the speaking object; when the angle is greater than 15°, the first preset weight w1 is set to 0.9, prioritizing proximity to the user gaze direction. The target position for updating the subtitles is then subjected to exponential smoothing before being input into the rendering queue to update the rendering queue.

[0146] If the angle difference between the target position of the updated subtitle and the target position of the subtitle before the update exceeds 6°, a smoothing algorithm is enabled to perform a gradual migration. For example, the migration position of the subtitle can be determined by formula (2):

[0147] P_t=P_old+α(P_new-P_old) (2)

[0148] Where P_t represents the migration position of the subtitle, P_old represents the target position of the subtitle before the update, P_new represents the target position of the subtitle update, and α represents the preset adjustment coefficient.

[0149] α can be 0.15, and the subtitle position is updated iteratively once per frame, completing the subtitle position migration in about 150 milliseconds, thus avoiding the subtitle jumping suddenly and affecting the user's subtitle reading experience.

[0150] In the rendering and interaction layer, subtitles are drawn with a refresh cycle of 16 milliseconds per frame, and gesture or touch operations are received. Users can quickly adjust the font and position of subtitles, hide or highlight subtitles of a specific speaker, or lock the current subtitle position in the AR scene. Gestures or touch operations are captured through the front-end interface to update subtitle rendering parameters in the local state machine or set subtitle rendering parameters in the local model, without uploading private data such as audio and video to the server.

[0151] In this embodiment, feature extraction of audio and video frames takes approximately 2 milliseconds, and unifying the timestamps of the two types of features takes 1 millisecond. Emotion inference takes 8 milliseconds, incremental speech transcription takes an average of 40 milliseconds, subtitle synthesis and layout takes 3 milliseconds, subtitle rendering takes less than 16 milliseconds, and the total terminal-to-end latency is between 70 and 150 milliseconds, varying with the audio's speaking speed.

[0152] In this embodiment, emotion recognition is achieved by fusing speech and facial images using a local graph attention network, enabling stable emotion discrimination even in noisy or occluded environments. The subtitles employ a "text-emotion-rendering instruction" triple structure, laying the foundation for multi-channel presentation and standard interfaces. The subtitle layout algorithm combines both gaze direction and speaker direction to dynamically adjust the subtitle position, reducing saccade distance by 30%–40%. The entire inference process is completed on the mobile terminal, and the emotion recognition model is updated using differential privacy federated learning, satisfying both the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), while also addressing cloud latency of 400–800 milliseconds.

[0153] In this embodiment, the network topology computing modes include a fully edge-side mode, an edge-cloud hybrid mode, and a near-edge mode. The fully edge-side mode, with a total model size of approximately 30MB, runs on the Neural Network Processing Unit (NPU) of a smartphone or glasses, suitable for privacy-sensitive scenarios. The edge-cloud hybrid mode performs emotion inference on the edge, and the generated caption triples are uploaded to the conference server, requiring a bandwidth of 2 kilobytes per second. The near-edge mode deploys a lightweight speech recognition model to edge computing nodes while maintaining a network connection; the smartphone only performs emotion inference and caption layout, further reducing latency to 60 milliseconds.

[0154] In this embodiment, when the speech signal-to-noise ratio is below 10 dB, the system reduces the prosodic feature weights and performs single-modal emotion recognition of facial images. When the facial occlusion rate of the facial image exceeds 40%, single-modal emotion recognition of audio is performed. If the speech signal-to-noise ratio is below 10 dB and the facial occlusion rate of the facial image exceeds 40%, only plain text subtitles are output. When the NPU utilization rate is above 80%, the system automatically switches to an 8-bit quantized emotion recognition model and reduces the emotion categories from 12 to 5 to save system resources.

[0155] In this embodiment, the system defaults to using Low-Rank Adaptation (LoRA) fine-tuning for the local emotion recognition model to adapt to the dialect. The gradient information of the emotion recognition model is uploaded after differential privacy noise addition. The server aggregates the gradient information weekly and distributes the optimized emotion recognition model. Users can enable "zero-upload" mode at any time to prevent the external transmission of audio and video data.

[0156] In this embodiment, the emotion recognition model can employ a multimodal self-attention mechanism network (Transformer architecture) or a dual-tower architecture, replacing the graph attention network. The single-camera device and inertial sensor used to locate the speaker's direction can be upgraded to dual-camera stereo ranging or ultra-wideband (UWB) tags. Eye tracking can be performed using an under-display Time of Flight (ToF) device or an inertial sensor. The caption rendering layer can be switched to a WebGraphics Library (WebGL) or a cross-platform development tool (Unity) overlay layer. A secure execution environment or homomorphic encryption aggregation can replace differential privacy, and the edge inference hardware can also use a Bluetooth compute stick or a Field Programmable Gate Array (FPGA) accelerator card. Through this modular and alternative design, the system can be deployed on mobile terminals such as smartphones and AR glasses, while retaining scalability for the algorithm, hardware, and network.

[0157] In this embodiment, speech-to-text transcription can be performed first, followed by emotion inference. Binding text to emotion during subtitle generation reduces inference waiting time and is suitable for ultra-high-speed speech scenarios. Multimodal streaming media data (such as heart rate photoelectric information and inertial gait) can serve as auxiliary signals for emotion inference, improving the robustness of emotion recognition. Facial expression action units can be estimated using depth cameras or millimeter-wave point clouds, and user gaze vectors can be inferred through inertial measurement models, making it suitable for low-end devices without dedicated eye-tracking sensors. Subtitle anchoring logic can be adjusted according to cognitive load or the priority of user-defined subtitle positions (e.g., subtitles fixed in the upper left corner of the interface). This embodiment can also be applied to scenarios such as e-sports live streaming, online education, court recording, and automatic movie subtitle generation to annotate the emotional information of subtitles.

[0158] In the aforementioned scenario of emotion-adaptive real-time caption generation and layout, through cross-modal fusion of speech prosody and facial expression units, the system can still output 12 categories and 5 levels of emotion labels even under noisy or partially occluded conditions. Compared to the convolutional neural network model, the on-device emotion inference accuracy is improved by 9.4 percentage points, and the caption comprehension accuracy for hearing-impaired users increases from 67% to 92%, achieving a complete presentation of emotional context. The dynamic layout engine adjusts the caption position in real time according to the principle of minimum eye saccades. In multi-person dialogue scenarios, the average eye saccade distance is reduced from 14.7° to 9.8°, the eye saccade amplitude is reduced by 33%, and the user's subjective fatigue score is reduced by two levels, significantly reducing the load of eye movement. The end-to-end on-device inference latency is 70-150 milliseconds, which is 4-6 times shorter than the round-trip latency of cloud transcription solutions. The inference power consumption after 4-bit quantization of the model increases by only 0.9 watts, allowing smartphones or AR glasses to be used continuously for more than 4 hours, achieving low latency and low power consumption end-to-end.

[0159] When high noise or high occlusion is detected, the system automatically switches to single-modal emotion inference or reverts to outputting plain text subtitles, maintaining a subtitle availability rate of over 99%. When the NPU load is too high, the system automatically reduces the dimensionality of emotion categories, implementing a robust degradation strategy. Federated learning combined with differential privacy noise pruning ensures that the gradient size of a single uploaded model is less than 50 kilobytes, balancing dialect adaptation and data compliance. In medical, judicial, and other scenarios, a "zero-upload" mode can be enabled to ensure that personal data is run locally in a closed manner, achieving data privacy and legal compliance. The subtitle triple (text-emotion-rendering instruction) is backward compatible with WebVTT and ARIA standards, and can be seamlessly embedded into various applications such as conferencing software, game live streaming, and cinema projection. The SDK size is less than 5 megabytes, improving secondary development efficiency and enabling easy integration and cross-scenario compatibility of subtitles. Through emotion perception, gaze-adaptive layout, and local privacy computing, the system comprehensively improves the integrity, comfort, real-time performance, and compliance of the subtitle experience, providing quantifiable technical value for hearing impairment assistance, cross-language communication, and communication in noisy environments.

[0160] The following description continues to illustrate the exemplary structure of the data processing apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the data processing device 455 of the memory 450 may include: a first acquisition module 4551, used to acquire voice data and facial images corresponding to a first object; an emotion prediction module 4552, used to predict the emotion category based on at least one of the voice data and facial images when it is determined that the emotion prediction conditions are met, and obtain an emotion prediction result; a data generation module 4553, used to generate text data based on the voice data, and generate subtitle data based on the emotion prediction result and the text data; a second acquisition module 4554, used to acquire eye-tracking data corresponding to a second object, and position information of the first object relative to the second object; and a position determination module 4555, used to determine the target position of the subtitle data based on the position information and eye-tracking data, and display the subtitle data at the target position.

[0161] In some embodiments, the emotion prediction module 4552 is further configured to determine the signal-to-noise ratio of the speech data and the occlusion rate of the facial image; when the signal-to-noise ratio of the speech data is greater than or equal to a first preset threshold, or the occlusion rate of the facial image is less than or equal to a second preset threshold, it is determined that the emotion prediction condition is met.

[0162] In some embodiments, the emotion prediction module 4552 is further configured to: predict the emotion category based on the speech data and the facial image when the signal-to-noise ratio of the speech data is greater than or equal to a first preset threshold and the occlusion rate of the facial image is less than or equal to a second preset threshold; predict the emotion category based on the facial image when the signal-to-noise ratio of the speech data is less than the first preset threshold and the occlusion rate of the facial image is less than or equal to the second preset threshold; and predict the emotion category based on the speech data when the signal-to-noise ratio of the speech data is greater than or equal to the first preset threshold and the occlusion rate of the facial image is greater than the second preset threshold.

[0163] In some embodiments, the emotion prediction module 4552 is further configured to extract features from the speech data and facial images respectively to obtain audio features and facial features; fuse the audio features and facial features to obtain fused features; use a graph attention network to predict the emotion category of the fused features to obtain an emotion category probability distribution; and perform emotion intensity mapping based on the emotion category probability distribution to obtain the emotion prediction result.

[0164] In some embodiments, the data generation module 4553 is further configured to determine the target display format and target subtitle color of the subtitle border corresponding to the emotion prediction result based on the emotion category corresponding to the emotion prediction result, wherein each emotion category corresponds to a preset display format and preset subtitle color of the subtitle border; and to render the text data using the target display format and target subtitle color of the subtitle border to obtain subtitle data.

[0165] In some embodiments, the data generation module 4553 is further configured to, when the vibration mode is detected to be in the activated state, determine the target vibration frequency corresponding to the terminal based on the emotion category corresponding to the subtitle data, and vibrate based on the target vibration frequency, wherein each emotion category corresponds to a preset vibration frequency; and when the voice reading mode is detected to be in the activated state, output the subtitle voice corresponding to the subtitle data.

[0166] In some embodiments, the eye-tracking data includes the eye center position. The position determination module 4555 is further configured to determine the unit vector pointing from the eye center position to a preset gaze point as a gaze vector; obtain at least one preset position for displaying subtitle data; for each preset position, determine a first distance between the gaze vector and the preset position, and determine a second distance between the position information and the preset position; use a first preset weight and a second preset weight to perform a weighted summation of the first distance and the second distance to obtain a summation result; and determine the preset position corresponding to the minimum summation result as the target position of the subtitle data.

[0167] In some embodiments, the position determination module 4555 is further configured to: determine updated position information when the position information of the first object relative to the second object changes; determine the updated target position of the subtitle data based on the updated position information and eye-tracking data; determine a third distance between the target position of the subtitle data and the updated target position of the subtitle data when the deviation angle between the target position of the subtitle data and the updated target position of the subtitle data is greater than a preset angle; and move the target position of the subtitle data to the updated target position of the subtitle data based on the third distance and a preset adjustment coefficient.

[0168] In some embodiments, the position determination module 4555 is further configured to adjust the font size of the subtitle data in response to an adjustment instruction for the subtitle font, thereby obtaining the adjusted subtitle data; or to fix the target position of the subtitle data in response to a locking instruction for the subtitle position, and display the subtitle data at the target position.

[0169] This application provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the data processing method provided in this application.

[0170] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the data processing method provided in this application, for example... Figure 3A The data processing method is shown.

[0171] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0172] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0173] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0174] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0175] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the voice data and facial image corresponding to the first object; When it is determined that the emotion prediction conditions are met based on the voice data and the facial image, the emotion category is predicted based on at least one of the voice data and the facial image to obtain the emotion prediction result. Text data is generated based on the speech data, and subtitle data is generated based on the emotion prediction result and the text data; Obtain eye movement data corresponding to the second object, and position information of the first object relative to the second object, wherein the eye movement data includes the center position of the eyeball; The unit vector pointing from the center of the eyeball to the preset fixation point is determined as the gaze vector; Obtain at least one preset position for displaying the subtitle data; for each preset position, determine a first distance between the gaze vector and the preset position, and determine a second distance between the position information and the preset position. Using a first preset weight and a second preset weight, the first distance and the second distance are weighted and summed to obtain the summation result; The preset position corresponding to the minimum summation result is determined as the target position of the subtitle data, and the subtitle data is displayed at the target position.

2. The method according to claim 1, characterized in that, The method further includes: Determine the signal-to-noise ratio of the speech data and the occlusion rate of the facial image; When the signal-to-noise ratio of the speech data is greater than or equal to a first preset threshold, or the occlusion rate of the facial image is less than or equal to a second preset threshold, the emotion prediction condition is determined to be met.

3. The method according to claim 2, characterized in that, When it is determined that the emotion prediction conditions are met based on the speech data and the facial image, the emotion category is predicted based on at least one of the speech data and the facial image to obtain the emotion prediction result, including: When the signal-to-noise ratio of the speech data is greater than or equal to the first preset threshold, and the occlusion rate of the facial image is less than or equal to the second preset threshold, emotion category prediction is performed based on the speech data and the facial image to obtain the emotion prediction result. When the signal-to-noise ratio of the speech data is less than the first preset threshold and the occlusion rate of the facial image is less than or equal to the second preset threshold, emotion category prediction is performed based on the facial image to obtain the emotion prediction result. When the signal-to-noise ratio of the speech data is greater than or equal to the first preset threshold, and the occlusion rate of the facial image is greater than the second preset threshold, emotion category prediction is performed based on the speech data to obtain the emotion prediction result.

4. The method according to claim 3, characterized in that, The process of predicting the emotion category based on the speech data and the facial image to obtain the emotion prediction result includes: Feature extraction is performed on the voice data and the facial image respectively to obtain corresponding audio features and facial features; The audio features and the facial features are fused to obtain the fused features; Using a graph attention network, the fused features are used to predict the emotion category, resulting in an emotion category probability distribution. Based on the probability distribution of the emotion categories, emotion intensity is mapped to obtain the emotion prediction result.

5. The method according to claim 1, characterized in that, The process of generating subtitle data based on the emotion prediction result and the text data includes: Based on the emotion category corresponding to the emotion prediction result, the target display format and target subtitle color of the subtitle border corresponding to the emotion prediction result are determined, wherein each emotion category corresponds to a preset display format and preset subtitle color of the subtitle border; The text data is rendered using the target display format of the subtitle border and the target subtitle color to obtain subtitle data.

6. The method according to claim 5, characterized in that, The method further includes: When the vibration mode is detected to be activated, the target vibration frequency corresponding to the terminal is determined based on the emotion category corresponding to the subtitle data, and vibration is performed based on the target vibration frequency, wherein each emotion category corresponds to a preset vibration frequency. When the voice reading mode is detected to be activated, the voice text corresponding to the subtitle data is output.

7. The method according to claim 1, characterized in that, The method further includes: When the position information of the first object relative to the second object changes, the position information is updated. Based on the updated location information and the eye-tracking data, the target location for updating the caption data is determined; When the deviation angle between the target position of the subtitle data and the update target position of the subtitle data is greater than a preset angle, a third distance between the target position of the subtitle data and the update target position of the subtitle data is determined; Based on the third distance and the preset adjustment coefficient, the target position of the subtitle data is moved to the updated target position of the subtitle data.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: In response to an adjustment command for the subtitle font, the font size of the subtitle data is adjusted to obtain the adjusted subtitle data; or In response to a lock command for the subtitle position, the target position of the subtitle data is fixed, and the subtitle data is displayed at the target position.

9. A data processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire the voice data and facial image corresponding to the first object; An emotion prediction module is used to predict the emotion category based on at least one of the voice data and the facial image when it is determined that the emotion prediction conditions are met, and to obtain an emotion prediction result. The data generation module is used to generate text data based on the voice data, and to generate subtitle data based on the emotion prediction result and the text data; The second acquisition module is used to acquire eye movement data corresponding to the second object, and position information of the first object relative to the second object, wherein the eye movement data includes the center position of the eyeball; The position determination module is used to determine the unit vector pointing from the center position of the eyeball to a preset gaze point as the gaze vector; obtain at least one preset position for displaying the subtitle data; for each preset position, determine a first distance between the gaze vector and the preset position, and determine a second distance between the position information and the preset position; use a first preset weight and a second preset weight to perform a weighted summation on the first distance and the second distance to obtain a summation result; determine the preset position corresponding to the minimum summation result as the target position of the subtitle data, and display the subtitle data at the target position.

10. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or the computer program are executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or the computer program are executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Intelligent glasses and translation method thereof

    CN108509430A

  • Video display method and device, equipment and storage medium

    CN116055792A

  • Voice interaction method and device, equipment and storage medium

    CN119673160A