Content interaction method and device based on laughter recognition, equipment, storage medium and program product

Abstract

The present disclosure provides a content interaction method and device based on laughter recognition, equipment, storage medium and program product. The method comprises: performing laughter recognition on audio data collected from the environment of an electronic device; in response to identifying that there is laughter in the audio data, extracting a first laughter feature corresponding to the identified first laughter; matching the first laughter feature with a second laughter feature stored in a circular buffer area of the electronic device to obtain the similarity of the first laughter feature and the second laughter feature, wherein the second laughter feature is a laughter feature corresponding to at least one second laughter extracted within a past preset time length; and triggering interaction with the currently played content of the electronic device based on the similarity of the first laughter feature and the second laughter feature.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and specifically to a content interaction method, device, electronic device, computer-readable storage medium, and computer program product based on laughter recognition. Background Technology

[0002] With the rapid development of online live streaming and video sharing platforms, interactive content has become an indispensable part of online streaming media scenarios. Various video platforms are constantly enriching their interactive content formats, promoting the deep integration of interactive content with video content, making interactive elements a crucial component of video content, influencing user viewing experience and platform user stickiness.

[0003] Content interaction features can bring users a diverse participation experience and many benefits: by participating in content interaction features, users can break away from the simple "watching" mode, actively participate in the currently playing content, and form a closer connection with the content, no longer being passive bystanders receiving the content; at the same time, rich interactive features can effectively break the monotony of the live broadcast room or watching videos alone, enhance the user's interest in watching the currently playing content, increase user viewing time and activity, and thus improve the platform's user retention rate and platform competitiveness.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a content interaction method, apparatus, device, computer-readable storage medium, and computer program product based on laughter recognition.

[0006] According to one aspect of this disclosure, a content interaction method based on laughter recognition is provided, comprising: performing laughter recognition on audio data collected from the environment of an electronic device; in response to the recognition of laughter in the audio data, extracting a first laughter feature corresponding to the recognized first laughter; matching the first laughter feature with a second laughter feature stored in a loop buffer of the electronic device to obtain a similarity between the first laughter feature and the second laughter feature, wherein the second laughter feature is a laughter feature corresponding to at least one second laughter extracted within a preset time period in the past; and triggering interaction with the currently playing content of the electronic device based on the similarity between the first laughter feature and the second laughter feature.

[0007] According to another aspect of this disclosure, an apparatus for a content interaction method based on laughter recognition is also provided, comprising: a laughter recognition unit configured to recognize laughter from audio data collected from the environment of an electronic device; a laughter feature extraction unit configured to extract a first laughter feature corresponding to a first laughter in response to the recognition of laughter in the audio data; a laughter feature matching unit configured to match the first laughter feature with a second laughter feature stored in a loop buffer of the electronic device to obtain a similarity between the first laughter feature and the second laughter feature, wherein the second laughter feature is a laughter feature corresponding to at least one second laughter extracted within a preset time period in the past; and a content interaction unit configured to trigger interaction with currently playing content on the electronic device based on the similarity between the first laughter feature and the second laughter feature.

[0008] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described above.

[0009] According to another aspect of this disclosure, a computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above-described method when executed by a processor.

[0010] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0011] The content interaction method based on laughter recognition provided in this disclosure can identify a user's laughter and determine whether to trigger interaction with the currently playing content by matching the similarity of laughter features within a preset time period. This can reduce the user's operational burden, improve the user's viewing experience, and prevent cheating behavior such as fake interaction.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1An exemplary flowchart of a content interaction method based on laughter recognition according to an embodiment of the present disclosure is shown.

[0015] Figures 2A to 2E A schematic diagram illustrating user interaction with currently playing content on an electronic device according to an embodiment of the present disclosure is shown.

[0016] Figure 3 An exemplary block diagram of an apparatus for a content interaction method based on laughter recognition according to an embodiment of the present disclosure is shown.

[0017] Figure 4 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0020] The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0021] It should be noted that, in any part of this disclosure involving the collection, storage, use, transmission, and processing of data, each stage strictly adheres to the laws, regulations, industry standards, and regulatory requirements of the data source, usage location, and relevant countries and regions to ensure the legality and compliance of data activities. In the collection stage, the purpose, method, and scope of collection are clearly communicated to the data subject in a prominent manner. Collection is conducted only after obtaining the data subject's legal authorization, ensuring that the collection process follows the "minimum necessary" principle and does not exceed the scope of data collection. In the storage stage, storage periods are limited, and data is promptly deleted or anonymized / encrypted after the storage purpose is achieved. In the usage stage, a strict data security protection mechanism is implemented, using field-level desensitization technology and processing the original data according to preset desensitization rules. For different types of data, multiple desensitization strategies, such as data generalization, data anonymization, and data encryption, are employed to effectively mitigate the risk of sensitive information leakage and ensure that all data used is securely processed and desensitized, comprehensively protecting the rights and interests of data subjects and data security. In the transmission and processing stages, the confidentiality and security of data are ensured during transmission and processing.

[0022] As mentioned earlier, with the rapid development of online live streaming and video sharing platforms, major live streaming platforms are increasing their efforts to explore interactive features and continuously innovating interactive formats in order to enhance user engagement, increase user stickiness, and thus improve platform competitiveness. Currently, relevant live streaming interactive technologies have taken on various forms, mainly including the following three categories.

[0023] The first is text-based bullet comments, which is the most basic and widely used interactive method. It allows users to edit custom text content while watching videos or live streams, and the text is displayed on the live stream or video screen in real time after being sent, enabling real-time text communication between users and between users and the streamer.

[0024] Secondly, there are interactive features such as likes / votes / PKs. Users can like or vote for programs or hosts by clicking the corresponding buttons on the screen. These interactions are usually limited in number to guide users to participate reasonably and enhance the atmosphere of the live stream.

[0025] Thirdly, there is the gift tipping feature, where users can purchase virtual gifts on the platform and send them to the streamers to express their support and affection. At the same time, streamers can receive incentives through gift revenue, which further enhances the quality of their program content.

[0026] However, the aforementioned live streaming interactive technologies still have many shortcomings, failing to meet users' needs for lightweight and low-barrier interaction. First, all of these interactive methods require manual operation from users. Whether it's editing and sending text comments, clicking like / vote buttons, or purchasing / sending virtual gifts, it requires users to divert their attention from watching the live stream. Manual interaction interrupts the viewing rhythm when focused on the content, affecting the user's immersive viewing experience. Second, text comments require users to actively edit and send them. For users who don't want to type or are not good at expressing themselves, the participation threshold is high, leading some users to be unwilling to participate in this type of interaction. Furthermore, existing interactive features are very cumbersome, requiring users to shift their attention from the current program to the interactive operation. The lack of a lightweight, low-barrier interactive feature that requires minimal manual operation makes it difficult to balance user viewing experience and interactive needs, and fails to fully mobilize the participation of all users. Finally, the stability of the interactive ecosystem also faces challenges. The lack of effective anti-cheating mechanisms in these technologies leads to frequent occurrences of various cheating behaviors. For example, in the gift-giving process, there are behaviors such as using script tools to register multiple accounts in bulk, using device emulators to make fake donations, or maliciously using multiple accounts to collect interaction data, which seriously interfere with the fairness of the interaction and have a negative impact on the overall health of the platform's ecosystem.

[0027] To address the aforementioned issues, this disclosure provides a content interaction method based on laughter recognition. This method identifies user laughter from ambient audio data and determines whether interaction with the currently playing content is triggered by matching the extracted laughter features with similarity features stored in a loop buffer. Firstly, by employing this method, users can interact simply by naturally expressing their emotions, eliminating the need for manual operation and effectively reducing user workload and distraction. Secondly, determining interaction based on the matching of extracted laughter features with those stored in the loop buffer ensures genuine interaction, prevents fake interactions, guards against cheating, and maintains a healthy live streaming ecosystem.

[0028] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0029] Figure 1 An exemplary flowchart of a content interaction method based on laughter recognition according to an embodiment of the present disclosure is shown.

[0030] At S101, laughter is recognized from audio data collected from the environment of the electronic device.

[0031] At S102, in response to the detection of laughter in the audio data, the first laughter feature corresponding to the first laughter detected is extracted.

[0032] At S103, the first laughter feature is matched with the second laughter feature stored in the loop buffer of the electronic device to obtain the similarity between the first laughter feature and the second laughter feature, wherein the second laughter feature is a laughter feature corresponding to at least one second laughter extracted within a preset time period in the past.

[0033] At S104, interaction with the currently playing content of the electronic device is triggered based on the similarity between the first laughter feature and the second laughter feature.

[0034] The method disclosed herein captures a user's natural laughter, matches the user's laughter characteristics with laughter characteristics stored in a loop buffer, and finally triggers interactive operations linked to the currently playing content based on the matching results. This achieves intelligent interaction without manual operation, lowers the barrier to user participation, and allows users to complete interactions without interrupting their viewing. Furthermore, it effectively prevents fake interactions, ensuring the authenticity of laughter-based interaction data.

[0035] Specific examples of this disclosure will be described in detail below.

[0036] At S101, laughter is recognized from audio data collected from the environment of the electronic device.

[0037] In some examples, laughter recognition algorithms can be used to analyze collected audio data in real time to identify laughter. Exemplary laughter recognition algorithms can include traditional machine learning algorithms such as those based on Hidden Markov Models (HMMs) and Support Vector Machines (SVMs), as well as deep learning-based laughter recognition algorithms such as those based on Long Short-Term Memory Networks (LSTMs), Convolutional Neural Networks (CNNs) or CRNNs (CNN+RNNs), to distinguish between laughter and non-laughter. These algorithms can be flexibly selected based on the performance of electronic devices and the real-time requirements of recognition, ensuring efficient completion of laughter recognition tasks in different scenarios.

[0038] In some examples, the user may be asked to grant the electronic device microphone permission in order to collect audio data from the device's environment.

[0039] In some examples, a laughter recognition algorithm may be used to identify whether laughter exists in the audio data in response to determining that the audio data in the environment exceeds a predetermined decibel and the frequency of the audio data matches the frequency of human voice.

[0040] Specifically, a reasonable preset decibel level can be used to perform coarse-grained pre-screening of the collected ambient audio data. For example, the preset decibel level can be dynamically adjusted according to the usage scenario of the electronic device; it can be appropriately lowered in quiet environments and correspondingly increased in noisy environments. Furthermore, a suitable range of human voice frequencies can be preset, for example, human voice frequencies are typically between 300Hz and 3400Hz. The laughter recognition algorithm is only used for subsequent laughter recognition when the collected audio data simultaneously meets the conditions of the ambient audio data exceeding the preset decibel level and the audio data frequency matching the human voice frequency. This operation can significantly reduce invalid recognition operations, avoid inputting non-human sounds (such as wind noise, electrical noise, etc.) into the laughter recognition algorithm, and reduce the computational burden on the electronic device. In addition, it can save the power and memory resources of the electronic device. By using the model only when the conditions are met, the effectiveness of laughter recognition is balanced with device battery life and performance consumption.

[0041] In some examples, to acquire higher quality audio data, the following echo cancellation operation can be performed: before laughter recognition is performed on the audio data acquired from the environment of the electronic device, the audio data corresponding to the content currently being played by the electronic device can be used as a reference signal, and the echo corresponding to the reference signal can be subtracted from the audio data in the acquired environment.

[0042] Specifically, when an electronic device plays content, its sound reflects off the environment, creating echoes that mix into the collected audio data and interfere with laughter recognition. By extracting the audio data corresponding to the currently playing content as a reference signal, the frequency, amplitude, and other audio data characteristics corresponding to the echoes can be determined. Furthermore, by subtracting the corresponding echoes from the collected ambient audio data, redundant interference can be effectively filtered out, allowing only the user's actual reaction to be extracted.

[0043] In some examples, to enhance the identified laughter, the following spectrum restoration operation can be performed: after laughter recognition is performed on audio data collected from the environment of an electronic device, in response to the recognition of laughter in the audio data, noise suppression can be performed on the collected audio data of the environment through a Wiener filter, and the contrast of the laughter can be improved.

[0044] Specifically, in audio data collected from the environment of electronic devices, even if laughter can be identified, environmental noise (such as wind noise, electrical noise, etc.) may still remain within the laughter, and the spectra of laughter and noise easily overlap, resulting in blurred laughter characteristics. The Wiener filter can precisely suppress the noise spectrum. Simultaneously, by adjusting audio spectrum parameters, the contrast between laughter and noise is improved, making the frequency, amplitude, and other features of laughter more prominent, providing a reliable foundation for subsequent extraction of the first laughter feature. Furthermore, the Wiener filter is a lightweight noise reduction tool, placing a low computational burden on electronic devices.

[0045] For the above examples, those skilled in the art can combine or use one or more of the above examples individually to achieve better recognition of laughter and enhance the effect of laughter, and this disclosure does not limit this.

[0046] At S102, in response to the detection of laughter in the audio data, the first laughter feature corresponding to the first laughter detected is extracted.

[0047] In response to the laughter recognition result in S101, feature extraction is performed on the confirmed first laugh. This step extracts first laugh features that can be used for matching from unstructured laughter segments. Laughter features may include at least one of the following: laughter duration, frequency distribution, energy distribution, voiceprint, and plosive beat. Therefore, this disclosure does not involve blind sampling of laughter, but focuses on extracting the aforementioned core laugh features. Laughter duration describes the duration of the first laugh. For example, endpoint detection (VAD) technology can be used to calculate the total duration from the onset of the laugh to the acoustic energy decaying to the background noise level. Frequency distribution describes the sound intensity of the first laugh in a specific frequency band. For example, the fundamental frequency and its harmonic distribution can be extracted by analyzing the spectral envelope of the laugh using Fast Fourier Transform (FFT). Energy distribution describes the intensity fluctuations of the first laugh, identifying whether the laugh gradually increases or suddenly erupts. For example, the root mean square (RMS) energy of the audio frame can be extracted, and its dynamic envelope in the time domain can be observed. Voiceprints can describe "who is laughing," encompassing unique acoustic features resulting from differences in the vocal tract and laryngeal cavity structure of the speaker. Plosive beats describe the intermittent exhalation bursts of laughter, allowing for the extraction of the repetition frequency and interval duration of plosive syllables such as "ha, ha" in laughter. The above techniques can be used individually or in combination to extract the first laughter feature. Although this disclosure has listed laughter features above, those skilled in the art can extract them based on any other laughter features known in the art.

[0048] At S103, the first laughter feature is matched with the second laughter feature stored in the loop buffer of the electronic device to obtain the similarity between the first laughter feature and the second laughter feature, wherein the second laughter feature is a laughter feature corresponding to at least one second laughter extracted within a preset time period in the past.

[0049] Similarity matching of the first and second laughter features can be performed based on algorithms such as cosine similarity, Euclidean distance, and Hamming distance.

[0050] This step introduces a circular buffer and performs similarity matching between the newly extracted first laughter feature and the previously extracted second laughter feature. Its core purpose is to prevent cheating behaviors, such as users repeatedly playing recorded laughter into their electronic devices or maliciously triggering the interaction repeatedly within a short period, thus ensuring the authenticity and uniqueness of the interaction trigger.

[0051] A loop buffer can be constructed as follows: a loop storage space can be allocated in the local memory of the electronic device to store historical laughter features extracted and stored within a preset duration up to the current moment. In some examples, the preset duration can be set to 30 seconds. This is because: First, for cheating methods such as looping recorded laughter on a mobile phone, to make the recorded laughter sound natural and non-repetitive, the cheater might record a laughter clip of about 10 seconds. Setting the preset duration to 30 seconds means that the loop buffer can cover at least 2 to 3 complete cheating loops. Even if the cheater records slightly longer material, a 30-second loop buffer is sufficient to capture the starting point of the second replay, so that the laughter based on cheating cannot trigger interaction with the content currently being played on the electronic device. Second, genuine laughter from the content currently being played on the electronic device usually lasts less than 10 seconds. In normal program viewing, the same joke usually does not appear twice in a short period of time. Therefore, if the preset duration is too short, such as 5 seconds, it may not be able to intercept slightly longer loop recordings; if the preset duration is too long, such as 10 minutes, if a user laughs out loud at different jokes within a short period, the limited preset duration of the loop buffer may lead to misjudgment, preventing the user from voting for the true experience. Although this disclosure shows a preset duration of 30 seconds, in practical applications, those skilled in the art can dynamically adjust the specific setting of the preset duration according to the specific scenario to meet real needs.

[0052] Furthermore, this circular buffer follows a first-in, first-out (FIFO) principle. In some embodiments, expired laughter features may be deleted after a preset time period. In some examples, at least one second laugh corresponds to a laughter feature with a timestamp corresponding to its storage time, and wherein, for each of the at least one second laughs, if the time difference between the timestamp of the laughter feature corresponding to the second laugh and the current time exceeds a preset time period, the laughter feature corresponding to the second laugh is deleted, and the storage space of the laughter feature corresponding to the second laugh is released.

[0053] Specifically, when storing each laughter feature in the circular buffer, the storage time can be recorded synchronously and used as the timestamp of that laughter feature. By assigning a unique timestamp to each laughter feature in the circular buffer, the laughter features can acquire temporal attributes. To achieve the first-in, first-out (FIFO) characteristic of the circular buffer and a dynamic window within a preset duration, the current time can be continuously or periodically retrieved, and the time difference between the current time and the storage time of each laughter feature (i.e., the timestamp of each laughter feature) can be calculated. If the time difference exceeds the preset duration, it means that the laughter feature has expired and should be deleted, releasing the corresponding storage space. In this way, the circular buffer only needs to maintain a small scale of laughter features (containing only laughter features from the past preset duration) to perform similarity matching between newly extracted laughter features and the laughter features in the circular buffer in subsequent steps. This design ensures that the system can capture recent repetitive operations while minimizing client memory overhead and avoiding privacy risks and computational burdens caused by long-term storage of audio features.

[0054] At S104, interaction with the currently playing content of the electronic device is triggered based on the similarity between the first laughter feature and the second laughter feature.

[0055] In some embodiments, a neural network-based decision model can be used to process the first laughter feature, the second laughter feature, and the similarity between the first laughter feature and the second laughter feature. A deep representation between the first laughter feature and the second laughter feature is mined using machine learning, and based on the aforementioned deep representation and similarity, a decision result is output regarding whether to trigger interaction with the currently playing content on the electronic device.

[0056] In some embodiments, interaction with the currently playing content on an electronic device can be triggered by comparing a preset threshold with similarity.

[0057] Specifically, the step may include: if the similarity between the first laughter feature and the second laughter feature is not less than a preset threshold, then no interaction with the currently playing content of the electronic device is triggered; if the similarity between the first laughter feature and the second laughter feature is less than the preset threshold, then interaction with the currently playing content of the electronic device is triggered.

[0058] As mentioned earlier, if the similarity between the newly extracted first laughter feature and the second laughter feature in the loop buffer is not less than a preset threshold, it means that highly similar laughter has been captured within a preset duration. In this case, the newly extracted first laughter feature may correspond to the second replay of the cheating laughter segment. Conversely, if the similarity between the newly extracted first laughter feature and the second laughter feature stored in the loop buffer is less than the preset threshold, it means that the newly identified first laughter is the user's genuine laughter, and interaction with the currently playing content on the electronic device can be triggered.

[0059] In some examples, the preset threshold can be set to around 0.95. However, this preset threshold is only an example, and those skilled in the art can flexibly set it according to actual needs. In some examples, multiple preset thresholds can also be set according to actual conditions. For example, a first threshold, a second threshold, and a third threshold can be preset, where the first threshold is less than the second threshold, and the second threshold is less than the third threshold. When the similarity is less than the first threshold, interaction with the currently playing content on the electronic device can be triggered. When the similarity is not less than the first threshold but less than the second threshold, the similarity can be recalculated using different similarity matching methods to reconfirm the similarity calculation results. Then, based on the two similarity calculation results, it can be jointly determined whether to trigger content interaction. When the similarity is not less than the second threshold, interaction with the currently playing content on the electronic device can be directly triggered.

[0060] In other examples, in response to the similarity between the first laughter feature and the second laughter feature being less than a preset threshold, this disclosure also includes storing the first laughter feature in a circular buffer. That is, at this time, not only is interaction with the currently playing content of the electronic device triggered, but the first laughter feature is also stored as a new valid laughter feature in the circular buffer, so that it can become a laughter feature that provides reference and matching for newly extracted laughter features in the next preset time period (i.e., the second laughter feature in the circular buffer).

[0061] In some examples, the preset storage space of the circular buffer corresponds to a preset duration, and storing the first laughter feature in the circular buffer includes: when the laughter features stored in the circular buffer reach the preset storage space, the earliest laughter feature stored in the circular buffer is overwritten with the first laughter feature.

[0062] Specifically, the preset storage space is determined in advance based on the upper limit of the laughter features that may occur within a preset duration. For example, if the preset storage space is determined based on the frequency of laughter, and the highest frequency of laughter within the preset duration is 15 times, then the circular buffer is allocated storage space corresponding to 15 laughter features.

[0063] Moreover, as mentioned earlier, the circular buffer exhibits a first-in, first-out (FIFO) characteristic. This means that the storage space in the circular buffer can be monitored in real time. Even if the stored laughter features reach the preset storage space (e.g., the storage space corresponding to 15 laughter features is full), no new memory is allocated; instead, an overwrite operation is performed. The oldest stored laughter feature can be overwritten with the newly extracted first laughter feature. In this way, although the circular buffer is physically finite, it logically forms a time window that always contains laughter features within the latest preset duration.

[0064] Furthermore, triggering interaction with the currently playing content on an electronic device can take many forms, such as sending gifts, liking, or saving. In some examples, triggering interaction with the currently playing content can include triggering a vote count for that content. That is, one (or more) laughs from a user can trigger one (or more) votes on the currently playing content. This approach maps one (or more) laughs to one (or more) votes, significantly lowering the barrier to user interaction and enhancing the immersive experience during playback.

[0065] In some examples, based on the vote count triggered by the electronic device's currently playing content, a visual representation of the vote corresponding to the vote count is displayed in the interactive area. Specifically, specific interactive areas can be preset in the electronic device's display interface, such as a pop-up window on the right side of the interface, a progress bar at the bottom, or as disclosed herein. Figure 2D The "+1" visual representation shown is illustrated (more details will be discussed later). Those skilled in the art can flexibly configure the voting visualization according to actual needs. The voting visualization informs users that their laughter has been successfully converted into a vote count, enhancing the certainty of the interaction and the user's immersion.

[0066] In some examples, in response to triggering interaction with the content currently being played on the electronic device, the intensity of the emotion corresponding to the first laugh can be determined; based on the intensity of the emotion corresponding to the first laugh, an emotional visualization corresponding to the intensity of the emotion is displayed in the interactive area of ​​the currently playing content.

[0067] Specifically, after the first laugh triggers interaction with the currently playing content on the electronic device, the intensity of the emotion corresponding to the first laugh can be determined. In some examples, methods for determining the intensity of the emotion corresponding to the laugh may include, for example, analyzing the amplitude of the laugh to determine its volume, analyzing the rate of change of the fundamental frequency to determine the fluctuation of the laugh, analyzing the duration of the laugh and the frequency of its bursts, or determining the intensity of the emotion corresponding to the first laugh based on relevant data of the characteristics of the first laugh. In some examples, emotion assessment algorithms can be used to map the above features to specific intensity ranges. For example, a high-decibel, long-duration laugh would be judged as a strong emotion, while a slight, low laugh would be judged as a weak emotion.

[0068] Based on the emotional intensity corresponding to the first laugh, the emotional intensity can be converted into a visual representation of the emotion in the interactive area in real time. The form of the emotional visualization can be related to the emotional intensity. For example, the higher the emotional intensity, the more intense the animation and the more vibrant the light effects in the interactive area. Furthermore, according to this disclosure… Figures 2A to 2E The lower right corner displays an emoji to visualize emotions; a blank expression is shown when the user is not laughing, and a smiling emoji is displayed when laughter is detected. Those skilled in the art can flexibly configure the emotion visualization according to actual needs.

[0069] In some examples, the first laugh may be preprocessed in response to triggering interaction with the content currently being played on the electronic device; the audio quality of the preprocessed first laugh may be determined using an audio quality assessment model; in response to determining that the audio quality of the preprocessed first laugh is higher than a predetermined quality threshold, and in response to user authorization, the display of the first comment in the content currently being played on the electronic device may be triggered, wherein the content of the first comment corresponds to the preprocessed first laugh.

[0070] Preprocessing the first laugh can include truncating the identified laughter audio data and retaining only the laughter portion. Preprocessing of the first laugh can also include methods to optimize its quality, such as removing residual environmental noise, echoes, and other redundant interference, while standardizing the audio format and adjusting the volume amplitude to ensure the laughter is clearly identifiable.

[0071] In some examples, audio quality assessment models are used to determine the audio quality of the preprocessed first laugh based on at least one of the following: audio clarity, vocal appeal, and emotional positivity. Specifically, for audio clarity, the model analyzes the signal-to-noise ratio, distortion, and frequency band integrity of the laugh after interference removal to determine if the laugh is clear and free of noise, ensuring that the details of the laugh are discernible. For vocal appeal, the model assesses the energy distribution, rhythm, and emotional fullness of the laugh to identify whether the laugh has a strong emotional resonance. For emotional positivity, the model combines emotional semantic analysis to determine whether the laugh conveys positive and joyful emotions, avoiding including negative or mechanically simulated laughter within the scope of effective interaction. By combining these dimensions, audio quality can be quantified. In some examples, audio quality assessment models can be implemented in various ways, such as CNN-based no-reference audio quality assessment models that can score the quality of laughter signals without the need for clean reference audio, focusing on evaluating audio clarity, distortion, and noise levels; LSTM-based temporal audio quality assessment models that can capture the continuous temporal features of laughter and comprehensively evaluate audio fluency, vocal appeal, and emotional positivity; and other models or algorithms that can be used to assess the quality of laughter audio.

[0072] In this process, in response to determining that the audio quality of the preprocessed first laugh is higher than a predetermined quality threshold, and in response to user authorization, the first bullet comment is triggered to be displayed in the currently playing content on the electronic device. The content of the first bullet comment corresponds to the preprocessed first laugh. Specifically, firstly, as mentioned above, the audio quality assessment model can determine the audio quality of the preprocessed first laugh. When the audio quality is higher than the predetermined quality threshold, it can be determined that the laugh can be used as a valid interactive signal for subsequent processing. Secondly, user authorization is obtained simultaneously, which can be achieved through pop-up confirmation, preset preference settings, etc., to ensure that the user allows their laugh to be converted into a bullet comment for display. After all the aforementioned conditions are met, the display of the first bullet comment can be triggered. In some examples, the corresponding first bullet comment content can be intelligently matched and generated based on parameters related to the characteristics of the first laugh, such as laugh duration, frequency distribution, energy distribution, etc. This first bullet comment content can correspond to the preprocessed first laugh and be presented in the form of text, dynamic graphics, or audio bullet comments. For example, the preprocessed first laugh can be displayed as an audio bullet comment, or a more dynamic visual effect can be matched based on the high pitch and shortness of the first laugh. The first bullet comment appears on top of the content currently being played on the electronic device and is synchronized with the playback content, without interrupting the playback process and enhancing the viewer's immersion.

[0073] Furthermore, to enhance interactive fun and protect user privacy, this disclosure also provides a method for generating and publishing voice-changing laughter videos with one click. In some examples, one or more preset voice-changing effects can be used to change the pre-processed first laugh to generate a voice-changing laugh; one or more preset video templates can be used to overlay the voice-changing laugh onto the preset video template to generate a voice-changing laugh video; and the voice-changing laugh video can be published in response to user confirmation.

[0074] Specifically, preset voice-changing effects can cover a variety of styles to suit different user preferences, such as cute cartoon voices, deep magnetic voices, and light and ethereal voices. For the first laugh after preprocessing, users can either be recommended suitable voice-changing effects or manually select one. The voice-changing process is fast and efficient, without changing the core rhythm and emotional expression of the laughter. It only optimizes and adjusts the timbre and pitch of the voice, preserving the original emotional core of the laughter while avoiding the leakage of the user's real voiceprint, thus balancing fun and privacy. In addition, preset video templates can be pre-set, including various types such as simple text templates, dynamic effect templates, and scene-based templates. The overlay process is completed automatically by the system. In some examples, the temporal characteristics of the voice-changing laughter can be extracted, and the frame sequence, switching time points, effect trigger nodes, and audio track reserved duration of the preset video template can be analyzed. Through a timeline alignment algorithm, the time of the voice-changing laughter is aligned with the time of the video template. After the voice-changing laughter video is generated and confirmed by the user, it can be published immediately. In some examples, a preview of the voice-changing laughter video can be generated, allowing users to quickly check the video effect. Once confirmed, the video can be published with a single click of the publish button. This method greatly simplifies the user's workflow, enabling users to quickly transform their laughter into shareable and interactive voice-changing laughter videos without needing any video editing or audio processing skills. This enhances the interactive experience, reduces operational costs, and is suitable for various application scenarios such as live streaming and short video playback.

[0075] Figures 2A to 2E A schematic diagram illustrating user interaction with currently playing content on an electronic device according to an embodiment of the present disclosure is shown.

[0076] Specifically, Figure 2A This shows the interface where the user accesses the currently playing program on the electronic device. It can be seen that... Figure 2A A large number of "hahaha" text comments floated at the top of the interface. Figure 2A The interactive area in the lower right corner displays anthropomorphic expressions, which is a combination of... Figure 1 The emotional visualization representation described in Method 100.

[0077] Figures 2B to 2CAs shown, a video platform can request microphone permissions from a user's electronic device to collect audio data from the device's environment. Additionally or alternatively, the user can be provided with a text prompt to convert laughter into a vote for the currently playing program. In some examples, this can be clicked... Figure 2B or Figure 2C The emotion visualization in the lower right corner represents a vote for the currently playing program. In other examples, as mentioned earlier, after obtaining microphone access to the user's electronic device, a vote for the currently playing program can be triggered based on the user's laughter.

[0078] Figure 2D As shown, after the user grants microphone permissions, a visual representation of the emotion corresponding to the emotion intensity is displayed in the interactive area. Specifically, in Figure 2D In the interactive area at the bottom right corner, anthropomorphic expressions appear... Figure 2A The expressionless face changes to a smiling face. Furthermore, based on the vote count triggered for the currently playing program, a visual representation of the vote count is displayed in the interactive area. Specifically, in Figure 2D In the bottom right interactive area, two "+1" voting icons appear, representing that the user successfully triggered two votes through laughter. In some cases, a content ranking list based on the voting results can also be displayed on the user interface. For example, the ranking list could be based solely on the results of laughter votes.

[0079] Figure 2E The first comment posted after user confirmation was displayed. Specifically, in Figure 2E The currently playing program displays the first comment corresponding to the pre-processed first laugh. This first comment is in the form of an audio comment and lasts for 10 seconds.

[0080] Figure 3 An exemplary block diagram of an apparatus 300 for a content interaction method based on laughter recognition, according to embodiments of the present disclosure, is shown. It can be utilized... Figure 3 The system shown is used to perform the combination Figure 1 Method 100 is described.

[0081] like Figure 3 As shown, the device 300 includes: a laughter recognition unit 301, a laughter feature extraction unit 302, a laughter feature matching unit 303, and a content interaction unit 304. In addition to these units, the device 300 may also include other components; however, since these components are not relevant to the content of this disclosure embodiment, their illustrations and descriptions are omitted here.

[0082] The device 300 includes a laughter recognition unit 301, configured to recognize laughter from audio data collected from the environment of the electronic device.

[0083] The device 300 includes a laughter feature extraction unit 302, configured to extract a first laughter feature corresponding to the first laughter detected in the audio data in response to the detection of laughter in the audio data.

[0084] The device 300 includes a laughter feature matching unit 303, configured to match a first laughter feature with a second laughter feature stored in a loop buffer of an electronic device to obtain the similarity between the first laughter feature and the second laughter feature, wherein the second laughter feature is a laughter feature corresponding to at least one second laughter extracted within a preset time period in the past.

[0085] The device 300 includes a content interaction unit 304, which is configured to trigger interaction with the content currently being played on the electronic device based on the similarity between a first laughter feature and a second laughter feature.

[0086] It should be understood that Figure 3 The various modules or units of the apparatus 300 shown can be used with reference to Figure 1 The steps in method 100 described correspond to each other. Therefore, the operations, features, and advantages described above for method 100 also apply to apparatus 300 and its included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0087] Although specific functions have been discussed with reference to specific modules above, it should be noted that the functions of the units discussed in this article can be divided into multiple units, and / or at least some functions of multiple units can be combined into a single unit.

[0088] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described above.

[0089] According to another aspect of this disclosure, a computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above-described method when executed by a processor.

[0090] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0091] See Figure 4The present invention describes a structural block diagram of an electronic device 400 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device can be different types of computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0092] like Figure 4 As shown, the electronic device 400 may include at least one processor 401, working memory 402, input unit 404, display unit 405, speaker 406, storage unit 407, communication unit 408 and other output units 409 that are capable of communicating with each other via system bus 403.

[0093] Processor 401 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 401 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Processor 401 may be configured to acquire and execute computer-readable instructions stored in working memory 402, storage unit 407, or other computer-readable media, such as program code of operating system 402a, program code of application program 402b, etc.

[0094] Working memory 402 and storage unit 407 are examples of computer-readable storage media for storing instructions that are executed by processor 401 to perform the various functions described above. Working memory 402 may include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, storage unit 407 may include hard disk drives, solid-state drives, removable media including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Working memory 402 and storage unit 407 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by processor 401 as a specific machine configured to perform the operations and functions described in the examples herein.

[0095] Input unit 404 can be any type of device capable of inputting information to electronic device 400. Input unit 404 can receive input digital or character information and generate key signal input related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone and / or remote control. Output unit can be any type of device capable of presenting information, and can include, but is not limited to, display unit 405, speaker 406 and other output units 409. Other output units 409 can include, but are not limited to, video / audio output terminals, vibrators and / or printers. Communication unit 408 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth™ devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0096] The application program 402b in the working memory 402 can be loaded to execute the various methods and processes described above, for example... Figure 1 Boxes S101-S104 in the diagram. For example, in some embodiments, the laughter-recognition-based content interaction method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 407. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 400 via storage unit 407 and / or communication unit 408. When the computer program is loaded and executed by processor 401, one or more steps of the laughter-recognition-based content interaction method described above can be performed. Alternatively, in other embodiments, processor 401 can be configured to perform the laughter-recognition-based content interaction method by any other suitable means (e.g., by means of firmware).

[0097] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0099] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0101] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0102] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0103] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0104] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of this disclosure is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A content interaction method based on laughter recognition, comprising: Laughter recognition from audio data collected from the environment of electronic devices; In response to the detection of laughter in the audio data, the first laughter feature corresponding to the first laughter detected is extracted; The first laughter feature is matched with the second laughter feature stored in the circular buffer of the electronic device to obtain the similarity between the first laughter feature and the second laughter feature, wherein the second laughter feature is a laughter feature corresponding to at least one second laughter extracted within a preset time period in the past. The interaction with the currently playing content of the electronic device is triggered based on the similarity between the first laughter feature and the second laughter feature.

2. The method according to claim 1, wherein, Interaction with the currently playing content of the electronic device is triggered based on the similarity between the first laughter feature and the second laughter feature, including: If the similarity between the first laughter feature and the second laughter feature is not less than a preset threshold, then the interaction with the currently playing content of the electronic device will not be triggered. If the similarity between the first laughter feature and the second laughter feature is less than the preset threshold, then the interaction with the currently playing content of the electronic device is triggered.

3. The method according to claim 2, wherein, In response to the fact that the similarity between the first laughter feature and the second laughter feature is less than the preset threshold, the method further includes storing the first laughter feature in the circular buffer.

4. The method according to claim 3, wherein, The preset storage space of the circular buffer corresponds to the preset duration. Furthermore, storing the first laughter feature in the circular buffer includes: When the number of laughter features stored in the circular buffer reaches the preset storage space, the first laughter feature is used to overwrite the earliest laughter feature stored in the circular buffer.

5. The method according to claim 1, wherein, The laughter feature corresponding to at least one second laugh has a timestamp corresponding to its storage time. Furthermore, the method further includes: For each of the at least one second laughs, if the time difference between the timestamp of the laugh feature corresponding to the second laugh and the current time exceeds the preset duration, then the laugh feature corresponding to the second laugh is deleted, and the storage space of the laugh feature corresponding to the second laugh is released.

6. The method according to claim 1, wherein the laughter characteristics include at least one of the following: laughter duration, frequency distribution, energy distribution, voiceprint, and plosive rhythm.

7. The method according to claim 1, wherein, Laughter recognition from audio data collected from the environment of electronic devices includes: In response to determining that the audio data in the environment exceeds a predetermined decibel and that the frequency of the audio data matches the human voice frequency, a laughter recognition algorithm is used to identify whether laughter exists in the audio data.

8. The method of claim 1, further comprising, before performing laughter recognition on audio data acquired from the environment of the electronic device: The audio data corresponding to the content currently being played by the electronic device is used as a reference signal, and the echo corresponding to the reference signal is subtracted from the collected audio data in the environment.

9. The method according to claim 1, further comprising, after performing laughter recognition on audio data collected from the environment of the electronic device: In response to the detection of laughter in the audio data, noise suppression is performed on the collected ambient audio data using a Wiener filter, and the contrast of the laughter is enhanced.

10. The method according to claim 1, further comprising: In response to triggering an interaction with the content currently being played on the electronic device, the intensity of the emotion corresponding to the first laugh is determined; Based on the emotional intensity corresponding to the first laugh, a visual representation of the emotion corresponding to the emotional intensity is displayed in the interactive area of ​​the currently playing content.

11. The method according to claim 1, wherein, Triggering interaction with the currently playing content on the electronic device includes: Trigger a vote count for the content currently being played on the electronic device.

12. The method of claim 11, further comprising: Based on the vote count triggered on the currently playing content of the electronic device, a voting visualization corresponding to the vote count is displayed in the interactive area.

13. The method according to claim 2, further comprising: In response to the interaction triggered by the electronic device currently playing content, the first laugh is preprocessed; The audio quality of the first laugh after preprocessing was determined using an audio quality assessment model. In response to determining that the audio quality of the preprocessed first laugh is higher than a predetermined quality threshold, and in response to user authorization, a first bullet comment is triggered to be displayed in the currently playing content of the electronic device, wherein the content of the first bullet comment corresponds to the preprocessed first laugh.

14. The method of claim 13, wherein the audio quality of the preprocessed first laugh is determined using an audio quality assessment model based on at least one of the following: audio clarity, vocal appeal, and emotional positivity.

15. The method of claim 13, further comprising: The pre-processed first laugh is altered using one or more preset voice-changing effects to generate a voice-changing laugh; Using one or more preset video templates, the altered laughter is superimposed on the preset video templates to generate an altered laughter video; In response to user confirmation of posting the altered laughter video.

16. An apparatus for a content interaction method based on laughter recognition, comprising: The laughter recognition unit is configured to recognize laughter from audio data collected from the environment of the electronic device; The laughter feature extraction unit is configured to extract the first laughter feature corresponding to the first laughter in response to the recognition of laughter in the audio data. The laughter feature matching unit is configured to match the first laughter feature with the second laughter feature stored in the circular buffer of the electronic device to obtain the similarity between the first laughter feature and the second laughter feature, wherein the second laughter feature is a laughter feature corresponding to at least one second laughter extracted within a preset time period in the past. The content interaction unit is configured to trigger interaction with the content currently being played by the electronic device based on the similarity between the first laughter feature and the second laughter feature.

17. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores a computer program that, when executed by the at least one processor, implements the method according to any one of claims 1-15.

18. A non-transitory computer-readable storage medium storing a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-15.

19. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-15.

Images