Earphone control method, open earphone, mobile terminal and storage medium

Abstract

The embodiment of the invention provides an earphone control method, an open earphone, a mobile terminal and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps that heart rate data and temperature data of a target object are obtained, then emotion type recognition is conducted on the heart rate data and the temperature data through a preset emotion state judgment model, and emotion state information of the target object is obtained. And sending the emotional state information to the mobile terminal, and finally playing target audio data determined by the mobile terminal according to the emotional state information. According to the embodiment of the invention, the functions of the open earphone can be increased, so that the requirements of a user on the intelligent earphone can be better met.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an earphone control method, an open earphone, a mobile terminal and a storage medium. Background Art

[0002] With the widespread use of smart headphones, they are closely connected with our daily lives. Some open-type smart Bluetooth headphones, due to their portable and non-in-ear characteristics, can be worn on the ears all day as a wearable device, without burden and without affecting daily life. However, in the related art, as a device that can be worn all day, open-type headphones have relatively single functions, are not smart enough, and cannot meet the growing and diverse needs of users. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an earphone control method, an open earphone, a mobile terminal and a storage medium, aiming to increase the functions of the open earphone to better meet the user's demand for smart earphones.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a headset control method, which is applied to a headset and includes:

[0005] Obtain the heart rate data and temperature data of the target object;

[0006] Performing emotion type recognition on the heart rate data and the temperature data through a preset emotion state judgment model to obtain the emotion state information of the target object;

[0007] sending the emotional state information to a mobile terminal;

[0008] The target audio data determined by the mobile terminal according to the emotional state information is played.

[0009] To achieve the above object, a second aspect of an embodiment of the present application provides an earphone control method, which is applied to a mobile terminal. The method includes:

[0010] Receiving emotional state information sent by the headset; wherein the emotional state information is obtained by the headset acquiring the heart rate data and temperature data of the target object, and identifying the emotional type of the heart rate data and the temperature data through a preset emotional state judgment model;

[0011] generating target audio data according to the emotional state information and the terminal operation mode information of the mobile terminal;

[0012] The target audio data is sent to the earphone.

[0013] In some embodiments, the terminal operation mode information includes a music playback mode and an emotional music recommendation mode; and generating target audio data according to the emotional state information and the terminal operation mode information of the mobile terminal includes:

[0014] Identifying the emotional state information;

[0015] In response to the terminal operation mode information being the music playback mode, and the emotional state information matching a preset negative emotion category, obtaining a target duration; wherein the target duration is the matching duration of the emotional state information corresponding to the negative emotion category;

[0016] If the target duration is greater than or equal to a preset duration threshold, setting the terminal operation mode information to the emotional music recommendation mode;

[0017] Acquire music preference data and historical listening data of the target object;

[0018] In response to the terminal operation mode information being the emotional music recommendation mode, the target audio data is generated according to the music preference data and the historical listening data.

[0019] In some embodiments, generating the target audio data according to the music preference data and the historical listening data includes:

[0020] Filtering candidate music data from the music preference data;

[0021] Sorting the candidate music data according to the historical listening data to generate a music playlist;

[0022] The candidate music data is used as the target audio data according to the music play list.

[0023] In some embodiments, the terminal operation mode information includes a standby mode and a conversation operation mode; and generating target audio data according to the emotional state information and the terminal operation mode information of the mobile terminal includes:

[0024] Identifying the emotional state information;

[0025] In response to the terminal operation mode information being the standby mode and the emotional state information matching a preset negative emotion category, generating a function prompt voice, and sending the function prompt voice to the headset to output the function prompt voice to the target object;

[0026] Acquire the input operation of the target object in response to the function prompt voice, and perform operation intention analysis on the input operation to obtain the demand information of the target object;

[0027] Setting the terminal operation mode information to the dialogue operation mode according to the demand information;

[0028] In response to the terminal operation mode information being the dialogue operation mode, the target audio data is generated according to original voice data input by the target object.

[0029] In some embodiments, generating the target audio data according to the original voice data input by the target object includes:

[0030] Performing speech recognition on the original speech data input by the target object to obtain the original speech text;

[0031] The original speech text is semantically interpreted by a preset large language model to obtain an answer text;

[0032] Speech generation is performed on the answer text to obtain the target audio data.

[0033] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application provides an open-type earphone, the open-type earphone comprising:

[0034] A heart rate detection module, used to detect the heart rate data of the target object;

[0035] A temperature detection module, used to detect the temperature data of the target object;

[0036] A control module, the control module is connected to the heart rate detection module and the temperature detection module respectively, and is used to identify the emotion type of the heart rate data and the temperature data through a preset emotion state judgment model to obtain the emotion state information of the target object;

[0037] a communication module, the communication module being connected to the control module and being used to send the emotional state information to a mobile terminal; the communication module being further used to receive data sent by the mobile terminal so that the control module receives target audio data determined by the mobile terminal according to the emotional state information;

[0038] An audio playing module, wherein the audio playing module is connected to the control module and is used to play the target audio data.

[0039] To achieve the above objective, a fourth aspect of an embodiment of the present application provides a mobile terminal, the mobile terminal comprising:

[0040] An information receiving module, used to receive emotional state information sent by the headset; wherein the emotional state information is obtained by the headset acquiring the heart rate data and temperature data of the target object, and identifying the emotional type of the heart rate data and the temperature data through a preset emotional state judgment model;

[0041] An audio data generating module, configured to generate target audio data according to the emotional state information and the terminal operation mode information of the mobile terminal;

[0042] The audio data sending module is used to send the target audio data to the earphone.

[0043] In some embodiments, the mobile terminal further includes a display module;

[0044] The information receiving module is also used to receive the heart rate data and the temperature data. The display module is connected to the information receiving module and is used to display at least one of the heart rate data, the temperature data and the emotional state information to the target object.

[0045] To achieve the above objectives, the fifth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect or the second aspect.

[0046] The headphone control method, open-type headphone, mobile terminal and storage medium proposed in the present application first obtain the heart rate data and temperature data of the target object to understand the user's physiological state. Then, the preset emotional state judgment model is used to analyze these data to accurately identify the emotional state information of the target object. This process gives the headphone the ability to perceive the user's emotions, so that the headphone is no longer just an audio playback device, and has a certain intelligent interaction foundation. Subsequently, the emotional state information is sent to the mobile terminal to achieve effective data transmission. Finally, the target audio data determined by the mobile terminal according to the emotional state information is played, and the user's current emotional needs are met through personalized audio content, which greatly enriches the function of the headphone. The method of this embodiment enables the open-type headphone to no longer be limited to simple audio playback, but to provide more intelligent and diversified services according to the user's emotional state, meet the growing needs of users, make up for the shortcomings of the open-type headphone in the related technology that has a single function and a low degree of intelligence, and improve the user's experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the earphone control method provided by an embodiment of the present application;

[0048] Figure 2is another flow chart of the earphone control method provided by an embodiment of the present application;

[0049] Figure 3 yes Figure 2 Flow chart of step S202 in FIG.

[0050] Figure 4 yes Figure 3 Flow chart of step S305 in FIG.

[0051] Figure 5 yes Figure 2 Flow chart of step S202 in FIG.

[0052] Figure 6 yes Figure 5 Flow chart of step S505 in FIG.

[0053] Figure 7 is a schematic diagram of the structure of an open-type earphone provided in an embodiment of the present application;

[0054] Figure 8 It is a structural diagram of a mobile terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] First, some nouns involved in this application are analyzed:

[0059] Open-Back earphones: Open-Back Bluetooth earphones are small and lightweight earphones that transmit sound to the ears through an open design without inserting the earphones into the ear canal. These earphones are mainly divided into two types: bone conduction earphones and air conduction earphones. Bone conduction earphones transmit sound waves directly to the skull through bone conduction technology, bypassing the eardrum to transmit sound. Users do not block their ears when using them, so they can hear ambient sounds at the same time. They are suitable for scenes such as running and cycling that require attention to the surrounding environment. Air conduction earphones, on the other hand, use a design in which the speaker fits the outside of the ear, and transmits sound to the ear canal through air, making it more comfortable and breathable to wear while retaining the perception of external sounds. The advantage of this type of earphone is that it is light to wear, does not compress the ear canal, and allows the user to perceive ambient sounds, making it particularly suitable for sports and commuting.

[0060] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0061] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0062] With the widespread use of smart headphones, they are closely connected with our daily lives. Some open-type smart Bluetooth headphones, due to their portable and non-in-ear characteristics, can be worn on the ears all day as a wearable device, without burden and without affecting daily life. However, in the related art, as a device that can be worn all day, open-type headphones have relatively single functions, are not smart enough, and cannot meet the growing and diverse needs of users.

[0063] Based on this, the embodiments of the present application provide an earphone control method, an open earphone, a mobile terminal and a storage medium, aiming to increase the functions of the open earphone to better meet the user's demand for smart earphones.

[0064] The earphone control method, open earphone, mobile terminal and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the earphone control method in the embodiments of the present application is described.

[0065] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0066] Figure 1 is an optional flow chart of the earphone control method provided in an embodiment of the present application. Figure 1 The earphone control method in the embodiment is applied to earphones and may include but is not limited to steps S101 to S106.

[0067] Step S101, obtaining the heart rate data and temperature data of the target object.

[0068] Step S102, using a preset emotion state judgment model to identify the emotion type of the heart rate data and the temperature data to obtain the emotion state information of the target object.

[0069] Step S103: sending the emotional state information to the mobile terminal.

[0070] Step S104: playing the target audio data determined by the mobile terminal according to the emotional state information.

[0071] In the steps S101 to S104 shown in the embodiment of the present application, the user's physiological state is first understood by acquiring the heart rate data and temperature data of the target object. Then, these data are analyzed using a preset emotional state judgment model to accurately identify the emotional state information of the target object. This process gives the headset the ability to perceive the user's emotions, so that the headset is no longer just an audio playback device, and has a certain intelligent interaction foundation. Subsequently, the emotional state information is sent to the mobile terminal to achieve effective data transmission. Finally, the target audio data determined by the mobile terminal based on the emotional state information is played, and the user's current emotional needs are met through personalized audio content, greatly enriching the function of the headset. The method of this embodiment enables open-type headphones to no longer be limited to simple audio playback, but to provide more intelligent and diversified services based on the user's emotional state, meet the growing needs of users, make up for the shortcomings of open-type headphones in related technologies that have a single function and low intelligence, and enhance the user's experience.

[0072] In step S101 of some embodiments, the target object refers to the user wearing the headset. A photoplethysmography (PPG) sensor can be set in the headset, and the sensor measures the heart rate data of the target object by detecting changes in the absorption of light by the blood. At the same time, a contact temperature sensor, such as a thermistor sensor, can also be set in the headset. When the sensor is close to the ear skin of the target object, the body surface temperature is measured by using the characteristic that the resistance of the thermistor changes with temperature. A non-contact infrared temperature sensor can also be used to infer the body surface temperature by detecting the intensity of infrared rays radiated by the human body, but it is not limited to this.

[0073] It is understandable that during the collection process, it is necessary to ensure that the temperature data collection time is synchronized with the heart rate data collection time or to collect them according to a certain time correspondence, so as to facilitate subsequent unified data analysis and processing. For example, the collection timestamps of the two can be set to the same timing reference to facilitate subsequent related queries.

[0074] In step S102 of some embodiments, the emotional state information refers to different emotional states of the target object, such as happiness, sadness, anger, calmness, etc. Heart rate variability (HRV) can usually be calculated based on heart rate data. This data refers to the volatility of the time interval between consecutive heartbeats (i.e., RR interval or the peak interval of the R wave in the electrocardiogram) within a certain period of time, reflecting the regularity and variation of the heartbeat, and further reflecting the different moods of people. Generally speaking, the higher the HRV value, the less stress, and the more relaxed and happy the mood, as shown in Table 1.

[0075] Emotional state HRV(ms) Pleasure 65 Relax 55 calm 45 disturbed 35 sad 18

[0076] Table 1

[0077] However, it is not accurate enough to query the corresponding emotional state of the target object only by measuring HRV. HRV will have certain differences due to factors such as gender and age of different individuals. In addition, under different emotions, the surface temperature of a person may also change to a certain extent. For example, when a person feels angry, the temperature of the face will rise to a certain extent. The emotional state judgment model provided in this embodiment is a pre-trained deep learning model, which can perform a comprehensive evaluation based on the two modal data of body temperature and heart rate, and accurately identify the emotional type of the target object. The heart rate data and temperature data of the target object obtained in step S101 are feature extracted to generate corresponding feature vectors, which are input into the trained emotional state judgment model in real time. The model calculates the probability value of the target object being in different emotional states based on the input feature vectors, for example, the probability of happy emotions is 0.6, the probability of sad emotions is 0.2, etc., and then the emotional state with the highest probability is used as the preliminary judgment result.

[0078] In step S103 of some embodiments, a suitable communication protocol is selected for data transmission according to the network connection capability of the mobile terminal and the application scenario requirements. If the mobile terminal and the data acquisition or processing device are in the same local area network environment, the Wi-Fi protocol can be used preferentially to transmit the emotional state information by using its higher data transmission rate and stability. If communication is required in a wide area network environment, such as the device interacting with a remote mobile terminal, a mobile network communication protocol such as 4G and 5G can be used to ensure that information can be sent to the mobile terminal anytime and anywhere.

[0079] Considering that emotional state information involves personal privacy, emotional state information can be encrypted during transmission. Common symmetric encryption algorithms or asymmetric encryption algorithms can be used to encrypt data. Only mobile terminals with corresponding decryption keys can decrypt and view received data. At the same time, identity authentication is required during the establishment of the communication link, such as through pre-shared keys or digital certificates between the device and the mobile terminal to verify the legitimacy of both parties and prevent information from being illegally obtained or tampered with.

[0080] In other embodiments, the heart rate data and temperature data may also be sent together to a mobile terminal or earphone box with a display screen. When the target object has a viewing requirement, the mobile terminal or earphone box can display the heart rate data and temperature data sent by the earphone.

[0081] In step S104 of some embodiments, the target audio data is audio data obtained by the mobile terminal analyzing the emotional state information.

[0082] See also Figure 2, Figure 2 is another optional flow chart of the earphone control method provided in an embodiment of the present application. Figure 2 The earphone control method in the embodiment is applied to a mobile terminal, and may include but is not limited to steps S201 to S203:

[0083] Step S201, receiving emotional state information sent by the earphone.

[0084] Step S202: Generate target audio data according to the emotional state information and the terminal operation mode information of the mobile terminal.

[0085] Step S203: sending the target audio data to the earphone.

[0086] In step S201 of some embodiments, the emotional state information is obtained by obtaining the heart rate data and temperature data of the target object through the earphone, and the emotion type is identified by the heart rate data and temperature data through a preset emotional state judgment model.

[0087] In step S202 of some embodiments, the terminal operation mode information refers to the current working mode of the mobile terminal, including music playing mode, emotional music recommendation mode, standby mode, dialogue operation mode and call mode. It should be noted that in this embodiment, when the target user communicates with others through the mobile terminal and the headset, that is, when the terminal operation mode information is the call mode, the mobile terminal regards the current call task as the highest priority task, so as to allow the target object to complete the current call. At this time, the mobile terminal will not respond to the emotional state information and perform other operations.

[0088] When the terminal operation mode information is other operation modes, the process of generating target audio data in response to different emotional state information will be further explained in the following embodiments.

[0089] In step S203 of some embodiments, the target audio data can be music files of different styles or preset voice information. In some embodiments, the target audio data is packaged according to the communication protocol requirements established with the earphone. For example, in the Bluetooth audio transmission protocol, the audio data is packaged according to the data format specified therein, and necessary protocol header information (including audio coding format, number of channels, sampling rate and other parameter descriptions) is added so that the earphone end can correctly identify and parse the received audio data. If the data volume of the target audio data is large and exceeds the maximum allowed number of bytes of a single transmission specified in the communication protocol, the audio data needs to be sub-packetized, each data packet is numbered in a certain order, and corresponding sub-packet identification and length information are added to facilitate the earphone end to reassemble and restore after receiving, so as to ensure the integrity and continuity of the audio data. After sending each packet of audio data, wait for the confirmation message from the headset. If no confirmation message is received within a certain period of time, it is considered that an error has occurred in the transmission of the data packet, and the data packet will be resent, and a reasonable retransmission limit will be set. For example, the number of retransmissions can be set to 3 times. If it still fails after 3 retransmissions, the target object is prompted to check the connection or headset status so that the headset can perform subsequent audio playback operations.

[0090] In steps S201 to S203 shown in the embodiment of the present application, the heart rate and temperature data of the target object are obtained through the earphone, and the emotional state is identified through the preset model. This process realizes the accurate perception of the target object's emotions, fully utilizes the advantage of the earphone being close to the target object, and converts physiological data into understandable emotional information. Then, the target audio data is generated by combining the emotional state information with the mobile terminal operation mode information. This multi-dimensional analysis and processing not only takes into account the user's current emotional needs, but also takes into account the terminal usage scenario, making the generated audio data more targeted and adaptable, greatly improving the personalization of the user experience. Finally, the target audio data is sent to the earphone to ensure that the target object can receive audio that matches its own emotions and the current scene in a timely manner. Whether it is in alleviating negative emotions, enhancing positive emotions, or adapting to audio needs in different usage scenarios, it can provide more intelligent services, which improves the overall interactive experience between users and devices.

[0091] See also Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S305:

[0092] Step S301, identifying emotional state information.

[0093] Step S302, in response to the terminal operation mode information being a music play mode, and the emotional state information matching a preset negative emotion category, obtaining a target duration.

[0094] Step S303: If the target duration is greater than or equal to the preset duration threshold, the terminal operation mode information is set to the emotional music recommendation mode.

[0095] Step S304, obtaining the music preference data and historical listening data of the target object.

[0096] Step S305 , in response to the terminal operation mode information being an emotional music recommendation mode, generating target audio data according to the music preference data and the historical listening data.

[0097] In step S301 of some embodiments, the identification in this embodiment refers to classifying the emotional state information, and the emotional state information can be roughly divided into two categories: positive emotion category and negative emotion category, where the positive emotion category can include happiness, pleasure and calmness, and the negative emotion category can include anxiety, sadness and anger.

[0098] In step S302 of some embodiments, the terminal operation mode information is a music playing mode, that is, the mobile terminal is currently selecting or controlling a music file through a music software (such as pausing, sliding progress, and switching songs, etc.), and the earphone plays the selected music file to the target object. The target duration is the matching duration of the emotional state information corresponding to the negative emotion category.

[0099] In step S303 of some embodiments, the preset duration threshold may be 3 minutes, 5 minutes or 10 minutes, and the present embodiment does not strictly limit its specific duration. When the target duration is greater than or equal to the preset duration threshold, the mobile terminal may determine that the target object has been in a negative emotional state for a certain period of time. At this point, the mobile terminal may adjust the music push rules and play order, and achieve an auxiliary effect of helping the target object to regulate emotions through precisely adapted music push. In this scenario, the operating mode of the mobile terminal is the emotional music recommendation mode.

[0100] It should be noted that a person's emotions may have multiple emotions within a period of time. When the mobile terminal generates the target audio data, the emotional state information is classified into positive emotion categories and negative emotion categories, which can help the mobile terminal determine whether it is necessary to perform an emotional regulation operation for the target object. Exemplarily, the target object currently has emotional fluctuations of anger and sadness, and the preset duration threshold is 3 minutes. If the mobile terminal does not classify the emotional state information, when the emotional state information is identified as anger, the target duration is started. If the target duration is 2 minutes, the emotional state information is identified as sadness, and the mobile terminal responds again, that is, the target duration is re-timed. If the target duration is 1 minute, the emotional state information is identified as anger again, and the timing is re-timed. For the mobile terminal, it is difficult for the mobile terminal to determine that anger and sadness are actually emotional states that require emotional regulation operations on the target object, resulting in continuous re-timing of the target duration, which may eventually lead to the inability to open or fail to open the emotional music recommendation mode in time. If the emotional state information is classified, the mobile terminal can classify anger and sadness into one category and accumulate the target duration, thereby achieving a timely response to the emotional regulation needs. From the perspective of human cognition, although anger and sadness are both negative emotions, there are substantial differences between the two. Therefore, in some scenarios where emotional state information needs to be displayed, the corresponding emotional state information is directly displayed so that the target object can understand his or her emotional state more clearly.

[0101] In step S304 of some embodiments, the music preference data may include songs that the target object has performed a collection operation on (e.g., a favorite playlist), singers that the target object listens to most, song styles that the target object listens to most, and song genres that the target object listens to most. The historical listening data may refer to the number of complete listening times or the cumulative playing time corresponding to each song listened to in a preset period of time in the past, but is not limited thereto.

[0102] In step S305 of some embodiments, see Figure 4 Step S305 may include but is not limited to steps S401 to S403:

[0103] Step S401, filtering and obtaining candidate music data from the music preference data.

[0104] Step S402: sorting the candidate music data according to the historical listening data to generate a music playlist.

[0105] Step S403: taking the candidate music data as the target audio data according to the music playlist.

[0106] In step S401 of some embodiments, the music preference data can be screened according to the emotion label of each song. The emotion label of a song is similar to the emotional state information, covering categories such as joy, sadness, and calmness. During specific screening, songs with joy and calm emotion labels in the music preference data are selected as candidate music data. Regarding the acquisition of song emotion labels, they can be pre-labeled by the music software platform, or the songs in the music preference data can be identified through a pre-trained model. When using the model for identification, the characteristics of the song such as rhythm, pitch, and lyrics text are used as input. After model analysis and processing, the emotion label corresponding to each song can be obtained.

[0107] In step S402 of some embodiments, the play counts corresponding to the candidate music data are queried in the historical listening data, and the candidate music data are sorted in descending order according to the play counts, and the resulting list is the music play list.

[0108] In step S403 of some embodiments, for music stored locally, the corresponding audio file path is found through file system operations according to the song information in the playlist to ensure that the file exists and is readable. For music stored in the cloud, the audio data is obtained from the cloud storage service using an API call.

[0109] Steps S401 to S403 shown in the embodiment of the present application select songs that better meet the user's emotional needs as candidate music data by screening based on the emotional tags of the songs in the music preference data. Then, the candidate music data is sorted in descending order according to the number of plays in the historical listening data to generate a music playlist, making full use of the target object's past listening behavior information, giving priority to songs that are more frequently listened to and of greater interest, and further improving the pertinence and personalization of music recommendations. Providing users with a coherent, smooth and personalized music service experience, taking advantage of the ability of music to help users regulate their emotions, and improving the efficiency of music resource utilization.

[0110] Steps S301 to S305 shown in the embodiment of the present application first accurately identify the emotional state information, and can keenly perceive the emotional category of the target object. Then, when it is detected that the terminal is in music playback mode and the target object is in a negative mood, the target duration is obtained, thereby realizing continuous detection and quantitative evaluation of the emotional state of the target object. When the target duration reaches the preset threshold and switches to the emotional music recommendation mode, the transition from the normal mode to the targeted service mode is realized. Subsequently, the music preference data and historical listening data of the target object are obtained, and the music preference information of the target object is deeply mined. Finally, the target audio data is generated based on these rich data, so that the recommended music not only fits the daily music preferences of the target object, but also plays a regulatory role for the current negative emotions, thereby enhancing the personalized service capabilities of the system and providing the target object with a more targeted and humane music service.

[0111] See also Figure 5 In some embodiments, the terminal operation mode information includes a standby mode and a conversation operation mode, and step S202 may also include but is not limited to steps S501 to S505:

[0112] Step S501, identifying emotional state information.

[0113] Step S502, in response to the terminal operation mode information being the standby mode and the emotional state information matching the preset negative emotion category, generating a function prompt voice, and sending the function prompt voice to the headset to output the function prompt voice to the target object.

[0114] Step S503: obtaining the input operation of the target object in response to the function prompt voice, and performing operation intention analysis on the input operation to obtain the demand information of the target object.

[0115] Step S504: setting the terminal operation mode information to a dialogue operation mode according to the demand information.

[0116] Step S505 , in response to the terminal operation mode information being the dialogue operation mode, generating target audio data according to the original voice data input by the target object.

[0117] In step S501 of some embodiments, this step is consistent with the relevant embodiments of step S301 and will not be described in detail here.

[0118] In step S502 of some embodiments, the standby mode refers to a state in which the mobile terminal and the headset are connected in communication, but no audio is played or a call is made. The function prompt voice refers to a voice that asks the target object whether the language model needs to communicate with it. For example, the function prompt voice can be "I detected that you are a little sad, do you need me to chat with you?" This embodiment does not strictly limit the specific content of the function prompt voice. In some embodiments, the function prompt voice can generate voice data with different timbres through different voice packages, and the target object can select the voice package according to his or her preferences.

[0119] In step S503 of some embodiments, the input operation may be a voice prompt of the target object's voice reply function, such as saying "OK" or "No", which is collected and acquired by the microphone on the headset. It may also be clicking on the interactive interface of the mobile terminal, or entering text in the dialog box of the interactive interface, which is not strictly limited in this embodiment. The demand information is information used to reflect whether the target object needs to start a conversation. For voice input, the operation intention analysis may be to use voice recognition technology to convert the voice into text. Then, natural language processing (NLP) technology, such as lexical analysis, syntactic analysis, semantic analysis, etc., is used to analyze the converted text. With the help of a pre-trained intention recognition model (such as an intention classification model based on a neural network), it is determined whether the target object wants to chat. In other embodiments, the operation intention analysis may also be directly determined based on the target object clicking a specific icon. For example, when the target object clicks the "Yes" icon, it can be determined that the demand information is that the target object needs a chat service.

[0120] In step S504 of some embodiments, when the demand information indicates that the target object needs a chat service, the terminal operation mode information is set to a conversation operation mode. When the demand information indicates that the target object does not need a chat service, the terminal operation mode information is not modified.

[0121] In step S505 of some embodiments, see Figure 6 Step S505 includes but is not limited to steps S601 to S603:

[0122] Step S601, performing speech recognition on the original speech data input by the target object to obtain the original speech text.

[0123] Step S602: semantically interpret the original speech text through a preset large language model to obtain an answer text.

[0124] Step S603, generate speech for the answer text to obtain target audio data.

[0125] In step S601 of some embodiments, the original voice data refers to the voice data input by the target object to the mobile terminal. A mature voice recognition engine can be used to perform voice recognition on the original voice data, and the corresponding text obtained is the original voice text.

[0126] In step S602 of some embodiments, the large language model can be a transformer model, an LSTM model, or a GPT-3 model, etc. The text output by the large language model for the original voice text is the answer text. Before the original voice text is input into the large language model, certain preprocessing is required, including removing noise in the text (such as special characters, irrelevant punctuation marks, etc.), normalizing the text (such as converting the text to a unified case format), correcting spelling errors, etc. Then, by calling the API interface of the large language model, the preprocessed original voice text is sent to the model. During the calling process, it is necessary to set appropriate parameters, such as the maximum generation length and the temperature coefficient (used to control the randomness of the generated text), etc., to control the output of the model. After the model receives the text, it will perform semantic interpretation according to its trained knowledge and algorithm, and generate the corresponding answer text.

[0127] In step S603 of some embodiments, text-to-speech technology is used to convert the answer text into speech. When performing speech synthesis, it is necessary to set at least one of the following parameters: speech rate (which can be adjusted according to the nature of the content and the user's habits, such as the news broadcast content can be appropriately accelerated, while the story telling content can be appropriately slowed down), intonation (such as rising tone, falling tone, etc., to enhance the expressiveness of the speech) and volume, etc. The answer text is sent to the speech synthesis engine, and the engine will generate corresponding speech audio data. After the audio data is generated, some optimization processing may be required, such as removing noise from the audio, performing audio equalization processing, etc., to improve the quality of the audio. Finally, the generated target audio data is format converted and encoded according to the requirements of the terminal device and the headset so that the headset can play smoothly.

[0128] Steps S601 to S603 shown in the embodiment of the present application first accurately convert the original speech input by the target object into text through speech recognition. Then, the original speech text is semantically interpreted using a preset large language model. The model can deeply understand the intention of the target object, generate high-quality answer text, provide valuable information and accurate responses to the target object, and significantly enhance the intelligent interaction ability and information processing ability of the system. Finally, the answer text is converted into speech through speech synthesis technology, which facilitates the target object to receive information in the most convenient way. The method of this embodiment realizes the full process automation and intelligence of voice interaction, greatly improves the experience of the target object, and provides a more convenient, efficient and intelligent interactive service for the target object.

[0129] Steps S501 to S505 shown in the embodiment of the present application first accurately identify the emotional state information, and can keenly perceive the emotional category of the target object. When it is detected that the terminal is in standby mode and the target object is in a negative mood, a function prompt voice is actively generated and sent to the headset to provide a help path for the target object, opening an interactive channel with the target object. Then, by obtaining the input operation of the target object and performing an in-depth intent analysis, the demand information is obtained to determine whether a conversation is needed. Then, the terminal is switched to the conversation operation mode according to the demand information. Finally, the target audio data is generated according to the original voice data of the target object to meet the chat needs of the target object when it is in a negative mood. The method of this embodiment realizes the intelligent processing of the entire process from emotion perception to demand response, greatly improves the experience of the target object, and provides a more personalized and humane service for the target object.

[0130] See also Figure 7 The embodiment of the present application further provides an open-type earphone, the earphone comprising:

[0131] The heart rate detection module 701 is used to detect the heart rate data of the target object.

[0132] The temperature detection module 702 is used to detect the temperature of the target object.

[0133] The control module 703 is connected to the heart rate detection module and the temperature detection module respectively, and is used to identify the emotion type of the heart rate data and the temperature data through a preset emotion state judgment model to obtain the emotion state information of the target object.

[0134] The communication module 704 is connected to the control module and is used to send the emotional state information to the mobile terminal. The communication module is also used to receive data sent by the mobile terminal so that the control module receives the target audio data determined by the mobile terminal according to the emotional state information.

[0135] The audio playing module 705 is connected to the control module and is used to play the target audio data.

[0136] The specific implementation of the open-type earphone is basically the same as the specific implementation of the earphone control method applied to the earphone, and will not be described in detail here. Due to the characteristic of the open-type earphone not being inserted into the ear, the target object can wear it all day without causing any burden.

[0137] See also Figure 8 The embodiment of the present application further provides a mobile terminal, which can implement the above-mentioned earphone control method, and the mobile terminal includes:

[0138] The information receiving module 801 is used to receive the emotional state information sent by the earphone. The emotional state information is obtained by the earphone acquiring the heart rate data and temperature data of the target object and identifying the emotional type of the heart rate data and temperature data through a preset emotional state judgment model.

[0139] The audio data generating module 802 is used to generate target audio data according to the emotional state information and the terminal operation mode information of the mobile terminal.

[0140] The audio data sending module 803 is used to send the target audio data to the earphone.

[0141] In some embodiments, the mobile terminal also includes a display module 804, the information receiving module 801 is also used to receive heart rate data and temperature data, and the display module 804 is connected to the information receiving module 801 and is used to display at least one of the heart rate data, temperature data and emotional state information to the target object.

[0142] In some embodiments, the mobile terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, an earphone box, etc.

[0143] The specific implementation of the mobile terminal is substantially the same as the specific embodiment of the earphone control method applied to the mobile terminal, and will not be described in detail herein.

[0144] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned headphone control method is implemented.

[0145] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0147] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0148] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0150] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0151] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0156] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for controlling a headset, characterized in that: Applied to headphones, the method comprises: Obtain the heart rate data and temperature data of the target object; Performing emotion type recognition on the heart rate data and the temperature data through a preset emotion state judgment model to obtain the emotion state information of the target object; sending the emotional state information to a mobile terminal; The target audio data determined by the mobile terminal according to the emotional state information is played.

2. A method for controlling an earphone, characterized in that: Applied to a mobile terminal, the method comprises: Receiving emotional state information sent by the headset; wherein the emotional state information is obtained by the headset acquiring the heart rate data and temperature data of the target object, and identifying the emotional type of the heart rate data and the temperature data through a preset emotional state judgment model; generating target audio data according to the emotional state information and the terminal operation mode information of the mobile terminal; The target audio data is sent to the earphone.

3. The method according to claim 2, characterized in that The terminal operation mode information includes a music playing mode and an emotional music recommendation mode; The generating target audio data according to the emotional state information and the terminal operation mode information of the mobile terminal includes: Identifying the emotional state information; In response to the terminal operation mode information being the music playback mode, and the emotional state information matching a preset negative emotion category, obtaining a target duration; wherein the target duration is the matching duration of the emotional state information corresponding to the negative emotion category; If the target duration is greater than or equal to a preset duration threshold, setting the terminal operation mode information to the emotional music recommendation mode; Acquire music preference data and historical listening data of the target object; In response to the terminal operation mode information being the emotional music recommendation mode, the target audio data is generated according to the music preference data and the historical listening data.

4. The method according to claim 3, characterized in that The generating the target audio data according to the music preference data and the historical listening data comprises: Filtering candidate music data from the music preference data; Sorting the candidate music data according to the historical listening data to generate a music playlist; The candidate music data is used as the target audio data according to the music play list.

5. The method according to claim 2, characterized in that: The terminal operation mode information includes a standby mode and a conversation operation mode; the generating target audio data according to the emotional state information and the terminal operation mode information of the mobile terminal includes: Identifying the emotional state information; In response to the terminal operation mode information being the standby mode and the emotional state information matching a preset negative emotion category, generating a function prompt voice, and sending the function prompt voice to the headset to output the function prompt voice to the target object; Acquire the input operation of the target object in response to the function prompt voice, and perform operation intention analysis on the input operation to obtain the demand information of the target object; Setting the terminal operation mode information to the dialogue operation mode according to the demand information; In response to the terminal operation mode information being the dialogue operation mode, the target audio data is generated according to original voice data input by the target object.

6. The method according to claim 5, characterized in that The step of generating the target audio data according to the original voice data input by the target object includes: Performing speech recognition on the original speech data input by the target object to obtain the original speech text; The original speech text is semantically interpreted by a preset large language model to obtain an answer text; Speech generation is performed on the answer text to obtain the target audio data.

7. An open-type headphone, characterized in that: The open-back headphones include: A heart rate detection module, used to detect the heart rate data of the target object; A temperature detection module, used to detect the temperature data of the target object; A control module, the control module is connected to the heart rate detection module and the temperature detection module respectively, and is used to identify the emotion type of the heart rate data and the temperature data through a preset emotion state judgment model to obtain the emotion state information of the target object; a communication module, the communication module being connected to the control module and being used to send the emotional state information to a mobile terminal; the communication module being further used to receive data sent by the mobile terminal so that the control module receives target audio data determined by the mobile terminal according to the emotional state information; An audio playing module, wherein the audio playing module is connected to the control module and is used to play the target audio data.

8. A mobile terminal, characterized in that: The mobile terminal comprises: An information receiving module, used to receive emotional state information sent by the headset; wherein the emotional state information is obtained by the headset acquiring the heart rate data and temperature data of the target object, and identifying the emotional type of the heart rate data and the temperature data through a preset emotional state judgment model; An audio data generating module, configured to generate target audio data according to the emotional state information and the terminal operation mode information of the mobile terminal; The audio data sending module is used to send the target audio data to the earphone.

9. The mobile terminal according to claim 8, characterized in that: The mobile terminal also includes a display module; The information receiving module is also used to receive the heart rate data and the temperature data. The display module is connected to the information receiving module and is used to display at least one of the heart rate data, the temperature data and the emotional state information to the target object.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the earphone control method according to claim 1 or the earphone control method according to any one of claims 2 to 6 is implemented.

Images