Interaction method and system, earphone and storage medium

Abstract

The invention discloses an interaction method and system, an earphone and a storage medium. The method comprises the steps that the earphone obtains target data through a target module configured by the earphone, the target data are sent to a preset background, the target data comprise voice data and image data, and the target module comprises a camera module and a voice recognition module; the preset background uses a preset large language model to determine a user intention according to the voice data, and sends response data to the earphone according to the user intention and the image data and / or the voice data; and the earphone broadcasts response voice according to the response data. According to the technical scheme, the scene seen by the human eyes is captured in real time, the multi-mode large language model is used for processing and analysis, the intention of the user is rapidly and accurately recognized, the response to the user is output, and rich interaction experience is provided for the user.

Description

Technical Field

[0001] The present invention relates to the technical field of smart earphones, and in particular to an interaction method, system, earphones and storage medium. Background Art

[0002] With the continuous development of smart wearable devices, people's lifestyles are gradually changing. With the rapid development of artificial intelligence technology and the Internet of Things, smart wearable devices are no longer limited to the fields of health and sports.

[0003] At present, smart headphones are a kind of wearable device integrated with intelligent systems. They not only have the audio playback function of traditional headphones, but also connect to mobile terminals such as smartphones through Bluetooth technology to achieve a variety of extended functions.

[0004] However, traditional smart headphones usually only support voice interaction, and the headphones can usually only perceive the environment through the microphone of the headphones. The services that the headphones can provide are not rich enough, which affects the user's interactive experience. Summary of the invention

[0005] The present invention provides an interactive method, system, earphone and storage medium to improve the interactive experience of users.

[0006] In a first aspect, the present invention provides an interaction method, comprising:

[0007] The headset obtains target data through a target module configured by itself, and sends the target data to a preset background, wherein the target data includes voice data and image data, and the target module includes a camera module and a voice recognition module;

[0008] The preset background determines the user intention according to the voice data using a preset large language model, and sends response data to the headset according to the user intention and the image data and / or the voice data;

[0009] The earphone broadcasts a response voice according to the response data.

[0010] In a second aspect, the present invention provides an interactive system, comprising:

[0011] The data acquisition module in the headset is used to acquire target data through the target module configured by itself and send the target data to the preset background, wherein the target data includes voice data and image data, and the target module includes a camera module and a voice recognition module;

[0012] A response data sending module in the preset background is used to determine the user intention according to the voice data using a preset large language model, and send the response data to the headset according to the user intention and the image data and / or the voice data;

[0013] The broadcast module in the earphone is used to broadcast the response voice according to the response data.

[0014] In a third aspect, the present invention provides an earphone, the earphone comprising: an earphone body and a camera module;

[0015] The headset body includes a battery, a voice recognition module, at least one processor, and a memory in communication with the at least one processor;

[0016] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the interaction method of the first aspect mentioned above.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement the interaction method of the first aspect when executed.

[0018] In the interactive solution provided by the present invention, the headset uses its own camera module and voice recognition module to collect voice data and image data, and sends the data to the background. The background uses a large language model to mine the user's intention from the data, and outputs response data to the headset according to the user's intention and the received data, so that the headset can broadcast the corresponding response voice. This solution not only supports voice interaction, but also uses the camera module to perceive the environment, realizes real-time capture of the scene seen by the human eye, and uses a multimodal large language model for processing and analysis, quickly and accurately identifies the user's intention and outputs a response to the user, providing users with a rich interactive experience.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1is a flowchart of an interaction method provided according to Embodiment 1 of the present invention;

[0022] Figure 2 is a flowchart of an interaction method provided according to Embodiment 2 of the present invention;

[0023] Figure 3 is a schematic diagram of the structure of an interactive system provided according to Embodiment 3 of the present invention;

[0024] Figure 4 It is a schematic diagram of the structure of an earphone provided according to Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In the description of the present invention, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are an "or" relationship. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1A flowchart of an interaction method is provided for the first embodiment of the present invention. This embodiment is applicable to the situation where the headset interacts with a preset background. The method can be executed by an interaction system. The interaction system can be implemented in the form of hardware and / or software. The interaction system can be configured in the headset. The headset can be composed of two or more physical entities or one physical entity.

[0029] like Figure 1 As shown, the first embodiment of the present invention provides an interaction method, which specifically includes the following steps:

[0030] S101. The headset obtains target data through a target module configured by itself, and sends the target data to a preset background, wherein the target data includes voice data and image data, and the target module includes a camera module and a voice recognition module.

[0031] In this embodiment, a camera module and a voice recognition module can be pre-configured in the headset, wherein the camera module can be used to obtain images, and the voice recognition module can be used to obtain voices. The headset can send the acquired target data to a mobile terminal (such as a mobile phone) through the network, and the mobile terminal then sends the information in the target data to a preset background through the network, wherein the mobile terminal can pre-configure an application corresponding to the headset to realize data interaction with the headset.

[0032] Exemplarily, the process of establishing a connection between the headset and the mobile phone may include:

[0033] 1) Turn on the Wi-Fi and Bluetooth on your phone, and open the headset APP;

[0034] 2) Turn on the headset, put the headset's WiFi into Station mode, and start Bluetooth broadcasting;

[0035] 3) Select the SmartSense device in the headset app and connect it via Bluetooth;

[0036] 4) After the Bluetooth connection is successful, turn on the mobile phone's Wifi hotspot so that the headset can actively connect to the mobile phone's hotspot via Wifi;

[0037] 5) After the headset is successfully connected to the mobile phone hotspot, a Websocket communication channel is established;

[0038] 6) The headset App obtains headset device information through Websocket, including MAC address, IP address, battery information, and version number.

[0039] At this point, the headset and the mobile phone have successfully established a communication connection and can begin transmitting control commands and audio and video stream data.

[0040] S102: The preset background determines the user intention according to the voice data using a preset large language model, and sends response data to the headset according to the user intention and the image data and / or the voice data.

[0041] S103: The earphone broadcasts a response voice according to the response data.

[0042] Optionally, the preset large language model may be one or more types of multimodal large language models, and the preset large language model may be ChatGPT (Chat Generative Pre-trained Transformer, a pre-trained language model based on the Transformer model), and / or GPT-4o. Multimodality can be understood as the large language model being able to process image data and speech data.

[0043] In this embodiment, the background can be pre-configured in the cloud, and a multimodal large language model can be configured in the background. The large language model can determine the user's intention based on the voice data, and output response data based on the user's intention and the image data and / or voice data collected by the headset, and send the response data to the headset through the mobile terminal. The headset can broadcast a response voice based on the response data.

[0044] For example, if the user intends to assist in reading, the large language model can identify and provide answers based on the image of the reading material captured by the headset. The preset background can also use the contextual understanding and reasoning memory capabilities of the large language model to output answers to questions raised by users based on voice data, so that users can ask questions through voice interaction and get responses in a timely manner, and have discussion-based communications with AI, thereby assisting users in understanding reading content without relying on traditional search methods such as mobile phones and computers. Compared with existing AI headsets or mobile phone AI camera apps, this method improves the efficiency and quality of information acquisition, and provides users with a convenient and efficient learning and reading experience.

[0045] Exemplarily, the preset background may also send the response data to the mobile terminal, and the mobile terminal may instruct the headset to broadcast the response voice corresponding to the response data through the network.

[0046] The interaction process between the headset, the mobile phone and the preset background may include:

[0047] 1) After the headset is awakened by triggering a designated button or by voice, the headset starts uploading voice data to the mobile phone and enters voice interaction mode;

[0048] 2) After the mobile phone sends the voice data to the cloud server (i.e., the preset background), the cloud server uses the preset large language model to identify the user's intention based on the voice data. For example, the voice data may include voice data containing questions such as "Look at what I have in my hand" or "Look at what is happening in front of me".

[0049] 3) The cloud server sends a photo or video command to the headset through the mobile phone. After receiving the command, the headset turns on the camera and captures the video image and uploads it to the cloud server;

[0050] 4) The cloud server uses the preset large language model to analyze and process the received voice data and image data, generates an audio stream (i.e., response data) and sends it to the mobile phone;

[0051] 5) After the mobile phone receives the audio stream data, it plays it on the earphone via Bluetooth. The preset large language model supports multiple rounds of conversations. By repeating steps 2) to 5), the user can continue to have a conversation with the preset large language model;

[0052] 6) End this round of conversation by triggering a designated button or by voice.

[0053] In the interactive method provided by the embodiment of the present invention, the headset uses the camera module and voice recognition module configured by itself to collect voice data and image data, and sends the data to the background. The background uses a large language model to mine the user's intention from the data, and outputs response data to the headset according to the user's intention and the received data, so that the headset can broadcast the corresponding response voice. This solution not only supports voice interaction, but also uses the camera module to perceive the environment, realizes real-time capture of the scene seen by the human eye, and uses a multimodal large language model for processing and analysis, quickly and accurately identifies the user's intention and outputs a response to the user, providing users with a rich interactive experience.

[0054] Optionally, the preset background sends response data to the headset according to the user intention, the image data and the voice data, including: if the user intends to search for a target object, the preset background generates the location of the target object according to the image data, the first historical image data and the voice data by using the preset large language model; the preset background sends first response data to the headset, wherein the first response data includes voice description data of the location.

[0055] Exemplarily, the headset can pre-acquire image data of the surrounding environment (i.e., the first historical image data), which may contain an image of the target object. When the user wants to find the target object, he can input voice into the headset. After the preset background recognizes that the user intends to search for the target object, the user can continue to input voice containing keyword information (such as appearance) of the target object into the headset, so that the preset background can use the preset large language model to determine the information of the target object according to the voice data, and determine the data where the target object image appears from the first historical image data and the image data sent by the headset this round, and use the data to locate the position of the target object. Among them, if the image of the target object does not exist in the first historical image data and the image data sent by the headset this round, the preset large language model can be preset to output a prompt audio to guide the user to obtain new image data by moving the current position. Finally, the preset background can send the first response data including the voice description data of the position to the headset. The advantage of this setting is that the large language model is used to memorize the location of the object, and the location of the object can be located in time when the user needs it. For special groups of people, such as the visually impaired or those with memory impairment, this method can provide an advanced perception-assistive tool that enables users to perceive their surroundings more accurately through voice feedback and obtain detailed voice feedback, thereby enhancing the user's environmental cognition ability and independent living ability.

[0056] Furthermore, before the preset background sends the first response data to the headset, the method further includes:

[0057] The preset background generates route planning data according to the position, wherein the route planning data is voice guidance data for reaching the position of the target object from the current position; wherein the preset background sends first response data to the headset, including: the preset background sends first response data to the headset, wherein the first response data also includes the route planning data.

[0058] Specifically, the preset background uses the preset large language model to plan a route from the current position to the location of the target object according to the location of the target object and the image data sent by the headset, and outputs the route planning data. The first response data sent by the preset background may include voice description data of the location of the target object and the route planning data, which helps the visually impaired perceive the environment.

[0059] Embodiment 2

[0060] Figure 2 This is a flowchart of an interactive method provided in the second embodiment of the present invention. The technical solution of the embodiment of the present invention is further optimized on the basis of the above-mentioned optional technical solutions, and a specific method of... is given.

[0061] Optionally, the preset background sends response data to the headset according to the user intention and the image data, including: if the user intends to consult the operation suggestions of the target device, the preset background uses the preset large language model to send second response data to the headset according to the image data, wherein the second response data includes voice data of the operation suggestions for the target device, and the image of the target device is in the image data. The advantage of this setting is that by sending the voice data of the operation suggestions of the target device to the headset, the user can quickly obtain the device operation suggestions, help the user solve the problems encountered in life, and further enhance the user's independent living ability.

[0062] Optionally, the preset background sends response data to the headset according to the user intention and the image data, including: if the user intends to consult cooking suggestions for the target food, the preset background uses the preset large language model to send third response data to the headset according to the image data, wherein the third response data includes voice data of cooking suggestions for the target food, and the image of the target food is in the image data. The advantage of this setting is that by sending voice data of cooking suggestions for the target food to the headset, the user can quickly acquire cooking knowledge.

[0063] Optionally, the preset background sends response data to the headset according to the user intention and the image data, including: if the user intention is teaching interaction, the preset background uses the preset large language model to generate initial interactive response data according to the image data and the voice data; the preset background sends fourth response data to the headset according to the initial interactive response data and historical voice data, wherein the historical voice data is the voice data of a preset character, and the fourth response data includes the voice data of the initial interactive response data, and the voice data of the initial interactive response data corresponds to the simulated voice of the preset character. The advantage of this setting is that by sending the interactive response voice data of the preset character's voice to the headset, the user's learning experience is enhanced and the user is helped to concentrate.

[0064] Optionally, the preset background sends response data to the headset according to the user intention, the image data and the voice data, including: if the user intention is to consult the target scene, the preset background determines the user portrait, and uses the preset large language model to send fifth response data to the headset according to the user portrait, the image data and the voice data, wherein the user portrait is determined according to the second historical image data and the historical voice data, and the fifth response data includes the voice explanation data of the target scene and the voice data of the recommended music, and the image of the target scene is in the image data. The advantage of this setting is that customized explanation content is generated, and the user experience is enhanced by sending voice explanation data and voice data of recommended music to the headset.

[0065] like Figure 2 As shown, an interaction method provided by Embodiment 2 of the present invention specifically includes the following steps:

[0066] S201. The headset obtains voice data and image data through a target module configured by itself, and sends the voice data and image data to a preset background.

[0067] Wherein, the target module includes a speech recognition module and a camera module.

[0068] S202: The preset background uses a preset large language model to determine the user intention according to the voice data.

[0069] S203, the preset background determines whether the user's intention is to search for the target object, if so, execute step 204, if not, execute step 206.

[0070] S204: The preset background generates the location of the target object according to the image data, the first historical image data and the voice data using the preset large language model, and generates route planning data according to the location description data.

[0071] S205: The preset background sends first response data to the headset, wherein the first response data includes the voice description data of the location and the route planning data. Execute step 214.

[0072] S206, the preset background determines whether the user intends to consult the operation suggestions of the target device, if so, executes step 207, if not, executes step 208.

[0073] S207: The preset background uses the preset large language model to send second response data to the headset according to the image data. The second response data includes voice data of operation suggestions for the target device, and the image of the target device is in the image data. Execute step 214.

[0074] Exemplarily, the target device may be a household appliance, a commercial appliance, or an industrial appliance. When the user wants to know how to operate the target device, the user may use the headset to capture an image of the target device, and then send the image data corresponding to the image to the preset background. The preset background may use a preset large language model to output a safe operation suggestion for the target device based on the image data, and then convert the operation suggestion into voice data, and send the voice data (i.e., the second response data) to the headset.

[0075] S208, the preset background determines whether the user intends to consult cooking suggestions for the target food, if so, execute step 209, if not, execute step 210.

[0076] S209, the preset background uses the preset large language model to send third response data to the headset according to the image data, wherein the third response data includes voice data of cooking suggestions for the target food, and the image of the target food is in the image data. Execute step 214.

[0077] For example, when the user wants to know how to cook a target food, the user can use the headset to take an image of the target food, and then send the image data corresponding to the image to the preset background. The preset background can use the preset large language model to output cooking suggestions (such as detailed cooking steps) for the target food based on the image data, and then convert the cooking suggestions into voice data, and send the voice data (i.e., the third response data) to the headset.

[0078] S210, the preset background determines whether the user's intention is teaching interaction, if so, executes step 211, if not, executes step 212.

[0079] S211, the preset background generates initial interactive response data based on the image data and the voice data using the preset large language model, and sends fourth response data to the headset based on the initial interactive response data and the historical voice data, wherein the historical voice data is the voice data of the preset character, the fourth response data includes the voice data of the initial interactive response data, and the voice corresponding to the voice data of the initial interactive response data is the simulated voice of the preset character. Execute step 214.

[0080] Exemplarily, when the user wants to conduct teaching interaction, the user can input interactive voice into the headset and use the headset to capture interactive images. For example, when children are doing outdoor outings and other activities, they can use the headset to capture environmental images and ask questions through the headset. The preset background large language model can output answers and explanations (i.e., initial interactive response data) based on the image data and voice data collected by the headset. Then, the preset large language model is used to determine the virtual voice parameters of the preset role (such as a preset person or a preset virtual character) based on the pre-collected voice data of the preset role, and the parameters can be used to generate voice data corresponding to the initial interactive response data, thereby enhancing the child's concentration on the explanation content output by the model.

[0081] S212, the preset background determines whether the user's intention is a consultation target scenario, if so, executes step 213, if not, executes step 202.

[0082] S213, the preset background determines the user portrait, and uses the preset large language model to send fifth response data to the headset according to the user portrait, the image data and the voice data, wherein the user portrait is determined based on the second historical image data and the historical voice data, the fifth response data includes voice explanation data of the target scene and voice data of recommended music, and the image of the target scene is in the image data.

[0083] Exemplarily, when a user is traveling, he wants to consult a target scene, where the target scene is a certain landscape in the current environment. The user can input a question voice into the headset and use the headset to take an image of the landscape. After determining the user's intention, the preset background can first use the preset large language model to determine the user portrait based on the second historical image data and historical voice data, such as the user portrait is a literary youth. Then, the preset large language model can be used to output the explanation content that conforms to the user portrait and the music recommended for the user to listen to based on the image data and voice data collected by the headset, and send the voice data of the explanation content (i.e., the voice explanation data of the target scene) and the voice data of the recommended music to the headset. For example, when the user portrait is a child, the preset large language model can generate easy-to-understand and vivid and interesting voice explanation data for the target scene, and recommend voice data for music.

[0084] Optionally, if the preset background determines whether the user's intention is translation, the preset large language model is used to send sixth response data to the headset based on the image data and the voice data, wherein the sixth response data includes a first translated voice of the target text in the image data and a second translated voice of the target voice in the voice data.

[0085] S214. The earphone broadcasts a response voice according to the response data.

[0086] Specifically, the response data includes first response data, second response data, third response data, fourth response data or fifth response data.

[0087] The interactive method provided by the embodiment of the present invention enables users to quickly obtain device operation suggestions by sending voice data of operation suggestions of the target device to the headset, and enables users to quickly obtain cooking knowledge by sending voice data of cooking suggestions of the target food to the headset, helping users solve problems encountered in life and further enhancing users' independent living ability. By sending interactive response voice data of preset character voices to the headset, the user's learning experience is enhanced and the user is helped to concentrate. Customized explanation content can also be generated, and the user experience is enhanced by sending voice explanation data and voice data of recommended music to the headset.

[0088] Embodiment 3

[0089] Figure 3 This is a schematic diagram of the structure of an interactive system provided by Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a data acquisition module 301 and a broadcast module 302 in the earphone, and a response data sending module 303 in the preset background, wherein:

[0090] A data acquisition module, used to acquire target data through a target module configured by itself, and send the target data to a preset background, wherein the target data includes voice data and image data, and the target module includes a camera module and a voice recognition module;

[0091] a response data sending module, configured to determine the user intention according to the voice data by using a preset large language model, and send response data to the headset according to the user intention and the image data and / or the voice data;

[0092] The broadcast module is used to broadcast the response voice according to the response data.

[0093] In the interactive system provided by the embodiment of the present invention, the headset uses the camera module and voice recognition module configured by itself to collect voice data and image data, and sends the data to the background. The background uses a large language model to mine the user's intention from the data, and outputs response data to the headset according to the user's intention and the received data, so that the headset can broadcast the corresponding response voice. This system not only supports voice interaction, but also uses the camera module to perceive the environment, realizes real-time capture of the scene seen by the human eye, and uses a multimodal large language model for processing and analysis, quickly and accurately identifies the user's intention and outputs the response to the user, providing users with a rich interactive experience.

[0094] Optionally, the response data sending module includes:

[0095] a position determination unit, configured to generate a position of the target object using the preset large language model according to the image data, the first historical image data and the voice data if the user intends to search for the target object;

[0096] The location description sending unit is used to send first response data to the headset, wherein the first response data includes the voice description data of the location.

[0097] Optionally, the response data sending module further includes:

[0098] A route planning unit is used to generate route planning data according to the location description data before the preset background sends the first response data to the headset, wherein the route planning data is voice guidance data for reaching the location of the target object from the current location.

[0099] Furthermore, the preset background sends first response data to the headset, including:

[0100] The preset background sends first response data to the headset, wherein the first response data also includes the route planning data.

[0101] Optionally, the response data sending module includes:

[0102] An operation suggestion sending unit is used for sending second response data to the headset based on the image data by using the preset large language model if the user intends to consult the operation suggestions of the target device, wherein the second response data includes voice data of the operation suggestions for the target device, and the image of the target device is in the image data.

[0103] Optionally, the response data sending module includes:

[0104] A cooking suggestion sending unit is used to send third response data to the headset based on the image data using the preset large language model if the user intends to consult for cooking suggestions for the target food, wherein the third response data includes voice data of cooking suggestions for the target food, and the image of the target food is in the image data.

[0105] Optionally, the response data sending module includes:

[0106] an initial interaction answer determination unit, configured to generate initial interaction response data based on the image data and the voice data by using the preset large language model if the user intends to interact with the teaching;

[0107] An interactive answer sending unit is used for the preset background to send fourth response data to the headset according to the initial interactive response data and the historical voice data, wherein the historical voice data is the voice data of a preset character, and the fourth response data includes the voice data of the initial interactive response data, and the voice corresponding to the voice data of the initial interactive response data is the simulated voice of the preset character.

[0108] Optionally, the response data sending module includes:

[0109] The voice explanation sending unit is used to determine the user portrait if the user intends to consult the target scene, and use the preset large language model to send fifth response data to the headset according to the user portrait, the image data and the voice data, wherein the user portrait is determined based on the second historical image data and the historical voice data, the fifth response data includes the voice explanation data of the target scene and the voice data of the recommended music, and the image of the target scene is in the image data.

[0110] The interactive system provided by the embodiment of the present invention can execute the interactive method provided by any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0111] Embodiment 4

[0112] Figure 4 A schematic diagram of the structure of an earphone 40 that can be used to implement an embodiment of the present invention is shown. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0113] like Figure 4 As shown, the headset 40 includes a headset body 401 and a camera module 402, and the headset body 401 includes a battery 403, a voice recognition module 404, at least one processor 405, and a memory 406 connected to the at least one processor 405, such as a read-only memory (ROM), a random access memory (RAM), etc. The memory stores a computer program that can be executed by at least one processor, and the processor can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) or the computer program loaded from the storage unit to the random access memory (RAM). Among them, the headset body can also include a speaker module, a WiFi module, a BLE module and a BT module.

[0114] The camera module in the headset is a detachable camera module, which can be connected to the headset body through a magnetic attraction component. The processor can be a variety of general and / or special processing components with processing and computing capabilities. The processor executes the various methods and processes described above, such as the steps performed by the headset in the interactive method.

[0115] The camera module is mainly used to collect video image data, and in order to reduce the transmission bandwidth, the video image needs to be compressed, and the compression method is MJPEG or H.264. In order to improve the image quality, the image can be processed by ISP, including but not limited to automatic white balance (AWB), automatic exposure (AE), automatic focus (AF), color interpolation, noise removal, bad pixel correction, lens shading correction (LSC), black level correction (BLC) and motion stabilization.

[0116] The speech recognition module is mainly used to collect audio data. In order to obtain clear speech signals, the speech can be processed at the front end, including but not limited to the use of multi-microphone environmental noise reduction (ENC), acoustic echo cancellation (AEC) and voice enhancement algorithms.

[0117] The speaker module is mainly used to play Bluetooth music and the audio stream of the response data output by the large language model. In order to improve the quality of the played audio, the audio signal needs to be tuned and actively noise-reduced, including but not limited to the use of algorithms such as EQ, DRC and ANC active noise reduction.

[0118] The WiFi module is mainly used to achieve video transmission. The video image stream will be transmitted to the mobile phone App or preset background in real time via WiFi. The mobile phone releases the WiFi hotspot, and the headset is connected as a Station, which can cover the user's outdoor and mobile application scenarios. Considering that the 2.4GHz frequency is seriously interfered by surrounding wireless devices, the headset WiFi module supports both 2.4GHz and 5GHz bands, and users can choose the band according to the network conditions.

[0119] The BLE module is mainly used to realize uplink transmission of voice data, encoded with the Opus compression algorithm. When only AI voice interaction is performed, the WiFi and camera modules can be turned off to save power and extend the battery life of the device.

[0120] BT module is mainly used to realize traditional Bluetooth audio playback and make and receive calls.

[0121] The headset can communicate with the preset background through the mobile phone, or directly with the preset background. When the headset needs to communicate with the preset background through the mobile phone, the app in the mobile phone is responsible for releasing the WiFi hotspot, establishing BLE, BT and WiFi connections with the headset, and uploading audio data and image data to the preset background in the cloud, and receiving the audio stream generated by the large language model in the preset background, and then playing it in the headset via Bluetooth.

[0122] The earphones provided above can be used to execute the steps performed by the earphones in the interaction method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0123] Embodiment 5

[0124] In the context of the present invention, a computer-readable storage medium may be a tangible medium, wherein the computer-executable instructions, when executed by a computer processor, are used to perform an interactive method, the method comprising:

[0125] The headset obtains target data through a target module configured by itself, and sends the target data to a preset background, wherein the target data includes voice data and image data, and the target module includes a camera module and a voice recognition module;

[0126] The preset background determines the user intention according to the voice data using a preset large language model, and sends response data to the headset according to the user intention and the image data and / or the voice data;

[0127] The earphone broadcasts a response voice according to the response data.

[0128] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by an instruction execution system, device or equipment or used with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] The computer device provided above can be used to execute the interactive method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0130] It is worth noting that in the embodiment of the above-mentioned interactive device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0131] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An interactive method, characterized in that: include: The headset obtains target data through a target module configured by itself, and sends the target data to a preset background, wherein the target data includes voice data and image data, and the target module includes a camera module and a voice recognition module; The preset background determines the user intention according to the voice data using a preset large language model, and sends response data to the headset according to the user intention and the image data and / or the voice data; The earphone broadcasts a response voice according to the response data.

2. The method according to claim 1, characterized in that The preset background sends response data to the headset according to the user intention, the image data and the voice data, including: If the user intends to search for a target object, the preset background generates the location of the target object according to the image data, the first historical image data and the voice data using the preset large language model; The preset background sends first response data to the headset, wherein the first response data includes voice description data of the location.

3. The method according to claim 2, characterized in that Before the preset background sends the first response data to the headset, the method further includes: The preset background generates route planning data according to the position, wherein the route planning data is voice guidance data for reaching the location of the target object from the current location; The preset background sending the first response data to the headset includes: The preset background sends first response data to the headset, wherein the first response data also includes the route planning data.

4. The method according to any one of claims 1 to 3, characterized in that The preset background sends response data to the headset according to the user intention and the image data, including: If the user intends to consult for operating suggestions for the target device, the preset background uses the preset large language model to send second response data to the headset based on the image data, wherein the second response data includes voice data of operating suggestions for the target device, and the image of the target device is in the image data.

5. The method according to claim 1, characterized in that: The preset background sends response data to the headset according to the user intention and the image data, including: If the user intends to inquire about cooking suggestions for the target food, the preset background uses the preset large language model to send third response data to the headset based on the image data, wherein the third response data includes voice data of cooking suggestions for the target food, and the image of the target food is in the image data.

6. The method according to claim 1, characterized in that The preset background sends response data to the headset according to the user intention and the image data, including: If the user intends to interact with the user through teaching, the preset background generates initial interactive response data based on the image data and the voice data using the preset large language model; The preset background sends fourth response data to the headset based on the initial interactive response data and the historical voice data, wherein the historical voice data is the voice data of a preset character, and the fourth response data includes the voice data of the initial interactive response data, and the voice corresponding to the voice data of the initial interactive response data is the simulated voice of the preset character.

7. The method according to claim 1, characterized in that The preset background sends response data to the headset according to the user intention, the image data and the voice data, including: If the user intends to consult the target scenario, the preset background determines the user portrait, and uses the preset large language model to send fifth response data to the headset based on the user portrait, the image data and the voice data, wherein the user portrait is determined based on the second historical image data and the historical voice data, the fifth response data includes voice explanation data of the target scene and voice data of recommended music, and the image of the target scene is in the image data.

8. An interactive system, characterized in that: include: The data acquisition module in the headset is used to acquire target data through the target module configured by itself and send the target data to the preset background, wherein the target data includes voice data and image data, and the target module includes a camera module and a voice recognition module; A response data sending module in the preset background is used to determine the user intention according to the voice data using a preset large language model, and send the response data to the headset according to the user intention and the image data and / or the voice data; The broadcast module in the earphone is used to broadcast the response voice according to the response data.

9. A headset, characterized in that: The earphone comprises: an earphone body and a camera module; The headset body includes a battery, a voice recognition module, at least one processor, and a memory in communication with the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the steps performed by the headset in the interaction method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the interaction method according to any one of claims 1 to 7 when executed.

Images