Voice reply method, device and system, storage medium and product

Abstract

The embodiment of the invention provides a voice reply method, device and system, a storage medium and a product, and relates to the technical field of computers, and the method comprises the steps: collecting first inquiry voice information; obtaining a target image collected by a camera; first reply voice information of the first inquiry voice information is obtained, the first reply voice information is information related to the target image, and the target image and the first inquiry voice information are processed by a pre-trained dialogue model to obtain the first reply voice information; and playing the first reply voice information. Firstly, the intelligent dialogue is realized by adopting the earphone, so that the efficiency of the intelligent dialogue is improved. In addition, the first reply voice information can be quickly determined by adopting the dialogue model. And furthermore, the first inquiry voice information is replied by obtaining the target image, so that reply to the problem proposed by the user for the real-time environment can be realized.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a voice response method, device, system, storage medium, and product. Background Technology

[0002] Intelligent dialogue is an interaction between users and smart devices that uses a question-and-answer format. It is widely used in scenarios such as voice assistants and intelligent customer service.

[0003] Currently, intelligent dialogue is mainly implemented through terminal devices. For example, users input voice or text information through the interface of a terminal device, and the terminal device outputs corresponding response information. This method suffers from low dialogue efficiency. Summary of the Invention

[0004] This disclosure provides a voice response method, apparatus, device, computer-readable storage medium, and product to address the technical problem of low efficiency in existing technologies that use terminal devices to achieve intelligent dialogue.

[0005] In a first aspect, embodiments of this disclosure provide a voice response method applied to an earphone device with a camera, the method comprising:

[0006] Collect the voice information of the first inquiry;

[0007] Acquire the target image captured by the camera;

[0008] The first response voice information is obtained from the first query voice information, wherein the first response voice information is information related to the target image, and the target image and the first query voice information are processed by a pre-trained dialogue model to obtain the first response voice information.

[0009] Play the first reply voice message.

[0010] Secondly, embodiments of this application provide an earphone device, including: the earphone device carrying a camera, the earphone device including:

[0011] The acquisition unit is used to acquire the first query voice information;

[0012] The acquisition unit is used to acquire the target image captured by the camera;

[0013] The processing unit is used to obtain the first response voice information of the first query voice information, wherein the first response voice information is information related to the target image, and the target image and the first query voice information are processed by a pre-trained dialogue model to obtain the first response voice information.

[0014] The playback unit is used to play the first response voice message.

[0015] Thirdly, embodiments of this application provide a voice response system, including an earphone device and a terminal device as described in the second aspect, wherein:

[0016] The headset device is used to collect the first query voice information; acquire the target image captured by the camera, and send the target image and the first query voice information to the terminal device.

[0017] The terminal device is used to process the target image and the first query voice information through a pre-trained dialogue model to obtain the first response voice information of the first query voice information, and send the first response voice information to the headset device;

[0018] The headset device is used to receive and play the first reply voice information sent by the terminal device.

[0019] Fourthly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0020] The memory stores the instructions that the computer executes;

[0021] The processor executes computer execution instructions stored in memory, causing at least one processor to perform the first aspect above and various possible voice response methods of the first aspect.

[0022] Fifthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the first aspect and various possible voice response methods described above.

[0023] In a sixth aspect, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect as described above and various possible voice response methods of the first aspect.

[0024] The voice response method, device, system, storage medium, and product provided in this embodiment include: acquiring first query voice information; acquiring a target image captured by a camera; obtaining first response voice information of the first query voice information, wherein the first response voice information is information related to the target image, and the target image and the first query voice information are processed by a pre-trained dialogue model to obtain the first response voice information; and playing the first response voice information. Firstly, this application uses headphones to achieve intelligent dialogue, improving the efficiency of intelligent dialogue. Furthermore, this application uses a dialogue model that can quickly determine the first response voice information. Moreover, by acquiring the target image to respond to the first query voice information, it is possible to respond to questions raised by users in real-time environments. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 An example diagram illustrating an application scenario of a voice response method provided in this application embodiment;

[0027] Figure 2 A flowchart illustrating a voice response method provided in an embodiment of this application;

[0028] Figure 3 A flowchart of another voice response method provided in the embodiments of this application;

[0029] Figure 4 This is a schematic diagram of the structure of an earphone device provided in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0032] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, the target user should be informed of the type, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and the authorization of the target user should be obtained.

[0033] It is understood that the above notification and target user authorization process is merely illustrative and does not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0034] In related technologies, if a user needs to inquire about information about an object in their surroundings, they need to use a mobile device with a camera to photograph the object, obtain an image, and then use a recognition application on the mobile device to scan the image. The relevant information about the object is then displayed on the mobile device's interface. If the user needs to know further information about the object, they need to search for it within the application based on that information. For example, if a user sees a bouquet of flowers in their current environment but doesn't know the name of the flower, they can photograph the flower with their mobile device, obtain an image containing the flower, and then use a recognition application on the mobile device to scan the image to obtain the flower's name. If the user then needs to know the flower's blooming season, planting location, etc., they need to enter a question about the flower into the application to get the corresponding answer. It is evident that the process of obtaining relevant information about an object is complex and inefficient.

[0035] To address the technical problem, this disclosure provides a voice response method. By deploying a camera on a headset device, controlling the camera to acquire a target image in real time, and the headset device collecting first inquiry voice information, a first response voice information is obtained from the target image and the first inquiry voice information. The first response voice information is then played through the headset device, allowing users to quickly and easily obtain the information they want and improving dialogue efficiency.

[0036] It should be noted that the voice response method, device, system, storage medium and product provided in this disclosure can be applied to any type of headphone device equipped with a camera.

[0037] The voice response method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0038] Figure 1 This diagram illustrates an application scenario of a voice response method provided in an embodiment of this application. Figure 1 As shown, a user wears an earphone device, which includes a first earphone and a second earphone. The user wears the first earphone in one ear and the second earphone in the other ear. The earphone device carries a camera; in this embodiment, either the first or second earphone carries a camera. Furthermore, the first and / or second earphone carries a microphone for capturing the user's inquiry voice information. It is understood that in this application, the camera on the earphone device captures a target image, and the microphone captures the user's inquiry voice information. Based on the target image and the user's inquiry voice information, a response voice information can be generated. This response voice information can be played through the earphone device, making the entire dialogue process simple and efficient, and improving the user experience.

[0039] Figure 1This is merely an exemplary application scenario and can be used in any headphone device with a camera, such as in-ear headphones, semi-in-ear headphones, over-ear headphones, wired headphones, TWS (True Wireless Stereo) headphones, etc. This application does not limit the type of headphone device.

[0040] Figure 2 A flowchart of a voice response method provided in this application embodiment, the voice response method may include the following steps:

[0041] S201. Collect the first inquiry voice information.

[0042] The first inquiry voice information is captured by the microphone of the headset device. It can be understood that one earpiece (either the first or second earpiece) of the headset device carries a microphone, which can capture the user's first inquiry voice information in real time.

[0043] It is understood that the first query voice information collected needs to be processed by the preset application using a dialogue model. In one embodiment, the preset application has been activated, and the first query voice information is any query voice information input by the user, such as "What's in front of me?", "What's this flower called?", "What kind of flower is this?", or "What color is this flower?", etc.

[0044] In another embodiment, the preset application is not activated, and the first inquiry voice information includes: the preset wake word of the preset application and the inquiry voice information, such as "XX, what's in front?", "XX, XX, what's in front?", "XX, take a look at what's in front?".

[0045] "XX" is the preset wake word for the preset application.

[0046] S202. Acquire the target image captured by the camera.

[0047] The camera captures at least one target image or a series of consecutive target images.

[0048] In one embodiment, before acquiring the target image captured by the camera, the method further includes: responding to a wake-up operation on the camera to wake up the camera; acquiring the target image captured by the camera includes: controlling the woken-up camera to acquire the target image.

[0049] It is understandable that if the camera has been activated and the first inquiry voice information is collected, then the camera will be controlled to collect the target image after the first inquiry voice information is collected.

[0050] The camera can be activated via a wake word or by touching a headset. The wake word can be the same as the default wake word for the app. Examples of wake words include "XX", "XX, take a look", "XX, scan", "XX, take another look", "XX, take another look", and "XX, take a closer look".

[0051] In another embodiment, acquiring the target image captured by the camera includes: determining the shooting intention of the first inquiry voice information through a recognition model; and controlling the camera to acquire the target image according to the shooting intention.

[0052] It's understandable that after collecting the initial voice query, the system first identifies a wake-up word for the camera within the query. If this word is present, the camera is activated. A recognition model analyzes the user's intent to capture the image. For example, if the initial query is "What is the name of this flower?", the intent is to photograph the flower. The camera can then be controlled to focus on the flower and capture the target image. By controlling the camera to capture the target image based on the user's intent, an accurate image can be obtained, which can then be used to provide an accurate response to the user.

[0053] The recognition model is a network model deployed on the headphone device.

[0054] In this embodiment of the application, the headphone device includes: a first headphone and a second headphone, the first headphone carrying a first camera, the second headphone carrying a second camera, and the target image including: a target image captured by the first camera and a target image captured by the second camera.

[0055] It is understood that, in one embodiment, at least one camera can be installed in only one earphone; for example, if only the first camera is installed in the first earphone, then only the first camera needs to capture the target image. In another embodiment, cameras can be installed in both earphones, then both earphone cameras can capture the target image.

[0056] The target image captured by the first camera and the target image captured by the second camera are the same object from different angles. This is used to determine the subsequent first response voice information, which can improve the accuracy of the first response voice information.

[0057] S203, Receive the first response voice message to the first inquiry voice message.

[0058] The first response voice information is information related to the target image. The target image and the first query voice information are processed by a pre-trained dialogue model to obtain the first response voice information.

[0059] The dialogue model is an LLM (Large Language Model), a pre-trained multimodal language model that processes the target image and the initial natural language query speech information to obtain the initial response speech information. This initial response speech information is related to the target image. For example, if the initial query speech information is "What's in front of me?", and the target image includes the word "flower", then the initial response speech information could be "There's a potted flower in front of me". If the initial query speech information is "What is this flower?", and the target image includes the word "rose", then the initial response speech information could be "This is a rose".

[0060] In one embodiment, deploying a dialogue model in the headset device to obtain the first response voice information of the first query voice information includes: inputting the target image and the first query voice information into a pre-trained dialogue model for processing to obtain the first response voice information of the first query voice information.

[0061] It is understandable that if a dialogue model is deployed on the headset device side, after the headset device collects the first query voice information and the target image, the headset device will input the first query voice information and the target image into the dialogue model for processing to obtain the first response voice information of the first query voice information.

[0062] In this embodiment, the input to the dialogue model can be a first query voice information in voice format, or the first query voice information can be converted into first query text information using STT (Speech To Text) technology. The first query text information is then input into the dialogue model, and the dialogue model can also output first response text information. The headset device uses TTS (Text To Speech) technology to convert the first response text information from text format to voice format to obtain the first response voice information.

[0063] In this embodiment, a dialogue application can be installed on the headset device side, and a dialogue model can be deployed in the cloud server. The headset device can upload the target image and the first query voice information to the dialogue model in the cloud service for processing to obtain the first response voice information.

[0064] In another embodiment, obtaining the first response voice information of the first inquiry voice information includes: sending a target image and the first inquiry voice information to a terminal device, wherein the terminal device processes the target image and the first inquiry voice information through a pre-trained dialogue model to obtain the first response voice information of the first inquiry voice information; and receiving the first response voice information sent by the terminal device.

[0065] It is understandable that if the dialogue model is deployed on the terminal device side, the headset device will send the collected target image and the first query voice information to the terminal device. The terminal device will process the target image and the first query voice information through the dialogue model to obtain the first response voice information.

[0066] In this embodiment, a dialogue application can be installed on the terminal device side, and a dialogue model can be deployed in the cloud server. The terminal device can upload the target image and the first query voice information to the dialogue model in the cloud service for processing through the dialogue application to obtain the first reply voice information, and then send the first reply voice information to the headset device for playback.

[0067] In summary, this application can use multiple methods to process the target image and the first query voice information through a dialogue model to obtain the first response voice information.

[0068] S204. Play the first reply voice message.

[0069] In this embodiment, the first response voice information can be played on one earphone (first earphone or second earphone), or the first response voice information can be played on both the first earphone and the second earphone.

[0070] In summary, this application deploys a camera on a headset device, controls the camera to acquire target images in real time, and the headset device collects first inquiry voice information. Based on the target image and the first inquiry voice information, a first response voice information is obtained, and the first response voice information is played through the headset device. This allows users to quickly and easily obtain the information they want, improving dialogue efficiency.

[0071] Figure 3 A flowchart of another voice response method provided in this application embodiment, the voice response method may include the following steps:

[0072] S301, collect the first inquiry voice information.

[0073] The specific implementation process of this step is described in S201 and will not be repeated here.

[0074] S302, acquire the target image captured by the camera.

[0075] The specific implementation process of this step is described in S202 and will not be repeated here.

[0076] S303, send the target image and the first query voice information to the terminal device.

[0077] In this embodiment of the application, the headset device and the terminal device are connected in communication. The headset device sends the target image and the first inquiry voice information to the terminal device via Bluetooth or WIFI (mobile hotspot). When the headset device sends the target image and the first inquiry voice information to the terminal device via WIFI, the headset device can connect to the hotspot of the terminal device.

[0078] Users can choose to use the camera on one earbud (either the first or the second earbud) to capture the target image, or they can choose to use the cameras on both earbuds (the first and the second earbud) to capture the target image.

[0079] In one embodiment, sending a target image to a terminal device includes: sending a target image captured by a first camera to the terminal device using a first earpiece; and sending a target image captured by a second camera to the terminal device using a second earpiece. When both earpieces are configured to capture target images, one earpiece (e.g., the first earpiece) can be designated as the master earpiece, and the other earpiece (e.g., the second earpiece) as the slave earpiece.

[0080] It can be understood that the first earpiece and the terminal device are connected in communication, and the second earpiece and the terminal device are also connected in communication. Therefore, both the first and second earpieces can independently send target images to the terminal device.

[0081] If the first earpiece has a microphone, it sends the target image captured by the first camera and a first interrogation voice message captured by the first earpiece to the terminal device; the second earpiece sends the target image captured by the second camera to the terminal device. If the second earpiece has a microphone, it sends the target image captured by the first camera to the terminal device, and the second earpiece sends the target image captured by the second camera and a second interrogation voice message captured by the second earpiece to the terminal device.

[0082] In this embodiment, the first and second earpieces communicate with the terminal device independently, without needing to consider the connection between the earpieces. The earpiece carrying the microphone is the main earpiece.

[0083] In another embodiment, sending a target image to a terminal device includes: a first earpiece receiving a target image captured by a second camera sent by a second earpiece; and using the first earpiece to send the target image captured by the first camera and the target image captured by the second camera to the terminal device.

[0084] It is understood that the first earpiece is communicatively connected to the terminal device, while the second earpiece is not. The first and second earpieces are communicatively connected. In this embodiment, the first earpiece is the master earpiece, and the second earpiece is the slave earpiece. The first and second earpieces are connected via Bluetooth. In this case, the second earpiece sends the target image captured by the second camera to the first earpiece, and the first earpiece sends the target images captured by the first and second cameras to the terminal device.

[0085] Furthermore, when the first earpiece and the terminal device are connected, the second earpiece and the terminal device are not connected, but the first earpiece and the second earpiece are connected, if the first earpiece has a microphone, then the first earpiece collects the first inquiry voice information and sends the first inquiry voice information to the terminal device. If the second earpiece has a microphone, then the second earpiece collects the first inquiry voice information, sends the first inquiry voice information to the first earpiece, and the first earpiece sends the first inquiry voice information to the terminal device.

[0086] In this embodiment, after the first earpiece receives the target image sent by the second earpiece, it then sends the target image to the terminal device. Only one earpiece needs to establish a communication connection with the terminal device. The data transmission link from one earpiece to the terminal device is relatively short.

[0087] Furthermore, in this application, the target image captured by the camera of the main earphone can be sent to the terminal device first, and then the target image captured by the camera of the earphone can be sent to the terminal device, or they can be sent simultaneously, which is not limited here.

[0088] Furthermore, the first query voice information is encoded before being sent to the terminal device, for example, using OPUS encoding (a voice encoding method). After the terminal device receives the encoded first query voice information, it decodes it using OPUS decoding (a voice decoding method) and then inputs it into the question-and-answer model for processing.

[0089] S304, receive the first reply voice message sent by the terminal device.

[0090] In one embodiment, a first earpiece receives a first reply voice message sent by a terminal device; the first earpiece forwards the first reply voice message to a second earpiece.

[0091] In the case where only the first earpiece and the terminal device are connected, and the first earpiece and the second earpiece are connected, the terminal device sends the first reply voice information to the first earpiece, and the first earpiece then forwards the first reply voice information to the second earpiece.

[0092] In another embodiment, a first headset is used to receive the first reply voice information sent by the terminal device; a second headset is used to receive the first reply voice information sent by the terminal device.

[0093] When the first and second earpieces have established a communication connection with the terminal device, the terminal device sends the first reply voice information to the first and second earpieces for playback.

[0094] S305, playing the first reply voice message.

[0095] In this embodiment, the terminal device can encode the first response voice information and send it to the headset device. After receiving the encoded first response voice information, the headset device decodes and plays the first response voice information.

[0096] The headset device plays the first response voice message through its speaker.

[0097] S306, collect the second inquiry voice information.

[0098] Understandably, users can continue to ask questions based on the first response voice message played by the headphone device, thus enabling multi-round dialogues.

[0099] S307, the second response voice information of the second inquiry voice information is obtained.

[0100] The second response voice information is information related to the target image. The target image and the second query voice information are processed by the dialogue model to obtain the second response voice information.

[0101] In this embodiment of the application, the user can implement polling dialogue for the target image. For example, if the first question voice information is "What is in front?", and the target image includes the corresponding "flower", then the first reply voice information can be "There is a potted flower in front", and the second question voice information is "The name of this flower". Then the second question voice information and the target image are input into the dialogue model for processing to obtain the second reply voice information.

[0102] The method for determining the second reply voice information is the same as that for the first reply voice information, and will not be repeated here.

[0103] S308, playing the second reply voice message.

[0104] In one alternative embodiment, after a multi-round dialogue with the target image has ended, the headset device can also collect other inquiry voice information and other images to start a new round of dialogue.

[0105] In this embodiment, the communication connection between the headset device and the terminal device enables the headset device to have intelligent dialogue capabilities, thereby improving the efficiency of intelligent dialogue and enhancing the user experience.

[0106] Figure 4 This is a schematic diagram of the structure of an earphone device provided in an embodiment of this application. The earphone device carries a camera, and the earphone device 40 may include the following units:

[0107] Acquisition unit 41 is used to acquire the first inquiry voice information;

[0108] Acquisition unit 42 is used to acquire the target image captured by the camera;

[0109] Processing unit 43 is used to obtain first response voice information of first query voice information, wherein the first response voice information is information related to the target image, and the target image and the first query voice information are processed by a pre-trained dialogue model to obtain the first response voice information.

[0110] Playback unit 44 is used to play the first reply voice information.

[0111] In some embodiments, it further includes: a wake-up unit (not shown), configured to wake up the camera in response to a wake-up operation on the camera before acquiring the target image captured by the camera;

[0112] The acquisition unit 42 is specifically used to control the awakened camera to acquire target images.

[0113] In some embodiments, the acquisition unit 42 is specifically used to determine the shooting intention of the first inquiry voice information through a recognition model; and to control the camera to acquire the target image according to the shooting intention.

[0114] In some embodiments, a dialogue model is deployed in the headset device. The processing unit 43 is specifically used to input the target image and the first query voice information into the pre-trained dialogue model for processing to obtain the first response voice information of the first query voice information.

[0115] In some embodiments, the processing unit 43 is specifically used to send a target image and a first query voice information to a terminal device. The terminal device is used to process the target image and the first query voice information through a pre-trained dialogue model to obtain a first response voice information of the first query voice information; and to receive the first response voice information sent by the terminal device.

[0116] In some embodiments, the headphone device includes: a first earphone and a second earphone, the first earphone carrying a first camera, the second earphone carrying a second camera, and the target image including: a target image captured by the first camera and a target image captured by the second camera.

[0117] In some embodiments, when the processing unit 43 sends a target image to the terminal device, it is specifically used to: send a target image captured by a first camera to the terminal device using a first headset; and send a target image captured by a second camera to the terminal device using a second headset.

[0118] In some embodiments, when the processing unit 43 sends a target image to the terminal device, it is specifically used to: receive a target image captured by a second camera sent by a second earpiece; and send the target image captured by the first camera and the target image captured by the second camera to the terminal device using the first earpiece.

[0119] In some embodiments, when the processing unit 43 receives the first reply voice information sent by the terminal device, it is specifically used to: receive the first reply voice information sent by the terminal device using a first earpiece; and forward the first reply voice information to a second earpiece using the first earpiece.

[0120] In some embodiments, when the processing unit 43 receives the first reply voice information sent by the terminal device, it is specifically used to: receive the first reply voice information sent by the terminal device using a first headset; and receive the first reply voice information sent by the terminal device using a second headset.

[0121] In some embodiments, after playing the first reply voice message

[0122] The acquisition unit 41 is also used to acquire the second inquiry voice information;

[0123] Processing unit 43 is further configured to obtain second response voice information of the second query voice information, wherein the second response voice information is information related to the target image, and the target image and the second query voice information are processed by the dialogue model to obtain the second response voice information;

[0124] Playback unit 44 is also used to play the second reply voice message.

[0125] To implement the above embodiments, this disclosure also provides a voice response system, including the aforementioned headset device and terminal device.

[0126] in:

[0127] The headset device is used to collect the first query voice information; acquire the target image captured by the camera, and send the target image and the first query voice information to the terminal device.

[0128] The terminal device is used to process the target image and the first query voice information through a pre-trained dialogue model to obtain the first response voice information of the first query voice information, and send the first response voice information to the headset device;

[0129] The headset device is used to receive and play the first reply voice information sent by the terminal device.

[0130] To implement the above embodiments, this disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the voice response method as described in any of the above embodiments.

[0131] To implement the above embodiments, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the voice response method as described in any of the above embodiments.

[0132] To implement the above embodiments, this disclosure also provides an electronic device, including: a processor and a memory;

[0133] The memory stores the instructions that the computer executes;

[0134] The processor executes computer execution instructions stored in memory, causing the processor to perform the voice response method as described in any of the above embodiments.

[0135] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device 50 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device described is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0136] like Figure 5As shown, the electronic device 50 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 51, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 52 or a program loaded from storage device 58 into random access memory (RAM) 53. The RAM 53 also stores various programs and data required for the operation of the electronic device 50. The processing unit 51, ROM 52, and RAM 53 are interconnected via a bus 54. An input / output (I / O) interface 55 is also connected to the bus 54.

[0137] Typically, the following devices can be connected to I / O interface 55: input devices 56 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 57 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 58 including, for example, magnetic tapes, hard disks, etc.; and communication devices 59. Communication device 59 allows electronic device 50 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 50 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0138] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 59, or installed from a storage device 58, or installed from a ROM 52. When the computer program is executed by the processing device 51, it performs the functions defined in the methods of embodiments of this disclosure.

[0139] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0140] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0141] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0142] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the target user's computer, partially on the target user's computer, as a standalone software package, partially on the target user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the target user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0145] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0146] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0147] In a first aspect, according to one or more embodiments of the present disclosure, a voice response method is provided, applied to an earphone device carrying a camera, the method comprising: acquiring first query voice information; acquiring a target image acquired by the camera; obtaining first response voice information of the first query voice information, wherein the first response voice information is information related to the target image, the target image and the first query voice information are processed by a pre-trained dialogue model to obtain the first response voice information; and playing the first response voice information.

[0148] According to one or more embodiments of this disclosure, before acquiring the target image captured by the camera, the method further includes: waking up the camera in response to a wake-up operation; acquiring the target image captured by the camera includes: controlling the woken-up camera to acquire the target image.

[0149] According to one or more embodiments of this disclosure, acquiring a target image captured by a camera includes: determining the shooting intent of a first interrogation voice message through a recognition model; and controlling the camera to acquire the target image according to the shooting intent.

[0150] According to one or more embodiments of this disclosure, deploying a dialogue model in an earphone device to obtain first response voice information of a first inquiry voice information includes: inputting a target image and the first inquiry voice information into a pre-trained dialogue model for processing to obtain first response voice information of the first inquiry voice information.

[0151] According to one or more embodiments of this disclosure, obtaining the first response voice information of the first inquiry voice information includes: sending a target image and the first inquiry voice information to a terminal device, wherein the terminal device processes the target image and the first inquiry voice information through a pre-trained dialogue model to obtain the first response voice information of the first inquiry voice information; and receiving the first response voice information sent by the terminal device.

[0152] According to one or more embodiments of this disclosure, the headphone device includes: a first headphone and a second headphone, the first headphone carrying a first camera, the second headphone carrying a second camera, and the target image including: a target image captured by the first camera and a target image captured by the second camera.

[0153] According to one or more embodiments of this disclosure, sending a target image to a terminal device includes: sending a target image captured by a first camera to the terminal device using a first headset; and sending a target image captured by a second camera to the terminal device using a second headset.

[0154] According to one or more embodiments of this disclosure, sending a target image to a terminal device includes: a first earpiece receiving a target image captured by a second camera sent by a second earpiece; and using the first earpiece to send the target image captured by the first camera and the target image captured by the second camera to the terminal device.

[0155] According to one or more embodiments of this disclosure, receiving a first reply voice message sent by a terminal device includes: receiving the first reply voice message sent by the terminal device using a first earpiece; and forwarding the first reply voice message to a second earpiece using the first earpiece.

[0156] According to one or more embodiments of this disclosure, receiving first reply voice information sent by a terminal device includes: receiving the first reply voice information sent by the terminal device using a first headset; and receiving the first reply voice information sent by the terminal device using a second headset.

[0157] According to one or more embodiments of this disclosure, after playing the first response voice information, the method further includes: acquiring second inquiry voice information; obtaining second response voice information of the second inquiry voice information, wherein the second response voice information is information related to the target image, and the target image and the second inquiry voice information are processed by a dialogue model to obtain the second response voice information; and playing the second response voice information.

[0158] Secondly, according to one or more embodiments of this disclosure, an earphone device is provided, the earphone device carrying a camera, the earphone device comprising:

[0159] The acquisition unit is used to acquire the first query voice information;

[0160] The acquisition unit is used to acquire the target image captured by the camera;

[0161] The processing unit is used to obtain the first response voice information of the first query voice information, wherein the first response voice information is information related to the target image, and the target image and the first query voice information are processed by a pre-trained dialogue model to obtain the first response voice information.

[0162] The playback unit is used to play the first response voice message.

[0163] Thirdly, according to one or more embodiments of this disclosure, a voice response system is provided, including an earphone device and a terminal device as described in the second aspect, wherein:

[0164] The headset device is used to collect the first query voice information; acquire the target image captured by the camera, and send the target image and the first query voice information to the terminal device.

[0165] The terminal device is used to process the target image and the first query voice information through a pre-trained dialogue model to obtain the first response voice information of the first query voice information, and send the first response voice information to the headset device;

[0166] The headset device is used to receive and play the first reply voice information sent by the terminal device.

[0167] Fourthly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0168] The memory stores the instructions that the computer executes;

[0169] At least one processor executes computer execution instructions stored in memory, causing at least one processor to perform the voice response method as described in the first aspect above and various possible designs of the first aspect.

[0170] Fifthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, implement the voice response method as described in the first aspect and various possible designs of the first aspect.

[0171] In a sixth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the voice response method as described in the first aspect above and various possible designs of the first aspect.

[0172] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0173] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0174] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A voice reply method, characterized by, The method is applied to a headset device carrying a camera, and the method comprises the following steps: collecting first inquiry voice information; acquiring target images collected by the camera; obtaining first reply voice information of the first inquiry voice information, wherein the first reply voice information is information related to the target images, and the target images and the first inquiry voice information are processed by a pre-trained dialogue model to obtain the first reply voice information; playing the first reply voice information.

2. The voice reply method of claim 1, wherein, Before the step of acquiring the target images collected by the camera, the method further comprises the following steps: in response to a wake-up operation on the camera, waking up the camera; the step of acquiring the target images collected by the camera comprises the following step:

3. The voice reply method of claim 1, wherein, controlling the woken-up camera to collect the target images. The step of acquiring the target images collected by the camera comprises the following steps: determining a shooting intention of the first inquiry voice information through a recognition model; 4. The voice reply method of claim 1, wherein, controlling the camera to collect the target images according to the shooting intention. The dialogue model is deployed in the headset device, and the step of obtaining the first reply voice information of the first inquiry voice information comprises the following step:

5. The voice reply method of claim 1, wherein, inputting the target images and the first inquiry voice information into a pre-trained dialogue model for processing to obtain the first reply voice information of the first inquiry voice information. The step of obtaining the first reply voice information of the first inquiry voice information comprises the following steps: sending the target images and the first inquiry voice information to a terminal device, wherein the terminal device is configured to process the target images and the first inquiry voice information through a pre-trained dialogue model to obtain the first reply voice information of the first inquiry voice information; 6. The voice reply method according to any one of claims 1 to 5, characterized in that, receiving the first reply voice information sent by the terminal device.

7. The voice reply method of claim 6, wherein, The headset device comprises a first headset and a second headset, the first headset carries a first camera, the second headset carries a second camera, and the target images comprise target images collected by the first camera and target images collected by the second camera. The step of sending the target images to a terminal device comprises the following steps: sending the target images collected by the first camera to the terminal device by using the first headset; 8. The voice reply method of claim 6, wherein, sending the target images collected by the second camera to the terminal device by using the second headset. The step of sending the target images to a terminal device comprises the following steps: the first headset receives the target images collected by the second camera and sent by the second headset; 9. The voice reply method of claim 6, wherein, the first headset sends the target images collected by the first camera and the target images collected by the second camera to the terminal device. The step of receiving the first reply voice information sent by the terminal device comprises the following steps: the first headset receives the first reply voice information sent by the terminal device; 10. The voice reply method of claim 6, wherein, the first headset forwards the first reply voice information to the second headset. The step of receiving the first reply voice information sent by the terminal device comprises the following steps: the first headset receives the first reply voice information sent by the terminal device; 11. The voice reply method according to any one of claims 1 to 5, characterized by, the second headset receives the first reply voice information sent by the terminal device. After the step of playing the first reply voice information, the method further comprises the following steps: collecting second inquiry voice information; obtain second reply voice information of the second inquiry voice information, wherein the second reply voice information is information related to the target image, and the target image and the second inquiry voice information are processed by the dialogue model to obtain the second reply voice information; play the second reply voice information.

12. An earphone device, characterized by The earphone device carries a camera, and the earphone device comprises: a collection unit configured to collect first inquiry voice information; an acquisition unit configured to acquire a target image collected by the camera; a processing unit configured to obtain first reply voice information of the first inquiry voice information, wherein the first reply voice information is information related to the target image, and the target image and the first inquiry voice information are processed by a pre-trained dialogue model to obtain the first reply voice information; a playing unit configured to play the first reply voice information.

13. A voice reply system characterized by The earphone device and the terminal device comprise the earphone device of claim 12, wherein: the earphone device is configured to collect first inquiry voice information, acquire a target image collected by the camera, and send the target image and the first inquiry voice information to the terminal device; the terminal device is configured to process the target image and the first inquiry voice information by a pre-trained dialogue model, obtain first reply voice information of the first inquiry voice information, and send the first reply voice information to the earphone device; the earphone device is configured to receive the first reply voice information sent by the terminal device and play the first reply voice information.

14. An electronic device, comprising: comprise: a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the voice reply method of any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the voice reply method of any one of claims 1 to 11 is realized.

16. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the voice reply method of any one of claims 1 to 11.

Images