Voice response processing method and device, equipment, medium and product

Abstract

The embodiment of the invention provides a voice response processing method and device, equipment, a medium and a product, which are applied to the technical field of voice interaction, and the method comprises the following steps: obtaining voice data of a current user and visual data corresponding to the voice data, generating a joint feature vector according to the voice data and the visual data, and sending the joint feature vector to a server; and performing user identity recognition on the current user according to the joint feature vector, and generating a personalized response strategy for the current user in combination with a user identity recognition result and the voice data so as to control equipment to respond according to the personalized response strategy. Through the embodiment of the invention, the identity of the user is recognized according to the collected voice and vision, so that personalized response is carried out according to the identity of the user, and the personalized response requirement of the user is met.

Description

Technical Field

[0001] The present invention relates to the field of voice interaction technology, and in particular to methods, devices, equipment, media and products for voice response processing. Background Art

[0002] With the development of voice interaction technology, in scenarios such as smart homes, users interact with devices through voice, bringing users a convenient and intelligent experience.

[0003] In existing voice interaction technologies, devices are usually only able to simply recognize the collected voice and then make interactive responses based on the voice recognition results, but are unable to meet the user's personalized response needs. Summary of the invention

[0004] In view of the above problems, a method, apparatus, device, medium and product for voice response processing are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A method for voice response processing, the method comprising:

[0006] Acquire voice data of the current user and visual data corresponding to the voice data;

[0007] generating a joint feature vector based on the speech data and the visual data;

[0008] According to the joint feature vector, performing user identity recognition on the current user;

[0009] Combining the result of user identification and the voice data, a personalized response strategy for the current user is generated to control the device to respond according to the personalized response strategy.

[0010] Optionally, combining the result of user identification and the voice data to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy includes:

[0011] Context information corresponding to the voice data is obtained, and a personalized response strategy for the current user is generated in combination with a result of user identification, the voice data, and the context information, so as to control the device to respond according to the personalized response strategy.

[0012] Optionally, the combining the result of user identification, the voice data, and the context information to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy includes:

[0013] Determining user preference information of the current user according to the result of user identity recognition;

[0014] Performing semantic analysis on the speech data;

[0015] In combination with the user preference information, the result of the semantic analysis, and the context information, a personalized response strategy for the current user is generated to control the device to respond according to the personalized response strategy.

[0016] Optionally, combining the user preference information, the result of the semantic analysis, and the context information to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy includes:

[0017] Combining the user preference information, the result of the semantic analysis, and the context information, generating a personalized response voice for the current user, and controlling the device to play the personalized response voice;

[0018] Alternatively, the user preference information, the result of the semantic analysis, and the context information are combined to generate a personalized response operation for the current user, and the device is controlled to execute the personalized response operation.

[0019] Optionally, it also includes:

[0020] Obtaining feedback information of the current user on the personalized response strategy;

[0021] The user preference information is adjusted according to the feedback information.

[0022] Optionally, generating a joint feature vector according to the speech data and the visual data includes:

[0023] generating speech features according to the speech data;

[0024] generating visual features based on the visual data;

[0025] Fusing the speech feature and the visual feature to obtain a fused feature;

[0026] A joint feature vector is generated according to the speech features, visual features, and fusion features.

[0027] Optionally, the performing user identity identification on the current user according to the joint feature vector includes:

[0028] Inputting the joint feature vector into a pre-trained data model to perform voiceprint matching and visual matching on the joint feature vector through the data model to obtain a result of user identity recognition;

[0029] Obtain a user identity recognition result output by the data model.

[0030] Optionally, it also includes:

[0031] Obtain unimodal sample data for speech or vision;

[0032] The unimodal sample data is used to train a preset data model so that the data model learns the unimodal feature extraction capability using a self-supervisory mechanism.

[0033] Optionally, it also includes:

[0034] Obtaining unimodal features for speech and unimodal features for vision;

[0035] The data model is trained using unimodal features for speech and unimodal features for vision, so that the data model learns cross-modal feature fusion capability using a self-supervisory mechanism.

[0036] Optionally, the device is a smart home device.

[0037] A device for voice response processing, the device comprising:

[0038] A data acquisition module, used to acquire the voice data of the current user and the visual data corresponding to the voice data;

[0039] A joint feature vector generation module, used to generate a joint feature vector according to the speech data and the visual data;

[0040] A user identification module, used to identify the user identity of the current user according to the joint feature vector;

[0041] The personalized response module is used to combine the result of user identity recognition and the voice data to generate a personalized response strategy for the current user, so as to control the device to respond according to the personalized response strategy.

[0042] An electronic device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method described above when executed by the processor.

[0043] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0044] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described above.

[0045] The embodiments of the present invention have the following advantages:

[0046] In an embodiment of the present invention, by acquiring the voice data of the current user and the visual data corresponding to the voice data, a joint feature vector is generated according to the voice data and the visual data, and the user identity of the current user is identified according to the joint feature vector. In combination with the result of the user identity identification and the voice data, a personalized response strategy for the current user is generated to control the device to respond according to the personalized response strategy, thereby realizing the identification of the user identity according to the collected voice and vision, and then making a personalized response according to the user identity, thereby meeting the user's personalized response needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0048] Figure 1 is a flowchart of a method for voice response processing provided by some embodiments of the present invention;

[0049] Figure 2a is a schematic diagram of a self-supervised learning mechanism provided by some embodiments of the present invention;

[0050] Figure 2b is a schematic diagram of a large model call and context analysis provided by some embodiments of the present invention;

[0051] Figure 2c is a flowchart of another method for voice response processing provided by some embodiments of the present invention;

[0052] Figure 3 is a flowchart of another method for voice response processing provided by some embodiments of the present invention;

[0053] Figure 4 is a flowchart of another method for voice response processing provided by some embodiments of the present invention;

[0054] Figure 5 is a flowchart of another method for voice response processing provided by some embodiments of the present invention;

[0055] Figure 6 It is a structural block diagram of a voice response processing device provided by some embodiments of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In scenarios such as smart homes, voice response systems have gone through the introduction of home automation, the Internet of Things, smart speaker central control, and AI assistants to meet the personalized needs of users. Voice response systems are moving towards seamless interaction, bringing users a smarter experience. However, voice response systems still need to be improved in terms of personalization, recognition accuracy, and response speed, especially in a home environment with multiple users.

[0058] In related technologies, voiceprint recognition can be used to identify user identities. Voiceprint recognition identifies and verifies user identities by analyzing individual voice characteristics. However, voiceprint recognition models are affected by the environment, language changes, and short-term voice performance, resulting in inaccurate user identity recognition.

[0059] In an embodiment of the present invention, accurate user identification is achieved through multimodal fusion technology combined with voice features and dynamic visual sensor data. The voiceprint model is dynamically updated by adopting a self-supervised learning mechanism to adapt to changes in user pronunciation. Based on a deep learning network, voice and visual feature extraction is optimized. At the same time, a large model is combined to generate personalized voice responses, thereby improving the system's interactive naturalness and multi-scenario adaptability.

[0060] The present invention is described below with reference to the accompanying drawings:

[0061] Reference Figure 1 , showing a step flow chart of a method for voice response processing provided by some embodiments of the present invention, which is applied to a voice response system, and the voice response system can be deployed in smart home devices.

[0062] Specifically, the following steps may be included:

[0063] Step 101, obtaining the voice data of the current user and the visual data corresponding to the voice data.

[0064] In actual applications, voice data can be collected through sensors such as microphones in the device, and visual data can be collected through sensors such as cameras in the device, wherein the visual data can be visual data collected at the same time as the voice data, that is, visual data when the user speaks.

[0065] In the embodiments of the present invention, visual data can not only supplement the deficiencies of voice signals and enhance the accuracy of identity recognition, but also perceive the user's emotions and intentions through the user's micro-expressions and postures, adjust the interaction method in real time, and combine with voice to help the system understand complex semantics and scenarios, thereby improving user experience.

[0066] In some examples, the user can first wake up the device with a wake-up word, and after the device is woken up, the device can be triggered to collect voice data and visual data.

[0067] Step 102: Generate a joint feature vector based on the speech data and the visual data.

[0068] After obtaining the speech data and the visual data, a joint feature vector integrating speech and vision can be obtained by extracting and fusing the features of the speech data and the visual data.

[0069] In some embodiments of the present invention, generating a joint feature vector based on the speech data and the visual data includes: generating speech features based on the speech data; generating visual features based on the visual data; fusing the speech features and the visual features to obtain fused features; generating a joint feature vector based on the speech features, visual features, and fused features.

[0070] In practical applications, speech features can be extracted from speech data, and visual features can be extracted from visual data. Then, the speech features and visual features can be fused to obtain fused features. Then, the speech features, visual features, and fused features can be organized into a joint feature vector.

[0071] Step 103: Perform user identity recognition on the current user according to the joint feature vector.

[0072] After obtaining the joint feature vector, the joint feature vector can be used to identify the current user. Not only can voiceprint recognition be performed using voice, and image recognition can be performed using vision, but voice and vision can also be integrated for more accurate recognition, thereby improving the accuracy of user identity recognition.

[0073] In some embodiments of the present invention, the user identity identification of the current user based on the joint feature vector includes: inputting the joint feature vector into a pre-trained data model to perform voiceprint matching and visual matching on the joint feature vector through the data model to obtain a result of user identity identification; and obtaining the result of user identity identification output by the data model.

[0074] As an example, the data model may be a neural network model.

[0075] In some embodiments of the present invention, it also includes: acquiring unimodal sample data for speech or vision; using the unimodal sample data to train a preset data model so that the data model adopts a self-supervisory mechanism to learn unimodal feature extraction capabilities.

[0076] The following combination Figure 2a An example of the self-supervised learning process is as follows:

[0077] For self-supervised feature extraction of speech and visual data, including unimodal feature extraction for speech and unimodal feature extraction for vision, the details are as follows:

[0078] For unimodal feature extraction for speech:

[0079] In practical applications, unimodal sample data for speech is obtained, and then the unimodal sample data is used to train the data model. For example, if the data model is a voiceprint recognition model, the data model can use a self-supervisory mechanism to learn the unimodal feature extraction capability for speech.

[0080] In the process of self-supervised speech feature extraction, through comparative learning, the speech data of the same user in different time periods are used as positive samples, and the speech data of different users are used as negative samples. By learning to bring positive samples closer and negative samples farther away, new voiceprint features are automatically generated.

[0081] In the embodiment of the present invention, by means of comparative learning, without relying on manually labeled data, the system continuously optimizes the voiceprint recognition model during the interaction process, so that it maintains accuracy in long-term use.

[0082] In some examples, an autoencoder structure can be used to learn voiceprint features on unlabeled data. By encoding and decoding the input speech data, the internal structural features of the data can be learned without labels, and more representative voiceprint features can be extracted for subsequent recognition and matching.

[0083] For unimodal feature extraction for vision:

[0084] In practical applications, unimodal sample data targeting vision is obtained, and then the unimodal sample data is used to train the data model. If the data model is a visual recognition model, the data model can use a self-supervisory mechanism to learn the unimodal feature extraction capability for vision.

[0085] In the process of self-supervised visual feature extraction, the system uses self-supervised learning to establish a self-supervisory mechanism within the data by detecting small changes in the user's facial expressions, lip movements, etc. For example, the system uses the "fill in the gaps" strategy to infer the missing parts using the learned features when some facial features are blocked, thereby strengthening the model's learning of complete visual features.

[0086] In some examples, for the analysis of lip movements, the system analyzes the user's lip movement trajectories in different speech segments through a self-supervised method of temporal consistency learning. The system then automatically generates target labels based on the relationship between consecutive time segments, thereby optimizing the model's mapping relationship between lip movements and pronunciation, further improving the matching effect of speech and vision.

[0087] As an example, target tags include action category tags, timestamp alignment tags, emotional state tags, location or action trajectory tags, interaction intention tags, and scene context tags. For example, after a user sends an instruction, the user's facial features are identified, and the user's emotions are identified through the emotional classification tag model. If the user is calm after replying, the process ends automatically. If the user is confused, further explanation and inquiry are triggered.

[0088] In the embodiment of the present invention, the generated tags can provide the ability of situational awareness, so that the voice interaction system can dynamically adjust the dialogue mode, device control and recognition accuracy. For example, through the tag, it is determined that the user is cooking, and the system infers that the user needs concise and clear instructions to complete the operation, and it is not suitable to interact for a long time; when the user is excited, he can get more positive suggestions.

[0089] In some embodiments of the present invention, it also includes:

[0090] Acquire unimodal features for speech and unimodal features for vision; use the unimodal features for speech and unimodal features for vision to train the data model so that the data model adopts a self-supervisory mechanism to learn cross-modal feature fusion capabilities.

[0091] In practical applications, self-supervised learning not only plays a role in feature extraction in a single modality, but also helps the system automatically learn the correlation between speech and visual features in multimodal feature fusion, thereby improving the effect of the fusion model. The data model uses a self-supervised mechanism to learn cross-modal feature fusion capabilities.

[0092] In some examples, the system automatically generates labels by using cross-modal contrastive learning to leverage the alignment between speech and visual data. The system trains the model through self-supervised learning, enabling it to learn the deep associations between speech and vision on unlabeled multimodal data.

[0093] In some examples, after extracting speech and visual features, the system uses joint self-supervised representation learning technology to map the features of these different modalities into the same feature space for unified learning. This enables the system to effectively solve the dimensionality and timing differences of multimodal data when fusing features, thereby improving the generalization ability and recognition accuracy of the model.

[0094] In some examples, in order to enable the voiceprint and visual models to dynamically adapt to user changes, an online update mechanism of the model is introduced in the self-supervised learning process, namely, a self-supervised update mechanism of the dynamic adaptive model.

[0095] Through self-supervised learning, new feature data is automatically generated during each user's voice interaction and visual capture process. These data are continuously added to the existing model through incremental learning methods, enabling the system to adaptively update the user's voiceprint and visual model, and realize incremental learning of unlabeled data.

[0096] The system has designed a self-supervised update strategy to ensure that the model maintains efficient and accurate recognition capabilities during continuous use. When new features differ greatly from existing features, the system will automatically trigger a self-update mechanism to dynamically adjust the model's parameters and structure to ensure its sensitivity to user changes.

[0097] Step 104: Generate a personalized response strategy for the current user in combination with the result of user identification and the voice data, so as to control the device to respond according to the personalized response strategy.

[0098] As an example, the device is a smart home device.

[0099] After obtaining the result of user identification, the result of semantic analysis of voice data can be used in combination with the corresponding result of user identification to generate a personalized response strategy for the current user, thereby providing different response strategies for different users.

[0100] In some embodiments of the present invention, the combination of the result of user identity recognition and the voice data to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy includes: obtaining context information corresponding to the voice data, and combining the result of user identity recognition, the voice data, and the context information to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy.

[0101] As an example, when processing user instructions, it is necessary to collect contextual information related to the current scene, including the user's historical interaction records, device status information, environmental information, and scene information.

[0102] In practical applications, by obtaining the contextual information corresponding to the voice data, when generating a personalized response strategy, not only the results of semantic analysis of the voice data and the corresponding user identity recognition results can be used, but also the voice information can be combined for analysis to generate a more accurate personalized response strategy.

[0103] In some embodiments of the present invention, the combination of the result of user identity identification, the voice data, and the context information to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy, including: determining the user preference information of the current user based on the result of user identity identification; performing semantic analysis on the voice data; and combining the user preference information, the result of the semantic analysis, and the context information to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy.

[0104] In practical applications, different users have different user preference information. The user preference information of the current user can be determined based on the result of user identity recognition, and then a personalized response strategy for the current user can be generated by combining the user preference information, the result of semantic analysis, and context information.

[0105] In some examples, a large model can be called to perform semantic analysis on voice data, in which the main natural language understanding task packages used are intent recognition (extracting user requests) and entity recognition (extracting key entities involved). Then, the context information is combined with the results of semantic analysis and user preference information through the large model to generate personalized response content. For example, if a child issues an instruction at night and needs to turn on the lights, the response content can be more interesting, turning on the night children's eye protection light, and asking whether nursery rhymes need to be played.

[0106] In some embodiments of the present invention, the user preference information, the result of the semantic analysis, and the context information are combined to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy, including: combining the user preference information, the result of the semantic analysis, and the context information to generate a personalized response voice for the current user, and controlling the device to play the personalized response voice.

[0107] In some embodiments of the present invention, the user preference information, the result of the semantic analysis, and the context information are combined to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy, including: combining the user preference information, the result of the semantic analysis, and the context information to generate a personalized response operation for the current user, and controlling the device to perform the personalized response operation.

[0108] In some embodiments of the present invention, the method further includes: obtaining feedback information of the current user on the personalized response strategy; and adjusting the user preference information according to the feedback information.

[0109] In actual applications, user preference information can be adjusted through user feedback on personalized response strategies, such as response confirmation, to achieve data update and continuous optimization. The large model records interaction data by updating user preference data and adjusting the personalized model for subsequent optimization.

[0110] For example, a user usually uses soft lighting at 7 o'clock in the evening. The intelligent system automatically adjusts the lighting at 7 o'clock. After the user comes in, he reports that the light is a little dim and wants to increase the brightness. The system records the user's new operation record and automatically adjusts the light to this brightness at 7 o'clock in the future as the user's regular preference.

[0111] The following combination Figure 2b The following is an example of the large model calling and context analysis process:

[0112] 1. Collect contextual information: When processing user instructions, it is necessary to collect contextual information related to the current scenario, including the user's historical interaction records, device status information, environmental information, and scenario information.

[0113] 2. Large model call: including natural language understanding (intent recognition, entity recognition), context information integration (time, space, user group, user preferences, etc.), and personalized content generation. Semantic analysis of user instructions is performed. The main natural language understanding task packages used are intent recognition (extracting user requests) and entity recognition (extracting key entities involved). Context information is combined with user instructions to generate personalized content.

[0114] 3. Scenario evaluation and feedback optimization: Scenario evaluation and adjustment are performed through device response confirmation and user preference matching.

[0115] 4. Personalized speech synthesis and output: Convert text into speech output through voice style selection and emotional speech synthesis.

[0116] 5. Data update and continuous optimization: After the large model is called and the response is generated, the interaction data is recorded by updating the user preference data and adjusting the personalized model for subsequent optimization.

[0117] In an embodiment of the present invention, by acquiring the voice data of the current user and the visual data corresponding to the voice data, a joint feature vector is generated according to the voice data and the visual data, and the user identity of the current user is identified according to the joint feature vector. In combination with the result of the user identity identification and the voice data, a personalized response strategy for the current user is generated to control the device to respond according to the personalized response strategy, thereby realizing the identification of the user identity according to the collected voice and vision, and then making a personalized response according to the user identity, thereby meeting the user's personalized response needs.

[0118] The following combination Figure 2c The present invention is exemplified as follows:

[0119] 1. The user issues a voice command.

[0120] 2. Voice and visual data collection, that is, collecting voice data and visual data.

[0121] 3. Multimodal feature extraction and fusion, including speech feature extraction and visual feature extraction. Speech features include pitch features, high-pitched features, rhythm features, Mel-frequency cepstral coefficients, linear prediction coding, and spectrogram. Visual features include mouth shape and movement trajectory, facial expression features, depth information, eye movement, and blinking frequency.

[0122] 4. Real-time voiceprint and feature matching: Real-time voiceprint and feature matching means that the system can not only match the user's current feature vector with the system database template in real time, but also update the model through self-supervised dynamic learning, so that it can be gradually optimized without clear labeling to adapt to changes in user characteristics. For example, one day the user's voice is hoarse due to a cold, the curvature of the corners of the mouth is different from before, and the voiceprint characteristics change slightly. The system recognizes its identity and updates the voiceprint model through self-supervised comparison. The system can quickly adapt to the user's current state and issue a more intelligent and personalized response.

[0123] 5. Large model calling and context analysis.

[0124] 6. Personalized response generation.

[0125] 7. Voice output and user interaction.

[0126] 8. Smart home device control.

[0127] 9. Data storage and dynamic optimization.

[0128] Reference Figure 3 , shows a flowchart of another method for voice response processing provided by some embodiments of the present invention, which may specifically include the following steps:

[0129] Step 301, obtaining the voice data of the current user and the visual data corresponding to the voice data.

[0130] In actual applications, voice data can be collected through sensors such as microphones in the device, and visual data can be collected through sensors such as cameras in the device, wherein the visual data can be visual data collected at the same time as the voice data, that is, visual data when the user speaks.

[0131] In the embodiments of the present invention, visual data can not only supplement the deficiencies of voice signals and enhance the accuracy of identity recognition, but also perceive the user's emotions and intentions through the user's micro-expressions and postures, adjust the interaction method in real time, and combine with voice to help the system understand complex semantics and scenarios, thereby improving user experience.

[0132] In some examples, the user can first wake up the device with a wake-up word, and after the device is woken up, the device can be triggered to collect voice data and visual data.

[0133] Step 302: Generate a joint feature vector based on the speech data and the visual data.

[0134] After obtaining the speech data and the visual data, a joint feature vector integrating speech and vision can be obtained by extracting and fusing the features of the speech data and the visual data.

[0135] Step 303: Perform user identity recognition on the current user according to the joint feature vector.

[0136] After obtaining the joint feature vector, the joint feature vector can be used to identify the current user. Not only can voiceprint recognition be performed using voice, and image recognition can be performed using vision, but voice and vision can also be integrated for more accurate recognition, thereby improving the accuracy of user identity recognition.

[0137] Step 304, obtaining context information corresponding to the voice data, and combining the result of user identification, the voice data, and the context information to generate a personalized response strategy for the current user, so as to control the device to respond according to the personalized response strategy.

[0138] As an example, when processing user instructions, it is necessary to collect contextual information related to the current scene, including the user's historical interaction records, device status information, environmental information, and scene information.

[0139] In practical applications, by obtaining the contextual information corresponding to the voice data, when generating a personalized response strategy, not only the results of semantic analysis of the voice data and the corresponding user identity recognition results can be used, but also the voice information can be combined for analysis to generate a more accurate personalized response strategy.

[0140] Reference Figure 4 , shows a flowchart of another method for voice response processing provided by some embodiments of the present invention, which may specifically include the following steps:

[0141] Step 401, obtaining the voice data of the current user and the visual data corresponding to the voice data.

[0142] In actual applications, voice data can be collected through sensors such as microphones in the device, and visual data can be collected through sensors such as cameras in the device, wherein the visual data can be visual data collected at the same time as the voice data, that is, visual data when the user speaks.

[0143] In the embodiments of the present invention, visual data can not only supplement the deficiencies of voice signals and enhance the accuracy of identity recognition, but also perceive the user's emotions and intentions through the user's micro-expressions and postures, adjust the interaction method in real time, and combine with voice to help the system understand complex semantics and scenarios, thereby improving user experience.

[0144] In some examples, the user can first wake up the device with a wake-up word, and after the device is woken up, the device can be triggered to collect voice data and visual data.

[0145] Step 402: Generate a joint feature vector based on the speech data and the visual data.

[0146] After obtaining the speech data and the visual data, a joint feature vector integrating speech and vision can be obtained by extracting and fusing the features of the speech data and the visual data.

[0147] Step 403: Perform user identity recognition on the current user according to the joint feature vector.

[0148] After obtaining the joint feature vector, the joint feature vector can be used to identify the current user. Not only can voiceprint recognition be performed using voice, and image recognition can be performed using vision, but voice and vision can also be integrated for more accurate recognition, thereby improving the accuracy of user identity recognition.

[0149] Step 404: Determine the user preference information of the current user according to the result of user identity recognition.

[0150] Step 405: perform semantic analysis on the speech data.

[0151] Step 406: Generate a personalized response strategy for the current user in combination with the user preference information, the result of the semantic analysis, and the context information, so as to control the device to respond according to the personalized response strategy.

[0152] As an example, when processing user instructions, it is necessary to collect contextual information related to the current scene, including the user's historical interaction records, device status information, environmental information, and scene information.

[0153] In practical applications, by obtaining the contextual information corresponding to the voice data, when generating a personalized response strategy, not only the results of semantic analysis of the voice data and the corresponding user identity recognition results can be used, but also the voice information can be combined for analysis to generate a more accurate personalized response strategy.

[0154] In practical applications, different users have different user preference information. The user preference information of the current user can be determined based on the result of user identity recognition, and then a personalized response strategy for the current user can be generated by combining the user preference information, the result of semantic analysis, and context information.

[0155] In some examples, a large model can be called to perform semantic analysis on voice data, in which the main natural language understanding task packages used are intent recognition (extracting user requests) and entity recognition (extracting key entities involved). Then, the context information is combined with the results of semantic analysis and user preference information through the large model to generate personalized response content. For example, if a child issues an instruction at night and needs to turn on the lights, the response content can be more interesting, turning on the night children's eye protection light, and asking whether nursery rhymes need to be played.

[0156] Reference Figure 5 , shows a flowchart of another method for voice response processing provided by some embodiments of the present invention, which may specifically include the following steps:

[0157] Step 501, obtaining the voice data of the current user and the visual data corresponding to the voice data.

[0158] In actual applications, voice data can be collected through sensors such as microphones in the device, and visual data can be collected through sensors such as cameras in the device, wherein the visual data can be visual data collected at the same time as the voice data, that is, visual data when the user speaks.

[0159] In the embodiments of the present invention, visual data can not only supplement the deficiencies of voice signals and enhance the accuracy of identity recognition, but also perceive the user's emotions and intentions through the user's micro-expressions and postures, adjust the interaction method in real time, and combine with voice to help the system understand complex semantics and scenarios, thereby improving user experience.

[0160] In some examples, the user can first wake up the device with a wake-up word, and after the device is woken up, the device can be triggered to collect voice data and visual data.

[0161] Step 502: Generate voice features based on the voice data.

[0162] Step 503: Generate visual features based on the visual data.

[0163] Step 504: fuse the speech feature and the visual feature to obtain a fused feature.

[0164] Step 505: Generate a joint feature vector based on the speech features, visual features, and fusion features.

[0165] In practical applications, speech features can be extracted from speech data, and visual features can be extracted from visual data. Then, the speech features and visual features can be fused to obtain fused features. Then, the speech features, visual features, and fused features can be organized into a joint feature vector.

[0166] Step 506: Perform user identity recognition on the current user according to the joint feature vector.

[0167] After obtaining the joint feature vector, the joint feature vector can be used to identify the current user. Not only can voiceprint recognition be performed using voice, and image recognition can be performed using vision, but voice and vision can also be integrated for more accurate recognition, thereby improving the accuracy of user identity recognition.

[0168] Step 507: Generate a personalized response strategy for the current user in combination with the result of user identification and the voice data, so as to control the device to respond according to the personalized response strategy.

[0169] After obtaining the result of user identification, the result of semantic analysis of voice data can be used in combination with the corresponding result of user identification to generate a personalized response strategy for the current user, thereby providing different response strategies for different users.

[0170] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0171] Reference Figure 6 , shows a schematic diagram of the structure of a voice response processing device provided by some embodiments of the present invention, which may specifically include the following modules:

[0172] The data acquisition module 601 is used to acquire the voice data of the current user and the visual data corresponding to the voice data;

[0173] A joint feature vector generating module 602, configured to generate a joint feature vector according to the speech data and the visual data;

[0174] A user identification module 603, configured to identify the user identity of the current user according to the joint feature vector;

[0175] The personalized response module 604 is used to combine the result of user identity recognition and the voice data to generate a personalized response strategy for the current user, so as to control the device to respond according to the personalized response strategy.

[0176] In some embodiments of the present invention, combining the result of user identification and the voice data to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy includes:

[0177] Context information corresponding to the voice data is obtained, and a personalized response strategy for the current user is generated in combination with a result of user identification, the voice data, and the context information, so as to control the device to respond according to the personalized response strategy.

[0178] In some embodiments of the present invention, the combining the result of user identification, the voice data, and the context information to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy includes:

[0179] Determining user preference information of the current user according to the result of user identity recognition;

[0180] Performing semantic analysis on the speech data;

[0181] In combination with the user preference information, the result of the semantic analysis, and the context information, a personalized response strategy for the current user is generated to control the device to respond according to the personalized response strategy.

[0182] In some embodiments of the present invention, combining the user preference information, the result of the semantic analysis, and the context information to generate a personalized response strategy for the current user to control the device to respond according to the personalized response strategy includes:

[0183] Combining the user preference information, the result of the semantic analysis, and the context information, generating a personalized response voice for the current user, and controlling the device to play the personalized response voice;

[0184] Alternatively, the user preference information, the result of the semantic analysis, and the context information are combined to generate a personalized response operation for the current user, and the device is controlled to execute the personalized response operation.

[0185] In some embodiments of the present invention, it also includes:

[0186] Obtaining feedback information of the current user on the personalized response strategy;

[0187] The user preference information is adjusted according to the feedback information.

[0188] In some embodiments of the present invention, generating a joint feature vector according to the speech data and the visual data includes:

[0189] generating speech features according to the speech data;

[0190] generating visual features based on the visual data;

[0191] Fusing the speech feature and the visual feature to obtain a fused feature;

[0192] A joint feature vector is generated according to the speech features, visual features, and fusion features.

[0193] In some embodiments of the present invention, the step of identifying the current user according to the joint feature vector includes:

[0194] Inputting the joint feature vector into a pre-trained data model to perform voiceprint matching and visual matching on the joint feature vector through the data model to obtain a result of user identity recognition;

[0195] Obtain a user identity recognition result output by the data model.

[0196] In some embodiments of the present invention, it also includes:

[0197] Obtain unimodal sample data for speech or vision;

[0198] The unimodal sample data is used to train a preset data model so that the data model learns the unimodal feature extraction capability using a self-supervisory mechanism.

[0199] In some embodiments of the present invention, it also includes:

[0200] Obtaining unimodal features for speech and unimodal features for vision;

[0201] The data model is trained using unimodal features for speech and unimodal features for vision, so that the data model learns cross-modal feature fusion capability using a self-supervisory mechanism.

[0202] In some embodiments of the present invention, the device is a smart home device.

[0203] In an embodiment of the present invention, by acquiring the voice data of the current user and the visual data corresponding to the voice data, a joint feature vector is generated according to the voice data and the visual data, and the user identity of the current user is identified according to the joint feature vector. In combination with the result of the user identity identification and the voice data, a personalized response strategy for the current user is generated to control the device to respond according to the personalized response strategy, thereby realizing the identification of the user identity according to the collected voice and vision, and then making a personalized response according to the user identity, thereby meeting the user's personalized response needs.

[0204] Some embodiments of the present invention further provide an electronic device, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the above method is implemented when the computer program is executed by the processor.

[0205] Some embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and the computer program implements the above method when executed by a processor.

[0206] Some embodiments of the present invention further provide a computer program product, including a computer program, which implements the above method when executed by a processor.

[0207] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0208] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0209] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0210] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0211] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0212] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0213] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0214] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0215] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the above elements.

[0216] The above describes in detail the provided voice response processing methods, devices, equipment, media and products. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for voice response processing, characterized in that: The method comprises: Acquire voice data of the current user and visual data corresponding to the voice data; generating a joint feature vector based on the speech data and the visual data; According to the joint feature vector, performing user identity recognition on the current user; Combining the result of user identification and the voice data, a personalized response strategy for the current user is generated to control the device to respond according to the personalized response strategy.

2. The method according to claim 1, characterized in that The combining the result of user identification and the voice data to generate a personalized response strategy for the current user, so as to control the device to respond according to the personalized response strategy, includes: Context information corresponding to the voice data is obtained, and a personalized response strategy for the current user is generated in combination with a result of user identification, the voice data, and the context information, so as to control the device to respond according to the personalized response strategy.

3. The method according to claim 2, characterized in that The combining the result of user identification, the voice data, and the context information to generate a personalized response strategy for the current user, so as to control the device to respond according to the personalized response strategy, includes: Determining user preference information of the current user according to the result of user identity recognition; Performing semantic analysis on the speech data; In combination with the user preference information, the result of the semantic analysis, and the context information, a personalized response strategy for the current user is generated to control the device to respond according to the personalized response strategy.

4. The method according to claim 3, characterized in that The combining the user preference information, the result of the semantic analysis, and the context information to generate a personalized response strategy for the current user, so as to control the device to respond according to the personalized response strategy, includes: Combining the user preference information, the result of the semantic analysis, and the context information, generating a personalized response voice for the current user, and controlling the device to play the personalized response voice; Alternatively, the user preference information, the result of the semantic analysis, and the context information are combined to generate a personalized response operation for the current user, and the device is controlled to execute the personalized response operation.

5. The method according to claim 3, characterized in that: Also includes: Obtaining feedback information of the current user on the personalized response strategy; The user preference information is adjusted according to the feedback information.

6. The method according to any one of claims 1 to 5, characterized in that: The step of generating a joint feature vector according to the speech data and the visual data comprises: generating speech features according to the speech data; generating a visual feature based on the visual data; Fusing the speech feature and the visual feature to obtain a fused feature; A joint feature vector is generated according to the speech features, visual features, and fusion features.

7. The method according to claim 6, characterized in that The performing user identity identification on the current user according to the joint feature vector includes: Inputting the joint feature vector into a pre-trained data model to perform voiceprint matching and visual matching on the joint feature vector through the data model to obtain a result of user identity recognition; Obtain a user identity recognition result output by the data model.

8. The method according to claim 7, characterized in that Also includes: Obtain unimodal sample data for speech or vision; The unimodal sample data is used to train a preset data model so that the data model learns the unimodal feature extraction capability using a self-supervisory mechanism.

9. The method according to claim 8, characterized in that Also includes: Obtaining unimodal features for speech and unimodal features for vision; The data model is trained using unimodal features for speech and unimodal features for vision, so that the data model learns cross-modal feature fusion capability using a self-supervisory mechanism.

10. The method according to claim 1, characterized in that The device is a smart home device.

11. A device for voice response processing, characterized in that: The device comprises: A data acquisition module, used to acquire the voice data of the current user and the visual data corresponding to the voice data; A joint feature vector generation module, used to generate a joint feature vector according to the speech data and the visual data; A user identification module, used to identify the user identity of the current user according to the joint feature vector; The personalized response module is used to combine the result of user identity recognition and the voice data to generate a personalized response strategy for the current user, so as to control the device to respond according to the personalized response strategy.

12. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 10 when executed by the processor.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images