Interaction right switching method and apparatus, electronic device, and storage medium

Abstract

The application provides an interactive right switching method and device, electronic equipment and a storage medium, wherein the method comprises: determining video data in an interactive scene and an initial interactive person indicated by the video data; determining lip data of each person in the interactive scene based on the video data; in the case that a wake-up word is detected, determining a potential interactive person outputting the wake-up word based on voice data corresponding to the wake-up word and the lip data of each person; and in the case that the potential interactive person and the initial interactive person are different persons, switching the interactive right of the initial interactive person based on an identity number of the potential interactive person. The application realizes interactive right switching in a multi-person scene, overcomes the defect that the interactive right is difficult to switch in the traditional scheme when multiple persons interact, can stably track each person, realizes accurate positioning of the potential interactive person, and stably switches the interactive right.

Description

Technical Field

[0001] This invention relates to the field of voice interaction technology, and in particular to a method, apparatus, electronic device, and storage medium for switching interaction rights. Background Technology

[0002] With the development of speech recognition technology, the scenarios in which it is applied are becoming more and more complex. From the initial recognition of Mandarin in quiet scenarios to the recognition of accents, minority languages, dialects and other languages ​​in high-noise and complex scenarios, it faces increasingly greater challenges and increasingly harsher environments.

[0003] Currently, multimodal technology is often used to solve the recognition problem in high-noise scenarios. It uses audio and video information as input and improves the accuracy of speech recognition by fusing audio and video. However, current multimodal technology is mostly used in single-person scenarios, such as car drivers and hospital self-service interactive terminals. It is widely used in single-person scenarios. However, for open multi-person scenarios, such as command and dispatch screens and interactive screens in public places, there are often more than one interaction object. In this case, multimodal technology in single-person scenarios cannot identify the role of different interaction objects or switch the interaction rights according to their interaction intentions. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for switching interaction rights, which solves the problem of difficulty in switching interaction rights when multiple people interact in the prior art, and realizes stable tracking of each person, accurate positioning of potential interaction personnel, and steady switching of interaction rights.

[0005] This invention provides a method for switching interaction rights, comprising:

[0006] Determine the video data in the interaction scenario, and the initial interaction personnel indicated by the video data;

[0007] Based on the video data, determine the lip data of each person in the interaction scenario;

[0008] If a wake word is detected, the potential interaction personnel who output the wake word are determined based on the voice data corresponding to the wake word and the lip data of each person.

[0009] If the potential interacting person and the initial interacting person are different people, the interaction rights of the initial interacting person are switched based on the identity number of the potential interacting person.

[0010] According to a method for switching interaction rights provided by the present invention, the method further includes, after switching the interaction rights of the initial interaction person based on the identity number of the potential interaction person, the method further includes:

[0011] Based on the video data, the image interaction intention of the potential interacting person is determined, and the image interaction intention includes at least one of gesture interaction intention, posture interaction intention, and action interaction intention;

[0012] And / or, based on the interactive voice in the interactive scenario, determine the voice interaction intent of the potential interactive personnel;

[0013] Interaction is performed based on the image interaction intent and / or the voice interaction intent.

[0014] According to a method for switching interaction rights provided by the present invention, determining the voice interaction intent of the potential interaction user based on the interactive voice in the interaction scenario includes:

[0015] Based on the lip data of the potential interacting person, speech separation and / or speech endpoint detection are performed on the interactive speech to obtain the effective speech of the potential interacting person.

[0016] Based on the lip data of the potential interacting person, speech recognition is performed on the valid speech, and intent recognition is performed on the recognized text obtained from the speech recognition to obtain the speech interaction intent of the potential interacting person.

[0017] According to a method for switching interaction rights provided by the present invention, the method further includes, after switching the interaction rights of the initial interaction person based on the identity number of the potential interaction person, the method further includes:

[0018] Based on the interactive voice in the interactive scenario, determine the voice interaction intention of the target interactive person. The target interactive person is a preset number of interactive persons who are in the interactive scenario and who are before the potential interactive persons.

[0019] Based on the video data, the facial data of the target interacting person is determined;

[0020] The interaction is conducted based on the facial data and voice interaction intent of the target user.

[0021] According to a method for switching interaction rights provided by the present invention, the method further includes, after switching the interaction rights of the initial interaction person based on the identity number of the potential interaction person, the method further includes:

[0022] Obtain the sound source localization results and visual localization results of the potential interacting persons;

[0023] Based on the sound source localization results and the visual localization results, the potential interacting personnel are targeted and tracked.

[0024] The sound source localization result is obtained by locating the sound source of the potential interactive person using a microphone array in the interactive scene, and the visual localization result is obtained by locating the visual person of the potential interactive person using a camera in the interactive scene.

[0025] According to a method for switching interaction rights provided by the present invention, determining the lip data of each person in the interaction scene based on the video data includes:

[0026] Human body tracking is performed based on each frame of the video data to obtain the human body regions of each person in the interactive scene in each frame of the video data.

[0027] Face detection is performed on each human body region to obtain the face region of each person in each frame image, and key point detection is performed on each face region to obtain the lip data of each person in each frame image.

[0028] According to the present invention, an interaction right switching method is provided, wherein human body tracking is performed based on each frame of the video data to obtain the human body regions of each person in the interaction scene in each frame of the video data, including:

[0029] Human detection is performed on each frame of the video data to obtain the human body region in each frame of the video data.

[0030] Based on the overlapping area of ​​each human body region in adjacent frame images, human body tracking is performed on each person corresponding to each human body region to obtain the human body region of each person in each frame image in the interactive scene.

[0031] The present invention also provides an interaction right switching device, comprising:

[0032] A determining unit is used to determine the video data in the interaction scenario and the initial interaction personnel indicated by the video data;

[0033] A face detection unit is used to determine the lip data of each person in the interaction scenario based on the video data.

[0034] The lip sound detection unit is used to determine the potential interactive personnel who output the wake word based on the speech data corresponding to the wake word and the lip data of each person when a wake word is detected.

[0035] The interaction right switching unit is used to switch the interaction right of the initial interaction person based on the identity number of the potential interaction person when the potential interaction person and the initial interaction person are different people.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the interaction right switching method as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the interaction right switching method as described above.

[0038] The interaction right switching method, device, electronic device, and storage medium provided by this invention perform human body tracking and face detection on each person in the interaction scenario using video data. It also combines lip data of each person with the speech data corresponding to the wake word to perform lip-sound consistency detection, obtaining the detection results. Based on the detection results, the potential interaction person outputting the wake word can be accurately located. The interaction right is switched to the initial interaction person using the identity number of the potential interaction person. This achieves interaction right switching in multi-person scenarios, overcoming the shortcomings of traditional solutions where interaction right is difficult to switch in multi-person interactions. It enables stable tracking of each person in multi-person scenarios, achieving accurate location of potential interaction persons and steady switching of interaction rights. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the interaction right switching method provided by the present invention;

[0041] Figure 2 This is one of the flowcharts illustrating the multimodal interaction process provided by the present invention;

[0042] Figure 3 This is a flowchart illustrating step 220 in the multimodal interaction process provided by the present invention;

[0043] Figure 4 This is the second flowchart illustrating the multimodal interaction process provided by the present invention;

[0044] Figure 5 This is a flowchart illustrating the directional tracking process provided by the present invention;

[0045] Figure 6 This is a flowchart illustrating step 120 of the interaction right switching method provided by the present invention;

[0046] Figure 7 This is a flowchart illustrating step 121 of the interaction right switching method provided by the present invention;

[0047] Figure 8 This is a schematic diagram of the interactive right switching device provided by the present invention;

[0048] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] With the development of speech recognition technology, its application scenarios are becoming increasingly complex. From the initial recognition of Mandarin in quiet scenarios to the recognition of accents, minority languages, and dialects in high-noise and complex scenarios, the challenges it faces are not only accents and languages, but more importantly, high noise in complex scenarios, i.e., how to achieve accurate speech recognition in high-noise and complex scenarios.

[0051] Noise interference can be broadly categorized as follows: environmental noise / human voice interference, interference from different signal-to-noise ratios, and interference between speakers from the same and different speaking regions. From the perspective of source, noise can be divided into environmental noise and man-made noise. Environmental noise is mostly relatively singular, commonly found in natural ambient sounds, music, knocking sounds, and friction sounds, and its speech spectrum is relatively uniform, showing a clear difference from human voice. However, the location of environmental noise sources is not fixed, often resulting in multi-point noise interference. Man-made noise is the primary form of interference, commonly seen in conferences as interference from non-main speakers or non-target speakers. This type of interference has a smaller spectral difference from the main or target speaker's speech, often making it difficult to distinguish.

[0052] Interference from non-homophonic regions (where the pickup device and the target speaker are in the same pickup area, while the non-target speaker is in a different pickup area) can be avoided through beam narrowing. However, interference from homophonic regions (where the target speaker, non-target speaker, and pickup device are in the same pickup area) is currently difficult to avoid using appropriate methods. Furthermore, as the signal-to-noise ratio of both types of noise and effective speech continues to decrease, the speech recognition environment will become increasingly harsh.

[0053] To address speech recognition challenges in noisy environments, current research often employs multimodal technologies based on audio and video. These technologies use audio and video as input, and the fusion of these inputs improves speech recognition accuracy while mitigating misidentification due to noise when the target speaker is not speaking. Therefore, multimodal technology has a natural advantage in noisy and complex scenarios. However, it is primarily suited for single-person scenarios, such as those involving a car driver, subway ticket machines, and hospital self-service terminals, where it has achieved widespread application. In these scenarios, the interaction is with a single person; even in queues, the interaction proceeds only after the previous person has finished interacting.

[0054] However, in open, multi-person scenarios, such as command and dispatch screens or assembly and interactive screens in public settings (which are combined with virtual avatars for interaction), there are often more than one potential interactive object on the screen. In this case, the aforementioned multimodal technology for single-person scenarios cannot switch the interaction rights according to the interaction intentions of different interactive objects.

[0055] Furthermore, current multimodal technologies can be divided into two parts: vision-based face detection and multimodal interaction based on the fusion of speech and video. The former detects potential faces in each frame of an image and returns the coordinates of the face's range. Then, using face alignment technology, it detects the feature contour points of the facial features within the face region and inputs them into the downstream multimodal interaction task. Multimodal interaction, on the other hand, fuses the face with the corresponding speech, simultaneously utilizing multimodal data from both speech and video to assist a single speech recognition task, thereby improving its recognition effect and accuracy.

[0056] However, the aforementioned multimodal technologies are only effective for the target speaker's face, meaning they are highly effective without needing to change the target speaker's identity. In open, multi-person scenarios, they clearly cannot identify the roles of different interacting individuals or switch interaction rights based on their interaction intentions.

[0057] Referring to traditional speech recognition frameworks, there are currently two main techniques for acquiring interaction rights: using wake words for interaction and locating the target speaker through sound source localization. The former involves outputting a wake word to activate the interactive device, thereby gaining interaction rights; the latter uses a microphone array to detect and locate a sound source in a quiet environment. However, its localization range is large, making precise localization impossible, and it requires a very high signal-to-noise ratio. In environments with significant noise interference, its localization accuracy is greatly reduced.

[0058] In summary, current multimodal technologies struggle to accurately identify and switch the interaction rights of the target speaker in open, multi-person scenarios when their roles change. The main reason for this is that:

[0059] First, it is difficult to reliably track a single target speaker: Current face detection-based tracking algorithms are prone to missed detections and false detections. When detecting all possible faces in a frame of an image, if the target turns or tilts its head, it is very easy to lose tracking. Even if the target turns its head back to normal later, it will only be marked as a new speaker when it is detected again.

[0060] Secondly, the interaction rights cannot be switched in multi-person scenarios: the current interaction technology is designed for a single interaction object and limits the position of the single interaction object, requiring that its position does not change significantly; however, in open multi-person scenarios, if there are multiple potential interaction objects at the same time, it is not possible to switch the interaction rights for different interaction objects.

[0061] Third, it is impossible to accurately locate potential interaction objects: Current methods for obtaining interaction rights based on monomodal speech, such as wake word interaction and sound source localization, can only define a general interaction range. Moreover, they have extremely high requirements for the signal-to-noise ratio of the environment. In high-noise and complex scenarios, it is difficult to accurately locate potential interaction objects in multi-person scenarios.

[0062] To address this, the present invention provides an interaction right switching method. This method utilizes video data from an interactive scenario to perform human body tracking and face detection on each individual within that scenario. It also combines lip data of each individual with voice data corresponding to the wake word to perform lip-phonetic consistency detection. The detection results accurately locate potential interactive personnel, thereby enabling the switching of interaction rights between different interactive personnel. This overcomes the shortcomings of traditional solutions that cannot perform role positioning and interaction right switching for different interactive personnel in multi-person scenarios. It allows for stable tracking of individuals in open multi-person scenarios, achieving accurate positioning of potential interactive personnel and a steady switching of interaction rights. Figure 1 This is a flowchart illustrating the interaction right switching method provided by the present invention, as follows: Figure 1 As shown, the execution subject of this method can be an interactive device or a server that directly controls the interactive device. The method includes:

[0063] Step 110: Determine the video data in the interaction scenario and the initial interaction personnel indicated by the video data;

[0064] Specifically, before switching the interaction rights, it is necessary to first determine the video data in the interaction scenario. The video data here refers to the continuous interactive image data in the interaction scenario collected in real time by the image acquisition device. The image acquisition device can be a camera, webcam, video camera, etc. It can be installed on the interaction device or independent of the interaction device. This embodiment of the invention does not make specific limitations on this.

[0065] Here, after acquiring the video data, it is also necessary to determine the first person to interact with the interactive device in the interactive scenario, i.e., the initial person to interact. This can be determined from the first few frames of the video data. Specifically, this can be done by performing face detection on the first few frames, selecting the face region with the highest confidence from all detection results, and using the corresponding person as the initial person to interact; or by selecting the face region with the largest area from all detection results that meet the detection threshold, and using the corresponding person as the initial person to interact. The detection threshold here can be a confidence threshold, an area threshold, or a combination of both, and its specific value can be set according to actual needs.

[0066] Step 120: Based on the video data, determine the lip data of each person in the interaction scenario;

[0067] Considering that traditional face tracking methods are prone to missed detections and false detections due to the target turning or tilting their head, resulting in tracking loss, this invention does not start with face tracking, but instead performs human body tracking, and then performs face detection on this basis. By binding human body tracking and face detection, the loss of the target and false detection (marking the target as a new interactive person) can be greatly avoided, ensuring stable tracking of people.

[0068] Specifically, after obtaining the video data in the interactive scenario, step 120 can be executed to determine the lip data of each person using the video data. The specific process includes the following steps:

[0069] Since faces can disappear in a few frames of video data due to head turning or tilting, making them difficult to detect and track, human body tracking is more stable and the human body area occupies a larger area, making it easier to solve the movement position of the corresponding person. Therefore, human body tracking can be performed based on each frame of the video data to determine the human body area of ​​each person in each frame of the interactive scene. Specifically, the human body tracking of each person in each frame of the video data can be achieved by using the area of ​​the human body region in each frame of the video data.

[0070] Subsequently, face detection can be performed on the human body regions in each frame of the image obtained by human body tracking. That is, face detection is performed on each person's region to obtain the face region of each person in each frame of the image. The key point contours of the facial features can be determined from these face regions, and the lip data of each person can be extracted from the key point contours for subsequent interaction right switching in multi-person scenarios and multimodal interaction tasks.

[0071] Here, the integration of human body tracking and face detection not only makes face detection simpler and more efficient—that is, face detection only needs to be performed within the face region in each frame of the fixed tracking image—it can significantly improve the accuracy of face detection while reducing interference. It also makes the tracking process more stable. Even if the face of the corresponding person is not detected in a few frames, because it is the human body that is being tracked, not the face, and the human body will not disappear into thin air, there will be no jumping of the tracking object or loss of tracking. Based on the overlap of the human body region in the consecutive frames, the movement trajectory of each person can be accurately and stably tracked, realizing effective tracking of each person in interactive scenarios.

[0072] It should be noted that the tracking here is not only for the interaction personnel, but also for other personnel in the interaction scenario. Tracking each person can lay the foundation for the accurate location of potential interaction personnel before the switching of interaction rights, provide the auxiliary information needed to locate potential interaction personnel, and provide key assistance for the smooth progress of the switching of interaction rights.

[0073] Step 130: If a wake word is detected, determine the potential interaction personnel for outputting the wake word based on the speech data corresponding to the wake word and the lip data of each person.

[0074] Specifically, after obtaining the lip data of each person in step 120, step 130 can be executed. If a wake-up word is detected, the potential interaction personnel for outputting the wake-up word are determined based on the speech data corresponding to the wake-up word and the lip data of each person. The specific process includes:

[0075] In an open, multi-person scenario, in addition to the current person interacting, there may be other people with interaction intentions. At this time, this person can wake up the interactive device by outputting a specific wake word, so that their interaction intention is made clear. Based on the accurate positioning of them, the interaction rights can be switched to enable them to have the ability to interact and interact with the interactive device.

[0076] First, during the human-computer interaction process, voice detection can be continuously performed on the interaction scenario. When a wake word is detected, the voice data corresponding to the wake word can be obtained. Here, the wake word is used to trigger the single-modal (voice) wake-up engine in the interactive device and throw out the wake-up state. It is a pre-set specific word, such as "Ai Jia Ni Hao" or "HeySiri".

[0077] The voice data corresponding to the wake-up word can be understood as the voice data at the wake-up time and the time before and after it. It can be collected by a voice acquisition device, such as a microphone or a microphone pen. It can be installed on the interactive device or it can be independent of the interactive device. This embodiment of the invention does not make specific limitations on this.

[0078] Then, the voice data corresponding to the wake word and the lip data of each person can be used to accurately locate the person with the intention to interact. Specifically, based on the voice data corresponding to the wake word and the lip data of each person, lip-sound consistency detection can be performed on each person to obtain the detection result. That is, the lip movements reflected by the lip data of each person before and after wake-up are compared with the lip movements corresponding to the pronunciation of the wake word in the voice data to verify whether the lip movements of each person correspond to the pronunciation rhythm of the wake word, thereby obtaining the detection result.

[0079] After that, the potential interactive personnel for outputting the wake word can be determined by referring to the detection results obtained from the lip-sound consistency detection. That is, the personnel corresponding to the lip data when the detection result is lip-sound consistency are regarded as potential interactive personnel with interactive intentions. In other words, the personnel whose lip movement amplitude and the lip amplitude corresponding to the wake word are consistent with the speech data and lip data are regarded as potential interactive personnel with interactive intentions, which can also be called interactive right switching personnel.

[0080] Compared to traditional wake-up interactions based on single voice and sound source localization, this method uses lip-verb consistency detection to achieve more accurate and precise role localization of potential interactors when switching roles, and has wider applicability, thus facilitating the subsequent switching of interaction rights.

[0081] Step 140: If the potential interactor and the initial interactor are different people, switch the interaction rights of the initial interactor based on the identity number of the potential interactor.

[0082] Specifically, after obtaining the lip-consonance detection result through the above steps, step 140 can be executed. Based on this detection result, the potential interaction user for the wake-up word is determined, and the interaction right is switched according to the identity number of the potential interaction user. The specific process includes:

[0083] First, determine whether the potential interaction personnel identified this time are the same person as the current interaction personnel (initial interaction personnel). Specifically, this can be done by using the identity labels of the potential interaction personnel and the initial interaction personnel to determine whether they correspond to the same person or belong to different people.

[0084] Furthermore, when the potential interacting person and the initial interacting person are different individuals, a switch of interaction rights is initiated. Specifically, when it is determined that they are not the same person, an identity label is assigned to the potential interacting person. This identity label is determined during the tracking process; that is, when tracking each person in the interaction scenario, each person is labeled with an identity document (ID) representing their identity. This identity label can be used to switch interaction rights, transferring the interaction rights from the initial interacting person's identity label to the potential interacting person's identity label. After the switch, the potential interacting person becomes the current interacting person, possessing interaction rights and capable of multimodal interaction with the interactive device.

[0085] The interaction right switching method provided by this invention uses video data in an interactive scenario to perform human body tracking and face detection on each person in the scenario. It also combines the lip data of each person with the speech data corresponding to the wake word to perform lip-sound consistency detection and obtain the detection results. Based on the detection results, the potential interactive person who outputs the wake word can be accurately located. By identifying the potential interactive person, the interaction right of the initial interactive person can be switched. This realizes the interaction right switching in a multi-person scenario, overcomes the defect of traditional solutions where the interaction right is difficult to switch in a multi-person interaction, and can stably track each person in a multi-person scenario, achieving accurate positioning of potential interactive persons and steady switching of interaction rights.

[0086] Based on the above embodiments, in step 130, based on the speech data corresponding to the wake word and the lip data of each person, lip-sound consistency detection is performed to obtain the detection result, including:

[0087] Based on the lip sound detection model, features are extracted from the speech data and lip data respectively, and lip sound consistency is detected based on the speech features and lip shape features obtained from the feature extraction to obtain the detection results for each person.

[0088] The lip sound detection model is trained based on the feature similarity between sample speech features from sample speech data and sample lip shape features from sample video data.

[0089] Specifically, step 130, which uses the speech data corresponding to the wake word and the lip data of each person to perform lip-sound consistency detection and obtain the detection result, includes the following:

[0090] Here, the process of performing lip-sound consistency detection using speech data and lip data can be achieved with the help of a lip-sound detection model. Specifically, the speech data corresponding to the wake word and the lip data of each person are first input into the lip-sound detection model. The lip-sound detection model then extracts features from the input speech data and lip data respectively. It extracts features related to the pronunciation of the wake word (e.g., pronunciation action, pronunciation rhythm, etc.) contained in the speech data, and features representing the lip shape and lip action of the corresponding person in the lip data, thereby obtaining the speech features of the speech data and the lip shape features of the lip data.

[0091] Next, based on the speech features of the speech data obtained from feature extraction and the lip shape features of each person's lip data, a lip sound detection model can be applied to perform lip sound consistency detection to obtain the detection results for each person. Specifically, the lip sound detection model compares the lip movements reflected by the lip shape features of each person before and after wake-up with the lip movements reflected in the pronunciation of the wake word represented by the speech features. By comparing the lip sound consistency, the person whose lip movement amplitude is consistent with the speech amplitude of the wake word is found. This person is the potential interactive person who outputs the wake word, and whether the lip sounds of each person are consistent is the detection result.

[0092] It is worth noting that, in order to ensure the efficiency of lip-sound consistency detection, in this embodiment of the invention, the lip data of each person can be screened before lip-sound consistency detection, that is, the lip data before and after wake-up is selected from all the lip data. The time before and after wake-up can be determined according to the time when the wake word is detected. Preferably, in this embodiment of the invention, the time when the wake word is detected and 2 seconds before and after that time are taken as the time before and after wake-up.

[0093] In addition, before inputting the speech data corresponding to the wake word and the lip data of each person into the lip sound detection model, sample speech data and sample video data can be used to pre-train the lip sound detection model. Unlike traditional methods that use multi-task learning for model training, this embodiment of the invention considers that multi-task learning requires complete sharing of abstract representation information between different modalities. If this condition is not met, the model cannot aggregate matching high-dimensional information representations, leading to training bias and poor detection performance. Therefore, the consistency between the speech amplitude represented by sample speech features and the lip movement amplitude represented by sample lip shape features is used for model training to obtain a trained lip sound detection model.

[0094] Specifically, during model training, firstly, a large amount of sample speech data and sample video data are collected. This sample data must include data with consistent lip sounds and data with different lip sounds. Then, the initial lip sound detection model can be used to extract features from the sample speech data and sample video data to determine the sample speech features of the sample speech data and the sample lip shape features of the sample video data. Subsequently, the initial lip sound detection model can be trained using the feature similarity between the sample speech features of the sample speech data and the sample lip shape features of the sample video data, thereby obtaining the trained lip sound detection model.

[0095] Among them, the sample speech data and sample video data with consistent lip sounds can come from the same audio and video data, which can be separated from the same audio and video data. The sample speech data and sample video data with different lip sounds can come from different time periods of the same audio and video data, or from different audio and video data. Here, it can be that the audio track of a certain audio and video data is stripped and then supplemented with data separated from another audio and video data, thus forming sample speech data and sample video data with different lip sounds.

[0096] Compared to traditional methods that use the error between predicted and labeled values ​​to drive parameter updates, the training method in this embodiment of the invention, which uses sample speech data and sample video data to reflect the consistency of lip movements, does not require the complete sharing of abstract representation information between different modalities. Furthermore, by applying the feature similarity between the sample speech features of the sample speech data and the sample lip shape features of the sample video data to train the initial lip sound detection model, the initial lip sound detection model can fully learn the proximity relationship between the sample speech features of the sample speech data and the sample lip shape features of the sample video data. This provides crucial assistance in improving the accuracy and precision of lip sound consistency detection.

[0097] In this embodiment of the invention, the comparative training based on feature similarity enables the initial lip sound detection model to determine the feature similarity between sample speech features and sample lip shape features based on whether the lip sounds are consistent. When the lip sounds are consistent, that is, when the sample speech data and sample video data can constitute positive sample data, the feature similarity between the sample speech features and sample lip shape features is maximized; conversely, when the lip sounds are different, that is, when the sample speech data and sample video data can constitute negative sample data, the feature similarity between the sample speech features and sample lip shape features is minimized.

[0098] Based on the above embodiments, the lip sound detection model is trained using the following steps:

[0099] Based on the initial lip sound detection model, feature extraction is performed on the sample speech data and sample video data respectively to obtain sample speech features and sample lip shape features;

[0100] From the sample speech data and sample video data, sample speech data and sample video data with consistent lip sounds are selected as positive sample data, and sample speech data and sample video data with different lip sounds are selected as negative sample data.

[0101] Based on the feature similarity between the sample speech features of the sample speech data in the positive sample data and the sample lip shape features of the sample video data, and the feature similarity between the sample speech features of the sample speech data in the negative sample data and the sample lip shape features of the sample video data, the parameters of the initial lip sound detection model are iterated to obtain the lip sound detection model.

[0102] Specifically, the training process of the lip sound detection model may include the following steps:

[0103] First, an initial lip-sound detection model can be used to extract features from sample speech data and sample video data respectively, so as to obtain sample speech features of sample speech data and sample lip shape features of sample video data. That is, the sample speech data and sample video data can be input into the initial lip-sound detection model, which will extract features related to pronunciation (e.g., pronunciation action, pronunciation rhythm, etc.) in the sample speech data and features representing lip shape, lip action, etc. in the sample video data, thereby obtaining sample speech features of sample speech data and sample lip shape features of sample video data.

[0104] Subsequently, referring to the lip-consonance labels between the sample speech data and the sample video data, sample data of the speech modality and sample data of the image modality can be selected from the sample speech data and sample video data to construct positive sample data and negative sample data. Here, the lip-consonance label represents whether there is lip-consonance consistency between the sample speech data and the sample video data. Specifically, sample speech data and sample video data with consistent lip-consonance can be selected from the sample speech data and sample video data as positive sample data, and sample speech data and sample video data with different lip-consonance can be selected as negative sample data.

[0105] Subsequently, the feature similarity between the sample speech features of the sample speech data in the positive sample data and the sample lip shape features of the sample video data, and the feature similarity between the sample speech features of the sample speech data in the negative sample data and the sample lip shape features of the sample video data can be determined. That is, the feature similarity between the sample speech features and the sample lip shape features in the positive sample data, and the feature similarity between the sample speech features and the sample lip shape features in the negative sample data can be calculated. Based on these two, the loss of the initial lip sound detection model in the contrastive training process can be determined, that is, the contrastive loss of the initial lip sound detection model.

[0106] It is worth noting that the feature similarity here can be expressed as cosine similarity, Euclidean distance, Minkowski distance, etc.; and preferably, the feature similarity in the embodiments of the present invention can be the lip movement amplitude similarity between features measured by Euclidean distance.

[0107] The initial training objective of the lip sound detection model is to maximize the feature similarity between the sample speech features and the sample lip shape features of the sample speech data and the sample video data when the lip sounds of the sample speech data and the sample video data are consistent (i.e., when they constitute positive sample data); and conversely, to minimize the feature similarity between the sample speech features and the sample lip shape features of the sample video data when the lip sounds of the sample speech data and the sample video data are different (i.e., when they constitute negative sample data).

[0108] Therefore, when the feature similarity between the sample speech features of the positive sample speech data and the sample lip shape features of the sample video data is high, and the feature similarity between the sample speech features of the negative sample speech data and the sample lip shape features of the sample video data is low, it can be determined that the contrast loss is small. Conversely, when the feature similarity between the sample speech features of the positive sample speech data and the sample lip shape features of the sample video data is low, and / or the feature similarity between the sample speech features of the negative sample speech data and the sample lip shape features of the sample video data is high, it can be determined that the contrast loss is large.

[0109] Then, based on this comparison loss, the parameters of the initial lip sound detection model can be iterated to obtain the lip sound detection model. This process is essentially adjusting the parameters of the initial lip sound detection model so that it can fully learn the mapping relationship between the sample data and its corresponding sample features during the adjustment process. In this way, the speech features and lip shape features corresponding to the speech data and lip data can be output based on this mapping relationship during the application.

[0110] Specifically, during parameter iteration, by comparing the loss and adjusting the model parameters, the adjusted initial lip sound detection model can achieve the highest possible feature similarity between the sample speech features and sample lip shape features corresponding to the output positive sample data when the input sample data belongs to positive sample data. Conversely, when the input sample data belongs to negative sample data, the feature similarity between the sample speech features and sample lip shape features corresponding to the output negative sample data can be minimized. Ultimately, a trained lip sound detection model can be obtained.

[0111] Based on the above embodiments, the formulas for calculating the feature similarity between sample speech features and sample lip shape features, as well as the contrast loss of the initial lip sound detection model, are as follows:

[0112] The feature similarity between sample speech features and sample lip shape features is represented as follows:

[0113]

[0114] In the formula, For sample speech data, For sample video data, This represents the sample speech features of the sample speech data. This represents the lip shape features of the sample video data. This represents the feature similarity between sample speech features and sample lip shape features, measured by Euclidean distance, where ||*|2 represents the L2 norm.

[0115] The formula for calculating the contrast loss of the initial lip sound detection model is as follows:

[0116]

[0117] In the formula, To compare the loss, W represents positive or negative sample data, i represents the i-th positive or negative sample data, and Y is the lip-consonance label, which is 0 or 1, where 0 indicates lip-consonance agreement and 1 indicates lip-consonance disagreement. This represents the feature similarity between sample speech features and sample lip shape features corresponding to all positive sample data measured by Euclidean distance, and the feature similarity between sample speech features and sample lip shape features corresponding to all negative sample data, where m is a constant.

[0118] Based on the above embodiments, Figure 2 This is one of the flowcharts illustrating the multimodal interaction process provided by the present invention, such as... Figure 2 As shown, in step 140, based on the identity tags of potential interacting personnel, the interaction rights of the initial interacting personnel are switched, and the process further includes:

[0119] Step 210: Based on the video data, determine the image interaction intent of potential interactors, including at least one of gesture interaction intent, posture interaction intent, and action interaction intent; and / or,

[0120] Step 220: Based on the interactive voice in the interactive scenario, determine the voice interaction intent of potential interactive personnel;

[0121] Step 230: Perform the interaction based on the image interaction intent and / or voice interaction intent.

[0122] Specifically, by using the identity tags of potential interactors, after switching the interaction rights of the initial interactor, multimodal interaction can be conducted with potential interactors. The specific process includes:

[0123] Step 210: First, the interaction intention of potential interactors can be determined by the video data in the interaction scene. The interaction intention can be one or more of the following: gesture interaction intention, action interaction intention, posture interaction intention. All such intentions can be represented by each frame of the video data. That is, gesture recognition, posture detection, action recognition, etc. can be performed on the human body area of ​​potential interactors in each frame of the image, and the image interaction intention of potential interactors can be determined based on the detection / recognition results.

[0124] Step 220: At the same time, the interactive intention of potential interactors can be determined by using the interactive voice in the interactive scenario. Specifically, the lip data of potential interactors can be determined first, and then the interactive voice in the interactive scenario can be recognized using this lip data. That is, the voice of potential interactors can be recognized only to obtain the recognized text. The semantic understanding of the recognized text can be performed to analyze the interactive intention, thereby obtaining the voice interaction intention.

[0125] Step 230: Then, interaction can be performed based on the image interaction intent, or based on the voice interaction intent, or by combining the voice interaction intent and the image interaction intent, so as to clearly define the interaction intent of the potential interaction user and thus interact based on this interaction intent, that is, responding to the interaction intent of the potential interaction user during the interaction process.

[0126] In this embodiment of the invention, by combining data from multiple levels to clarify the interaction intent of potential users, a better understanding of interaction intent is achieved, realizing a comprehensive understanding of intent and contributing to the improvement of the interaction experience in the human-computer interaction process.

[0127] Based on the above embodiments, Figure 3 This is a flowchart illustrating step 220 in the multimodal interaction process provided by the present invention, as shown below. Figure 3 As shown, step 220 includes:

[0128] Step 221: Based on the lip data of potential interactors, perform speech separation and / or speech endpoint detection on the interactive speech to obtain the effective speech of potential interactors;

[0129] Step 222: Based on the lip data of the potential interacting person, perform speech recognition on the valid speech, and perform intent recognition on the recognized text obtained from the speech recognition to obtain the speech interaction intent of the potential interacting person.

[0130] Specifically, step 220, the process of determining the voice interaction intent of a potential user by means of interactive voice, may include the following steps:

[0131] Step 221: First, the lip data of the potential interacting personnel can be determined from the lip data of each person by using the identity number of the potential interacting personnel. That is, the lip data corresponding to the identity number can be found from the lip data of each person by using the identity number of the potential interacting personnel as an index. This lip data is the lip data of the potential interacting personnel.

[0132] Then, based on the lip data, combined with the interactive speech, multimodal data can be used to perform speech separation tasks. That is, the lip data of potential interactors can be referenced to perform speech separation on the interactive speech in the interactive scenario to obtain the effective speech of potential interactors. The effective speech here is actually the separated speech of potential interactors. Specifically, additional lip information, such as lip shape and lip movements, can be provided by using lip data to assist in speech separation, so as to separate the speech of potential interactors from the interactive speech in the interactive scenario, thereby obtaining the separated speech of potential interactors. This avoids noise interference in complex scenarios and ensures the separation effect.

[0133] Here, the lip information reflected by the lip data of potential interactors is used as an aid, which can effectively solve the dependence on direction and angle in traditional speech separation technology, overcome the defects of incomplete separation and incomplete stripping, and can completely separate the speech of different speakers in the same direction. That is, it can separate the speech of potential interactors and speakers in the same direction and same area, and obtain the clean speech of potential interactors, thus achieving a good noise reduction purpose.

[0134] Multimodal data can also be used for speech endpoint detection. That is, multimodal data can be applied to multimodal VAD (Voice Activity Detection) tasks. Specifically, this can be done by using the lip data of potential interactors to perform speech endpoint detection on the interactive speech of the interactors, thereby obtaining the effective speech of the potential interactors. In fact, the lip data of potential interactors is used to assist in speech segmentation for the separation of speech, so as to cut out the speech segments of potential interactors from the interactive speech. That is, by using speech endpoint detection, the beginning and end endpoints of the effective speech segments of potential interactors that may be contained in the interactive speech are determined, and the effective speech is output for subsequent speech recognition.

[0135] Here, the lip information reflected by the lip data of potential users is used to assist in speech segmentation. Instead of cutting out all valid human voice segments as valid speech segments, only the valid speech segments of potential users are cut out, avoiding interference from other voices. This results in valid speech containing only the speech segments of potential users, solving the problem that traditional single-model speech VAD can only distinguish between human and non-human voices and cannot reject the speech of non-target users.

[0136] Furthermore, a speech endpoint detection task can be performed on the basis of the separation task. That is, the speech endpoint detection can be performed on the separated speech of the potential interacting person using the lip data of the potential interacting person, so as to obtain the effective speech of the potential interacting person. Specifically, the speech segment of the potential interacting person is cut out from the separated speech using the lip data of the potential interacting person. That is, the beginning and end endpoints of the effective speech segment of the potential interacting person that may be contained in the separated speech are determined by the speech endpoint detection, so as to output the effective speech for subsequent speech recognition.

[0137] Step 222: Then, the lip data of the potential interacting person can be used to perform speech recognition on their effective speech to obtain the recognized text. Specifically, the lip data of the potential interacting person can be used as a reference to perform speech recognition on the interactive speech in order to identify the speech of the potential interacting person from the mixed / overlapping speech in the complex scene, avoid noise interference, ensure recognition accuracy, and finally obtain the recognized text of the potential interacting person.

[0138] Here, multimodal speech recognition for potential interacting persons essentially utilizes lip information represented by the lip data of potential interacting persons, such as lip shape and lip movements, to assist in speech recognition tasks. In open multi-person scenarios, it can identify the speech of potential interacting persons from mixed / overlapping speech, that is, it only recognizes the speech of potential interacting persons, avoiding interference from environmental noise and other human noise, and ensuring recognition accuracy.

[0139] After that, intent recognition can be performed on the identified text of potential users to determine their voice interaction intent, so that interaction can be carried out based on the voice interaction intent. Specifically, semantic understanding can be performed on the identified text first to parse the voice interaction intent of potential users and the interaction slot information in the interaction process, such as extracting specific navigation route stops, means of transportation, navigation destination, etc., in preparation for response.

[0140] Based on the above embodiments, Figure 4 This is the second flowchart illustrating the multimodal interaction process provided by the present invention, as shown below. Figure 4 As shown, in step 140, based on the identity tags of potential interacting personnel, the interaction rights of the initial interacting personnel are switched, and the process further includes:

[0141] Step 410: Based on the interactive voice in the interactive scenario, determine the voice interaction intent of the target interactive personnel. The target interactive personnel are a preset number of interactive personnel who are in the interactive scenario and who are ahead of the potential interactive personnel.

[0142] Step 420: Based on the video data, determine the facial data of the target interaction person;

[0143] Step 430: Based on the facial data and voice interaction intent of the target user, conduct the interaction.

[0144] Specifically, in step 140, after switching the interaction rights of the initial interaction person using the identity tags of potential interaction persons, in addition to multimodal interaction with potential interaction persons, it is also possible to continuously track several interaction persons before the potential interaction person and interact with them based on their interaction intentions and facial data. The specific process includes the following steps:

[0145] Step 410: First, it is necessary to determine the target interaction personnel to be tracked. Here, the target interaction personnel are the preset number of interaction personnel who are in the interaction scenario and are the potential interaction personnel. The preset number can be determined according to the specific scenario, device computing power, actual needs, etc., and can be 2, 3, 5, etc. Then, the lip data of the target interaction personnel can be determined from the lip data of each person by using the identity number. Based on this lip data, the voice interaction intention of the target interaction personnel can be determined through the interactive voice. The process of determining the voice interaction intention is basically the same as the process of determining the voice interaction intention of the potential interaction personnel described above, and will not be repeated here.

[0146] Step 420: Then, the facial data of the target interactive person can be determined through video data. Specifically, it can be done by performing face detection on the human body area of ​​the target interactive person in each frame of the image to determine whether the face of the target interactive person can be detected from the image at the corresponding time, as well as the orientation information, position information, etc. of the face.

[0147] Step 430: After that, the interaction can be carried out by combining the voice interaction intent of the target interaction person and the facial data. Specifically, when the face of the target interaction person is detected and the orientation information and / or location information indicate that the target interaction person is facing the interaction device (or the device interaction screen), the voice interaction intent is responded to. That is, when the target interaction person is facing the interaction device, the voice interaction intent is responded to.

[0148] Correspondingly, if the face of the target interacting person is not detected, or if the face of the target interacting person is detected but the orientation information and / or location information indicate that the target interacting person is not facing the interactive device (or the device's interactive screen), their voice interaction intent is ignored, that is, if the target interacting person is not facing the interactive device, their voice interaction intent is not responded to.

[0149] Based on the above embodiments, Figure 5 This is a flowchart illustrating the targeted tracking process provided by the present invention, as shown below. Figure 5 As shown, based on the identity tags of potential interacting personnel, the interaction rights of the initial interacting personnel are switched, and the process also includes:

[0150] Step 510: Obtain the sound source localization results and visual localization results for potential interacting personnel;

[0151] Step 520: Based on the sound source localization results and the visual localization results, perform targeted tracking of potential interaction personnel;

[0152] The sound source localization result is obtained by locating the sound source of the person interacting with the microphone array in the interaction scenario, and the visual localization result is obtained by locating the potential person interacting with the camera in the interaction scenario.

[0153] Specifically, by using the identification tags of potential interactors, after switching the interaction rights of the initial interactor, it is possible to perform targeted tracking of potential interactors. The specific process includes:

[0154] To avoid the problem of low accuracy in sound source localization based on single-modal data in traditional solutions, which can only locate a general range, this embodiment of the invention, after switching interaction rights, can perform targeted tracking of potential interaction personnel from both sound source and visual perspectives to ensure the stability and accuracy of the interaction process. This reduces interference while ensuring the stability and accuracy of the tracking process.

[0155] Step 510: First, it is necessary to determine the sound source localization result and the visual localization result for the potential interaction personnel. The sound source localization result is obtained by using a microphone array in the interaction scene, while the visual localization result is determined by using a camera in the interaction scene. That is, the potential interaction personnel can be located by sound source using a microphone array to obtain their sound source localization result, and the potential interaction personnel can be located by visual localization using a camera to obtain their visual localization result.

[0156] Step 520: Then, by combining the sound source localization results and the visual localization results, the location information of potential interacting personnel can be determined, and based on this location information, targeted tracking of potential interacting personnel can be carried out. Here, targeted tracking of potential interacting personnel can ensure the stability of the tracking process and prevent tracking loss. Moreover, unlike other tracking methods, targeted tracking distinguishes the tracking process by prioritizing key points, making the priorities clearer, the execution stronger, and the execution effect better.

[0157] It is worth noting that while targeted tracking is performed on potential interaction personnel, the tracking of other personnel in the interaction scenario is not ignored. This ensures the tracking of each person in the scenario. However, unlike the fixed tracking of potential interaction personnel, other personnel are tracked through a tracking method that combines body tracking and face detection.

[0158] Based on the above embodiments, Figure 6 This is a flowchart illustrating step 120 of the interaction right switching method provided by the present invention, as follows: Figure 6 As shown, step 120 includes:

[0159] Step 121: Perform human body tracking based on each frame of the video data to obtain the human body regions of each person in each frame of the interactive scene.

[0160] Step 122: Perform face detection on each human body region to obtain the face region of each person in each frame image, and perform key point detection on each face region to obtain the lip data of each person in each frame image.

[0161] Considering that traditional solutions are prone to missed detections, false detections, or even tracking loss due to occasional irregular head turns or rotations, this invention addresses this issue by taking into account that while faces may temporarily disappear due to head turns or rotations, the human body does not vanish into thin air. Therefore, during tracking, not only face tracking is considered, but also the tracking of the human body of each individual. Furthermore, human body tracking and face detection are linked, which can improve tracking efficiency while reducing interference from face detection and achieving stable tracking of each individual in interactive scenarios.

[0162] Specifically, step 120, the process of obtaining the lip data of each person in the interactive scene based on the video data, may include the following steps:

[0163] Step 121: First, human body tracking can be performed based on each frame of the video data to obtain the human body region of each person in each frame. Specifically, the human body tracking of each person in each frame can be achieved by using the area of ​​the human body region in each frame. That is, the consistency of the corresponding person in the human body region in adjacent frame images can be evaluated by the overlap of the area of ​​the human body region in adjacent frame images. In short, the person corresponding to the two human body regions with the highest overlap of area in adjacent frame images is determined to be the same person, or the person corresponding to two human body regions with an overlap of area greater than the area threshold in adjacent frame images is regarded as the same person. In this way, the human body tracking of each person is achieved, and the human body region of each person in each frame image is obtained.

[0164] Here, the area threshold is a preset value used to determine whether two human body areas belong to the same person at the area level. The specific value can be set according to the actual situation and actual needs, for example, it can be 80%, 85%, 90%, etc.

[0165] Step 122: Next, face detection can be performed on each human body region to obtain the face region of each person in each frame image. Key point detection can be performed on each face region to obtain the lip data of each person in each frame image. Specifically, key point detection technology can be used to detect the key point contours of the corresponding facial features of each person from each face region, so that the lip data of each person can be extracted from the key point contours for subsequent interaction right switching in multi-person scenarios and multimodal interaction tasks.

[0166] In this embodiment of the invention, for human body tracking and face detection of various people in interactive scenarios, stable tracking of different people in multiple scenarios can be achieved. This solves the problem of difficulty in tracking people in open multi-person scenarios due to large changes in position and occasional irregular head turning or turning. It can accurately track each person, and can track back even if the face is briefly lost, and can maintain a fixed face ID.

[0167] Based on the above embodiments, Figure 7 This is a flowchart illustrating step 121 of the interaction right switching method provided by the present invention, as follows: Figure 7 As shown, step 121 includes:

[0168] Step 121-1: Perform human detection based on each frame of the video data to obtain the human body region in each frame of the video data;

[0169] Step 121-2: Based on the overlapping area of ​​each human body region in adjacent frame images, perform human body tracking on each person corresponding to each human body region to obtain the human body region of each person in each frame image in the interactive scene.

[0170] Specifically, step 121, which involves human body tracking based on each frame of the video data to obtain the human body region of each person in each frame, includes the following steps:

[0171] Step 121-1: First, human detection can be performed based on each frame of the video data to obtain the human body region in each frame. Specifically, it can be to detect and identify possible human body regions in each frame of the video data to obtain all possible human body regions in each frame and the confidence level of each human body region. The confidence level indicates the probability that the corresponding region belongs to a real human body region. The higher the confidence level, the more likely it is to belong to a real human body region, and vice versa. Human body detection here can be implemented by conventional human body detection algorithms, human body detection models, etc. This embodiment of the invention does not make specific limitations on this.

[0172] Step 122-2: Then, the overlap of the area of ​​each human body region in adjacent frame images can be calculated, that is, the overlap area of ​​each human body region. Based on this overlap area, it can be used to evaluate whether the person corresponding to the corresponding human body region is the same person, thereby realizing human body tracking for each person. That is, the people corresponding to the two human body regions with the largest overlap area in adjacent frame images can be regarded as the same person, or the people corresponding to two human body regions with an overlap area greater than the area threshold in adjacent frame images can be regarded as the same person. Finally, the human body region of each person in each frame image can be obtained.

[0173] The overall flow of the interaction right switching method provided by this invention includes the following steps:

[0174] First, determine the video data in the interaction scenario, and the initial interaction personnel indicated by the video data;

[0175] Subsequently, based on the video data, the lip data of each person in the interactive scene is determined. Specifically, this can be done by performing human body tracking based on each frame of the video data to obtain the human body region of each person in each frame of the interactive scene; performing face detection on each human body region to obtain the face region of each person in each frame of the image; and performing key point detection on each face region to obtain the lip data of each person in each frame of the image.

[0176] Specifically, the method involves human body tracking based on each frame of video data to obtain the human body regions of each person in each frame of the interactive scene. This includes: human body detection based on each frame of video data to obtain the human body regions in each frame of the image; and human body tracking based on the overlapping area of ​​each human body region in adjacent frame images to obtain the human body regions of each person in each frame of the interactive scene.

[0177] Subsequently, upon detecting the wake word, the potential interaction personnel for outputting the wake word are determined based on the speech data corresponding to the wake word and the lip data of each person.

[0178] Subsequently, when the potential interactors and the initial interactors are different individuals, the interaction rights of the initial interactors are switched based on the identity labels of the potential interactors.

[0179] Furthermore, based on the identity identifier of the potential interacting personnel, the interaction rights of the initial interacting personnel are switched. This process also includes: determining the image interaction intent of the potential interacting personnel based on video data, where the image interaction intent includes at least one of gesture interaction intent, posture interaction intent, and action interaction intent; and / or, determining the voice interaction intent of the potential interacting personnel based on the interactive voice in the interaction scenario; and performing the interaction based on the image interaction intent and / or voice interaction intent.

[0180] Among them, determining the voice interaction intent of potential interactors based on interactive voice in interactive scenarios includes: performing speech separation and / or speech endpoint detection on the interactive voice based on the lip data of potential interactors to obtain the effective voice of potential interactors; performing speech recognition on the effective voice based on the lip data of potential interactors, and performing intent recognition on the recognized text obtained from speech recognition to obtain the voice interaction intent of potential interactors.

[0181] Furthermore, based on the identity tags of potential interactors, the interaction rights of the initial interactors are switched. This process also includes: determining the voice interaction intent of the target interactors based on the interactive voice in the interaction scenario, where the target interactors are a preset number of interactors who are in the interaction scenario and precede the potential interactors; determining the facial data of the target interactors based on video data; and conducting the interaction based on the facial data and voice interaction intent of the target interactors.

[0182] Furthermore, based on the identity labels of potential interactors, the interaction rights of the initial interactors are switched. This process also includes: obtaining the sound source localization results and visual localization results of the potential interactors; and performing directional tracking of the potential interactors based on the sound source localization results and visual localization results. The sound source localization results are obtained by locating the potential interactors based on the microphone array in the interaction scene, and the visual localization results are obtained by locating the potential interactors based on the camera in the interaction scene.

[0183] The method provided in this invention uses video data from an interactive scenario to perform human body tracking and face detection on each person in the scenario. It also combines the lip data of each person with the speech data corresponding to the wake word to perform lip-sound consistency detection and obtain the detection results. Based on the detection results, the potential interactive person who outputs the wake word can be accurately located. By identifying the potential interactive person, the interaction rights of the initial interactive person can be switched. This realizes the switching of interaction rights in multi-person scenarios, overcoming the defect of traditional solutions where it is difficult to switch interaction rights when multiple people interact. It can stably track each person in a multi-person scenario, achieve accurate positioning of potential interactive persons, and steadily switch interaction rights.

[0184] The interaction right switching device provided by the present invention is described below. The interaction right switching device described below and the interaction right switching method described above can be referred to in correspondence.

[0185] Figure 8 This is a schematic diagram of the interactive right switching device provided by the present invention, as shown below. Figure 8 As shown, the device includes:

[0186] The determining unit 810 is used to determine the video data in the interactive scenario and the initial interactive personnel indicated by the video data;

[0187] The face detection unit 820 is used to determine the lip data of each person in the interaction scene based on the video data;

[0188] The lip sound detection unit 830 is used to determine the potential interactive person who outputs the wake word based on the speech data corresponding to the wake word and the lip data of each person when a wake word is detected.

[0189] The interaction right switching unit 840 is used to switch the interaction right of the initial interaction person based on the identity number of the potential interaction person when the potential interaction person and the initial interaction person are different people.

[0190] The interaction right switching device provided by this invention performs human body tracking and face detection on each person in the interaction scenario using video data. It also performs lip-phonetic consistency detection by combining the lip data of each person and the voice data corresponding to the wake word. The detection results can accurately locate the potential interaction person who outputs the wake word. By identifying the potential interaction person, the interaction right is switched to the initial interaction person. This realizes the switching of interaction right in multi-person scenarios, overcoming the defect of traditional solutions where it is difficult to switch interaction right in multi-person interaction. It can stably track each person in multi-person scenarios, achieve accurate positioning of potential interaction persons, and steadily switch interaction right.

[0191] Based on the above embodiments, the device further includes a multimodal interaction unit, used for:

[0192] Based on the video data, the image interaction intention of the potential interacting person is determined, and the image interaction intention includes at least one of gesture interaction intention, posture interaction intention, and action interaction intention;

[0193] And / or, based on the interactive voice in the interactive scenario, determine the voice interaction intent of the potential interactive personnel;

[0194] Interaction is performed based on the image interaction intent and / or the voice interaction intent.

[0195] Based on the above embodiments, the multimodal interaction unit is used for:

[0196] Based on the lip data of the potential interacting person, speech separation and / or speech endpoint detection are performed on the interactive speech to obtain the effective speech of the potential interacting person.

[0197] Based on the lip data of the potential interacting person, speech recognition is performed on the valid speech, and intent recognition is performed on the recognized text obtained from the speech recognition to obtain the speech interaction intent of the potential interacting person.

[0198] Based on the above embodiments, the multimodal interaction unit is used for:

[0199] Based on the interactive voice in the interactive scenario, determine the voice interaction intention of the target interactive person. The target interactive person is a preset number of interactive persons who are in the interactive scenario and who are before the potential interactive persons.

[0200] Based on the video data, the facial data of the target interacting person is determined;

[0201] The interaction is conducted based on the facial data and voice interaction intent of the target user.

[0202] Based on the above embodiments, the device further includes a directional tracking unit, used for:

[0203] Obtain the sound source localization results and visual localization results of the potential interacting persons;

[0204] Based on the sound source localization results and the visual localization results, the potential interacting personnel are targeted and tracked.

[0205] The sound source localization result is obtained by locating the sound source of the potential interactive person using a microphone array in the interactive scene, and the visual localization result is obtained by locating the visual person of the potential interactive person using a camera in the interactive scene.

[0206] Based on the above embodiments, the face detection unit 820 is used for:

[0207] Human body tracking is performed based on each frame of the video data to obtain the human body regions of each person in the interactive scene in each frame of the video data.

[0208] Face detection is performed on each human body region to obtain the face region of each person in each frame image, and key point detection is performed on each face region to obtain the lip data of each person in each frame image.

[0209] Based on the above embodiments, the face detection unit 820 is used for:

[0210] Human detection is performed on each frame of the video data to obtain the human body region in each frame of the video data.

[0211] Based on the overlapping area of ​​each human body region in adjacent frame images, human body tracking is performed on each person corresponding to each human body region to obtain the human body region of each person in each frame image in the interactive scene.

[0212] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9As shown, the electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, communications interface 920, and memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute an interaction right switching method. This method includes: determining video data in the interaction scenario and the initial interaction person indicated by the video data; determining the lip data of each person in the interaction scenario based on the video data; determining the potential interaction person who outputs the wake-up word based on the voice data corresponding to the wake-up word and the lip data of each person when a wake-up word is detected; and switching the interaction right of the initial interaction person based on the identity number of the potential interaction person when the potential interaction person and the initial interaction person are different.

[0213] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0214] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the interaction right switching method provided by the above methods, the method comprising: determining video data in an interaction scenario and an initial interaction person indicated by the video data; determining lip data of each person in the interaction scenario based on the video data; determining a potential interaction person who outputs the wake word based on the voice data corresponding to the wake word and the lip data of each person when a wake word is detected; and switching the interaction right of the initial interaction person based on the identity number of the potential interaction person when the potential interaction person and the initial interaction person are different persons.

[0215] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the interaction right switching method provided by the above methods. The method includes: determining video data in an interaction scenario and an initial interaction person indicated by the video data; determining lip data of each person in the interaction scenario based on the video data; determining a potential interaction person who outputs the wake word based on the speech data corresponding to the wake word and the lip data of each person when a wake word is detected; and switching the interaction right of the initial interaction person based on the identity identifier of the potential interaction person when the potential interaction person and the initial interaction person are different persons.

[0216] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0217] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for switching interaction rights, characterized in that, include: Determine the video data in the interaction scenario, and the initial interaction personnel indicated by the video data; Based on the video data, determine the lip data of each person in the interaction scenario; Upon detecting a wake word, lip-phone consistency detection is performed based on the speech data corresponding to the wake word and the lip data of each person to obtain a detection result. Based on the detection result, the potential interaction person who outputs the wake word is determined. The lip-phone consistency detection compares the lip movements reflected in the lip data of each person before and after wake-up with the lip movements corresponding to the pronunciation of the wake word in the speech data to verify whether the lip movements of each person correspond to the pronunciation rhythm of the wake word. If the potential interacting person and the initial interacting person are different people, the interaction rights of the initial interacting person are switched based on the identity number of the potential interacting person.

2. The interaction right switching method according to claim 1, characterized in that, The process of switching interaction rights for the initial interaction user based on the identity identifier of the potential interaction user further includes: Based on the video data, the image interaction intention of the potential interacting person is determined, and the image interaction intention includes at least one of gesture interaction intention, posture interaction intention, and action interaction intention; And / or, based on the interactive voice in the interactive scenario, determine the voice interaction intent of the potential interactive personnel; Interaction is performed based on the image interaction intent and / or the voice interaction intent.

3. The interaction right switching method according to claim 2, characterized in that, Determining the voice interaction intent of the potential user based on the interactive voice in the interactive scenario includes: Based on the lip data of the potential interacting person, speech separation and / or speech endpoint detection are performed on the interactive speech to obtain the effective speech of the potential interacting person. Based on the lip data of the potential interacting person, speech recognition is performed on the valid speech, and intent recognition is performed on the recognized text obtained from the speech recognition to obtain the speech interaction intent of the potential interacting person.

4. The interaction right switching method according to any one of claims 1 to 3, characterized in that, The process of switching interaction rights for the initial interaction user based on the identity identifier of the potential interaction user further includes: Based on the interactive voice in the interactive scenario, determine the voice interaction intention of the target interactive person. The target interactive person is a preset number of interactive persons who are in the interactive scenario and who are before the potential interactive persons. Based on the video data, the facial data of the target interacting person is determined; The interaction is conducted based on the facial data and voice interaction intent of the target user.

5. The interaction right switching method according to any one of claims 1 to 3, characterized in that, The process of switching interaction rights for the initial interaction user based on the identity identifier of the potential interaction user further includes: Obtain the sound source localization results and visual localization results of the potential interacting persons; Based on the sound source localization results and the visual localization results, the potential interacting personnel are targeted and tracked. The sound source localization result is obtained by locating the sound source of the potential interactive person using a microphone array in the interactive scene, and the visual localization result is obtained by locating the visual person of the potential interactive person using a camera in the interactive scene.

6. The method for switching interaction rights according to any one of claims 1 to 3, characterized in that, The step of determining the lip data of each person in the interaction scenario based on the video data includes: Human body tracking is performed based on each frame of the video data to obtain the human body regions of each person in the interactive scene in each frame of the video data. Face detection is performed on each human body region to obtain the face region of each person in each frame image, and key point detection is performed on each face region to obtain the lip data of each person in each frame image.

7. The interaction right switching method according to claim 6, characterized in that, The human body tracking based on each frame of the video data to obtain the human body regions of each person in the interactive scene in each frame of the video data includes: Human detection is performed on each frame of the video data to obtain the human body region in each frame of the video data. Based on the overlapping area of ​​each human body region in adjacent frame images, human body tracking is performed on each person corresponding to each human body region to obtain the human body region of each person in each frame image in the interactive scene.

8. An interaction right switching device, characterized in that, include: A determining unit is used to determine the video data in the interaction scenario and the initial interaction personnel indicated by the video data; A face detection unit is used to determine the lip data of each person in the interaction scenario based on the video data. The lip-sound detection unit is used to perform lip-sound consistency detection based on the speech data corresponding to the wake-up word and the lip data of each person when a wake-up word is detected, obtain the detection result, and determine the potential interaction person who outputs the wake-up word based on the detection result. The lip-sound consistency detection compares the lip movements reflected by the lip data of each person before and after wake-up with the lip movements corresponding to the pronunciation of the wake-up word in the speech data to verify whether the lip movements of each person correspond to the pronunciation rhythm of the wake-up word. The interaction right switching unit is used to switch the interaction right of the initial interaction person based on the identity number of the potential interaction person when the potential interaction person and the initial interaction person are different people.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the interaction right switching method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the interaction right switching method as described in any one of claims 1 to 7.

Images