Multi-mode voice wake-up method and device and electronic equipment

Abstract

The embodiment of the invention provides a multi-mode voice wake-up method and device and electronic equipment. The method is applied to the technical field of intelligent voice, and comprises the steps of providing a basis for subsequent accurate processing by acquiring voice, gesture and mouth shape multi-mode input data and adding a timestamp, avoiding a high misjudgment rate of a single voice mode in a complex environment, greatly improving wake-up accuracy, and when a mode conflict exists, improving wake-up accuracy. By identifying the confidence of each modal input data and combining the historical data to make a decision, the possibility of wrong wake-up operation caused by a single modal identification error is reduced, the reliability of wake-up decision is improved, if modal conflicts do not exist, the wake-up scene of the target equipment is accurately determined according to the multi-modal input data, and the wake-up efficiency of the target equipment is improved. Through accurate identification and adaptive processing of different wake-up scenes, the target device can be stably and accurately waken up in various complex environments, and the environmental adaptability of the target device is effectively enhanced.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent voice technology, and in particular to a multimodal voice wake-up method, device, and electronic device. Background Technology

[0002] With the deep penetration of internet technology and the rapid iteration of artificial intelligence algorithms, intelligent voice products have been widely integrated into all aspects of daily life, from smart speakers and wearable devices to in-vehicle voice assistants and tourist attraction navigation terminals. Their application scenarios continue to expand, and users are placing higher demands on the smoothness of product interaction and the accuracy of response. Among these, wake-up success rate, as a key indicator for measuring the core performance of intelligent voice products, directly affects user experience and product competitiveness, making its optimization and upgrading increasingly important.

[0003] However, current mainstream voice wake-up technologies still suffer from several pain points that urgently need to be addressed: 1. Limited recognition methods: Most products rely on a single voice modality for wake-up judgment, triggering a response solely through voiceprint feature matching or fixed wake-up word recognition. 2. Inaccurate recognition when there is severe environmental noise interference. 3. In multimodal input, voice and image are not synchronized, leading to inaccurate recognition. These problems not only reduce the practicality of intelligent voice products but also restrict their large-scale application in fields with high requirements for interactive reliability, such as mobile internet tourism and industrial IoT. Summary of the Invention

[0004] This disclosure provides a multimodal voice wake-up method, apparatus, and electronic device, which solves the technical problem of inaccurate voice wake-up in existing technologies.

[0005] According to a first aspect of this disclosure, a multimodal voice wake-up method is provided. The method includes: acquiring multimodal input data, the multimodal input data including voice data, gesture data, and lip-reading data, the multimodal input data being timestamped; performing multimodal fusion analysis on the multimodal input data to determine whether a modal conflict scenario exists; if a modal conflict exists, identifying the confidence level of the multimodal input data, and performing a wake-up operation on a target device based on the confidence level identification result and historical data of the multimodal input data; if no modal conflict exists, determining the wake-up scenario of the target device based on the multimodal input data, and performing a corresponding wake-up operation on the target device according to different wake-up scenarios.

[0006] In addition to the aspects described above and any possible implementation, a further implementation is provided, which determines the wake-up scenario of the target device based on the multimodal input data, including: if the speech confidence of the speech data is less than a first preset confidence threshold, then identifying the noise decibel value of the current environment and calculating the time offset between each modal input data; if the noise decibel value is less than or equal to a preset decibel threshold, and all time offsets are less than or equal to a preset time threshold, then determining the current scenario as a first wake-up scenario; if the noise decibel value is less than or equal to the preset decibel threshold, and any of the time offsets is greater than the preset time threshold, then determining the current scenario as a second wake-up scenario; if the noise decibel value is greater than the preset decibel threshold, and all time offsets are less than or equal to the preset time threshold, then determining the current scenario as a third wake-up scenario; if the noise decibel value is greater than the preset decibel threshold, and any of the time offsets is greater than the preset time threshold, then determining the current scenario as a fourth wake-up scenario.

[0007] As one possible implementation, calculating the time offset between each modal input data includes: extracting the time series of the speech data, denoted as a first time series; extracting the time series of the gesture data, denoted as a second time series; and extracting the time series of the lip-sync data, denoted as a third time series; using the first time series as a reference series, finding the optimal matching path of the first time series through a dynamic time warping algorithm to obtain a first matching path; finding the optimal matching path of the second time series through a dynamic time warping algorithm to obtain a second matching path; finding the optimal matching path of the third time series through a dynamic time warping algorithm to obtain a third matching path; calculating the time offset between the first matching path and the second matching path to obtain a first time offset; and calculating the time offset between the first matching path and the third matching path to obtain a second time offset.

[0008] As one possible implementation, in the first wake-up scenario, the target device is given a corresponding wake-up operation according to different wake-up scenarios, including: recognizing the gesture data to obtain a first gesture confidence score, and recognizing the lip shape data to obtain a first lip reading confidence score; weighting the speech confidence score, the first gesture confidence score, and the first lip reading confidence score according to a pre-set first weight allocation strategy to obtain a total confidence score; if the total confidence score is greater than a second pre-set confidence threshold, the target device is woken up.

[0009] As one possible implementation, in the second wake-up scenario, the target device is woken up according to different wake-up scenarios, including: if any time offset is greater than a preset time threshold, the time offset greater than the preset time threshold is recorded as an abnormal time offset; a penalty coefficient for the corresponding modal input data is obtained based on the abnormal time offset, and recorded as a first penalty coefficient; the confidence of the modal input data is corrected by the first penalty coefficient to obtain a first corrected confidence of the modal input data; the voice confidence and the first corrected confidence of the modal input data are weighted according to a preset first weight allocation strategy to obtain a total confidence; if the total confidence is greater than a second preset confidence threshold, the target device is woken up.

[0010] As one possible implementation, in the third wake-up scenario, the target device is woken up according to different wake-up scenarios, including: recognizing the gesture data to obtain a second gesture confidence score, and recognizing the lip shape data to obtain a second lip reading confidence score; adjusting the first weight allocation strategy according to the noise decibel value to obtain a second weight allocation strategy; weighting the speech confidence score of the speech data, the gesture confidence score of the second gesture data, and the lip reading confidence score of the second lip shape data according to the second weight allocation strategy to obtain a total confidence score; if the total confidence score is greater than a second preset confidence threshold, the target device is woken up.

[0011] As one possible implementation, in the fourth wake-up scenario, the target device is woken up according to different wake-up scenarios, including: if any time offset is greater than a preset time threshold, the time offset greater than the preset time threshold is recorded as an abnormal time offset; a penalty coefficient for the corresponding modal input data is obtained based on the abnormal time offset, and recorded as a second penalty coefficient; the confidence of the modal input data is corrected by the second penalty coefficient to obtain a second corrected modal input data confidence; the voice confidence and the second corrected modal input data confidence are weighted according to the second weight allocation strategy to obtain a total confidence; if the total confidence is greater than a second preset confidence threshold, the target device is woken up.

[0012] As one possible implementation, identifying the confidence level of the multimodal input data and performing a wake-up operation on the target device based on the confidence level identification result and historical data of the multimodal input data includes: identifying the multimodal input data to obtain the current speech confidence level, the current gesture confidence level, and the current lip-reading confidence level; searching for the corresponding historical speech accuracy rate, historical gesture accuracy rate, and historical lip-reading accuracy rate in the historical data for the current scenario; obtaining the final speech probability based on the current speech confidence level and the historical speech accuracy rate; obtaining the final gesture probability based on the current gesture confidence level and the historical gesture accuracy rate; obtaining the final lip-reading probability based on the current lip-reading confidence level and the historical lip-reading accuracy rate; determining the modal input data corresponding to the maximum final probability as the decision result, and performing a wake-up operation on the target device based on the decision result.

[0013] According to a second aspect of this disclosure, a multimodal voice wake-up device is provided. The device includes: an acquisition module for acquiring multimodal input data, the multimodal input data including voice data, gesture data, and lip-sync data, the multimodal input data being timestamped; a conflict determination module for performing multimodal fusion analysis on the multimodal input data to determine whether a modal conflict scenario exists; a first wake-up module for, if a modal conflict exists, identifying the confidence level of the multimodal input data, and performing a wake-up operation on a target device based on the confidence level identification result and historical data of the multimodal input data; and a second wake-up module for, if no modal conflict exists, determining the wake-up scenario of the target device based on the multimodal input data, and performing a corresponding wake-up operation on the target device according to different wake-up scenarios.

[0014] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0015] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods according to the first and / or second aspects of this disclosure.

[0016] The beneficial effects of this disclosure are as follows: By acquiring multimodal input data of voice, gestures, and lip movements and adding timestamps, a foundation is provided for subsequent accurate processing, avoiding the high misjudgment rate of a single voice modality in complex environments, and significantly improving wake-up accuracy. When modal conflicts exist, by identifying the confidence level of each modal input data and combining it with historical data for decision-making, the possibility of erroneous wake-up operations due to single modal recognition errors is reduced, and the reliability of wake-up decisions is improved. If there are no modal conflicts, the wake-up scenario of the target device is accurately determined based on the multimodal input data. Through this accurate identification and adaptive processing of different wake-up scenarios, the target device can be stably and accurately woken up in various complex environments, effectively enhancing the environmental adaptability of the target device and broadening the application scope of intelligent voice products.

[0017] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0019] Figure 1 A schematic diagram of an exemplary implementation environment architecture in which embodiments of the present disclosure can be implemented is shown;

[0020] Figure 2 A flowchart of a multimodal voice wake-up method according to an embodiment of the present disclosure is shown;

[0021] Figure 3 A block diagram of a multimodal voice wake-up device according to an embodiment of the present disclosure is shown;

[0022] Figure 4 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0024] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0025] With the deep penetration of internet technology and the rapid iteration of artificial intelligence algorithms, intelligent voice products have been widely integrated into all aspects of daily life. Among them, wake-up success rate, as a key indicator for measuring the core performance of intelligent voice products, directly affects user experience and product competitiveness, making its optimization and upgrading increasingly important.

[0026] A common approach to voice wake-up is to train a dedicated small speech recognition model, improving the wake-up success rate through extensive training. However, once a certain amount of input is reached, the improvement becomes minimal. Current mainstream solutions such as Amazon Alexa, Google Assistant, and Baidu's Xiaodu smart speaker still rely on isolated modal processing, failing to achieve effective collaboration of cross-modal features.

[0027] In this disclosure, by combining voice, gestures, and lip movements, that is, through the effective coordination of cross-modal features, the success rate of waking up the target device is improved.

[0028] The following describes an exemplary operating environment for implementing the embodiments of this disclosure provided in this application. Figure 1 A schematic diagram of an exemplary implementation environment architecture 100 in which embodiments of the present disclosure can be implemented is shown. The implementation environment architecture 100 includes an image acquisition device 110, a voice acquisition device 120, and a target device 130. As an example, gesture data and lip-reading data are acquired by the image acquisition device 110, and voice data is acquired by the voice acquisition device 120. The target device 130 performs multimodal fusion analysis on the multimodal input data to determine whether a modal conflict scenario exists. If a modal conflict exists, the confidence level of the multimodal input data is identified, and a wake-up operation is performed on the target device based on the confidence level identification result and historical data of the multimodal input data. If no modal conflict exists, the wake-up scenario of the target device is determined based on the multimodal input data, and a corresponding wake-up operation is performed on the target device according to different wake-up scenarios. By acquiring multimodal input data of voice, gestures, and lip-reading and adding timestamps, a foundation for subsequent accurate processing is provided, avoiding the high misjudgment rate of a single voice modality in complex environments, and significantly improving the wake-up accuracy.

[0029] Among them, the image acquisition device 110 can be a camera or other camera; the voice acquisition device 120 can be a microphone or other microphone; and the target device 130 can be a mobile phone, wearable device, laptop computer or other devices.

[0030] Figure 2 A flowchart of a multimodal voice wake-up method 200 according to an embodiment of the present disclosure is shown. Figure 2 The method shown can be executed by a computing device or a processing device. For example, it can be performed by... Figure 1 The target terminal 130 or the processor of terminal 130 executes.

[0031] Step S210: Obtain multimodal input data, which includes voice data, gesture data, and lip-reading data, and the multimodal input data is timestamped.

[0032] Step S220: Perform multimodal fusion analysis on the multimodal input data to determine whether there are modal conflict scenarios.

[0033] In step S230, if a modal conflict exists, the confidence level of the multimodal input data is identified, and the target device is woken up based on the confidence level identification result and the historical data of the multimodal input data.

[0034] In some embodiments, identifying the confidence level of multimodal input data and waking up the target device based on the confidence level identification result and historical data of the multimodal input data includes: identifying the multimodal input data to obtain the current speech confidence level, the current gesture confidence level, and the current lip reading confidence level; searching for the corresponding historical speech accuracy rate, historical gesture accuracy rate, and historical lip reading accuracy rate in the current scenario in the historical data; obtaining the final speech probability based on the current speech confidence level and the historical speech accuracy rate; obtaining the final gesture probability based on the current gesture confidence level and the historical gesture accuracy rate; obtaining the final lip reading probability based on the current lip reading confidence level and the historical lip reading accuracy rate; determining the modal input data corresponding to the maximum final probability as the decision result, and waking up the target device based on the decision result.

[0035] In this conflict scenario, historical data represents the historical accuracy of each modality (speech, lip reading, and gesture), which is the probability that a particular modality will correctly identify a given situation in similar past scenarios, derived from statistical analysis of a large number of historical samples. Its core function is to provide a reliable benchmark for decision-making in conflict scenarios, avoiding misjudgments caused by relying solely on the credibility of a single current recognition (which may be affected by transient noise and interference). Historical data (the historical accuracy of each modality) is one of the core inputs to the Bayesian probabilistic model. Its role is to provide a "prior reliability benchmark" for decision-making in conflict scenarios, significantly reducing the risk of misjudgments due to single recognition errors through dual verification of "historical performance" and "current confidence."

[0036] As an example, if the speech recognition result is inconsistent with the lip reading (the speech output is "stop", while the lip reading is "start"), then a modal conflict is determined. The confidence score for the recognized speech is 0.4, the probability of the lip reading is 0.9, and the probability of the gesture is 0.68. Assuming historical data: P(speech) = 0.7, P(lip reading) = 0.95, P(gesture) = 0.8, the final probability of speech = 0.7 × 0.4 = 0.28, the final probability of lip reading = 0.95 × 0.9 = 0.855, and the final probability of gesture = 0.8 × 0.68 = 0.544. The Bayesian probability model decision result = argmax[0.28, 0.855, 0.544]. The final decision result adopts lip reading (0.855).

[0037] In step S240, if no modal conflict exists, the wake-up scenario of the target device is determined based on the multimodal input data, and corresponding wake-up operations are performed on the target device according to different wake-up scenarios. Through this accurate identification and adaptive processing of different wake-up scenarios, the target device can be stably and accurately woken up in various complex environments, effectively enhancing the environmental adaptability of the target device.

[0038] In some embodiments, determining the wake-up scenario of the target device based on multimodal input data includes: if the voice confidence of the voice data is less than a first preset confidence threshold, identifying the noise decibel value of the current environment and calculating the time offset between each modal input data; if the noise decibel value is less than or equal to a preset decibel threshold, and all time offsets are less than or equal to a preset time threshold, determining the current scenario as a first wake-up scenario; if the noise decibel value is less than or equal to a preset decibel threshold, and any time offset is greater than a preset time threshold, determining the current scenario as a second wake-up scenario; if the noise decibel value is greater than a preset decibel threshold, and all time offsets are less than or equal to a preset time threshold, determining the current scenario as a third wake-up scenario; if the noise decibel value is greater than a preset decibel threshold, and any time offset is greater than a preset time threshold, determining the current scenario as a fourth wake-up scenario.

[0039] This approach involves scenario segmentation by considering multimodal data, environmental noise, and temporal differences between different modal data. Single-modal data (such as pure voice) is easily affected by interference, leading to false triggers, such as similar words on television or background noise in a noisy environment. Furthermore, due to network latency and other factors, there can be asynchrony between input data from different modalities. Therefore, temporal alignment compensation is considered. In different scenarios, users' wake-up needs and response methods for devices vary significantly. Multimodal scenario differentiation allows devices to "adjust behavior on demand." By accurately distinguishing different wake-up scenarios through multimodal data, the wake-up service becomes more tailored to user needs, ultimately achieving a more accurate, more demand-responsive, and more efficient interactive experience.

[0040] In some embodiments, calculating the time offset between different modal input data includes: extracting a time series of speech data, denoted as a first time series; extracting a time series of gesture data, denoted as a second time series; and extracting a time series of lip-sync data, denoted as a third time series; using the first time series as a reference series, finding the optimal matching path for the first time series using a dynamic time warping algorithm to obtain a first matching path; finding the optimal matching path for the second time series using a dynamic time warping algorithm to obtain a second matching path; and finding the optimal matching path for the third time series using a dynamic time warping algorithm to obtain a third matching path; calculating the time offset between the first matching path and the second matching path to obtain a first time offset; and calculating the time offset between the first matching path and the third matching path to obtain a second time offset. By identifying the time offset between different modal data, it is easier to match different modal data in the time dimension, ensuring that they correspond to the same event or scene segment, thereby improving the accuracy of scene understanding.

[0041] Specifically, for asynchronous modal data (such as a gesture delay of 200ms), a sliding window interpolation method is used to align the timing. The DTW algorithm is used to calculate the timing offset penalty coefficient η (0.8≤η≤1) to correct the confidence level. The current confidence level = original confidence level × η. Therefore, after identifying the asynchrony between different modal data, the offset penalty coefficient is used to correct the modal data with offset, thereby improving the accuracy of scene recognition and, consequently, the accuracy of voice wake-up.

[0042] As an example, voice data is collected, and the start / end time of each voice slice is marked to obtain a first time series (voice time series) T_voice = [t1, t2, t3, ..., tn]; gesture data is collected, and the time of each gesture slice is marked to obtain a first time series (gesture time series) T_gesture = [t1 + 0.2s, t2 + 0.2s, ..., tn + 0.2s]; and lip-reading data is collected, and the time of each lip-reading slice is marked to obtain a third time series (lip-reading time series) T_mouthe = [t1 + 0.02s, t2 + 0.02s, ..., tn + 0.02s]; where, the first time series ( Using a speech time series as the baseline sequence, the optimal matching path for the gesture time series, the speech time series, and the lip-reading time series is found through the DTW (Dynamic Time Warping) algorithm. The time offset between the optimal matching paths of the gesture and speech time series is calculated, as well as the time offset between the first and second matching paths, yielding the first time offset. Similarly, the time offset between the first and third matching paths is calculated, yielding the second time offset. For example, if the DTW algorithm identifies that "the gesture data is delayed by 200ms compared to the speech data," the first time offset is 200ms. Similarly, if it identifies that "the lip-reading data is delayed by 20ms compared to the speech data," the second time offset is 20ms.

[0043] In some embodiments, if the voice confidence level of the voice data is greater than or equal to a first preset confidence threshold, the target device is woken up.

[0044] As an example, the first preset confidence threshold is 0.7, the preset decibel threshold is 60 dB, and the preset time threshold is 100 milliseconds. If the speech confidence of the speech data is greater than the first preset confidence threshold (0.7), the target device is woken up by the speech data. If the speech confidence of the speech data is less than or equal to the first preset confidence threshold (0.7), the noise decibel value of the current environment is detected, for example, the noise decibel value of the current environment is 50 dB, and the time offset between each modal input data is calculated. If all calculated time offsets are less than or equal to the preset time threshold (100 milliseconds), since the noise decibel value of the current environment (50 dB) is less than the preset decibel threshold (60 dB), the current scene is determined to be the first wake-up scene.

[0045] As an example, the first preset confidence threshold is 0.7, the preset decibel threshold is 60 dB, and the preset time threshold is 100 ms. If the speech confidence of the speech data is less than or equal to the first preset confidence threshold (0.7), the noise decibel value of the current environment is detected, for example, if the noise decibel value of the current environment is 50 dB, and the time offset between each modal input data is calculated. If any time offset is found to be greater than the preset time threshold (100 ms), for example, if the calculated time offset is 120 ms, since the noise decibel value of the current environment (50 dB) is less than the preset decibel threshold (60 dB), the current scene is determined to be the second wake-up scene.

[0046] As an example, the first preset confidence threshold is 0.7, the preset decibel threshold is 60 dB, and the preset time threshold is 100 milliseconds. If the speech confidence of the speech data is less than or equal to the first preset confidence threshold (0.7), the noise decibel value of the current environment is detected. For example, if the noise decibel value of the current environment is 70 dB, the time offset between each modal input data is calculated. If all calculated time offsets are less than or equal to the preset time threshold (100 milliseconds), since the noise decibel value of the current environment (70 dB) is less than the preset decibel threshold (60 dB), the current scene is determined to be the third wake-up scene.

[0047] As an example, the first preset confidence threshold is 0.7, the preset decibel threshold is 60 dB, and the preset time threshold is 100 ms. If the speech confidence of the speech data is less than or equal to the first preset confidence threshold (0.7), the noise decibel value of the current environment is detected. For example, if the noise decibel value of the current environment is 70 dB, the time offset between each modal input data is calculated. If any calculated time offset is greater than the preset time threshold (100 ms), for example, if the calculated time offset is 120 ms, since the noise decibel value of the current environment (70 dB) is greater than the preset decibel threshold (60 dB), the current scene is determined to be the fourth wake-up scene.

[0048] After determining the voice wake-up scenario, customized wake-up rules are developed for different scenarios. The customized wake-up rules for different wake-up scenarios are described below:

[0049] In some embodiments, under a first wake-up scenario, a corresponding wake-up operation is performed on the target device according to different wake-up scenarios, including: recognizing gesture data to obtain a first gesture confidence score, and recognizing lip-reading data to obtain a first lip-reading confidence score; weighting the speech confidence score, the first gesture confidence score, and the first lip-reading confidence score according to a pre-set first weight allocation strategy to obtain a total confidence score; if the total confidence score is greater than a second pre-set confidence threshold, the target device is woken up. Through this accurate recognition and adaptive processing of different wake-up scenarios, the target device can be woken up stably and accurately in various complex environments, effectively enhancing the environmental adaptability of the target device.

[0050] As an example, the second preset confidence threshold is 0.6. In the first wake-up scenario, the user says "Open navigation," resulting in a voice confidence score of 0.6. The camera captures the lip movement of "Open navigation," resulting in a lip reading confidence score of 0.9. There is no obvious hand gesture, resulting in a gesture confidence score of 0.3. Since the noise level in the current environment is lower than the preset threshold, the default weighting strategy for this environment includes: a weight of 0.4 for voice confidence, a weight of 0.4 for lip reading confidence, and a weight of 0.2 for gesture confidence. The total confidence score is then calculated using the weights set above: 0.4 × 0.6 + 0.4 × 0.9 + 0.2 × 0.3 = 0.66. Because 0.66 is greater than the second preset confidence threshold (0.6), the target device is woken up.

[0051] In some embodiments, under the second wake-up scenario, the target device is woken up according to different wake-up scenarios, including: if any time offset is greater than a preset time threshold, the time offset greater than the preset time threshold is recorded as an abnormal time offset; the penalty coefficient of the corresponding modal input data is obtained based on the abnormal time offset and recorded as the first penalty coefficient; the modal input data confidence is corrected by the first penalty coefficient to obtain the first corrected modal input data confidence; the voice confidence and the first corrected modal input data confidence are weighted and calculated according to a preset first weight allocation strategy to obtain the total confidence; if the total confidence is greater than a second preset confidence threshold, the target device is woken up. Through this accurate identification and adaptive processing of different wake-up scenarios, the target device can be woken up stably and accurately in various complex environments, effectively enhancing the environmental adaptability of the target device.

[0052] As an example, the second preset confidence threshold is 0.6. In the second wake-up scenario, when the user says "Open navigation," the voice confidence is 0.6; the camera captures the lip movement of "Open navigation," resulting in a lip reading confidence of 0.9; and the user simultaneously makes a "query" gesture (e.g., pointing to the screen), resulting in a gesture confidence of 0.7. Due to a 100ms delay in the gesture data caused by the camera frame rate, the DTW algorithm calculates an offset Δt = 100ms, with a penalty coefficient η = 0.9 (moderate offset). After correction, the gesture confidence is 0.7 × 0.9 = 0.63. Since the noise level in the current environment is less than the preset decibel threshold, the default weighting strategy for this environment includes: a weight of 0.4 for voice confidence, a weight of 0.4 for lip reading confidence, and a weight of 0.2 for gesture confidence. The weighted calculation is then performed according to the above settings, resulting in a total confidence of 0.4 × 0.6 + 0.4 × 0.9 + 0.2 × 0.63 = 0.726. Since 0.726 is greater than the second preset confidence threshold (0.6), the target device is woken up.

[0053] In some embodiments, under the third wake-up scenario, the target device is woken up according to the different wake-up scenarios, including: recognizing gesture data to obtain a second gesture confidence score, and recognizing lip shape data to obtain a second lip reading confidence score; adjusting a first weight allocation strategy according to the noise decibel value to obtain a second weight allocation strategy; weighting the speech confidence score of the speech data, the gesture confidence score of the second gesture data, and the lip reading confidence score of the second lip shape data according to the second weight allocation strategy to obtain a total confidence score; if the total confidence score is greater than a second preset confidence threshold, the target device is woken up. Through this accurate recognition and adaptive processing of different wake-up scenarios, the target device can be woken up stably and accurately in various complex environments, effectively enhancing the environmental adaptability of the target device.

[0054] As an example, the second preset confidence threshold is 0.6. In the third wake-up scenario, the user says "Open navigation," with a voice confidence of 0.3. The camera captures the lip movement of "Open navigation," resulting in a lip reading confidence of 0.9. There is no obvious hand gesture, so the gesture confidence is 0.3. Since the noise level in the current environment exceeds the preset threshold, the weighting strategy for this environment is adjusted as follows: a weight of 0.2 is set for voice confidence, a weight of 0.6 is set for lip reading confidence, and a weight of 0.2 is set for gesture confidence. The weighted calculation is then performed according to the above settings, resulting in a total confidence of 0.2 × 0.6 + 0.6 × 0.9 + 0.2 × 0.3 = 0.72. Since 0.72 is greater than the second preset confidence threshold (0.6), the target device is woken up.

[0055] In some embodiments, under the fourth wake-up scenario, the target device is woken up according to the different wake-up scenarios, including: if any time offset is greater than a preset time threshold, the time offset greater than the preset time threshold is recorded as an abnormal time offset; the penalty coefficient of the corresponding modal input data is obtained based on the abnormal time offset, and recorded as the second penalty coefficient; the confidence of the modal input data is corrected by the second penalty coefficient to obtain the second corrected confidence of the modal input data; the voice confidence and the confidence of the second corrected modal input data are weighted according to the second weight allocation strategy to obtain the total confidence; if the total confidence is greater than the second preset confidence threshold, the target device is woken up. Through this accurate identification and adaptive processing of different wake-up scenarios, the target device can be woken up stably and accurately in various complex environments, effectively enhancing the environmental adaptability of the target device.

[0056] As an example, the second preset confidence threshold is 0.6. In the fourth wake-up scenario, when the user says "Open navigation," the voice confidence is 0.3; the camera captures the lip movement of "Open navigation," resulting in a lip reading confidence of 0.9; and the user simultaneously makes a "query" gesture (e.g., pointing to the screen), resulting in a gesture confidence of 0.7. Due to a 100ms delay in the gesture data caused by the camera frame rate, the DTW algorithm calculates an offset Δt = 100ms, with a penalty coefficient η = 0.9 (moderate offset). After correction, the gesture confidence is 0.7 × 0.9 = 0.63. Since the noise level in the current environment exceeds the preset decibel threshold, the default weighting strategy for this environment includes: a weight of 0.2 for voice confidence, a weight of 0.6 for lip reading confidence, and a weight of 0.2 for gesture confidence. The weighted calculation is then performed according to the above settings, resulting in a total confidence of 0.2 × 0.3 + 0.6 × 0.9 + 0.2 × 0.63 = 0.726. Since 0.726 is greater than the second preset confidence threshold (0.6), the target device is woken up.

[0057] Therefore, reinforcement learning is used to dynamically adjust the weights of each modality, emphasizing voiceprint features in quiet environments and strengthening lip reading or gesture signals in noisy scenes to improve wake-up accuracy. Furthermore, through cross-modal contrastive learning, a shared semantic space is constructed to align multimodal features (such as audio-lip sequence), and temporal differences are compressed using the DTW algorithm to reduce recognition bias caused by feature misalignment, further aligning with the recognition scenario and thus improving wake-up accuracy.

[0058] According to embodiments of this disclosure, by acquiring multimodal input data of voice, gestures, and lip movements and adding timestamps, a foundation is provided for subsequent accurate processing. This avoids the high misjudgment rate of a single voice modality in complex environments, significantly improving wake-up accuracy. When modal conflicts exist, by identifying the confidence level of each modal input data and combining it with historical data for decision-making, the possibility of erroneous wake-up operations due to single modal recognition errors is reduced, improving the reliability of wake-up decisions. If no modal conflicts exist, the wake-up scenario of the target device is accurately determined based on the multimodal input data. Through this accurate identification and adaptive processing of different wake-up scenarios, the target device can be stably and accurately woken up in various complex environments, effectively enhancing the environmental adaptability of the target device and broadening the application scope of intelligent voice products.

[0059] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0060] The above is an introduction to the method embodiments. The following describes the present disclosure further through device embodiments.

[0061] Figure 3 A block diagram of a multimodal voice wake-up device 300 according to an embodiment of the present disclosure is shown. Figure 3As shown, the multimodal voice wake-up device 300 includes: an acquisition module 310, a conflict judgment module 320, a first wake-up module 330, and a second wake-up module 340. The acquisition module acquires multimodal input data, including voice data, gesture data, and lip-sync data, and the multimodal input data is timestamped. The conflict judgment module performs multimodal fusion analysis on the multimodal input data to determine if a modal conflict scenario exists. The first wake-up module, if a modal conflict exists, identifies the confidence level of the multimodal input data and performs a wake-up operation on the target device based on the confidence level identification result and historical data of the multimodal input data. The second wake-up module, if no modal conflict exists, determines the wake-up scenario of the target device based on the multimodal input data and performs a corresponding wake-up operation on the target device according to the different wake-up scenarios. By acquiring multimodal input data of voice, gestures, and lip movements and adding timestamps, a foundation is laid for subsequent accurate processing. This avoids the high misjudgment rate of a single voice modality in complex environments, significantly improving wake-up accuracy. When modal conflicts exist, the confidence level of each modal input data is identified and decisions are made in conjunction with historical data, reducing the possibility of erroneous wake-up operations due to single modal recognition errors and improving the reliability of wake-up decisions. If there are no modal conflicts, the wake-up scenario of the target device is accurately determined based on the multimodal input data. Through this accurate identification and adaptive processing of different wake-up scenarios, the target device can be stably and accurately woken up in various complex environments, effectively enhancing the environmental adaptability of the target device.

[0062] In some embodiments, the multimodal voice wake-up device includes a scene determination module 350, configured to: identify the noise decibel value of the current environment and calculate the time offset between each modal input data if the voice confidence of the voice data is less than a first preset confidence threshold; determine the current scene as a first wake-up scene if the noise decibel value is less than or equal to a preset decibel threshold and all time offsets are less than or equal to a preset time threshold; determine the current scene as a second wake-up scene if the noise decibel value is less than or equal to a preset decibel threshold and any time offset is greater than a preset time threshold; determine the current scene as a third wake-up scene if the noise decibel value is greater than a preset decibel threshold and all time offsets are less than or equal to a preset time threshold; and determine the current scene as a fourth wake-up scene if the noise decibel value is greater than a preset decibel threshold and any time offset is greater than a preset time threshold.

[0063] In some embodiments, the first wake-up module is further configured to extract the time series of voice data, denoted as the first time series; extract the time series of gesture data, denoted as the second time series; and extract the time series of lip-shape data, denoted as the third time series; using the first time series as a reference series, find the optimal matching path of the first time series through a dynamic time warping algorithm to obtain the first matching path; find the optimal matching path of the second time series through a dynamic time warping algorithm to obtain the second matching path; and find the optimal matching path of the third time series through a dynamic time warping algorithm to obtain the third matching path; calculate the time offset between the first matching path and the second matching path to obtain the first time offset; and calculate the time offset between the first matching path and the third matching path to obtain the second time offset.

[0064] In some embodiments, the first wake-up module is further configured to recognize gesture data to obtain a first gesture confidence level, and recognize lip shape data to obtain a first lip reading confidence level; to perform weighted calculation on the speech confidence level, the first gesture confidence level, and the first lip reading confidence level according to a pre-set first weight allocation strategy to obtain a total confidence level; and to wake up the target device if the total confidence level is greater than a second pre-set confidence threshold.

[0065] In some embodiments, the first wake-up module is further configured to: if any time offset is greater than a preset time threshold, record the time offset greater than the preset time threshold as an abnormal time offset; obtain the penalty coefficient of the corresponding modal input data based on the abnormal time offset, and record it as the first penalty coefficient; correct the modal input data confidence using the first penalty coefficient to obtain the first corrected modal input data confidence; perform weighted calculation on the voice confidence and the first corrected modal input data confidence according to a preset first weight allocation strategy to obtain the total confidence; and wake up the target device if the total confidence is greater than a second preset confidence threshold.

[0066] In some embodiments, the first wake-up module is further configured to recognize gesture data to obtain a second gesture confidence level, and recognize lip shape data to obtain a second lip reading confidence level; adjust a first weight allocation strategy according to the noise decibel value to obtain a second weight allocation strategy; perform weighted calculation on the speech confidence level of the speech data, the gesture confidence level of the second gesture data, and the lip reading confidence level of the second lip shape data according to the second weight allocation strategy to obtain a total confidence level; if the total confidence level is greater than a second preset confidence threshold, then wake up the target device.

[0067] In some embodiments, the first wake-up module is further configured to: if any time offset is greater than a preset time threshold, record the time offset greater than the preset time threshold as an abnormal time offset; obtain the penalty coefficient of the corresponding modal input data based on the abnormal time offset, and record it as a second penalty coefficient; correct the modal input data confidence using the second penalty coefficient to obtain a second corrected modal input data confidence; perform weighted calculation on the voice confidence and the second corrected modal input data confidence according to a second weight allocation strategy to obtain a total confidence; and wake up the target device if the total confidence is greater than a second preset confidence threshold.

[0068] In some embodiments, the second wake-up module is used to recognize multimodal input data to obtain the current speech confidence, current gesture confidence, and current lip reading confidence; to search for the corresponding historical speech accuracy, historical gesture accuracy, and historical lip reading accuracy in the current scenario in historical data; to obtain the final speech probability based on the current speech confidence and historical speech accuracy; to obtain the final gesture probability based on the current gesture confidence and historical gesture accuracy; and to obtain the final lip reading probability based on the current lip reading confidence and historical lip reading accuracy; to determine the modal input data corresponding to the maximum final probability as the decision result, and to perform a wake-up operation on the target device based on the decision result.

[0069] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0070] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0071] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0072] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0073] Electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in ROM 402 or a computer program loaded into RAM 403 from storage unit 408. RAM 403 can also store various programs and data required for the operation of electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. I / O interface 405 is also connected to bus 404.

[0074] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0075] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the multimodal voice wake-up method. For example, in some embodiments, the multimodal voice wake-up method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the multimodal voice wake-up method described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the multimodal voice wake-up method by any other suitable means (e.g., by means of firmware).

[0076] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0077] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0078] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0079] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0080] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0081] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0082] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0083] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multimodal voice wake-up method, characterized in that, include: Acquire multimodal input data, which includes voice data, gesture data, and lip-reading data, and the multimodal input data is timestamped. Perform multimodal fusion analysis on the multimodal input data to determine whether there are modal conflict scenarios; If a modal conflict exists, the confidence level of the multimodal input data is identified, and the target device is woken up based on the confidence level identification result and the historical data of the multimodal input data. If there is no modal conflict, the wake-up scenario of the target device is determined based on the multimodal input data, and the target device is wake-up operated accordingly based on the different wake-up scenarios.

2. The method according to claim 1, characterized in that, The wake-up scenario of the target device is determined based on the multimodal input data, including: If the speech confidence of the speech data is less than the first preset confidence threshold, then the noise decibel value of the current environment is identified, and the time offset between each modal input data is calculated. If the noise decibel value is less than or equal to a preset decibel threshold, and all time offsets are less than or equal to a preset time threshold, then the current scene is determined as the first wake-up scene. If the noise decibel value is less than or equal to a preset decibel threshold, and any of the time offsets is greater than a preset time threshold, then the current scene is determined as the second wake-up scene. If the noise decibel value is greater than the preset decibel threshold, and all time offsets are less than or equal to the preset time threshold, then the current scene is determined as the third wake-up scene. If the noise decibel value is greater than a preset decibel threshold, and any of the time offsets is greater than a preset time threshold, then the current scene is determined as the fourth wake-up scene.

3. The method according to claim 2, characterized in that, Calculate the time offset between the input data of each modality, including: The time series of the extracted speech data is denoted as the first time series; the time series of the extracted gesture data is denoted as the second time series; and the time series of the extracted lip-sync data is denoted as the third time series. Using the first time series as the reference series, the optimal matching path of the first time series is found through the dynamic time warping algorithm to obtain the first matching path; and the optimal matching path of the second time series is found through the dynamic time warping algorithm to obtain the second matching path; and the optimal matching path of the third time series is found through the dynamic time warping algorithm to obtain the third matching path. Calculate the time offset between the first matching path and the second matching path to obtain a first time offset, and calculate the time offset between the first matching path and the third matching path to obtain a second time offset.

4. The method according to claim 3, characterized in that In the first wake-up scenario, the target device is given a corresponding wake-up operation according to the different wake-up scenarios, including: The gesture data is identified to obtain a first gesture confidence level, and the lip shape data is identified to obtain a first lip reading confidence level; The confidence scores of the speech, the first gesture, and the first lip reading are weighted according to a pre-set first weight allocation strategy to obtain the total confidence score. If the total confidence level is greater than the second preset confidence threshold, then the target device is woken up.

5. The method according to claim 4, characterized in that, In the second wake-up scenario, the target device is given a corresponding wake-up operation according to the different wake-up scenarios, including: If any time offset is greater than the preset time threshold, the time offset that is greater than the preset time threshold is recorded as an abnormal time offset. The penalty coefficient for the corresponding modal input data is obtained based on the abnormal time offset, and is denoted as the first penalty coefficient. The confidence level of the modal input data is corrected by the first penalty coefficient to obtain the first corrected confidence level of the modal input data; The speech confidence and the first corrected modal input data confidence are weighted and calculated according to a pre-set first weight allocation strategy to obtain the total confidence. If the total confidence level is greater than the second preset confidence threshold, then the target device is woken up.

6. The method according to claim 2, characterized in that, In the third wake-up scenario, the target device is wake-up according to the different wake-up scenarios, including: The gesture data is identified to obtain a second gesture confidence level, and the lip shape data is identified to obtain a second lip reading confidence level; The first weight allocation strategy is adjusted according to the noise decibel value to obtain the second weight allocation strategy; The speech confidence score of the speech data, the gesture confidence score of the second gesture data, and the lip reading confidence score of the second lip shape data are weighted and calculated according to the second weight allocation strategy to obtain the total confidence score. If the total confidence level is greater than the second preset confidence threshold, then the target device is woken up.

7. The method according to claim 6, characterized in that, In the fourth wake-up scenario, the target device is wake-up according to the different wake-up scenarios, including: If any time offset is greater than the preset time threshold, the time offset that is greater than the preset time threshold is recorded as an abnormal time offset. The penalty coefficient for the corresponding modal input data is obtained based on the abnormal time offset, and is denoted as the second penalty coefficient. The confidence level of the modal input data is corrected by the second penalty coefficient to obtain the second corrected confidence level of the modal input data; The speech confidence and the second corrected modal input data confidence are weighted according to the second weight allocation strategy to obtain the total confidence. If the total confidence level is greater than the second preset confidence threshold, then the target device is woken up.

8. The method according to claim 1, characterized in that, Identifying the confidence level of the multimodal input data, and performing a wake-up operation on the target device based on the confidence level identification result and historical data of the multimodal input data, including: The multimodal input data is identified to obtain the current speech confidence, current gesture confidence, and current lip reading confidence. Search the historical data for the corresponding speech accuracy, gesture accuracy, and lip reading accuracy in the current scenario. The final probability of speech is obtained based on the current speech confidence and the speech history accuracy; the final probability of gesture is obtained based on the current gesture confidence and the gesture history accuracy; and the final probability of lip reading is obtained based on the current lip reading confidence and the lip reading history accuracy. The modal input data corresponding to the maximum final probability is determined as the decision result, and the target device is woken up based on the decision result.

9. A multimodal voice wake-up device, characterized in that, include: The acquisition module is used to acquire multimodal input data, which includes voice data, gesture data, and lip-reading data, and the multimodal input data is timestamped. The conflict determination module is used to perform multimodal fusion analysis on the multimodal input data to determine whether there are modal conflict scenarios. The first wake-up module is used to identify the confidence level of the multimodal input data if a modal conflict exists, and to perform a wake-up operation on the target device based on the confidence level identification result and the historical data of the multimodal input data. The second wake-up module is used to determine the wake-up scenario of the target device based on the multimodal input data if there is no modal conflict, and to perform corresponding wake-up operations on the target device according to different wake-up scenarios.

10. An electronic device, comprising: at least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of claims 1-8.

Images