Speech segmentation method and apparatus, and home appliance

By using a combination of short-time energy and pitch period in speech data processing to generate an array to determine the target segmentation point, the accuracy problem of speech segmentation is solved and the precision of speech recognition is improved.

CN117198273BActive Publication Date: 2026-11-10FOSHAN SHUNDE MIDEA ELECTRONICS TECH CO LTD +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202210609285.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-31
Publication Date
2026-11-10
Estimated Expiration
2042-05-31

AI Technical Summary

Technical Problem

In speech data processing, existing technologies are prone to omissions or misidentifications in speech segmentation due to environmental interference, resulting in insufficient accuracy.

Method used

By combining the short-time energy and pitch period of the audio frame, a first array and a second array are generated to determine the target segmentation point of the target speech audio frame sequence in the audio data, thereby segmenting the speech audio frame sequence.

Benefits of technology

It improves the accuracy of speech segmentation, enhances the precision of speech recognition, and reduces the impact of environmental noise and non-speech audio.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117198273B_ABST
    Figure CN117198273B_ABST
Patent Text Reader

Abstract

The application discloses a voice segmentation method and device and household electrical equipment. The method comprises the following steps: obtaining audio data to be segmented; determining target segmentation points of each target voice audio frame sequence in the audio data based on the short-time energy of the audio frame in the audio data and the pitch period of the audio frame in the audio data; and segmenting each target voice audio frame sequence in the audio data according to the target segmentation points. The technical scheme provided by the application can improve the accuracy of voice segmentation in the audio data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of speech data processing technology, and in particular relates to a speech segmentation method, device and household appliance. Background Technology

[0002] In speech data processing scenarios, such as speech segmentation in audio data, conventional endpoint detection methods are currently commonly used. However, when there is significant environmental interference, this can easily lead to missed or incorrect recognition. Therefore, improving the accuracy of speech segmentation in audio data is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] The embodiments of this application provide a speech segmentation method, apparatus, and home appliance, which can at least to some extent improve the accuracy of speech segmentation in audio data.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to a first aspect of the embodiments of this application, a speech segmentation method is provided, the method comprising: acquiring audio data to be segmented; determining target segmentation points for each target speech audio frame sequence in the audio data based on the short-time energy of audio frames in the audio data and the pitch period of audio frames in the audio data; and segmenting each target speech audio frame sequence in the audio data according to the target segmentation points.

[0006] In some embodiments of this application, based on the foregoing scheme, determining the target segmentation point of each target speech audio frame sequence in the audio data based on the short-time energy of the audio frames in the audio data and the pitch period of the audio frames in the audio data includes: determining a first array based on the short-time energy of the audio frames in the audio data, the first array being used to characterize the distribution of spoken audio frames in the audio data; determining a second array based on the pitch period of the audio frames in the audio data, the second array being used to characterize the distribution of speech audio frames in the audio data; and determining the target segmentation point of each target speech audio frame sequence in the audio data using the first array and the second array.

[0007] In some embodiments of this application, based on the foregoing scheme, determining the first array based on the short-time energy of audio frames in the audio data includes: marking audio frames in the audio data with short-time energy higher than a preset energy value as having a sound tag; marking audio frames in the audio data with short-time energy lower than or equal to the preset energy value as having a silent tag; and determining the first array based on the sound tags and the silent tags.

[0008] In some embodiments of this application, based on the foregoing scheme, determining the first array based on the audio tag and the silent tag includes: generating the first array based on the audio tag or silent tag of each audio frame and the frame number of each audio frame in the audio data.

[0009] In some embodiments of this application, based on the foregoing scheme, generating a first array based on the audio tags or silent tags of each audio frame and the frame number of each audio frame in the audio data includes: determining audio frames in the audio data that correspond to the audio tags and have consecutive frame numbers, as an audio frame sequence; determining audio frames in the audio data that correspond to the silent tags and have consecutive frame numbers, as a silent audio frame sequence; and generating a first array based on the start and end frame numbers of the audio frame sequence and the start and end frame numbers of the silent audio frame sequence.

[0010] In some embodiments of this application, based on the foregoing scheme, determining the second array based on the pitch period of the audio frames in the audio data includes: labeling audio frames with periodic pitches in the audio data with speech tags; labeling audio frames without periodic pitches in the audio data with non-speech tags; and determining the second array based on the speech tags and the non-speech tags.

[0011] In some embodiments of this application, based on the foregoing scheme, determining the second array based on the voice tags and the non-voice tags includes: generating the second array based on the voice tags or non-voice tags of each audio frame and the frame number of each audio frame in the audio data.

[0012] In some embodiments of this application, based on the foregoing scheme, generating a second array based on the speech tags or non-speech tags of each audio frame and the frame sequence number of each audio frame in the audio data includes: determining audio frames in the audio data that correspond to speech tags and have consecutive frame sequence numbers, as a speech audio frame sequence; determining audio frames in the audio data that correspond to non-speech tags and have consecutive frame sequence numbers, as a non-speech audio frame sequence; and generating a second array based on the start and end frame sequence numbers of the speech audio frame sequence and the start and end frame sequence numbers of the non-speech audio frame sequence.

[0013] In some embodiments of this application, based on the foregoing scheme, the average pitch period of each audio frame in the speech audio frame sequence is the same.

[0014] In some embodiments of this application, based on the foregoing scheme, before generating the second array based on the start frame number and end frame number of the speech audio frame sequence and the start frame number and end frame number of the non-speech audio frame sequence, the method further includes: adjusting the speech audio frame sequence with fewer than a preset threshold number of audio frames into a non-speech audio frame sequence.

[0015] In some embodiments of this application, based on the foregoing scheme, each audio frame in the audio data includes multiple pitch periods. Determining a second array based on the pitch periods of the audio frames in the audio data includes: calculating the average pitch period of each audio frame in the audio data; identifying audio frames in the audio data with the same average pitch period and consecutive frame numbers as a speech audio frame sequence; identifying audio frames in the audio data with different average pitch periods and consecutive frame numbers as a non-speech audio frame sequence; and generating a second array based on the start and end frame numbers of the speech audio frame sequence and the start and end frame numbers of the non-speech audio frame sequence.

[0016] In some embodiments of this application, based on the foregoing scheme, determining the target segmentation point of each target speech audio frame sequence in the audio data using the first array and the second array includes: determining a first segmentation point of the spoken audio frame sequence in the audio data using the first array; and determining a second segmentation point of each target speech audio frame sequence in the audio data defined by the first segmentation point using the second array, as the target segmentation point.

[0017] In some embodiments of this application, on the one hand, short-time energy can avoid the influence of environmental noise on the determination of the target speech audio frame sequence in the audio data; on the other hand, pitch period can avoid the influence of non-speech audio in the environment on the determination of the target speech audio frame sequence in the audio data. Based on this, this application simultaneously determines the target segmentation point of each target speech audio frame sequence in the audio data based on short-time energy and pitch period, and segments each target speech audio frame sequence in the audio data according to the target segmentation point, which can improve the accuracy of speech segmentation in audio data, and thus improve the accuracy of speech recognition of audio data.

[0018] According to a second aspect of the embodiments of this application, a speech segmentation apparatus is provided, the apparatus comprising: an acquisition unit for acquiring audio data to be segmented; a determination unit for determining target segmentation points for each target speech audio frame sequence in the audio data based on the short-time energy of audio frames in the audio data and the pitch period of audio frames in the audio data; and a segmentation unit for segmenting each target speech audio frame sequence in the audio data according to the target segmentation points.

[0019] According to a third aspect of the embodiments of this application, a home appliance is provided, the home appliance including one or more processors and one or more memories, the one or more memories storing at least one piece of program code, the at least one piece of program code being loaded and executed by the one or more processors to implement the method described in any embodiment of the first aspect above.

[0020] The beneficial effects of the embodiments of the second to third aspects described above can be referred to the beneficial effects of the first aspect and the embodiments of the first aspect described above, and will not be repeated here.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0023] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown;

[0024] Figure 2 A flowchart of the speech segmentation method in an embodiment of this application is shown;

[0025] Figure 3 A detailed flowchart illustrating the determination of target segmentation points for each target speech audio frame sequence in the audio data in an embodiment of this application is shown.

[0026] Figure 4 A detailed flowchart illustrating how the first array is determined based on the short-time energy of audio frames in the audio data, as shown in an embodiment of this application, is presented.

[0027] Figure 5 The spectrograms in the embodiments of this application are shown;

[0028] Figure 6 This document illustrates a detailed flowchart of how, in an embodiment of this application, a first array is generated based on the audio tags or silent tags of each audio frame, and the frame number of each audio frame in the audio data.

[0029] Figure 7 A detailed flowchart illustrating how the second array is determined based on the pitch period of audio frames in the audio data in an embodiment of this application is shown.

[0030] Figure 8 The spectrograms in the embodiments of this application are shown;

[0031] Figure 9 This document illustrates a detailed flowchart of how a second array is generated based on speech tags or non-speech tags for each audio frame and the frame number of each audio frame in the audio data, as described in an embodiment of this application.

[0032] Figure 10 This document illustrates another detailed flowchart of determining the second array based on the pitch period of the audio frames in the audio data, as shown in an embodiment of this application.

[0033] Figure 11 This document illustrates a detailed flowchart of how, in an embodiment of this application, target segmentation points are determined for each target speech audio frame sequence in the audio data using the first array and the second array.

[0034] Figure 12 The overall flowchart of the speech segmentation method in the embodiments of this application is shown;

[0035] Figure 13 A block diagram of the speech segmentation device in an embodiment of this application is shown;

[0036] Figure 14 A schematic diagram of the structure of a household appliance in an embodiment of this application is shown. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0039] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0041] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "multiple" means two or more.

[0042] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solutions of the embodiments of this application can be applied is shown.

[0043] like Figure 1 As shown, the system architecture may include home appliances 101, a network 102, and a server (e.g., the cloud) 103. The network 102 serves as a medium for providing a communication link between the home appliances 101 and the server 103. The network 102 may include various connection types, such as wired communication links, wireless communication links, etc.

[0044] In this application, when the home appliance 101 receives audio data of a user's voice command (such as "I want to turn on the air conditioner") in its environment (e.g., inside a room), it sends the audio data to the server 103. The server 103 needs to perform voice segmentation on the audio data to identify the user's intent. In one embodiment of this application, after receiving the audio data, the server 103 can determine the target segmentation point of each target speech audio frame sequence in the audio data based on the short-time energy and pitch period of the audio frames in the audio data. Then, according to the target segmentation point, it segments each target speech audio frame sequence in the audio data to complete the voice segmentation of the audio data.

[0045] In this application, on the one hand, short-time energy can avoid the influence of environmental noise on the determination of the target speech audio frame sequence in the audio data; on the other hand, pitch period can avoid the influence of non-speech audio in the environment on the determination of the target speech audio frame sequence in the audio data. Based on this, this application simultaneously determines the target segmentation point of each target speech audio frame sequence in the audio data based on short-time energy and pitch period, and segments each target speech audio frame sequence in the audio data according to the target segmentation point, which can improve the accuracy of speech segmentation in audio data, and thus improve the accuracy of speech recognition of audio data.

[0046] It should be noted that the voice segmentation method provided in this application embodiment can be executed by the server 103, and correspondingly, the voice segmentation device is generally set in the server 103. However, in other embodiments of this application, the home appliance itself may also have similar functions to the server 103, thereby executing the voice segmentation scheme provided in this application embodiment.

[0047] It should also be noted that, Figure 1 The number of home appliances, networks, and servers shown is merely illustrative. Depending on the implementation requirements, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0048] It's important to explain that cloud computing, as described above, is a computing model that distributes computing tasks across a resource pool comprised of numerous computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" appear infinitely scalable and can be accessed, used on demand, and expanded at any time. By establishing a cloud computing resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform), various types of virtual resources are deployed within the resource pool for external customers to choose from. The cloud computing resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0049] It should also be noted in this application that the home appliances mentioned in this application may be air conditioners, washing machines, refrigerators, robot vacuum cleaners, etc.

[0050] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0051] Figure 2 A flowchart of a speech segmentation method according to an embodiment of this application is shown. This speech segmentation method can be executed by a device with computing processing capabilities, such as a... Figure 1 The appliance 101 or server 103 shown is used to execute this. (Refer to...) Figure 2 As shown, this speech segmentation method includes at least steps 200 to 400, which are detailed below:

[0052] In step 200, the audio data to be segmented is obtained.

[0053] The technical solution proposed in this application can be used in scenarios where home appliances are controlled by voice. In this scenario, the audio data can be voice commands issued by the user to the home appliances. For example, the user can issue a voice command to the air conditioner saying "Please turn on the air conditioner" or to the refrigerator saying "Please lower the temperature of the freezer compartment by one degree".

[0054] Continue to refer to Figure 2 In step 300, the target segmentation point of each target speech audio frame sequence in the audio data is determined based on the short-time energy of the audio frames in the audio data and the pitch period of the audio frames in the audio data.

[0055] In this application, it should be noted that short-time energy can refer to the output value of the square of the speech signal in the audio data after passing through a linear filter. Because the speech signal changes with time, its energy also changes with time. Therefore, when calculating the energy of a digitized speech signal, the overall energy is not calculated, but the energy of each frame is calculated. In this application, an audio frame generally includes multiple sampling points, and the energy of each audio frame is obtained by substituting the energy of each sampling point into the energy calculation and summing them. In scenarios where home appliances are controlled by voice, the short-time energy of audio frames generated by sound sources far away from the appliances is less than the short-time energy of audio frames in the user's voice commands.

[0056] In this application, it should be noted that ordinary sound is composed of a series of vibrations of different frequencies and amplitudes emitted by a sound-producing body. Among these vibrations, there is a vibration with the lowest frequency, and the sound produced by it is the fundamental tone. The period of the fundamental tone is generally the time of one complete wavelength of the mechanical wave. When a person produces voiced sounds, the airflow passes through the glottis, causing the vocal cords to vibrate in a relaxation-oscillation manner, generating a quasi-periodic pulse of airflow. This airflow excites the vocal tract to produce voiced sounds, also known as spoken speech. It carries most of the energy in speech, and the frequency of this vocal cord vibration is called the fundamental frequency, and the corresponding period is the fundamental tone period.

[0057] In this application, the target speech audio frame sequence can refer to a sequence of consecutively numbered audio frames, and one target speech audio frame sequence corresponds to the speech of one word spoken by the user. For example, the speech "lower the temperature" would generally include four target speech audio frame sequences.

[0058] Continue to refer to Figure 2 In step 400, the audio data is segmented into individual target speech audio frame sequences according to the target segmentation point.

[0059] In this application, after determining the target segmentation point, the audio data can be segmented into individual target speech audio frame sequences, and then the speech intent can be identified based on the audio data corresponding to each target speech audio frame sequence.

[0060] Next, this application will be approved. Figure 3 For example Figure 2 Step 300, which involves determining the target segmentation point for each target speech audio frame sequence in the audio data based on the short-time energy of the audio frames and the pitch period of the audio frames in the audio data, will be further explained below:

[0061] See Figure 3 This document illustrates a detailed flowchart of the process for determining target segmentation points for each target speech audio frame sequence in the audio data, as described in an embodiment of this application. Specifically, it includes steps 310 to 320:

[0062] In step 310, a first array is determined based on the short-time energy of the audio frames in the audio data. The first array is used to characterize the distribution of the audio frames in the audio data.

[0063] In one embodiment of this application, a first array is determined based on the short-time energy of audio frames in the audio data, which can be done according to... Figure 4 Perform the steps shown.

[0064] See Figure 4 This document illustrates a detailed flowchart of how, in an embodiment of this application, a first array is determined based on the short-time energy of audio frames in the audio data. Specifically, it includes steps 311 to 313:

[0065] Step 311: In the audio data, audio frames with short-term energy higher than a preset energy value are marked with audio tags.

[0066] Step 312: In the audio data, audio frames with short-term energy values ​​lower than or equal to a preset energy value are marked with a silent tag.

[0067] Step 313: Determine the first array based on the audio tags and the silent tags.

[0068] To enable those skilled in the art to better understand this application, the following will be combined with Figure 5 The spectrogram shown is explained below.

[0069] See Figure 5 The image shows a spectrogram from an embodiment of this application.

[0070] Figure 5 The spectrogram 500 corresponding to a segment of audio data is shown. As can be seen from the spectrogram, the audio data includes invalid sound segments and valid sound segments. Invalid sound segments can be some noise (such as the sound of a pet or the sound of a machine outside the room), while valid sound segments can be the sounds that need to be recognized for the voice intent, such as the voice command "turn down the temperature" issued by the user.

[0071] In this embodiment, as Figure 5 The audio data shown includes multiple audio frames. Audio frames with short-term energy higher than a preset energy value are marked with a sound tag, while audio frames with short-term energy lower than or equal to the preset energy value are marked with a silent tag.

[0072] In this application, a sound tag can be represented by the number "1", and a silent tag can be represented by the number "0". For example, if the audio data includes 10 audio frames, the short-time energy of the 1st, 2nd, 9th, and 10th audio frames is lower than a preset energy value, and the short-time energy of the 3rd to 8th audio frames is higher than the preset energy value, then the tag array for each audio frame in the entire audio data can be obtained as [0 0 1 1 1 1 1 1 1 1 0 0].

[0073] It should be noted that audio tags and silent tags can also be represented in other forms, and this application does not limit them.

[0074] In such Figure 4 In one embodiment of step 313 shown, a first array is determined based on the audio tag and the silent tag. Specifically, the first array can be generated based on the audio tag or silent tag of each audio frame and the frame number of each audio frame in the audio data.

[0075] In this embodiment, a first array is generated based on the audio tag or silent tag of each audio frame and the frame number of each audio frame in the audio data, and can be generated according to the following... Figure 6 Perform the steps shown.

[0076] See Figure 6This document illustrates a detailed flowchart of the generation of a first array based on the audio frame's audio tag or silent tag, and the frame number of each audio frame in the audio data, as described in this embodiment of the application. Specifically, it includes steps 3131 to 3133:

[0077] Step 3131: Determine the audio frames with consecutive frame numbers corresponding to the audio tag in the audio data, and use them as the audio frame sequence.

[0078] Step 3132: Determine the audio frames with consecutive frame numbers corresponding to the silent tag in the audio data, and use them as the silent audio frame sequence.

[0079] Step 3133: Generate a first array based on the start frame number and end frame number of the audio frame sequence with sound, and the start frame number and end frame number of the silent audio frame sequence.

[0080] First, refer to Figure 5 Each audio frame in the audio data corresponds to a frame number. The starting frame number of the entire audio data is 0, and the ending frame number is f. In practical situations, such as... Figure 5 The audio data shown includes invalid audio segments from the 0th audio frame to the ath audio frame, invalid audio segments from the ath audio frame to the eth audio frame, and invalid audio segments from the eth audio frame to the fth audio frame.

[0081] In such Figure 5 In the audio data shown, through, as Figure 6 The scheme shown can determine the audio frame sequence with sound as "a, a+1, a+2, ..., e-2, e-1, e", and the audio frame sequence without sound as "1, 2, 3, ..., a-2, a-1" and "e+1, e+2, ..., f-2, f-1, f".

[0082] Furthermore, based on the start and end frame numbers of the spoken audio frame sequence and the start and end frame numbers of the silent audio frame sequence, the generated first array can be "[0 1 0], [0 ae], [a ef]", where:

[0083] The “0” in “[0 1 0]” can represent a silent audio frame sequence, and the “1” in “[0 1 0]” can represent a sound audio frame sequence.

[0084] In “[0 ae]”, “0” can represent the starting frame number of the first silent audio frame sequence, “a” in “[0 ae]” can represent the starting frame number of the sound audio frame sequence, and “e” in “[0 ae]” can represent the starting frame number of the second silent audio frame sequence.

[0085] In “[aef]”, “a” can represent the ending frame number of the first silent audio frame sequence, “e” can represent the ending frame number of the sound audio frame sequence, and “f” can represent the ending frame number of the second silent audio frame sequence.

[0086] In this application, the distribution of audible and silent audio frames in the audio data can be determined by the first array, thereby avoiding the influence of silent audio frames (i.e., interference sounds) on speech segmentation processing, improving the accuracy of speech segmentation in audio data, and further improving the accuracy of speech recognition of audio data.

[0087] Continue to refer to Figure 3 In step 320, a second array is determined based on the pitch period of the audio frames in the audio data. The second array is used to characterize the distribution of speech audio frames in the audio data.

[0088] In one embodiment of this application, the second array is determined based on the pitch period of the audio frames in the audio data, and can be determined according to the following... Figure 7 Perform the steps shown.

[0089] See Figure 7 This document illustrates a detailed flowchart of how, in an embodiment of this application, a second array is determined based on the pitch period of audio frames in the audio data. Specifically, it includes steps 321 to 323:

[0090] Step 321: In the audio data, audio frames with periodic pitches are labeled with speech tags.

[0091] Step 322: In the audio data, audio frames that do not have a periodic pitch are marked with non-speech tags.

[0092] Step 323: Determine the second array based on the voice tags and the non-voice tags.

[0093] To enable those skilled in the art to better understand this application, the following will be combined with Figure 8 The spectrogram shown is explained below.

[0094] See Figure 8 The image shows a spectrogram from an embodiment of this application. It should be noted that... Figure 8 The acoustic spectrum shown is Figure 5 The sound spectrum shown is the same sound spectrum.

[0095] Figure 8The spectrogram 800 corresponding to a segment of audio data is shown. As can be seen from the spectrogram, the audio data includes invalid sound segments and speech segments with "temperature turned down". The invalid sound segments can be some noise (such as the sound of a pet or the sound of a machine outside the room), while the speech segments with "temperature turned down" can be the sounds that need to be recognized for speech intent.

[0096] In this embodiment, as Figure 8 The audio data shown includes multiple audio frames, each containing multiple pitch periods. Based on this, audio frames with periodic pitches (i.e., the pitch periods in each audio frame are the same or similar) can be labeled as speech tags, while audio frames without periodic pitches can be labeled as non-speech tags.

[0097] In this application, speech tags can be represented by the number "1", and non-speech tags can be represented by the number "0". For example, the audio data includes 10 audio frames, of which the 1st, 8th, 9th, and 10th audio frames do not have a periodic pitch, while the 2nd to 7th audio frames have a periodic pitch. Therefore, the tag array for each audio frame in the entire audio data can be obtained as [0 1 1 1 1 1 1 1 1 0 0 0].

[0098] It should be noted that voice tags and non-voice tags can also be represented in other forms, and this application does not limit them.

[0099] In such Figure 7 In one embodiment of step 323 shown, a second array is determined based on the voice tags and the non-voice tags. Specifically, the second array can be generated based on the voice tags or non-voice tags of each audio frame and the frame number of each audio frame in the audio data.

[0100] In this embodiment, a second array is generated based on the speech tags or non-speech tags of each audio frame and the frame number of each audio frame in the audio data, and can be generated according to the following... Figure 9 Perform the steps shown.

[0101] See Figure 9 The diagram illustrates a detailed flowchart of how a second array is generated based on voice or non-voice tags for each audio frame and the frame number of each audio frame in the audio data, as described in this application embodiment.

[0102] Specifically, steps 3131 to 3133 are included:

[0103] Step 3231: Determine the audio frames with consecutive frame numbers corresponding to the voice tags in the audio data, and use them as the voice audio frame sequence.

[0104] Step 3232: In the audio data, determine the audio frames that correspond to non-speech tags and have consecutive frame numbers, and use them as the non-speech audio frame sequence.

[0105] Step 3233: Generate a second array based on the start and end frame numbers of the speech audio frame sequence and the start and end frame numbers of the non-speech audio frame sequence.

[0106] In this embodiment, the average pitch period of each audio frame in the speech audio frame sequence is the same.

[0107] It should be noted that the average pitch period refers to the average value of the periods of each pitch in an audio frame.

[0108] In this application, since the pitch period of the speech corresponding to the same character is the same, and the pitch period of the speech corresponding to different characters is different, based on this, according to the technical feature of the same average pitch period, the speech audio frame sequence corresponding to each character can be further determined in the audio frames corresponding to the speech tags and with consecutive frame numbers.

[0109] Furthermore, in this embodiment, before generating the second array based on the start and end frame numbers of the speech audio frame sequence and the start and end frame numbers of the non-speech audio frame sequence, speech audio frame sequences with fewer than a preset threshold number of audio frames can be adjusted to non-speech audio frame sequences.

[0110] Since a sequence of audio frames corresponding to a text will not contain only one or a few audio frames, the advantage of adjusting audio frame sequences with fewer than a preset threshold of audio frames to non-audio frame sequences is that it can avoid the influence of audio frame sequences that do not correspond to real speech on the processing of audio data, thereby improving the accuracy of speech segmentation in audio data, and thus improving the accuracy of speech recognition of audio data.

[0111] To enable those skilled in the art to better understand this application, the following will continue to combine... Figure 8 Please provide an explanation.

[0112] First, refer to Figure 8 In practical situations, such as Figure 8 The audio data shown includes invalid audio segments from the 0th audio frame to the ath audio frame, invalid audio segments from the ath audio frame to the eth audio frame, and the “lower temperature” speech segments from the eth audio frame to the fth audio frame.

[0113] In such Figure 8 In the audio data shown, through, as Figure 9The scheme shown can determine the speech audio frame sequence as “a, a+1, a+2, ..., e-2, e-1, e”, and the non-speech audio frame sequence as “1, 2, 3, ..., a-2, a-1” and “e+1, e+2, ..., f-2, f-1, f”.

[0114] Furthermore, based on the start and end frame numbers of the speech audio frame sequence and the start and end frame numbers of the non-speech audio frame sequence, the generated second array can be "[0 1 1 1 1 0], [0 ab cde], [abcdef]", where:

[0115] The “0” in “[0 1 1 1 1 0]” can represent a non-speech audio frame sequence, and the “1” in “[0 1 1 1 1 0]” can represent a speech audio frame sequence.

[0116] In “[0 abcde]”, “0” can represent the starting frame number of the first non-speech audio frame sequence; “a” in “[0a bc de]” can represent the starting frame number of the first speech audio frame sequence (i.e., the speech segment corresponding to “warm”); “b” in “[0 abcde]” can represent the starting frame number of the second speech audio frame sequence (i.e., the speech segment corresponding to “degree”); “c” in “[0 abcde]” can represent the starting frame number of the third speech audio frame sequence (i.e., the speech segment corresponding to “tone”); “d” in “[0 abcde]” can represent the starting frame number of the fourth speech audio frame sequence (i.e., the speech segment corresponding to “low”); and “e” in “[0 abcde]” can represent the starting frame number of the second non-speech audio frame sequence.

[0117] In “[abcdef]”, “a” represents the end frame number of the first non-speech audio frame sequence; “b” represents the end frame number of the first speech audio frame sequence (corresponding to the “warm” speech segment); “c” represents the end frame number of the second speech audio frame sequence (corresponding to the “degree” speech segment); “d” represents the end frame number of the third speech audio frame sequence (corresponding to the “tone” speech segment); “e” represents the end frame number of the fourth speech audio frame sequence (corresponding to the “low” speech segment); and “f” represents the end frame number of the second non-speech audio frame sequence.

[0118] In another embodiment of this application, such as Figure 3 Step 320, as shown, determines the second array based on the pitch period of the audio frames in the audio data. This can also be done according to... Figure 10 Perform the steps shown.

[0119] See Figure 10 This document illustrates another detailed flowchart of how, in an embodiment of this application, a second array is determined based on the pitch period of audio frames in the audio data. Specifically, it includes steps 324 to 327:

[0120] Step 324: Calculate the average pitch period of each audio frame of the audio data.

[0121] Step 325: Determine audio frames with the same average pitch period and consecutive frame numbers from the audio data, and use them as a speech audio frame sequence.

[0122] Step 326: Identify audio frames in the audio data that have different average pitch periods and consecutive frame numbers as non-speech audio frame sequences.

[0123] Step 327: Generate a second array based on the start and end frame numbers of the speech audio frame sequence and the start and end frame numbers of the non-speech audio frame sequence.

[0124] Before step 327 as shown in 10, a speech audio frame sequence with fewer than a preset threshold number of audio frames can also be adjusted to a non-speech audio frame sequence.

[0125] In this application, the distribution of speech audio frames and non-speech audio frames in the audio data can be determined by the second array, thereby avoiding the influence of non-speech audio frames (i.e., sounds other than speech) on speech segmentation processing, improving the accuracy of speech segmentation in audio data, and further improving the accuracy of speech recognition of audio data.

[0126] Continue to refer to Figure 3 In step 330, the target segmentation points of each target speech audio frame sequence in the audio data are determined by the first array and the second array.

[0127] In one embodiment of this application, the target segmentation points of each target speech audio frame sequence in the audio data are determined using the first array and the second array, which can be done according to the following... Figure 11 Perform the steps shown.

[0128] See Figure 11 This document illustrates a detailed flowchart of how, in an embodiment of this application, target segmentation points for each target speech audio frame sequence in the audio data are determined using the first array and the second array. Specifically, it includes steps 331 to 332:

[0129] Step 331: Using the first array, determine the first segmentation point of the audio frame sequence in the audio data.

[0130] Step 332: Using the second array, determine the second segmentation point of each target speech audio frame sequence in the audio data defined by the first segmentation point, and use it as the target segmentation point.

[0131] In this embodiment, for example, a piece of audio data includes 1000 audio frames, wherein the first array is "[01 0], [0 100 900], [100 900 1000]", and the second array is "[0 1 1 1 1 1 0], [0 50 100 400 600 800 900], [50 100 400 600 800 900 1000]".

[0132] First, the first dividing point of the audio frame sequence in the audio data is determined to be the 100th audio frame and the 900th audio frame (that is, the sequence between the 100th audio frame and the 900th audio frame is the audio frame sequence).

[0133] Then, in the audio data defined by the sequence between the 100th and 900th audio frames, the second segmentation point of each target speech audio frame sequence is determined as the target segmentation point, namely, the 100th, 400th, 600th, 800th, and 900th audio frames, for a total of 5 target segmentation points. Specifically, the 100th and 400th audio frames define one target speech audio frame sequence, the 400th and 600th audio frames define another, the 600th and 800th audio frames define yet another, and the 800th and 900th audio frames define yet another.

[0134] As can be seen, in this application, by using a first array to characterize the distribution of spoken audio frames in the audio data and a second array to characterize the distribution of speech audio frames in the audio data, the target segmentation points of each target speech audio frame sequence in the audio data are determined. This avoids the influence of environmental noise on the determination of the target speech audio frame sequence in the audio data, and also avoids the influence of non-speech audio in the environment on the determination of the target speech audio frame sequence in the audio data. This improves the accuracy of speech segmentation in audio data, reduces the probability of speech misrecognition, improves the accuracy of speech recognition of audio data, and enhances the user interaction experience.

[0135] To enable those skilled in the art to better understand this application, the following will be combined with Figure 12This application will be described in its entirety.

[0136] See Figure 12 The diagram 1200 shows the overall flowchart of the speech segmentation method in the embodiments of this application.

[0137] like Figure 12 As shown, audio data is detected using cepstral endpoint detection and endpoint detection VAD, respectively. In endpoint detection VAD, it is determined whether the short-time energy of audio frames in the audio data exceeds a preset energy value. Audio frames exceeding the preset energy value are marked as "1", and those not exceeding the preset energy value are marked as "0", further obtaining a first array to represent the sequence of spoken audio frames. In cepstral endpoint detection, it is determined whether the pitch of audio frames in the audio data has periodicity. Audio frames with periodic pitch are marked as "1", and those without periodic pitch are marked as "0", further obtaining a second array to represent the sequence of speech audio frames. Then, based on the first and second arrays, a target speech audio frame sequence with a certain number of frames is determined. Finally, based on the target speech audio frame sequence with a certain number of frames, speech segmentation is performed on the audio data.

[0138] The following describes an apparatus embodiment of this application, which can be used to execute the speech segmentation method of the first aspect of the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the speech segmentation method of the first aspect of this application.

[0139] See Figure 13 The diagram shows a block diagram of the speech segmentation device in an embodiment of this application.

[0140] like Figure 13 As shown, the speech segmentation device 1300 according to an embodiment of this application includes: an acquisition unit 1301, a determination unit 1302, and a segmentation unit 1303.

[0141] The acquisition unit 1301 is used to acquire the audio data to be segmented; the determination unit 1302 is used to determine the target segmentation point of each target speech audio frame sequence in the audio data based on the short-time energy of the audio frames in the audio data and the pitch period of the audio frames in the audio data; and the segmentation unit 1303 is used to segment each target speech audio frame sequence in the audio data according to the target segmentation point.

[0142] In some embodiments of this application, based on the foregoing scheme, the determining unit 1302 includes: a first determining unit, configured to determine a first array based on the short-time energy of audio frames in the audio data, the first array being used to characterize the distribution of spoken audio frames in the audio data; a second determining unit, configured to determine a second array based on the pitch period of audio frames in the audio data, the second array being used to characterize the distribution of speech audio frames in the audio data; and a third determining unit, configured to determine target segmentation points for each target speech audio frame sequence in the audio data using the first array and the second array.

[0143] In some embodiments of this application, based on the foregoing scheme, the first determining unit is configured to: mark audio frames with short-term energy higher than a preset energy value as having a sound tag in the audio data; mark audio frames with short-term energy lower than or equal to the preset energy value as having a silent tag in the audio data; and determine a first array based on the sound tags and the silent tags.

[0144] In some embodiments of this application, based on the foregoing scheme, the first determining unit is further configured to generate a first array based on the audio tag or silent tag of each audio frame and the frame number of each audio frame in the audio data.

[0145] In some embodiments of this application, based on the foregoing scheme, the first determining unit is further configured to: determine, in the audio data, audio frames corresponding to the audible tags and with consecutive frame numbers, as an audible audio frame sequence; determine, in the audio data, audio frames corresponding to the silent tags and with consecutive frame numbers, as a silent audio frame sequence; and generate a first array based on the start and end frame numbers of the audible audio frame sequence and the start and end frame numbers of the silent audio frame sequence.

[0146] In some embodiments of this application, based on the foregoing scheme, the second determining unit is configured to: label audio frames with periodic pitches in the audio data as speech tags; label audio frames without periodic pitches in the audio data as non-speech tags; and determine a second array based on the speech tags and the non-speech tags.

[0147] In some embodiments of this application, based on the foregoing scheme, the second determining unit is further configured to generate a second array based on the speech tag or non-speech tag of each audio frame and the frame number of each audio frame in the audio data.

[0148] In some embodiments of this application, based on the foregoing scheme, the second determining unit is further configured to: determine audio frames in the audio data that correspond to voice tags and have consecutive frame numbers, as a voice audio frame sequence; determine audio frames in the audio data that do not correspond to voice tags and have consecutive frame numbers, as a non-voice audio frame sequence; and generate a second array based on the start and end frame numbers of the voice audio frame sequence and the start and end frame numbers of the non-voice audio frame sequence.

[0149] In some embodiments of this application, based on the foregoing scheme, the average pitch period of each audio frame in the speech audio frame sequence is the same.

[0150] In some embodiments of this application, based on the foregoing scheme, the second determining unit is further configured to: before generating the second array based on the start frame number and end frame number of the speech audio frame sequence and the start frame number and end frame number of the non-speech audio frame sequence, adjust the speech audio frame sequence with fewer than a preset threshold to a non-speech audio frame sequence.

[0151] In some embodiments of this application, based on the foregoing scheme, each audio frame in the audio data includes multiple pitch periods, and the second determining unit is configured to: calculate the average pitch period of each audio frame of the audio data; determine audio frames in the audio data with the same average pitch period and consecutive frame numbers as a speech audio frame sequence; determine audio frames in the audio data with different average pitch periods and consecutive frame numbers as a non-speech audio frame sequence; and generate a second array based on the start and end frame numbers of the speech audio frame sequence and the start and end frame numbers of the non-speech audio frame sequence.

[0152] In some embodiments of this application, based on the foregoing scheme, the third determining unit is configured to: determine a first segmentation point of the audio frame sequence in the audio data using the first array; and determine a second segmentation point of each target speech audio frame sequence in the audio data defined by the first segmentation point using the second array, as the target segmentation point.

[0153] Based on the same inventive concept, this application also provides a household appliance, see reference. Figure 14 The diagram shows a structural schematic of a home appliance in an embodiment of this application. The home appliance includes one or more memories 1404, one or more processors 1402, and at least one computer program (program code) stored in the memory 1404 and executable on the processor 1402. When the processor 1402 executes the computer program, it implements the speech segmentation method as described above.

[0154] Among them, Figure 14 In this document, a bus architecture (represented by bus 1400) is used. Bus 1400 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 1402 and memory represented by memory 1404. Bus 1400 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 1405 provides an interface between bus 1400 and receiver 1401 and transmitter 1403. Receiver 1401 and transmitter 1403 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 1402 is responsible for managing bus 1400 and general processing, while memory 1404 can be used to store data used by processor 1402 during operation.

[0155] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored as one or more instructions or codes on or transmitted via a computer-readable medium. Other examples and embodiments are within the scope and spirit of this application and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hardwired, or any combination thereof. Furthermore, the functional units may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0157] The units described as separate components may or may not be physically separate. Similarly, the components of the control device may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0158] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0159] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A speech segmentation method, characterized in that, The method includes: Obtain the audio data to be segmented; Based on the short-time energy of the audio frames in the audio data, a first array is determined, which is used to characterize the distribution of the audio frames in the audio data. Based on the pitch period of the audio frames in the audio data, a second array is determined, which is used to characterize the distribution of speech audio frames in the audio data; Using the first array and the second array, the target segmentation point of each target speech audio frame sequence in the audio data is determined. The target speech audio frame sequence refers to a sequence of consecutive frame numbers composed of several audio frames. One target speech audio frame sequence corresponds to the speech of one word spoken by the user. The audio data is segmented into individual target speech audio frame sequences according to the target segmentation points.

2. The method according to claim 1, characterized in that, The determination of the first array based on the short-time energy of audio frames in the audio data includes: Audio frames with short-term energy values ​​higher than a preset energy value are marked with audio tags in the audio data; Audio frames with short-term energy values ​​lower than or equal to a preset energy value are marked with a silent tag in the audio data; Based on the audio tags and the silent tags, a first array is determined.

3. The method according to claim 2, characterized in that, The determination of the first array based on the audio tags and the silent tags includes: A first array is generated based on the audio tag or silent tag of each audio frame and the frame number of each audio frame in the audio data.

4. The method according to claim 3, characterized in that, The first array, generated based on the audio tags or silent tags of each audio frame and the frame number of each audio frame in the audio data, includes: In the audio data, determine the audio frames corresponding to the audio tag and whose frame numbers are consecutive, and use them as the audio frame sequence. In the audio data, identify the audio frames with consecutive frame numbers corresponding to the silent tag, and use them as the silent audio frame sequence. A first array is generated based on the start and end frame numbers of the audio frame sequence and the start and end frame numbers of the silent audio frame sequence.

5. The method according to claim 1, characterized in that, The step of determining the second array based on the pitch period of the audio frames in the audio data includes: Audio frames with periodic pitches are labeled with speech tags in the audio data; Audio frames that do not have a periodic pitch are marked with non-speech tags in the audio data; A second array is determined based on the voice tags and the non-voice tags.

6. The method according to claim 5, characterized in that, The step of determining the second array based on the speech tags and the non-speech tags includes: A second array is generated based on the speech or non-speech tags of each audio frame and the frame number of each audio frame in the audio data.

7. The method according to claim 6, characterized in that, The second array, generated based on the speech or non-speech tags of each audio frame and the frame number of each audio frame in the audio data, includes: In the audio data, determine the audio frames with consecutive frame numbers corresponding to the speech tags, and use them as the speech audio frame sequence. In the audio data, determine the audio frames that correspond to non-speech tags and have consecutive frame numbers, and use them as the non-speech audio frame sequence. A second array is generated based on the start and end frame numbers of the speech audio frame sequence and the start and end frame numbers of the non-speech audio frame sequence.

8. The method according to claim 7, characterized in that, The average pitch period of each audio frame in the speech audio frame sequence is the same.

9. The method according to claim 7, characterized in that, Before generating the second array based on the start and end frame numbers of the speech audio frame sequence and the start and end frame numbers of the non-speech audio frame sequence, the method further includes: Speech audio frame sequences with fewer than a preset threshold number of audio frames are adjusted to non-speech audio frame sequences.

10. The method according to claim 1, characterized in that, Each audio frame in the audio data includes multiple pitch periods, and the determination of the second array based on the pitch periods of the audio frames in the audio data includes: Calculate the average pitch period of each audio frame of the audio data; Audio frames with the same average pitch period and consecutive frame numbers are identified from the audio data and used as a speech audio frame sequence. Audio frames with different average pitch periods and consecutive frame numbers are identified from the audio data and are used as a non-speech audio frame sequence. A second array is generated based on the start and end frame numbers of the speech audio frame sequence and the start and end frame numbers of the non-speech audio frame sequence.

11. The method according to claim 1, characterized in that, The step of determining the target segmentation points for each target speech audio frame sequence in the audio data using the first array and the second array includes: The first segmentation point of the audio frame sequence is determined in the audio data using the first array. Using the second array, a second segmentation point is determined in the audio data defined by the first segmentation point, which serves as the target segmentation point.

12. A speech segmentation device, characterized in that, The device includes: The acquisition unit is used to acquire the audio data to be segmented; The determining unit is configured to determine a first array based on the short-time energy of audio frames in the audio data, the first array being used to characterize the distribution of spoken audio frames in the audio data; and to determine a second array based on the pitch period of audio frames in the audio data, the second array being used to characterize the distribution of speech audio frames in the audio data; and to determine target segmentation points for each target speech audio frame sequence in the audio data using the first array and the second array, wherein the target speech audio frame sequence refers to a sequence of consecutively numbered frame sequences composed of several audio frames, and one target speech audio frame sequence corresponds to the speech of one word spoken by the user; The segmentation unit is used to segment the audio data into individual target speech audio frame sequences according to the target segmentation point.

13. A household appliance, characterized in that, The method includes one or more processors and one or more memories, wherein at least one piece of program code is stored in the one or more memories, and the at least one piece of program code is loaded and executed by the one or more processors to implement the method as claimed in any one of claims 1-11.

Citation Information

Patent Citations

  • Voice endpoint detection method and device, computer equipment and storage medium

    CN110931048A