Human-computer interaction method and system for speech recognition

Abstract

The present application relates to the field of data processing technology, and in particular to a human-computer interaction method and system for speech recognition. The method includes: performing short-time Fourier transform on audio data within a target time period to obtain a spectrum diagram, and gray-scaling the spectrum diagram to obtain a grayscale image; screening out multiple target horizontal areas from the grayscale image; sliding the grayscale image using a preset sliding window to determine a target time domain interval from the time period of the grayscale image; determining an initial spectrum area whose frequency value is within a predetermined fundamental frequency range from the grayscale image, and determining the target spectrum area where the human voice is located based on the initial spectrum area, the target time domain interval, and the multiple target horizontal areas, so as to perform human-computer interaction based on the target spectrum area. Through the above technical solution, the efficiency of voice interaction between users and electronic devices can be guaranteed.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a human-computer interaction method and system for speech recognition. Background Art

[0002] By receiving voice commands from users, electronic devices can achieve human-computer interaction between the electronic devices and users; for example, through voice commands input by users, users can control furniture equipment such as air conditioners, lamps, and curtains; or, students can interact with electronic devices through voice, and the electronic devices can provide voice teaching or oral practice for students, enabling students to learn in a self-service manner; or, users can interact with electronic devices through voice to query information in a relatively portable manner.

[0003] When a user is interacting with an electronic device through voice, there may be some ambient noise in the environment in which the electronic device is located; this may make it difficult for the electronic device to recognize the user's voice commands from the received audio, thereby affecting the efficiency and experience of the user's human-computer interaction with the electronic device.

[0004] For example, a user may need to repeatedly input the same voice command before the electronic device can successfully recognize the user's voice command from the received audio; or, based on the received audio, the electronic device may recognize one of the user's voice commands as another different voice command, affecting the user's interaction efficiency or operating experience.

[0005] Under the influence of ambient noise, the efficiency or experience of voice interaction between users and electronic devices is limited. Therefore, it is necessary to process the audio received by the electronic device so that the processed audio can better reflect the user's voice commands, thereby ensuring the efficiency of voice interaction between users and electronic devices. Summary of the Invention

[0006] To ensure the efficiency of voice interaction between users and electronic devices, the present application provides a human-computer interaction method and system for voice recognition.

[0007] According to a first aspect of an embodiment of the present application, a human-computer interaction method for speech recognition is provided, comprising: obtaining audio data received within a target time period, performing a short-time Fourier transform on the audio data to obtain a spectrogram, and gray-scaling the spectrogram to obtain a grayscale image; dividing the grayscale image into multiple horizontal regions of equal height, and screening multiple concentrated target horizontal regions from a set of horizontal regions using the concentration characteristics of adjacent horizontal regions and the average grayscale values ​​of the horizontal regions; the width of the horizontal region is equal to the width of the grayscale image; sliding a preset sliding window in the grayscale image, and when the average grayscale value of the image region where the preset sliding window is located is less than a preset grayscale threshold, using the time period where the image region is located as a candidate time period, thereby determining a target time domain interval based on the multiple candidate time periods; determining an initial spectrum region from the grayscale image whose frequency value is within a predetermined fundamental frequency range, and determining a target spectrum region where human voice is located based on the initial spectrum region, the target time domain interval, and the multiple target horizontal regions, thereby performing human-computer interaction based on the target spectrum region.

[0008] In this way, by processing the audio data of the target time period, a target spectrum area that avoids the influence of environmental noise can be obtained. Therefore, when human-computer interaction is performed according to the target spectrum area, the accuracy of human-computer interaction between the user and the electronic device can be guaranteed.

[0009] Optionally, the concentration characteristics of adjacent horizontal regions and the average grayscale value of the horizontal regions are used to screen out multiple concentrated target horizontal regions from the horizontal region set, including: using the average grayscale value of the horizontal regions to perform binary classification on the horizontal regions, and obtaining a set of horizontal regions corresponding to the category with the largest average grayscale value in the binary classification results; based on the vertical concentration characteristics of the horizontal regions in the horizontal region set, multiple horizontal regions clustered in the horizontal region set in the vertical direction are used as multiple target horizontal regions.

[0010] Optionally, the target time domain interval is determined based on multiple candidate time periods, including: determining multiple time period combinations from multiple candidate time periods based on the time interval between two adjacent candidate time periods; including multiple candidate time periods with similar time intervals between two adjacent candidate time periods in the same time period combination; and taking the time domain interval corresponding to the time period combination with the largest time span among the multiple time period combinations as the target time domain interval.

[0011] In this way, the target time domain interval for the user to speak can be filtered out from the target time period based on the characteristics of the user's breathing when inputting voice, thereby avoiding the influence of the time period when the user is not speaking or breathing.

[0012] Optionally, the target spectrum area where the human voice is located is determined based on the initial spectrum area, the target time domain interval and multiple target horizontal areas, including: taking the image area in the multiple target horizontal areas of the grayscale image located in the initial spectrum area as the candidate image area; taking the image area in the candidate image area whose time domain is located in the target time domain interval as the target spectrum area where the human voice is located.

[0013] In this way, the target spectrum region can be determined simply and effectively, and the influence of environmental noise can be avoided as much as possible.

[0014] Optionally, the target spectrum area where the human voice is located is determined based on the initial spectrum area, the target time domain interval and multiple target horizontal areas, including: determining the background sound area from the grayscale image based on the initial spectrum area, the target time domain interval and multiple target horizontal areas; the background sound area does not overlap with the initial spectrum area or the multiple target horizontal areas, and the background sound area is located in time domain intervals other than the target time domain interval; based on the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, determining the target spectrum area where the human voice is located from the image area overlapping with at least one of the initial spectrum area, the target time domain interval and the multiple target horizontal areas.

[0015] Optionally, based on the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, the target spectrum area where the human voice is located is determined from the image area overlapping with the initial spectrum area, the target time domain interval and at least one of the multiple target horizontal areas, including: taking the image area where the target time domain interval in the grayscale image is located as the first image area, and forming the multiple target horizontal areas into a second image area; for the target pixel point in the grayscale image, taking the number of times the target pixel point is included in the first image area, the initial spectrum area and the second image area as the overlap of the target pixel point; based on the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, selecting pixel points with matching overlap to form the target spectrum area.

[0016] In this way, according to the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, pixel points with matching overlap are selected to form the target spectrum area, so that the obtained target spectrum area can match the influence degree of the environmental noise.

[0017] Optionally, based on the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, pixel points with matching overlap are selected to form a target spectrum area, including: determining a matching target overlap among multiple overlaps based on a ratio interval of the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area; and forming the target spectrum area with pixel points in the grayscale image with an overlap greater than or equal to the target overlap.

[0018] Optionally, performing human-computer interaction according to the target spectrum region includes: performing short-time inverse Fourier transform on the target spectrum region to obtain a target audio signal, and parsing the target audio signal to obtain an audio instruction, so as to perform human-computer interaction according to the audio instruction.

[0019] In this way, since the target spectrum area avoids the influence of environmental noise, the target spectrum area is subjected to short-time inverse Fourier transform to obtain the target audio signal to achieve human-computer interaction, which can improve the efficiency or experience of human-computer interaction between the user and the electronic device.

[0020] Optionally, the method also includes: dividing the target horizontal area into multiple horizontal segments obtained by using a preset time length, for the target horizontal segment among the multiple horizontal segments; taking the ratio of the average grayscale value of the target horizontal segment to the average grayscale value of the image area with the same time period in the grayscale image and a frequency value within a predetermined fundamental frequency range as the characteristic value of the target horizontal segment; and reorganizing the horizontal segments in the target horizontal area whose characteristic values ​​are greater than or equal to a preset threshold into the target horizontal area.

[0021] According to a second aspect of an embodiment of the present application, a human-computer interaction system for speech recognition is provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the steps of the human-computer interaction method for speech recognition provided in the first aspect of the present application are implemented.

[0022] The technical solution provided by the embodiments of the present application may include the following beneficial effects: obtaining the audio data received within the target time period, performing short-time Fourier transform on the audio data to obtain a spectrum diagram, and gray-scaling the spectrum diagram to obtain a grayscale image; determining multiple target horizontal areas from the grayscale image, and the frequency characteristics and energy characteristics of the obtained target horizontal areas can be more consistent with the characteristics of human voice; using a preset sliding window to slide in the grayscale image to determine the target time domain interval, and the target time domain interval can be more consistent with the time domain characteristics of human voice; based on the initial spectrum area, the target time domain interval and multiple target horizontal areas, the target spectrum area obtained can be more consistent with the characteristics of human voice; performing human-computer interaction based on the target spectrum area can better avoid the interference of ambient sound on the human-computer interaction process, thereby improving the efficiency and accuracy of human-computer interaction.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart showing a human-computer interaction method for speech recognition according to an exemplary embodiment;

[0025] Figure 2 The figure is a schematic structural diagram of a human-computer interaction system for speech recognition according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] First, a brief introduction is given to the application scenarios of the embodiments of the present application. In the application scenarios of the present application, the user can speak a sound carrying voice commands, and the electronic device can parse the voice commands from the sound emitted by the user, thereby realizing human-computer interaction between the user and the electronic device.

[0027] However, when there is ambient noise in the scene where the user is speaking, it may be difficult for the electronic device to parse the audio containing the user's voice and the ambient sound to identify the user's intention. Figure 1 The incorrect voice commands may cause a certain deviation between the operation performed by the electronic device and the operation the user wants the electronic device to perform. The user may need to increase the volume and input the voice again to enable the electronic device to perform the desired operation, which affects the user's experience or efficiency of human-computer interaction with the electronic device.

[0028] In order to solve the above technical problems, the present invention provides a human-computer interaction method and system for speech recognition. Figure 1 is a flowchart of a human-computer interaction method for speech recognition according to an exemplary embodiment, which can be executed by an electronic device, such as Figure 1 As shown, the method includes the following steps.

[0029] In step S101, audio data received within a target time period is acquired, short-time Fourier transform is performed on the audio data to obtain a spectrogram, and grayscale processing is performed on the spectrogram to obtain a grayscale image.

[0030] The electronic device can obtain audio data within the target time period through the set audio receiving module; the target time period can be a time period with the current moment as the end moment, or it can be a time period before the current moment; the length of the target time period can be specified in advance or in real time according to the actual needs of the user, and the embodiment of the present application does not impose any restrictions on the specific length of the target time period.

[0031] Electronic devices with which users interact with computers through voice may include, for example, mobile phones, tablet computers, televisions, guidance devices, and teaching terminals.

[0032] Short-time Fourier transform (STFT) can process audio data. In the spectrum graph obtained by performing STFT on the audio data within the target time period, the horizontal axis of the spectrum graph represents time, the vertical axis of the spectrum graph represents the different frequencies of the audio signal, and the color or brightness at different positions in the spectrum graph represents the energy or amplitude of the audio data at the corresponding time and frequency.

[0033] In a scenario where a user interacts with an electronic device through voice, the user's voice and the ambient noise differ at least in frequency distribution or energy. Therefore, by converting the audio data within a target time period into a spectrum graph, it is possible to avoid interference from ambient noise in the audio data acquired by the electronic device.

[0034] By gray-scaling the spectrum graph to obtain a grayscale image, the frequency information, time domain information of the audio data and the energy information of the audio signal can be retained in the obtained grayscale image, and compared with the color spectrum image, the amount of calculation required for the operation process can be reduced.

[0035] In step S102 , the grayscale image is divided into multiple horizontal regions of the same height, and multiple concentrated target horizontal regions are screened from the horizontal region set using the concentration characteristics of adjacent horizontal regions and the average grayscale value of the horizontal regions.

[0036] The grayscale image is obtained by grayscale processing the spectrogram. Therefore, the longitudinal or vertical direction of the grayscale image corresponds to the frequency change of the spectrogram, so that the height change in the grayscale image corresponds to the change of the audio signal in the same time domain in the spectrogram at different frequencies.

[0037] The height of the lateral area may correspond to a preset frequency span; for example, the preset frequency span may be equal to 50 Hz, so that the height of the lateral area may correspond to a frequency span of 50 Hz.

[0038] Compared with the ambient noise in the user's environment, the frequency range of the user's voice is more concentrated; and compared with the ambient noise in the user's environment, the energy or amplitude of the user's voice has better continuity. Therefore, the difference in audio characteristics between ambient noise and human voice can be used to screen out multiple target horizontal areas from the horizontal area set.

[0039] In one embodiment, the concentration characteristics of adjacent horizontal regions and the average grayscale value of the horizontal regions are used to screen out a plurality of concentrated target horizontal regions from a horizontal region set, including: using the average grayscale value of the horizontal regions to perform binary classification on the horizontal regions, and obtaining a set of horizontal regions corresponding to the category with the largest average grayscale value in the binary classification result; based on the vertical concentration characteristics of the horizontal regions in the horizontal region set, a plurality of horizontal regions clustered in the vertical direction in the horizontal region set are used as a plurality of target horizontal regions.

[0040] When a user interacts with an electronic device through voice, the accuracy of the human-computer interaction may be affected by ambient noise with lower energy, and the average grayscale value of the horizontal area can reflect the average energy level of the audio in the horizontal area. Therefore, by using the average grayscale value of the horizontal area to perform binary classification on the horizontal area, multiple horizontal areas can be classified into categories corresponding to ambient noise with lower energy and categories corresponding to human voices with higher energy.

[0041] Multiple horizontal regions corresponding to categories with larger average grayscale values ​​in the binary classification results can be combined into a horizontal region set; in the obtained horizontal region set, since the frequency range of environmental noise has higher randomness and dispersion, the horizontal regions corresponding to environmental noise in the horizontal region set are more dispersed in the vertical axis direction of the grayscale image, and the horizontal regions corresponding to human voices in the horizontal region set are more concentrated in the vertical axis direction of the grayscale image. Therefore, the vertical concentration characteristics of the horizontal regions can be used to achieve a preliminary distinction between environmental noise and human voices.

[0042] Based on the longitudinal concentration characteristics of the lateral regions in the lateral region set, multiple lateral regions clustered in the longitudinal direction in the lateral region set are used as multiple target lateral regions, which can include: determining the interval distance between two lateral regions adjacent in the longitudinal direction in the lateral region set; clustering the lateral regions in the lateral region set based on the interval distance between two lateral regions adjacent in the longitudinal direction to obtain multiple lateral region clusters; and the lateral regions in the cluster with the smallest interval distance between two lateral regions adjacent in the longitudinal direction can be used as multiple target lateral regions.

[0043] For example, the horizontal area set may include 9 horizontal areas A1, A2, A3, A4, A5, A6, A7, A8 and A9 with ascending frequency ranges; if based on the interval distance between two adjacent horizontal areas in the vertical direction, it can be determined that the horizontal areas A1, A2, A3, A4 and A5 are concentrated, while A6, A7, A8 and A9 are more dispersed in the vertical direction, then the horizontal areas A1, A2, A3, A4 and A5 can be used as the determined multiple target horizontal areas.

[0044] In this way, multiple target horizontal regions that are more consistent with human voices can be screened out from all horizontal regions of the grayscale image through the concentration characteristics of the horizontal regions on the vertical axis and the average grayscale value of the horizontal regions.

[0045] In step S103, a preset sliding window is used to slide in the grayscale image. When the average grayscale value of the image area where the preset sliding window is located is less than the preset grayscale threshold, the time period where the image area is located is used as a candidate time period to determine the target time domain interval based on multiple candidate time periods.

[0046] The duration of the preset sliding window can be specified in advance or in real time, or it can be determined based on the duration of a single voice command when the user previously inputs voice; the step size of the preset sliding window when sliding in the grayscale image is less than or equal to the duration of the preset sliding window.

[0047] Within the time period of the audio acquired by the electronic device, there may be some sub-time periods in which the user performs voice input, and there may also be some sub-time periods in which the user does not perform voice input. In the grayscale image, the energy of the audio when the user performs voice input is usually higher than the energy of the audio when the user does not perform voice input.

[0048] By using the average grayscale value of the image area where the preset sliding window is located, it can be determined whether the user has made a voice input within the time period corresponding to the image area where the preset sliding window is located, so as to better determine the time period when the user actually made a voice input.

[0049] When the average grayscale value of the image area where the preset sliding window is located is less than the preset grayscale threshold, there is a high probability that the user did not make any voice input during the time period where the image area is located. When the user did not make any voice input, it may be that the user was in the breathing time period for voice input.

[0050] Compared with environmental noise, which has more random duration and frequency, the user's breathing time interval when performing voice input is more consistent. Or, in order to ensure that the user can complete breathing during voice input, the time intervals between adjacent voice commands entered are usually more consistent when performing relatively continuous voice input.

[0051] For example, when users practice speaking through electronic devices, they may read aloud or recite paragraphs. In order to ensure sufficient breathing time intervals, users usually breathe intermittently while reading. The fact that the breathing time intervals between users' voices are highly consistent can be used to distinguish between environmental noise and human voices in the time domain.

[0052] The preset grayscale threshold may be determined based on ranking information of pixels in the grayscale image. For example, the preset grayscale threshold may be equal to the grayscale value of a pixel whose grayscale value ranks at 50% in the grayscale image.

[0053] The average grayscale value of the image area where the preset sliding window is located is less than the preset grayscale threshold, which may mean that the average grayscale value of the image area where the preset sliding window is located is less than the preset value after normalization; the normalization process can be achieved by dividing the average grayscale value by the maximum grayscale value in the grayscale image, and the preset value can be equal to 0.5, for example.

[0054] In one embodiment, a target time domain interval is determined based on multiple candidate time periods, including: determining multiple time period combinations from multiple candidate time periods based on the time interval between two adjacent candidate time periods; including multiple candidate time periods with similar time intervals between two adjacent candidate time periods in the same time period combination; and taking the time domain interval corresponding to the time period combination with the largest time span among the multiple time period combinations as the target time domain interval.

[0055] Since the time intervals between the time periods when the user takes a breath are more consistent, multiple candidate time periods with similar time intervals and adjacent to each other can be determined into the same time period combination based on the time interval between two adjacent candidate time periods; among all time period combinations, the time period combination with the largest time span is more likely to correspond to the complete time period for the user to perform voice input.

[0056] The time domain interval corresponding to the time period combination with the largest time span among multiple time period combinations is used as the target time domain interval. The target time domain interval can reduce the time periods in which the user does not perform voice input while including the time periods in which the user actually performs voice input as much as possible, so as to improve the efficiency of human-computer interaction through voice.

[0057] In this way, based on the obtained multiple candidate time periods, a target time domain interval that meets the intermittent characteristics of the user's speaking can be obtained, so as to use the time domain interval to eliminate the interference of environmental noise on the human-computer interaction process.

[0058] In step S104, an initial spectrum region whose frequency value is within a predetermined fundamental frequency range is determined from the grayscale image, and a target spectrum region where the human voice is located is determined based on the initial spectrum region, the target time domain interval, and multiple target lateral regions, so as to perform human-computer interaction based on the target spectrum region.

[0059] The predetermined fundamental frequency range is determined based on the frequency range of the human voice when the user speaks; for example, the frequency range of the human voice is mainly concentrated between 85Hz and 400Hz, and depending on the age group or gender of the user, the fundamental frequency of different users may be in different intervals within the predetermined fundamental frequency range.

[0060] For example, the fundamental frequency of an adult male's voice is usually between 85Hz and 180Hz; the fundamental frequency of an adult female's voice is usually between 165Hz and 255Hz; and the fundamental frequency of a child's voice is usually between 250Hz and 400Hz.

[0061] Compared with the frequency range of the human voice when the user is speaking, the frequency range of the ambient noise has a stronger uncertainty. Therefore, using the predetermined fundamental frequency range can better eliminate the interference of the ambient noise outside the frequency range of the human voice.

[0062] Since the initial spectrum area is determined based on a predetermined fundamental frequency range that conforms to the frequency range of human voice, the target time domain interval is the time period when the user speaks, determined based on the characteristics of human voice breathing, and the multiple target horizontal areas are determined based on the concentration characteristics of the frequency band of human voice and the energy difference with the ambient noise, therefore, the initial spectrum area, the target time domain interval and the multiple target horizontal areas can reflect the different characteristics of the human voice features in the spectrum image.

[0063] Since the initial spectrum area, target time domain interval and multiple target horizontal areas can reflect the different characteristics of the human voice features in the spectrum image, the target spectrum area where the human voice is located is determined based on the initial spectrum area, target time domain interval and multiple target horizontal areas. The obtained target spectrum area can be more consistent with the spectrum area when the user inputs voice, thereby achieving the elimination of the spectrum area of ​​environmental noise.

[0064] Since the obtained target spectrum area can better avoid the influence of environmental noise, when using the target spectrum area for human-computer interaction, it can improve the accuracy and efficiency of the user's human-computer interaction with electronic devices through voice, thereby improving the user's experience of human-computer interaction with electronic devices through voice.

[0065] In one embodiment, a target spectrum region where the human voice is located is determined based on an initial spectrum region, a target time domain interval, and multiple target horizontal regions, including: taking an image region in the multiple target horizontal regions of a grayscale image located in the initial spectrum region as a candidate image region; and taking an image region in the candidate image region whose time domain is located in the target time domain interval as the target spectrum region where the human voice is located.

[0066] The multiple target horizontal areas obtained can avoid environmental noise with relatively sparse energy distribution in the audio data of the target time period, or can avoid noise areas with highly random frequency distribution, and the initial spectrum area is determined according to the frequency range of human voice. Therefore, the candidate image area can avoid the influence of environmental noise that may exist in the target time period from the two dimensions of frequency and spectrum distribution.

[0067] Since the target time domain interval is a time period that matches the time interval characteristics of human voice ventilation, the image area in the candidate image area whose time domain is within the target time domain interval is used as the target spectrum area where the human voice is located. This can further avoid the influence of possible ambient noise in the target time period from the dimension of time period, and achieve more accurate screening of the spectrum area of ​​the human voice.

[0068] In this way, the three features of frequency, time period and spectrum distribution in the grayscale image of the corresponding spectrogram can be comprehensively considered to obtain a target spectrum region that is more consistent with the human voice.

[0069] In one embodiment, a target spectrum region where a human voice is located is determined based on an initial spectrum region, a target time domain interval, and multiple target lateral regions, including: determining a background sound region from a grayscale image based on the initial spectrum region, the target time domain interval, and multiple target lateral regions; the background sound region does not overlap with the initial spectrum region or the multiple target lateral regions, and the background sound region is located in time domain intervals other than the target time domain interval; and determining the target spectrum region where the human voice is located from an image region overlapping with at least one of the initial spectrum region, the target time domain interval, and the multiple target lateral regions based on a ratio of a total area of ​​the grayscale image to an area of ​​the background sound region.

[0070] The background sound area does not overlap with the initial spectrum area or multiple target lateral areas, and the background sound area is located in time domain intervals other than the target time domain interval. Therefore, the background sound area is more likely to correspond to the environmental noise that may exist in the target time period.

[0071] Since the background sound area corresponds to the environmental noise that may exist in the target time period, and the total area of ​​the grayscale image corresponds to all the audio in the target time period, the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area can reflect the extent to which the human-computer interaction process in the target time period is not affected by environmental noise.

[0072] The target spectrum area where the human voice is located is determined based on the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area. This can take into account the degree to which the voice is not affected by environmental noise during the target time period and adaptively determine the matching target spectrum area to ensure the accuracy of human-computer interaction through voice.

[0073] In one embodiment, based on the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, a target spectrum area where the human voice is located is determined from an image area overlapping with at least one of the initial spectrum area, the target time domain interval, and a plurality of target lateral areas, including: taking the image area where the target time domain interval in the grayscale image is located as the first image area, and forming the plurality of target lateral areas into a second image area; for a target pixel point in the grayscale image, taking the number of times the target pixel point is included in the first image area, the initial spectrum area, and the second image area as the overlap of the target pixel point; and based on the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, selecting pixels with matching overlap to form the target spectrum area.

[0074] The target pixel point can be any pixel point in the grayscale image; since the first image area, the initial spectrum area, and the second image area correspond to three image areas in the grayscale image, respectively, and there may be overlap between these three image areas, the coincidence degree of the target pixel point may be any one of 0, 1, 2, and 3.

[0075] The greater the overlap of the target pixel points, the more the time period, frequency, and spectrum distribution characteristics of the local area where the target pixel points are located conform to the characteristics of human voice, and the more likely the audio information carried by the target pixel points is the key speech information of the human voice within the target time period.

[0076] The ratio of the total area of ​​the grayscale image to the area of ​​the background sound region can reflect the degree to which the target time period is not affected by environmental noise. Therefore, the larger the ratio of the total area of ​​the grayscale image to the area of ​​the background sound region, the greater the degree to which the target time period is not affected by environmental noise. This can make the target spectrum region include pixels with less overlap, and make the target spectrum region include more pixels carrying human voices, thereby avoiding the omission of voice commands.

[0077] On the contrary, the smaller the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, the greater the degree of influence of the environmental noise during the target time period. The target spectrum area can be made to include pixel points with a greater degree of overlap, and the target spectrum area can be made to include more pixel points carrying key voice information of human voice, so as to ensure the success rate of users interacting with electronic devices through voice.

[0078] In this way, according to the degree of influence of environmental noise in the target time period, pixel points with matching overlap can be adaptively selected to form the target spectrum area, so that the human-computer interaction process between the user and the electronic device can adapt to the degree of influence of environmental noise.

[0079] In one embodiment, pixel points with matching overlaps are selected to form a target spectrum region based on the ratio of the total area of ​​the grayscale image to the area of ​​the background sound region, including: determining a matching target overlap from a plurality of overlaps based on a ratio interval of the ratio of the total area of ​​the grayscale image to the area of ​​the background sound region; and forming the target spectrum region from pixel points in the grayscale image with an overlap greater than or equal to the target overlap.

[0080] Since the overlap of pixels in a grayscale image is equal to any one of 0, 1, 2, and 3; and the greater the overlap of pixels, the more likely the position of the pixel in the grayscale image and the pixel value of the pixel carry the user's key voice information; on the contrary, the smaller the overlap of pixels, the more likely the position of the pixel in the grayscale image and the pixel value of the pixel correspond to environmental noise.

[0081] Different ratio intervals can correspond to different degrees of overlap, respectively. The target overlap that matches multiple degrees of overlap can be determined based on the ratio interval of the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area and the preset correspondence. The preset correspondence is used to characterize the correspondence between different degrees of overlap and different ratio intervals.

[0082] According to the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, three ratio intervals can be divided; for example, the value range of the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area can be divided into a first ratio interval, a second ratio interval and a third ratio interval with gradually increasing values.

[0083] When the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area is within the first ratio range, it indicates that the impact of the ambient noise during the target time period is relatively large. To ensure the effect of human-computer interaction, only pixels with an overlap degree of 3 in the grayscale image may be selected to form the target spectrum area.

[0084] When the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area is within the second ratio range, it indicates that the impact of the ambient noise during the target time period is moderate. In order to ensure both the effect of human-computer interaction and the number of voice commands, pixel points with an overlap of 3 and pixel points with an overlap of 2 in the grayscale image can be selected to form the target spectrum area.

[0085] When the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area is within the third ratio interval, it indicates that the impact of the ambient noise during the target time period is relatively small. Pixels with an overlap of 3, 2, and 1 in the grayscale image can be selected to form the target spectrum area.

[0086] In this way, according to the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, pixel points with matching overlap are selected to form the target spectrum area, so that the obtained target spectrum area can adapt to the degree of influence of environmental noise in the target time period.

[0087] In one embodiment, performing human-computer interaction according to a target spectrum region includes: performing an inverse short-time Fourier transform on the target spectrum region to obtain a target audio signal, and parsing the target audio signal to obtain an audio instruction, so as to perform human-computer interaction according to the audio instruction.

[0088] The inverse short-time Fourier transform (ITF) can convert spectrum information into audio signals. The target spectrum region obtained after screening is subjected to an inverse short-time Fourier transform to obtain the target audio signal. Since most noise and other irrelevant information have been removed from the target spectrum region, the obtained target spectrum signal can more clearly reflect the user's voice.

[0089] Electronic devices can obtain audio instructions by parsing the target audio signal to perform human-computer interaction according to the audio instructions; performing human-computer interaction according to audio instructions may refer to executing operations corresponding to voice instructions, or controlling other devices to perform operations corresponding to audio instructions.

[0090] In this way, since the target audio signal can more clearly reflect the user's voice, the human-computer interaction result obtained by the electronic device can be more consistent with the user's actual voice command when performing human-computer interaction based on the target audio signal.

[0091] In one embodiment, a target horizontal region is divided into multiple horizontal segments obtained by using a preset time length, and a target horizontal segment among the multiple horizontal segments is selected; the ratio of the average grayscale value of the target horizontal segment to the average grayscale value of the image region with the same time period in the grayscale image and a frequency value within a predetermined fundamental frequency range is used as a characteristic value of the target horizontal segment; and the horizontal segments in the target horizontal region whose characteristic values ​​are greater than or equal to a preset threshold are reassembled into the target horizontal region.

[0092] An image region in the grayscale image that is within a predetermined fundamental frequency range is more likely to carry audio information of the user's voice within the target time period.

[0093] The energy of environmental noise is highly random and dispersed, so that the average grayscale value of the horizontal segment corresponding to the environmental noise is usually greater than the average grayscale value of the horizontal segment corresponding to the human voice. Therefore, the ratio of the average grayscale value of the target horizontal segment to the average grayscale value of the image area with the same time period in the grayscale image and a frequency value within the predetermined fundamental frequency range can reflect the probability that the target horizontal segment meets the energy characteristics of the human voice.

[0094] When the characteristic value of the target horizontal segment is greater than or equal to the preset threshold, it means that the target horizontal segment is more consistent with the characteristics of the human voice in the audio; the horizontal segments in the target horizontal area whose characteristic values ​​are greater than or equal to the preset threshold are reorganized into the target horizontal area, which can eliminate the horizontal segments of environmental noise other than the human voice, so that the obtained target horizontal area can better extract the audio information of the human voice in the subsequent processing steps.

[0095] Figure 2 FIG. 1 is a structural diagram of a human-computer interaction system 1000 for speech recognition according to an exemplary embodiment. Figure 2 The human-computer interaction system 1000 for speech recognition includes: a processor 1100 and a memory 1200, wherein the memory 1200 stores computer program instructions, and when the computer program instructions are executed by the processor 1100, all or part of the steps of the human-computer interaction method for speech recognition in this application are implemented.

[0096] Those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein, and the description and examples are to be considered merely as exemplary.

[0097] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A human-computer interaction method for speech recognition, characterized in that: include: Acquire audio data received within a target time period, perform short-time Fourier transform on the audio data to obtain a spectrogram, and perform grayscale processing on the spectrogram to obtain a grayscale image; The grayscale image is divided into multiple horizontal regions of the same height. The concentration characteristics of adjacent horizontal regions and the average grayscale value of the horizontal regions are used to screen multiple concentrated target horizontal regions from the horizontal region set. The width of the horizontal region is equal to the width of the grayscale image. A preset sliding window is used to slide in the grayscale image. When the average grayscale value of the image area where the preset sliding window is located is less than a preset grayscale threshold, the time period where the image area is located is used as a candidate time period, so as to determine the target time domain interval based on the multiple candidate time periods. An initial spectrum region whose frequency value is within a predetermined fundamental frequency range is determined from a grayscale image, and a target spectrum region where the human voice is located is determined based on the initial spectrum region, a target time domain interval, and multiple target lateral regions, so as to perform human-computer interaction based on the target spectrum region.

2. The human-computer interaction method for speech recognition according to claim 1, characterized in that: By using the concentration characteristics of adjacent horizontal regions and the average grayscale value of the horizontal regions, multiple concentrated target horizontal regions are screened out from the horizontal region set, including: The horizontal regions are classified into two categories using the average grayscale value of the horizontal regions, and the set of horizontal regions corresponding to the category with the largest average grayscale value in the binary classification results is obtained; According to the vertical concentration characteristics of the horizontal regions in the horizontal region set, a plurality of horizontal regions clustered in the vertical direction in the horizontal region set are used as a plurality of target horizontal regions.

3. The human-computer interaction method for speech recognition according to claim 1, characterized in that: Determine the target time domain interval based on multiple candidate time periods, including: Determine multiple time period combinations from multiple candidate time periods based on the time interval between two adjacent candidate time periods; the same time period combination includes multiple candidate time periods with similar time intervals between two adjacent candidate time periods; The time domain interval corresponding to the time period combination with the largest time span among the multiple time period combinations is used as the target time domain interval.

4. The human-computer interaction method for speech recognition according to claim 1, characterized in that: The target spectrum region where the human voice is located is determined based on the initial spectrum region, the target time domain interval, and multiple target horizontal regions, including: taking an image region located in an initial spectrum region in a plurality of target lateral regions of the grayscale image as a candidate image region; The image region in the candidate image region whose time domain is within the target time domain interval is used as the target spectrum region where the human voice is located.

5. The human-computer interaction method for speech recognition according to claim 1, characterized in that: The target spectrum region where the human voice is located is determined based on the initial spectrum region, the target time domain interval, and multiple target horizontal regions, including: Determine a background sound region from the grayscale image based on the initial spectrum region, the target time domain interval, and the multiple target lateral regions; the background sound region does not overlap with the initial spectrum region or the multiple target lateral regions, and the background sound region is located in a time domain interval other than the target time domain interval; According to the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, a target spectrum area where the human voice is located is determined from an image area overlapping with at least one of the initial spectrum area, the target time domain interval, and the plurality of target lateral areas.

6. The human-computer interaction method for speech recognition according to claim 5, characterized in that: Determining a target spectral region where the human voice is located from an image region overlapping with the initial spectral region, the target time domain interval, and at least one of the plurality of target lateral regions based on a ratio of a total area of ​​the grayscale image to an area of ​​the background sound region includes: The image region where the target time domain interval is located in the grayscale image is used as the first image region, and the plurality of target lateral regions are combined into a second image region; For a target pixel in the grayscale image, the number of times the target pixel is included in the first image area, the initial spectrum area, and the second image area is used as the coincidence degree of the target pixel; According to the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, pixels with matching overlap are selected to form the target spectrum area.

7. The human-computer interaction method for speech recognition according to claim 6, characterized in that: Based on the ratio of the total area of ​​the grayscale image to the area of ​​the background sound area, pixels with matching overlap are selected to form the target spectrum area, including: determining a matching target coincidence degree among a plurality of coincidence degrees according to a ratio interval of a ratio of a total area of ​​the grayscale image to an area of ​​the background sound region; The pixel points in the grayscale image whose overlap degree is greater than or equal to the target overlap degree form the target spectrum area.

8. The human-computer interaction method for speech recognition according to claim 1, characterized in that: Human-computer interaction based on the target spectrum area, including: The target audio signal is obtained by performing short-time inverse Fourier transform on the target spectrum area, and the target audio signal is analyzed to obtain audio instructions so as to perform human-computer interaction according to the audio instructions.

9. The human-computer interaction method for speech recognition according to claim 1, characterized in that: The method further comprises: A plurality of horizontal segments obtained by dividing the target horizontal area by a preset time length, targeting a target horizontal segment among the plurality of horizontal segments; The ratio of the average grayscale value of the target horizontal segment to the average grayscale value of the image region with the same time period and a frequency value within a predetermined fundamental frequency range in the grayscale image is used as the characteristic value of the target horizontal segment; The horizontal segments in the target horizontal region whose characteristic values ​​are greater than or equal to a preset threshold are reassembled into the target horizontal region.

10. A human-computer interaction system for speech recognition, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the human-computer interaction method for speech recognition according to any one of claims 1 to 9 is implemented.

Images