An offline recording intelligent voice recognition analysis system

Abstract

The application relates to the technical field of intelligent speech recognition and analysis, and discloses an intelligent speech recognition and analysis system for offline recording. The system comprises a recording data acquisition module, a feature hierarchical analysis module, a semantic matching module, an intention decision module, a central processing unit and the like. The recording data acquisition module divides original audio data streams according to preset voiceprint classification rules; the feature hierarchical analysis module extracts different semantic feature sets; the semantic matching module maps the feature sets into intention classification labels; the intention decision module calls a target execution scheme; and the central processing unit verifies parameters to generate a final control instruction. In addition, the system is also provided with a voice resource scheduling module, an instruction conversion module and a voice archiving module, which are used for device scheduling, instruction conversion and data storage. The system can effectively process offline complex recording data, accurately recognize voice intentions, improve the intelligent level of voice interaction, and is suitable for various offline voice application scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent speech recognition and analysis, in particular to an intelligent speech recognition and analysis system for offline recording. BACKGROUND

[0002] With the booming development of artificial intelligence technology, speech recognition and analysis technology has been widely applied in many fields. From early simple voice command recognition to today's complex voice dialogue interaction and voice content analysis, this technology has been constantly innovated, bringing great convenience to people's life and work. However, in the current field of speech recognition and analysis, the processing of offline recording still faces many challenges.

[0003] In offline environments, the complexity of recorded data far exceeds that of online. First, there are diverse noise disturbances in offline scenarios. In public places such as shopping malls, stations, etc., the noise of the crowd, the roar of the traffic tools, etc. will seriously affect the accuracy of speech recognition. Even in relatively quiet indoor environments, there may be running sounds of electrical appliances, outdoor environmental noise, etc. These noises not only reduce the quality of the speech signal, but also may cause the speech recognition system to misidentify the noise as speech content, resulting in incorrect recognition results. For example, recording speech in a restaurant, the collision sound of tableware and the sound of people talking together make it difficult to clearly distinguish the target speech.

[0004] Secondly, existing speech recognition systems have a single classification and analysis method for speech data when processing offline recordings. Most systems can only simply recognize speech content and cannot deeply mine the rich information behind the speech, such as the identity of the speaker, the recording scene, and the emotional state, etc. This greatly limits the application scenarios of speech recognition and makes it difficult to meet the growing diverse needs of users. Taking conference recording as an example, traditional systems can only record the content of speeches, cannot automatically identify different speakers, and cannot perceive the emotional changes of speakers, which obviously lacks the ability to grasp the conference highlights and discussion atmosphere.

[0005] In addition, offline recording involves different types of recording devices, and the performance and parameters of these devices vary greatly, resulting in inconsistent formats and quality of the original audio data collected. The sampling rate, bit rate, and channel number of different devices vary, which brings many inconveniences to subsequent speech processing. Moreover, existing speech recognition and analysis systems often lack effective adaptation mechanisms for these device differences, and the recognition accuracy and analysis effect will fluctuate greatly when processing data collected by different devices.

[0006] Meanwhile, current voice recognition analysis systems also have deficiencies in intent decision and response execution. They usually make decisions based on simple preset rules and cannot make flexible and intelligent decisions according to specific voice semantics and complex scene conditions. This leads to the fact that the response of the system may not accurately meet the needs of the user in actual application, and the interactive experience is poor. For example, in the smart home control scene, when the user issues control instructions in different tones and expressions, the system may not accurately understand the user's intent, resulting in control errors or failure to respond. SUMMARY

[0007] The purpose of the present application is to provide an offline recording intelligent voice recognition analysis system to solve the problems raised in the background art.

[0008] To achieve the above purpose, the present application provides the following technical solution: an offline recording intelligent voice recognition analysis system, the system comprising:

[0009] A recording data acquisition module is configured to obtain an original audio data stream of an offline scene and divide the original audio data stream into a basic voice data segment and an extended voice data segment based on a preset voiceprint classification rule.

[0010] A feature hierarchical analysis module is configured to extract voiceprint feature of the basic voice data segment to generate a first semantic feature set and extract emotion fluctuation feature of the extended voice data segment to generate a second semantic feature set.

[0011] A semantic matching module is configured to map the first semantic feature set and the second semantic feature set to corresponding intent classification labels after joint coding according to a preset semantic mapping table, and use the intent classification labels as the analysis parameters of the current voice.

[0012] An intent decision module is configured to call a target execution scheme in a preset strategy library based on the intent classification label and use the target execution scheme as the response parameter of the current voice.

[0013] A central processing unit is configured to send the original audio data stream to the feature hierarchical analysis module, send the first semantic feature set and the second semantic feature set to the semantic matching module, and perform logical verification on the analysis parameters and the response parameters to generate a final control instruction.

[0014] Preferably, the emotion fluctuation feature extraction of the extended voice data segment by the feature hierarchical analysis module comprises:

[0015] The continuous speech frames in the extended speech data segment are divided into valid speech segments and invalid speech segments, and short-time energy spectrum features of the valid speech segments are extracted based on a preset time-frequency conversion window to generate a time-frequency feature matrix;

[0016] The background noise segments in the extended speech data segment are subjected to fundamental frequency tracking processing, waveform distortion features of each fundamental frequency period are extracted, and a fundamental frequency feature vector is constructed;

[0017] The time-frequency feature matrix and the fundamental frequency feature vector are subjected to multi-dimensional feature fusion to generate the second semantic feature set.

[0018] Preferably, the preset voiceprint classification rule includes a basic acoustic type set and an extended acoustic type set; the basic acoustic type set contains a speech content identifier, an environmental noise identifier, and a device fingerprint identifier; the extended acoustic type set contains a user identity identifier and a recording scene identifier, and each acoustic identifier corresponds to an independent data analysis protocol.

[0019] Preferably, the system further includes a recording interface module, which is used to realize the physical connection of the recording data acquisition module, the feature hierarchical analysis module, the semantic matching module, and the intent decision module with a local recording device;

[0020] The recording data acquisition module divides an original audio data stream based on the preset voiceprint classification rule, which includes:

[0021] The original waveform data is acquired in real time from the local recording device through the recording interface module, and the head features of the original waveform data are matched according to the identifiers in the basic acoustic type set to separate a basic audio segment;

[0022] The tail features of the original waveform data are scanned according to the identifiers in the extended acoustic type set to extract an extended audio segment;

[0023] The basic audio segment and the extended audio segment are written into a basic voiceprint buffer area and an extended voiceprint buffer area respectively after being synchronized according to sample point serial numbers.

[0024] Preferably, when the preset semantic mapping table adopts a linear matching model, the intent classification label is a linear mapping result of the normalized fusion value of the first semantic feature set and the second semantic feature set;

[0025] When the preset semantic mapping table adopts a nonlinear matching model, the intent classification label is a discrete semantic label obtained by discriminating the nonlinear combination result of the first semantic feature set and the second semantic feature set through a classification function.

[0026] Preferably, a voice resource scheduling module connected with the central processing unit is further included, and the voice resource scheduling module is connected with a local voice device database through the recording interface module;

[0027] The voice resource scheduling module is configured to match an available device list from the local voice device database according to a device calling request in the final control instruction, and generate a device deployment topology graph to optimize a voiceprint matching path.

[0028] Preferably, the voice resource scheduling module generating the device deployment topology graph comprises:

[0029] obtaining a recording space plane model, and labeling real-time voiceprint feature points of each device in the available device list in the plane model;

[0030] calculating an optimal matching path from each device feature point to a target voice area based on a path search algorithm, and performing priority arrangement on the available device list according to path matching degrees;

[0031] superimposing the optimal matching path and the priority arrangement into the plane model to generate a real-time updated device deployment topology graph.

[0032] Preferably, when the central processing unit performs logical verification on the parsed parameters and the response parameters, a double verification mechanism of an integrity check rule and a logical consistency detection rule is adopted, the integrity check rule is used to confirm parameter validity, and the logical consistency detection rule is used to eliminate semantic conflicts between parameters.

[0033] Preferably, an instruction conversion module connected with the central processing unit is further included, and the instruction conversion module is configured to convert the final control instruction into a voice driving signal, and send the voice driving signal to a target voice device through the recording interface module to trigger an execution action.

[0034] Preferably, a voice archiving module connected with the central processing unit is further included, and the voice archiving module is configured to store the original audio data stream, the first semantic feature set, the second semantic feature set, the intent classification label and the final control instruction, and generate a complete voice processing record chain according to a recording time sequence.

[0035] Compared with the prior art, the application has the following beneficial effects:

[0036] In terms of processing complex offline environments, the system classifies raw audio data streams into basic speech data segments and extended speech data segments by presetting voiceprint classification rules. This classification method can specifically process different types of data, greatly improving the processing capability of complex offline recordings. In offline scenes with a large amount of background noise, the basic speech data segment can focus on the speech content itself, while the extended speech data segment can perform fundamental frequency tracking processing on the background noise segment, extract waveform distortion features, and combine the time-frequency feature matrix extracted by time-frequency transformation to realize multi-dimensional feature fusion and generate a second semantic feature set. In this way, even in the case of severe noise interference, the system can more accurately capture speech features and improve the accuracy of speech recognition, avoiding recognition errors caused by noise.

[0037] In terms of mining deep information of speech data, the feature hierarchical analysis module of the system can not only extract voiceprint feature from basic speech data segment to generate first semantic feature set, but also extract emotion fluctuation feature from extended speech data segment to generate second semantic feature set. This enables the system to deeply mine rich information behind the speech, such as the identity of the speaker, the recording scene, and the emotional state. In a conference scenario, different speakers can be accurately identified by voiceprint feature, and emotion fluctuation feature can reflect the emotional changes of the speakers, which helps to fully grasp the conference focus and discussion atmosphere, and provides more rich and accurate data support for subsequent conference summary and decision-making.

[0038] In the face of differences brought by different recording devices, the recording interface module of the system realizes physical connection with local recording devices, and during data collection, it can match head features and scan tail features according to the identifiers in the basic acoustic type set and the extended acoustic type set, respectively, separate and extract basic audio segments and extended audio segments, and then write them into the corresponding cache area. This way effectively adapts to the diversified data collected by different recording devices, ensuring that the system can stably and efficiently process data regardless of the recording device used, improving the system's versatility and compatibility.

[0039] In terms of intelligence in intent decision and response execution, the semantic matching module maps the two semantic feature sets to intent classification labels after joint encoding according to the preset semantic mapping table, and the intent decision module calls the target execution scheme in the preset strategy library based on this. This process realizes the intelligent conversion from speech semantics to actual execution scheme. Moreover, the central processing unit adopts a dual verification mechanism of integrity verification rules and logical consistency detection rules to perform logical verification on the analysis parameters and response parameters, generating accurate final control instructions. In the smart home control scenario, when users issue control instructions in different tones and expressions, the system can accurately understand the user's intent and quickly respond correctly, improving the user's interaction experience.

[0040] In addition, the voice resource scheduling module of the system can match the available device list from the local voice device database according to the device calling request in the final control instruction, and generate a device deployment topology graph to optimize the voiceprint matching path. This greatly improves the efficiency and accuracy of the system when calling devices, ensuring the efficiency of voice interaction. At the same time, the voice archiving module stores the original audio data stream, each semantic feature set, intent classification label and final control instruction in chronological order, generates a complete voice processing record chain, and facilitates subsequent data query, analysis and backtracking, providing strong support for the optimization and improvement of the system. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The working principle diagram of the offline recording intelligent voice recognition analysis system described in the application;

[0042] Figure 2 The flowchart for the feature hierarchical analysis module to extract the emotional fluctuation features of the extended voice data segment;

[0043] Figure 3 The flowchart for the preset voiceprint classification rule composition and data analysis protocol association;

[0044] Figure 4 The flowchart for the audio data stream division of the recording data acquisition module according to the preset rules;

[0045] Figure 5 The flowchart for generating intent classification labels based on different semantic mapping models. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0047] Please refer to Figures 1-5 The application provides an offline recording intelligent voice recognition analysis system, and the specific implementation manner is described in detail below.

[0048] The system as a whole includes a recording data acquisition module, a feature hierarchical analysis module, a semantic matching module, an intent decision module and a central processing unit.

[0049] The function of the audio data collection module is to obtain the original audio data stream of offline scenes and divide it into basic voice data segments and extended voice data segments based on preset voiceprint classification rules. This module collects original audio data streams from various offline scenes such as conference rooms, classrooms, and homes with the help of professional recording equipment or terminal devices with integrated recording functions. The collected audio data is divided according to the preset voiceprint classification rules. For example, by analyzing the acoustic characteristics of the audio, according to the identifiers in the basic acoustic type set and the extended acoustic type set, the part containing basic information such as voice content, environmental noise, and device fingerprints is divided into basic voice data segments, while the part containing extended information such as user identity and recording scene is divided into extended voice data segments.

[0050] The feature hierarchical analysis module receives the basic voice data segments and extended voice data segments from the audio data collection module. For the basic voice data segments, the module extracts voiceprint texture features to generate a first semantic feature set. This process identifies unique texture features related to voice content by analyzing acoustic features of the voice, such as frequency, amplitude, etc. For the extended voice data segments, the module extracts emotion fluctuation features to generate a second semantic feature set. For example, by analyzing the speech rate, tone changes, etc., the emotional state of the speaker is determined, and these emotion-related features are extracted to form the second semantic feature set.

[0051] The semantic matching module jointly encodes the first semantic feature set and the second semantic feature set according to the preset semantic mapping table. The result of joint encoding is mapped to the corresponding intent classification label, which serves as the analysis parameter of the current voice. The preset semantic mapping table can be a pre-trained model, which is trained with a large amount of voice data to determine the corresponding intent classification label for different semantic feature combinations.

[0052] The intent decision module calls the target execution scheme in the preset strategy library based on the intent classification label. The preset strategy library stores multiple execution schemes for different intent classification labels. The module selects the most suitable target execution scheme based on the current intent classification label and uses it as the response parameter of the current voice.

[0053] The central processing unit plays a core control role in the entire system. It sends the original audio data stream to the feature hierarchical analysis module and sends the first semantic feature set and the second semantic feature set to the semantic matching module. In addition, the central processing unit performs logical verification on the analysis parameters and response parameters, confirms parameter validity through integrity verification rules, eliminates semantic conflicts between parameters using logical consistency detection rules, and finally generates the final control instruction.

[0054] The following five embodiments further elaborate on the specific implementation of the present application.

[0055] Embodiment 1:

[0056] This embodiment relates to the specific way of extracting emotion fluctuation features of extended speech data segments by the feature hierarchical analysis module in the system.

[0057] In processing the extended speech data segment, the continuous speech frames in the extended speech data segment are divided to distinguish valid speech segments and invalid speech segments. This division is based on the energy, frequency and other features of the speech signal. For example, speech frames with high energy and normal frequency range are determined as valid speech segments, while speech frames with low energy or abnormal frequency are determined as invalid speech segments.

[0058] Based on the preset time-frequency transformation window, the short-time energy spectrum features of the valid speech segments are extracted. The preset time-frequency transformation window is set according to the actual application scene and requirements. Through the window, the valid speech segments are analyzed segment by segment, the short-time energy spectrum of each segment is calculated, and then a time-frequency feature matrix is generated. This time-frequency feature matrix contains the energy distribution information of the valid speech segments at different times and frequencies.

[0059] For the background noise segment in the extended speech data segment, the fundamental frequency tracking processing is performed. In this process, the waveform distortion features of each fundamental frequency period are extracted. Since the background noise will interfere with the speech signal, causing distortion of the speech waveform, by extracting these distortion features and constructing them into a fundamental frequency feature vector.

[0060] The time-frequency feature matrix and the fundamental frequency feature vector are fused in multiple dimensions. The fusion method can use splicing, weighted summation and other methods to organically combine two different types of features to generate a second semantic feature set. This second semantic feature set integrates the time-frequency features of the speech and the influence features of the background noise on the speech waveform, and more comprehensively reflects the emotion fluctuation features of the extended speech data segment.

[0061] In an actual offline meeting scene, professional recording equipment is used in the meeting site to record the whole process of the meeting. The recording data acquisition module continuously acquires the original audio data stream and transmits it to the feature hierarchical analysis module.

[0062] In the feature hierarchical analysis module processing the extended speech data segment, the first step is to divide the valid speech segment and invalid speech segment. In the meeting, when the participants speak, the speech signal energy is relatively high, and the frequency is also in the normal human speech range, and these speech frames are determined as valid speech segments; while when there is a short pause, paper turning sound or slight equipment current sound in the conference site, the corresponding speech frame energy is low or the frequency is abnormal, which is determined as invalid speech segment. For example, when participant A is elaborating his point of view, the speech part is the valid speech segment, and the occasional short pause is the invalid speech segment.

[0063] After the division, the short-time energy spectrum feature of the valid speech segment is extracted based on the preset time-frequency conversion window to generate a time-frequency feature matrix. Assuming that the length of the preset time-frequency conversion window is 0.1 second, the valid speech segment of participant A is analyzed in 0.1 second intervals. For each segment, the energy distribution at different frequencies is calculated. For example, in a certain 0.1 second speech segment, the energy is higher in the frequency range of 200Hz-500Hz, and lower in other frequency ranges. By analyzing a plurality of such 0.1 second speech segments, a time-frequency feature matrix reflecting the energy distribution of the valid speech segment at different times and frequencies can be generated.

[0064] Then, the fundamental frequency tracking processing is performed on the background noise segment in the extended speech data segment. In the conference process, the air conditioner running sound in the conference room, the traffic noise from the distant street and the like belong to the background noise. Taking the air conditioner running sound as an example, the waveform distortion characteristics of each fundamental frequency period are analyzed by the fundamental frequency tracking technology. Since the frequency and amplitude of the air conditioner running sound will fluctuate to a certain extent, the speech waveform is distorted, and the distortion characteristics such as the peak value and valley value change of the waveform are extracted, and a fundamental frequency feature vector is constructed.

[0065] Finally, the time-frequency feature matrix and the fundamental frequency feature vector are fused in multiple dimensions to generate a second semantic feature set. Here, the fusion is performed in the splicing manner, i.e. the time-frequency feature matrix and the fundamental frequency feature vector are connected together in a certain order to form a new feature set, which is the second semantic feature set. Through such fusion, the time-frequency features of the speech and the influence characteristics of the background noise on the speech waveform can be integrated, and the emotional fluctuation characteristics of the extended speech data segment can be more comprehensively reflected. For example, when participant A is speaking with excitement, the speech speed is accelerated, the tone is raised, and the change of the background noise also affects the speech, and these information are all contained in the generated second semantic feature set, which provides rich data support for the subsequent semantic analysis and intention judgment.

[0066] Embodiment 2:

[0067] The embodiment mainly describes the specific process of preset voiceprint classification rule and the audio data collection module based on the rule to divide the original audio data stream.

[0068] The preset voiceprint classification rule includes a basic acoustic type set and an extended acoustic type set. The basic acoustic type set covers speech content identification, environmental noise identification, and device fingerprint identification. Speech content identification is used to identify specific speech information in the speech, environmental noise identification is used to distinguish noise characteristics in different environments, and device fingerprint identification can identify device characteristics used to collect audio. The extended acoustic type set includes user identity identification and recording scene identification, and each acoustic identification corresponds to an independent data analysis protocol.

[0069] When dividing the original audio data stream, the audio data collection module acquires real-time original waveform data from the local recording device through the recording interface module. The recording interface module realizes the physical connection between the system and the local recording device, ensuring stable data transmission. After obtaining the original waveform data, the head characteristics of the original waveform data are matched according to the identification in the basic acoustic type set. For example, by identifying specific frequency, amplitude, and other characteristic patterns in the waveform data header, basic audio segments related to speech content, environmental noise, and device fingerprint are separated.

[0070] Then, according to the identification in the extended acoustic type set, the tail characteristics of the original waveform data are scanned to extract the extended audio segment. For example, according to the specific acoustic pattern in the tail characteristics, the user identity identification and the recording scene identification are identified, and the corresponding extended audio segment is extracted.

[0071] Finally, the separated basic audio segment and extended audio segment are synchronized according to the sample point sequence number to ensure the time sequence consistency of the data. After synchronization, they are written into the basic voiceprint buffer area and the extended voiceprint buffer area, respectively, providing an ordered data basis for subsequent feature extraction and analysis.

[0072] Suppose in a classroom environment, a teaching activity is being conducted, and a recording device is installed in the classroom to record the teacher's teaching and the interaction between the teacher and the students. The original audio data stream formed by these sounds is acquired by the audio data collection module.

[0073] In the preset voiceprint classification rule, the speech content identifier is used to identify the specific content of the speech of the teacher and the student. For example, the teacher says "Today we learn Newton's first law", and the specific pronunciation, vocabulary combination and other characteristics in this speech are the embodiment of the speech content identifier; the environmental noise identifier can identify the environmental sound in the classroom, such as the humming sound of the fan rotating and the sound of moving the table and chair, and these sounds have their unique frequency and amplitude characteristics; the device fingerprint identifier can distinguish the specific recording device in the classroom, and each device will leave unique signal characteristics in the audio acquisition process, such as weak signal interference generated by specific electronic components.

[0074] The user identity identifier in the extended acoustic type set can identify which teacher or student is speaking by analyzing the tone, pronunciation habit and other characteristics of the speech in this classroom scenario. For example, the pronunciation habit of teacher A is moderate speed and some words are pronounced with a unique tone, which constitutes the user identity identifier of teacher A. The recording scene identifier identifies that this is a classroom teaching scene, and the acoustic reflection characteristics of the classroom space and the sound reverberation caused by the density of personnel are part of the recording scene identifier. Each acoustic identifier corresponds to an independent data analysis protocol to ensure that relevant information can be accurately extracted and analyzed.

[0075] The recording data acquisition module starts to divide the original audio data stream. It acquires the original waveform data from the recording device in the classroom in real time with the help of the recording interface module. The recording interface module establishes a stable physical connection between the system and the recording device to ensure the reliability of data transmission. After obtaining the original waveform data, the head characteristics of the original waveform data are matched according to the identifiers in the basic acoustic type set. For example, by identifying the starting frequency, amplitude change mode and other characteristics of the waveform data head, it is determined which part belongs to the speech content of the teacher or the student, and which is environmental noise or signal generated by the device itself, so as to separate the basic audio segment. In this process, if the waveform head presents clear speech frequency characteristics and the amplitude is within the normal speech range, it can be judged as the basic audio segment related to the speech content.

[0076] According to the identifiers in the extended acoustic type set, the tail characteristics of the original waveform data are scanned to extract the extended audio segment. For example, by analyzing the specific tone characteristics of the tail of the waveform data, it is determined whether the speaker is teacher A or a student, which extracts the extended audio segment related to the user identity identifier; according to the sound reverberation characteristics of the tail and the change of the environmental noise, it is determined that this is recorded in the classroom scene, which extracts the extended audio segment related to the recording scene identifier.

[0077] The separated base audio segment and the extended audio segment are synchronized according to the sampling point sequence number. This is to ensure the consistency of the two audio segments in time sequence and ensure the accuracy of subsequent analysis. After synchronization, the base audio segment is written into the base voiceprint cache area, and the extended audio segment is written into the extended voiceprint cache area. In this way, an ordered and clearly classified data basis is provided for subsequent feature extraction and analysis, facilitating further in-depth processing of the voice data by the system, such as analyzing the teaching effect of teachers and the participation of students.

[0078] Embodiment 3:

[0079] This embodiment focuses on different types of preset semantic mapping tables and the generation method of intent classification labels.

[0080] When the preset semantic mapping table adopts a linear matching model, the first semantic feature set and the second semantic feature set are first normalized and fused. Normalization is to eliminate the differences in numerical range and dimension between different feature sets, so that they can be fused and calculated on the same scale. The fusion method can be simple addition or weighted addition, and the normalized fusion value is obtained.

[0081] Then, the normalized fusion value is input into the linear mapping model. The linear mapping model is trained according to a large amount of voice data, which determines the linear relationship between the normalized fusion value and the intent classification label. Through this linear mapping, the corresponding intent classification label is obtained, which is one of the analysis parameters of the current voice.

[0082] When the preset semantic mapping table adopts a non-linear matching model, the first semantic feature set and the second semantic feature set are non-linearly combined. Non-linear combination can adopt various ways, such as neuron connection in neural network, which combines two feature sets in a complex non-linear relationship.

[0083] Next, the non-linear combination result is discriminated by a classification function. The classification function is trained and can determine the discrete semantic label to which the non-linear combination result belongs according to the feature pattern of the non-linear combination result, and this discrete semantic label is the intent classification label. This non-linear matching model can more flexibly handle complex semantic relationships and improve the accuracy of intent classification.

[0084] Suppose in an intelligent home environment, the user interacts with the smart sound box through voice, and when the preset semantic mapping table adopts a linear matching model:

[0085] When the user says "turn on the living room light" to the smart speaker, the audio data collection module collects this voice, and the feature hierarchical analysis module extracts the voiceprint feature of the basic voice data segment to generate the first semantic feature set, denoted as F1, which contains the features corresponding to the key information "living room light" and "turn on" in the voice. Meanwhile, the emotion fluctuation feature of the extended voice data segment is extracted to generate the second semantic feature set, denoted as F2, such as the degree of urgency and the tone of the user's speech.

[0086] First, F1 and F2 are normalized. The purpose of normalization is to make different feature sets have the same scale, which is convenient for subsequent calculation. Here, the maximum-minimum normalization method is adopted, and the formula is: where X represents the original feature value, X min is the minimum value of the feature set, X max is the maximum value of the feature set, and X norm is the normalized feature value. Each feature value in F1 and F2 is normalized to obtain the normalized feature sets F 1n and F 2n , which are then fused. Here, simple addition fusion is used to obtain the normalized fusion value S, i.e., S = F 1n +F 2n .

[0087] The preset linear mapping model is trained according to a large amount of smart home voice command data. Assuming that the linear mapping relationship is Y = aS + b, where Y is the intent classification label, and a and b are coefficients determined by training. In this example, Y corresponds to the intent classification label "turn on the living room light device", which becomes the analysis parameter of the current voice, and the subsequent system will perform corresponding operations based on this analysis parameter.

[0088] When the preset semantic mapping table adopts a nonlinear matching model:

[0089] Similarly, the user says "turn on the living room light", and the extraction process of F1 and F2 remains unchanged. F1 and F2 are combined non-linearly, such as through a simple neural network structure. This neural network has several neurons, and the feature values in F1 and F2 are input to the neurons. The neurons perform complex nonlinear calculations through specific weights and activation functions.

[0090] After the nonlinear combination, the result is discriminated by a classification function. Assuming that the classification function is a function based on a decision tree algorithm, the decision tree makes layer-by-layer judgments according to various feature patterns in the nonlinear combination result. For example, first, it judges whether the feature related to "light" is contained, and then it judges whether the action is "turn on" or "turn off", etc., and finally outputs the discrete semantic label "turn on the living room light device" as the intent classification label, so as to determine the user's intent and then make the smart speaker execute the corresponding light-on operation.

[0091] Embodiment 4:

[0092] This embodiment focuses on the function of the voice resource scheduling module and the specific process of generating the device deployment topology graph.

[0093] The voice resource scheduling module is connected with the central processing unit and connected with the local voice device database through the recording interface module. When receiving the device calling request in the final control instruction of the central processing unit, the voice resource scheduling module starts to work.

[0094] It matches the available device list from the local voice device database. The local voice device database stores the information of various voice devices, including the type, function, current state, etc. of the device. The voice resource scheduling module filters out the available devices that meet the conditions according to the requirements in the device calling request, such as the functional requirements of the device, the use scenario requirements, etc., to form the available device list.

[0095] When generating the device deployment topology graph, the recording space plane model is first obtained. The recording space plane model can be obtained by field measurement, map data or preset space template, etc., which reflects the layout and size information of the recording space.

[0096] In the plane model, the real-time voiceprint feature points of each device in the available device list are labeled. These voiceprint feature points are determined according to the voiceprint features of the device under the current location and environment, and are obtained by analyzing the test signal emitted by the device or the actually collected voiceprint data.

[0097] Then, the optimal matching path from each device's feature point to the target voice area is calculated based on the path search algorithm. The path search algorithm can use common Dijkstra algorithm or A* algorithm, etc., according to the space layout and signal propagation characteristics, to calculate the best propagation path of the signal from the device to the target voice area. The available device list is prioritized according to the path matching degree, and the device with high path matching degree is placed in front.

[0098] Finally, the optimal matching path and priority ranking are superimposed into the plane model to generate a real-time updated device deployment topology. This device deployment topology can intuitively show the positions of various available devices in the recording space, the optimal path to the target voice area, and the priority of the device, providing a visual basis for optimizing the voiceprint matching path.

[0099] Suppose that in a large shopping mall monitoring center scenario, a set of voice monitoring and device scheduling systems based on the technology of the present application is deployed, and multiple recording devices are installed in the shopping mall monitoring center for real-time collection of sound information in each area of the shopping mall. When an abnormal sound, such as a quarrel or an alarm, occurs in a certain area, the central processing unit will send a device calling request to the voice resource scheduling module according to the processing result of the voice recognition analysis system.

[0100] After receiving the request, the voice resource scheduling module first matches the available device list from the local voice device database. The local voice device database stores detailed information of all relevant voice devices in the shopping mall, including the location, model, and current working state of each broadcast speaker, monitoring camera (partially with voice function), etc. Suppose that the abnormal sound occurs in the clothing area on the third floor of the shopping mall, and the voice resource scheduling module filters out the broadcast speakers and monitoring cameras closest to the area and in normal working state according to the location information "clothing area on the third floor of the shopping mall" in the device calling request to form an available device list. For example, broadcast speakers A1 and A2 and monitoring cameras C3 and C4 are included in the list.

[0101] Next, the device deployment topology is generated. The first step is to obtain the recording space plane model. In this shopping mall scenario, the architectural drawing data of the shopping mall can be used to construct the plane model, which accurately shows the layout of each floor of the shopping mall, the location of the passageway, the distribution of the stores, and the installation location of each voice device, etc.

[0102] In the plane model, the real-time voiceprint feature points of each device in the available device list are labeled. Due to the complexity of the shopping mall environment, the sound propagation characteristics are different at different locations, and the voiceprint features received by the device will also be different. By sending a specific test signal to each device and analyzing the reflected signal received by the device and the actual collected environmental voiceprint data, the real-time voiceprint feature points of each device are determined. For example, A1 broadcast speaker is located near the entrance of the clothing area, and its voiceprint feature points reflect the unique characteristics of sound propagation at this location, including signal strength, frequency attenuation, etc.; C3 monitoring camera is installed in the corner of the clothing area, and its voiceprint feature points contain the influence of acoustic reflection and shielding in the corner on the sound.

[0103] Then, the optimal matching path from each device's feature point to the target voice area (i.e., the clothing area on the third floor of the mall where abnormal sound occurs) is calculated based on a path search algorithm. Here, the A algorithm is used for path calculation. The A algorithm takes into account factors such as the distance from the device to the target area and the sound propagation loss on the path. For the A1 broadcast speaker, the algorithm will calculate multiple possible paths from its location to the target point in the clothing area according to the layout information of the mall, and evaluate the comprehensive cost of each path to determine the optimal path, which may be the route along the nearest channel and avoiding large obstacles; for the C3 monitoring camera, the optimal path from its installation location to the target area is also calculated, which may need to bypass some shelves and walls and other obstructions. The available device list is prioritized according to the path matching degree, with a high path matching degree indicating that the signal transmission from the device to the target area is better and less interference. For example, after calculation, the path matching degree of the A1 broadcast speaker is the highest, so it is ranked first; the path matching degree of the C3 monitoring camera is second, and so on.

[0104] Finally, the optimal matching path and priority ranking are superimposed into the plane model to generate a real-time updated device deployment topology graph. In the generated topology graph, the staff of the monitoring center can intuitively see the location of each available device in the mall, the optimal path to the target area, and the priority order of the devices. This allows the staff to quickly understand which devices can more effectively respond to abnormal situations, such as using the A1 broadcast speaker to play warning information and using the C3 monitoring camera to obtain live footage, so that timely measures can be taken to handle abnormal situations in the mall, achieving efficient monitoring and management of the mall environment.

[0105] Embodiment 5:

[0106] This embodiment mainly explains the specific working process of the instruction conversion module and the voice archiving module.

[0107] The instruction conversion module is connected to the central processing unit, and its function is to convert the final control instructions generated by the central processing unit into voice driving signals. The final control instructions are control signals generated after logical verification, which contain operation requirements for target voice devices. The instruction conversion module converts the final control instructions into corresponding voice driving signals according to the type and interface specification of the target voice device. For example, if the target voice device is a smart speaker, the instruction conversion module will convert the control instructions into voice driving signals that comply with the communication protocol of the smart speaker.

[0108] The converted voice driving signal is sent to the target voice device through the recording interface module, triggering the target voice device to perform corresponding actions. The recording interface module ensures that the voice driving signal can be accurately and stably transmitted to the target voice device, so that the target voice device can play voice, perform operations, and other actions according to the instruction requirements.

[0109] The voice archiving module is also connected to the central processing unit and is used to store the original audio data stream, the first semantic feature set, the second semantic feature set, the intent classification label, and the final control instruction. It stores these data in sequence according to the recording time sequence, forming a complete voice processing record chain. In this way, when subsequent voice processing processes need to be queried and analyzed, relevant data can be easily obtained from the voice archiving module. For example, when optimizing the voice recognition algorithm or troubleshooting, by checking the historical voice processing record chain, the data situation at different stages can be understood, providing a basis for improvement and debugging.

[0110] Suppose in an intelligent office scenario, employee Xiao Zhang uses the company's smart voice assistant to arrange and collaborate on work tasks. When Xiao Zhang says "Help me schedule a meeting with Mr. Li tomorrow afternoon at 3 pm and prepare the relevant project materials," the system modules begin to work together.

[0111] After completing the recognition, analysis, and logical verification of the voice data, the central processing unit generates the final control instruction. This instruction contains specific operation requirements for scheduling the meeting and preparing the project materials, such as the time of the meeting, the participants, and the specific information of the project materials that need to be prepared.

[0112] The instruction conversion module receives the final control instruction from the central processing unit. Since the communication protocol used by the company's meeting reservation system and the data management system is different from the format of the instruction generated by the central processing unit, the instruction conversion module needs to convert the final control instruction into a voice-driven signal that the corresponding system can recognize. For example, the meeting reservation system uses a specific API interface to receive reservation information, and the instruction conversion module will re-encode and package the meeting time (tomorrow afternoon 3 o'clock), participants (customer Li), and other information in the control instruction according to the format required by the API interface, and convert it into a voice-driven signal that conforms to the communication protocol of the meeting reservation system; for the data management system, the instruction conversion module will also convert the instruction about preparing project data into a signal format that can be recognized by the data management system. After conversion, the instruction conversion module sends these voice-driven signals to the corresponding target voice devices through the recording interface module, i.e. the relevant terminal devices of the meeting reservation system and the data management system, triggering them to perform corresponding actions. After receiving the signal, the meeting reservation system automatically creates a meeting reservation record for customer Li at 3 o'clock tomorrow afternoon in the system; the data management system starts collecting and organizing the project data designated by Xiao Zhang according to the instruction.

[0113] At the same time, the voice archiving module starts working. The voice archiving module is connected with the central processing unit, and it will store the key data generated during the entire voice processing process. First, the original audio data stream, i.e. the voice data collected by the recording device when Xiao Zhang speaks the instruction, is saved completely for possible subsequent review and analysis. Then the first semantic feature set and the second semantic feature set, which contain the voiceprint feature and the emotional fluctuation feature extracted from Xiao Zhang's voice, provide the basis for understanding Xiao Zhang's intention and emotional state. The intention classification label is also stored, which in this example may be "meeting reservation and data preparation", which clearly identifies the core intention of Xiao Zhang's voice instruction. Finally, the final control instruction generated by the central processing unit is also stored in the voice archiving module. The voice archiving module stores these data in sequence according to the recording time sequence, forming a complete voice processing record chain. For example, in the subsequent work, if there is a meeting reservation error or incomplete data preparation, the management personnel can trace back the entire voice processing process by querying the record chain in the voice archiving module, check whether it is a voice recognition error or a mistake in the instruction conversion or execution process, so as to quickly locate the problem and make corrections, and also provide actual data support for optimizing the intelligent voice assistant system.

[0114] It is to be understood that the terminology used herein such as first and second, and the like, is only used to distinguish one entity or action from another entity or action, and does not necessarily require or imply any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0115] While embodiments of the present application have been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application. The scope of the application is defined by the appended claims and their equivalents.

Claims

1. An intelligent speech recognition and analysis system for offline recordings, characterized in that, include: The recording data acquisition module is used to acquire the raw audio data stream of the offline scene, and divide the raw audio data stream into a basic voice data segment and an extended voice data segment based on a preset voiceprint classification rule; The feature layering and parsing module is used to extract voiceprint features from the basic speech data segment to generate a first semantic feature set, and to extract emotional fluctuation features from the extended speech data segment to generate a second semantic feature set. The semantic matching module is used to jointly encode the first semantic feature set and the second semantic feature set according to a preset semantic mapping table and map them to the corresponding intent classification label, and use the intent classification label as the parsing parameter of the current speech; The intent decision module is used to call the target execution scheme in the preset strategy library based on the intent classification label, and use the target execution scheme as the response parameter of the current speech; The central processing unit is used to send the original audio data stream to the feature hierarchical parsing module, send the first semantic feature set and the second semantic feature set to the semantic matching module, and also to perform logical verification on the parsing parameters and the response parameters to generate the final control command; The feature hierarchical parsing module performs emotional fluctuation feature extraction on the extended speech data segment, including: The continuous speech frames in the extended speech data segment are divided into valid speech segments and invalid speech segments. Short-time energy spectrum features are extracted from the valid speech segments based on a preset time-frequency transformation window to generate a time-frequency feature matrix. The background noise segment in the extended speech data segment is subjected to fundamental frequency tracking processing, and the waveform distortion features of each fundamental frequency cycle are extracted and a fundamental frequency feature vector is constructed. The time-frequency feature matrix and the fundamental frequency feature vector are fused in multiple dimensions to generate the second semantic feature set.

2. The speech recognition and analysis system according to claim 1, characterized in that, The preset voiceprint classification rules include a basic acoustic type set and an extended acoustic type set; the basic acoustic type set includes voice content identifiers, environmental noise identifiers, and device fingerprint identifiers; the extended acoustic type set includes user identity identifiers and recording scene identifiers, and each acoustic identifier corresponds to an independent data parsing protocol.

3. The speech recognition and analysis system according to claim 1, characterized in that, When the preset semantic mapping table adopts a linear matching model, the intent classification label is the linear mapping result of the normalized fusion values ​​of the first semantic feature set and the second semantic feature set. When the preset semantic mapping table adopts a nonlinear matching model, the intent classification label is a discrete semantic label that is judged by a classification function based on the nonlinear combination result of the first semantic feature set and the second semantic feature set.

4. The speech recognition and analysis system according to claim 1, characterized in that, It also includes a voice resource scheduling module connected to the central processing unit, which is connected to a local voice device database through a recording interface module; The voice resource scheduling module is used to match the list of available devices from the local voice device database according to the device call request in the final control command, and generate a device deployment topology map to optimize the voiceprint matching path.

5. The speech recognition and analysis system according to claim 4, characterized in that, The voice resource scheduling module generates a device deployment topology diagram including: Obtain a recording space planar model, and annotate the real-time audioprint feature points of each device in the list of available devices in the planar model; The optimal matching path from the feature points of each device to the target speech region is calculated based on the path search algorithm, and the list of available devices is prioritized according to the path matching degree. The optimal matching path and the priority order are superimposed on the planar model to generate a real-time updated device deployment topology map.

6. The speech recognition and analysis system according to claim 1, characterized in that, When the central processing unit performs logical verification on the parsed parameters and response parameters, it adopts a dual verification mechanism of integrity verification rules and logical consistency detection rules. The integrity verification rules are used to confirm the validity of the parameters, and the logical consistency detection rules are used to eliminate semantic conflicts between parameters.

7. The speech recognition and analysis system according to claim 1, characterized in that, It also includes an instruction conversion module connected to the central processing unit. The instruction conversion module is used to convert the final control instruction into a voice drive signal and send the voice drive signal to the target voice device through the recording interface module to trigger the execution action.

8. The speech recognition and analysis system according to claim 1, characterized in that, It also includes a voice archiving module connected to the central processing unit. The voice archiving module is used to store the original audio data stream, the first semantic feature set, the second semantic feature set, the intent classification label, and the final control command, and to generate a complete voice processing record chain according to the recording time sequence.

Images