A machine learning-based whole-process engineering consulting method and system

Abstract

The application belongs to the technical field of computers and provides a whole-process engineering consultation method and system based on machine learning, which comprises the following steps: when a user initiates a consultation request, input consultation voice is acquired through a microphone device and converted into an electrical signal, the electrical signal is positioned and segmented through short-time Fourier transform, the electrical signal is divided into multiple signal frames based on the segmented points, classification features in each signal frame are calculated, and the signal frames are merged based on the classification features, the electrical signal is divided into multiple voice segments according to the merging result, and all the voice segments are input into different recognition models to output consultation information. This method can be used in cooperation with existing large-scale multi-language models to provide accurate language-level segmented input, improve cross-language recognition robustness, and significantly improve segmentation accuracy and interpretability in long-voice multi-language scenarios without the need for experience thresholds.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a whole-process engineering consulting method based on machine learning. Background Technology

[0002] Currently, most engineering consulting systems adopt a text-based interaction method. This involves the consulting party submitting requirements, technical questions, or site conditions in text form via computer or mobile terminal, which are then manually interpreted, retrieved, and responded to by engineering technicians. This method places high demands on the consulting party's professional expression and text input abilities, easily leading to incomplete descriptions, semantic ambiguity, or omissions of key information. This results in engineers needing to ask follow-up questions for clarification, leading to low consulting efficiency. Furthermore, in complex engineering scenarios, consulting content often includes numerous parameters, operating conditions, and specification clauses. Typing these details word by word is costly and time-consuming, hindering rapid feedback from on-site personnel in urgent or high-load working environments. Therefore, introducing speech recognition technology to automatically transcribe the consulting party's speech input into text has become a trend to improve engineering consulting efficiency and lower the barrier to entry. However, existing speech recognition solutions generally treat the entire speech segment as a single input for end-to-end recognition, making it difficult to effectively handle long-duration, multilingual, or multi-dialect engineering consultation speech. On the one hand, when consultants continuously switch languages ​​(Chinese and other languages) or naturally transition between different engineering sub-problems in a single speech, existing models often fail to accurately distinguish between different languages ​​or topic boundaries, resulting in mixed recognition results and unclear semantic structures, which is not conducive to subsequent automatic parsing and engineering knowledge matching. On the other hand, traditional speech activity detection or simple energy-based segmentation methods can only roughly distinguish between audible and silent segments, and cannot finely segment speech segments based on language features or acoustic statistical features, making it difficult for downstream semantic understanding modules to optimize processing for single languages ​​and single semantic units. Summary of the Invention

[0003] The purpose of this invention is to propose a machine learning-based full-process engineering consulting method to solve one or more technical problems existing in the prior art, and at least provide a beneficial option or create conditions.

[0004] To achieve the above objectives, according to one aspect of the present invention, a machine learning-based full-process engineering consulting method is provided, the machine learning-based full-process engineering consulting method comprising the following steps: S100: When a user initiates a consultation request, the input consultation voice is acquired through a microphone device and converted into an electrical signal. S200 uses short-time Fourier transform to locate segmentation points in the electrical signal and divides the electrical signal into multiple signal frames based on these segmentation points; S300, calculate the classification features in each signal frame, and merge the signal frames based on the classification features; S400 divides the electrical signal into multiple speech segments based on the merging result, inputs all speech segments into different recognition models, and outputs consultation information.

[0005] Furthermore, the method also includes step S500, which involves inputting information into a decision tree model to output consultation results.

[0006] Furthermore, in S100, when the user initiates a consultation request, the method of acquiring the input consultation voice through the microphone device and converting it into an electrical signal specifically includes: after receiving the consultation request initiated by the user, the user controls the sensor with conversion function on the microphone to acquire the input consultation voice, and at the same time converts it into a digital signal through the analog-to-digital converter installed on the sensor, and obtains an electrical signal after the converted digital signal is processed by noise reduction filtering.

[0007] Existing multilingual speech recognition technologies largely rely on large-scale end-to-end models to directly recognize entire speech segments. For example, Google's Universal Speech Model (USM) covers over 300 languages ​​with a single model, but it primarily targets sentence-level recognition tasks and has limited ability to characterize language boundaries in long, multilingual mixed speech. While Meta's Massively Multilingual Speech (MMS) and multilingual models based on wav2vec2.0 or XLS-R can scale to hundreds or even thousands of languages, they also lack fine-grained modeling of language switching locations within long speech segments, merely implying language differences in end-to-end parameters. Models like OpenAI Whisper also directly convert raw speech to text, making it difficult to explicitly output stable and interpretable language segmentation results. Therefore, this invention proposes the following method: Furthermore, in S200, the method of locating segmentation points in the electrical signal through short-time Fourier transform and dividing the electrical signal into multiple signal frames based on these segmentation points specifically includes: first, enhancing the high-frequency components in the electrical signal by passing it through a first-order high-pass filter, wherein the first-order high-pass filter is:

[0008] Where X'(t) represents the t-th signal after passing through a first-order high-pass filter, X(t) represents the original t-th signal, and a is a parameter with a value range of [0.85,1]. The signal passing through the filter is divided into multiple raw frames with a length of 20ms. Each raw frame corresponds to a segment of electrical signal. Each raw frame is weighted by a window function, specifically a Hamming window function. Each weighted electrical signal is then subjected to a Fourier transform to obtain the corresponding spectral information. All spectral information is used to construct a spectrogram, and the envelope of the spectrogram is plotted. The envelope is used as the speech curve of the input speech. The speech curve is marked with the start and end times of each word's pronunciation using an endpoint detection algorithm as segmentation points. Based on these segmentation points, the speech curve is divided into multiple signal frames.

[0009] Furthermore, in S300, the method for calculating the classification features in each signal frame and merging the signal frames based on the classification features specifically includes: traversing all the curve segments contained in the signal frames sequentially along the signal acquisition time from the starting point of the speech curve; for each traversed curve segment, the maximum value of the amplitude in the curve is recorded as Vmax, and the minimum value is recorded as Vmin; the absolute value of the difference between the maximum and minimum values ​​is taken as the transition value in the curve segment; the length of the line segment between the frequency point corresponding to the maximum value and the frequency point corresponding to the minimum value in the curve segment is taken as the frequency difference of the curve segment; and the ratio of the transition value to the frequency difference is taken as the classification feature of the curve segment. The difference between Mandarin and dialect intonation is mainly reflected in the tone, exhibiting a clear exclusive characteristic. That is, the tonal variation patterns of different dialects show significant differences in frequency and amplitude fluctuations. Previous comparative studies, through statistical analysis of dialect pronunciation fundamental tones, have found that in Mandarin, the tone profiles of other fundamental tones besides the standard tone exhibit significant variations, forming distinct tone patterns. In contrast, dialect pronunciations lack this clear tone pattern distinction. This is used as a fundamental distinguishing feature between dialects and Mandarin. The classification feature in the above method consists of two parameters: the transition value, representing the largest frequency difference in the original frame, reflects the frequency distribution span of the sound in the original frame and can be used to evaluate tone stability; and the transition value maps this distribution span to specific tone frequencies in the original frame. By integrating these two parameters, the classification feature quantifies the variation features contained in the corresponding sound signal of the original frame through a ratio. Based on the classification feature, the pronunciation features of Mandarin and dialects can be effectively distinguished, thereby achieving accurate language type determination.

[0010] Using i as the sequence number of the original frame, Fram i This represents the classification feature in the i-th original frame. Starting from the first original frame, the next original frame is traversed in the order of signal acquisition time. If the classification feature in the current original frame is greater than the average of the classification features in all previously traversed original frames, the signal end point in the current original frame is marked as the segmentation point. The electrical signal is divided into multiple speech segments according to the position of the marked point. The above methods use the differences in the distribution of sound frequencies corresponding to the pronunciation tones of different languages ​​to mark the segmentation points between different languages ​​in the electrical signal of a language. The speech segments marked by these segmentation points contain signals from different languages. Inputting different speech segments into different recognition models can achieve efficient speech-to-text conversion, greatly improving consultation efficiency. However, the speech segments marked by the above methods do not specifically reflect the differences between the corresponding languages; that is, it is unclear which speech segment belongs to which language. Therefore, model selection remains difficult. Furthermore, because dialects contain a large number of sub-languages, and these sub-languages ​​contain many variations, it is difficult to specifically classify the signal into different language speech segments using classification methods. Existing technologies include multilingual recognition models, which are mainly suitable for text conversion between different languages. While solving the problem of language mixing, the input speech often experiences timeouts due to excessively long input speech duration. Although training speech during model training is typically short, effective speech with clear features and minimal noise, the quality of input speech varies due to user diversity in actual use. Current techniques improve recognition efficiency by segmenting long speech into segments, but this segmentation method still relies on different language features for classification. When the distribution of different languages ​​in an input speech segment shows significant differences, this method is prone to overfitting. To address this issue, this invention proposes a simple classification method based on the fundamental differences in the fundamental pitch of different languages ​​as reflected in the spectrogram, thus classifying the input speech.

[0011] Within the range of values ​​of i, iterate through all signal frames for classification features, signal frame fundamental frequency, fundamental frequency change rate, pitch span, and resonance amplitude.

[0012] The fundamental frequency of the signal frame can be obtained using the pitch function in MATLAB or the librosa.yin function in Python.

[0013] Furthermore, the fundamental frequency change rate is the absolute value of the difference between the fundamental frequencies of adjacent signal frames.

[0014] Specifically, for any given signal frame, the fundamental frequency change rate is the absolute value of the difference between the fundamental frequency of the current signal frame and the fundamental frequency of the most recently traversed signal frame.

[0015] Furthermore, the pitch span is the absolute value of the difference between the highest and lowest fundamental frequencies of the signal frame within the time window, and the time window length is 5 to 10 frames.

[0016] Furthermore, the resonance amplitude is calculated by taking the polynomial roots of the results after LPC linear prediction of all signal frames into a sequence and arranging them in order of magnitude. The positive frequencies of the paired conjugate complex roots are then obtained.

[0017] For the i-th signal frame, a feature vector Xi is constructed. This feature vector contains the classification features of the signal frame, the fundamental frequency, the rate of change of the fundamental frequency, the pitch span, and the resonance amplitude. The same method is used to construct the feature vectors of all signal frames, forming a one-dimensional feature vector sequence {Xi}, i∈[0,N]. Using the end point of any signal frame as a segmentation point, the sequence {Xi} is divided into K segments. Each segment contains... One signal frame, This invention introduces a segmentation scheme S, which is used to represent the boundary positions of each speech segment. Specifically, the segmentation scheme S is represented as an ordered set of boundary indices: ,in =0, And for any satisfy ,and , Indicates the first division The first speech segment, also representing the sequence number of the end point of the signal frame, is formed by the sequence of frames between adjacent boundary indices in all signal frames. The frame index range corresponding to each audio segment is: to ,Include Given 10 signal frames, assuming that the signal frames of the same language follow a Gaussian distribution, calculate the mean of each segment as 1. The covariance matrix is ; Segmentation scheme S is used to divide all signal frames into K consecutive speech segments.

[0018] Furthermore, this invention determines the optimal segmentation scheme S by calculating the complexity of all segmentation schemes, specifically expressed as:

[0019] in Indicates interval [ The cost of internal speech segments, and These represent the index values ​​of the two endpoints of the speech segment. Used to characterize the stability of acoustic features within the speech segment. To correspond with the number of speech segments Related redundancy assessment values.

[0020] The above expression is used to comprehensively measure the statistical consistency within each speech segment and the complexity caused by the number of speech segments. The interval [ As a cost to the speech segments within this interval, since speech segments are actually composed of arbitrary continuous speech frames, the characteristics of a speech frame are measured by a feature vector. The above method involves dividing a speech segment into multiple speech frames and then combining all adjacent speech frames to obtain all possible segments of a speech segment. Then, the complexity of all the segmentation results is calculated. Since the classification features, signal frame fundamental frequency, fundamental frequency change rate, pitch span, and resonance amplitude are different between different languages, the complexity of speech segments composed of speech frames from different languages ​​calculated by the above formula will be too large. However, if only the complexity of the combined speech segments is considered, the segmentation scheme tends to be too simplistic, that is, the segmentation result is likely to be a speech segment composed of a single speech frame. Such a segmentation result is obviously meaningless. Therefore, to avoid such problems, the parameter λ is added to the formula. The function of parameter λ is to constrain the number of segmentation points. The above expression is a comprehensive evaluation of the complexity of the segmentation result by combining the two parameters.

[0021] Furthermore, The expression is as follows:

[0022] in Indicates the first The t-th signal frame in a speech segment Indicates the first The number of speech frames contained in a speech segment, where T is the transpose symbol; If in the interval [ When all speech frames included in the formula belong to the same language, the feature vectors of the speech frames follow a Gaussian distribution. Partially used to determine speech segments [ Whether the distribution of feature vectors of the speech frames contained in the [data] is divergent is used to measure the similarity of the various speech frames within a speech segment. Represented as a speech segment [ The determinant of the calculated covariance matrix. The squared Mahalanobis distance is used to measure speech segments. The above expression is used to determine the deviation of a single speech frame from the overall speech frame, and is used to determine the degree of deviation of each segment of speech. Whether the distribution pattern of the feature vectors of the overall speech frames in the expression is consistent, whether there are individual speech frames that deviate from the whole, and the more languages ​​contained in the divided speech segments, the larger the output result of the expression will be. The output result of the expression will be used as a measure of the consistency and stability of the divided speech segments.

[0023] Furthermore, The specific derivation process is as follows: ; Where d is the dimension of the feature vector, and r is the index of the dimension in the feature vector. Indicates the first The number of speech frames contained in a speech segment. Indicates the first Dimensional features in The sequence of values ​​on each signal frame, Var(·) is the variance calculation function, used to calculate the variance of the eigenvalues ​​in each dimension of the eigenvector of the signal frame; since the segmentation scheme only considers the complexity within the speech segment, based on the principle of minimizing complexity, it is easy to result in the segmented speech containing only one speech frame. To avoid this, the above method is used to balance the complexity of the segmentation result. The principle reflected in the above method is that a small amount of data does not conform to a Gaussian distribution, so... This approach is not accepted and requires a new parameter to evaluate this extreme case separately, specifically when the number of speech frames contained in the segmented speech is extremely small. Even if the calculation results are ideal, the dataset is too small, so it is considered... The calculation results are not representative, but in the above methods, since the dataset is small, n k The feature dimension is also very small, and the feature dimension is a constant, so the result of the above expression will be extremely large. That is to say, when there are multiple speech segments in the segmentation scheme that contain only a few speech frames, we need to consider whether the system has fallen into such an extreme case when segmenting. At the same time, the above method not only considers the number of speech frames in the speech segment, but also the feature distribution of a small number of speech frames. By calculating the distribution variance of the feature values ​​of the speech frames, the large variance indicates an uneven distribution, thus avoiding the limitation of only considering the number of speech frames in the speech segment.

[0024] All possible segmentation schemes are identified through combination, and the model complexity of each segmentation scheme is calculated. The segmentation scheme with the lowest complexity is selected as the optimal segmentation scheme.

[0025] The above method converts long-term speech signals into frame-level acoustic feature sequences and performs a global optimization search using a Gaussian statistical model and objective function at the end of all frames to obtain a segmentation scheme. Compared with traditional segmentation methods that rely on empirical thresholds or simple energy changes, this method can adaptively determine language boundaries based on the statistical consistency of intra-segment features and the significant differences between inter-segment features without the need for manual threshold setting. This effectively avoids over-segmentation or under-segmentation, significantly improving the segmentation accuracy and stability of long speech in multilingual scenarios, and providing more accurate and robust structured input for subsequent language recognition, dialect analysis, and speech recognition models.

[0026] Furthermore, in S400, the method of dividing the electrical signal into multiple speech segments based on the merging result and inputting all speech segments into different recognition models to output consulting information specifically includes: dividing the electrical signal into multiple speech segments based on the optimal segmentation scheme, and sequentially inputting each target speech segment into a pre-trained multilingual speech recognition model to obtain the text content and language identifier corresponding to each speech segment; combining the engineering domain knowledge base and preset rules to perform semantic parsing and intent recognition on the text content; extracting key information related to engineering design parameters, construction schemes, safety risks, equipment selection, cost control, etc. from the recognition results; generating corresponding engineering consulting information and outputting it.

[0027] Furthermore, the multilingual speech recognition model can be an end-to-end model based on an encoder-decoder structure, a multilingual model based on self-supervised pre-training, or a combination of one or more models among cloud-based speech recognition service models based on an RNN-T structure.

[0028] Furthermore, the engineering domain knowledge base may include an engineering specification and standard library built on a relational database, an equipment and process association library stored in the form of a knowledge graph, and an engineering accident and safety case library based on ontology modeling, etc.

[0029] Furthermore, the preset rules may include IF-THEN form expert rules predefined by engineering experts, text classification and entity extraction models based on statistical learning or deep learning, risk assessment rules based on fuzzy logic, and device association reasoning strategies based on graph neural networks, etc.

[0030] This invention also provides a machine learning-based end-to-end engineering consulting system, the system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program in a unit of the following system: A signal conversion unit is used to acquire input speech via a sensor on the microphone and convert it into a digital signal; The frequency domain conversion unit is used to convert digital signals into frequency domain signals through Fourier transform and divide them into signal frames; The signal segmentation unit is used to calculate the classification features in the converted frequency domain signal frame and divide the digital signal into multiple speech segments based on the classification features; The information extraction unit is used to extract consultation information from the sealed speech segments through the model.

[0031] Beneficial Effects: This invention introduces a global optimization segmentation mechanism based on frame-level acoustic feature statistical consistency at the acoustic front end. By establishing segmentation schemes at all frame end points and minimizing the sum of intra-segment Gaussian statistical cost and segmentation complexity, it adaptively obtains the boundary positions of each language in long-term speech, achieving explicit structured modeling of multilingual mixed speech. This approach can be used in conjunction with existing large-scale multilingual ASR models, providing them with accurate language-level segmentation input and improving cross-language recognition robustness. Furthermore, it can significantly improve the segmentation accuracy and interpretability in long-speech multilingual scenarios without requiring empirical thresholds. Attached Figure Description

[0032] Figure 1 The diagram shows a flowchart of a machine learning-based end-to-end engineering consulting method. Figure 2 The diagram shows the structure of a machine learning-based end-to-end engineering consulting system. Detailed Implementation

[0033] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention. Example

[0034] Figure 1 The diagram shows a flowchart of a machine learning-based end-to-end engineering consulting method.

[0035] Reference Figure 1 This invention proposes a machine learning-based full-process engineering consulting method, which includes the following steps: S100: When a user initiates a consultation request, the input consultation voice is acquired through a microphone device and converted into an electrical signal. S200 uses short-time Fourier transform to locate segmentation points in the electrical signal and divides the electrical signal into multiple signal frames based on these segmentation points; S300, calculate the classification features in each signal frame, and merge the signal frames based on the classification features; S400 divides the electrical signal into multiple speech segments based on the merging result, inputs all speech segments into different recognition models, and outputs consultation information.

[0036] Furthermore, the method also includes step S500, which involves inputting information into a decision tree model to output consultation results.

[0037] Furthermore, in S100, when the user initiates a consultation request, the method of acquiring the input consultation voice through the microphone device and converting it into an electrical signal specifically includes: after receiving the consultation request initiated by the user, the user controls the sensor with conversion function on the microphone to acquire the input consultation voice, and at the same time converts it into a digital signal through the analog-to-digital converter installed on the sensor, and obtains an electrical signal after the converted digital signal is processed by noise reduction filtering.

[0038] Furthermore, in S200, the method of locating segmentation points in the electrical signal through short-time Fourier transform and dividing the electrical signal into multiple signal frames based on these segmentation points specifically includes: first, enhancing the high-frequency components in the electrical signal by passing it through a first-order high-pass filter, wherein the first-order high-pass filter is:

[0039] Where X'(t) represents the t-th signal after passing through a first-order high-pass filter, X(t) represents the original t-th signal, and a is a parameter with a value of 0.9.

[0040] The signal passing through the filter is divided into multiple raw frames with a length of 20ms. Each raw frame corresponds to a segment of electrical signal. Each raw frame is weighted by a window function, specifically a Hamming window function. Each weighted electrical signal is then subjected to a Fourier transform to obtain the corresponding spectral information. All spectral information is used to construct a spectrogram, and the envelope of the spectrogram is plotted. The envelope is used as the speech curve of the input speech. The speech curve is marked with the start and end times of each word's pronunciation using an endpoint detection algorithm as segmentation points. Based on these segmentation points, the speech curve is divided into multiple signal frames.

[0041] Furthermore, in S300, the method for calculating the classification features in each signal frame and merging the signal frames based on the classification features specifically includes: traversing all the curve segments contained in the signal frames sequentially along the signal acquisition time from the starting point of the speech curve; for each traversed curve segment, the maximum value of the amplitude in the curve is recorded as Vmax, and the minimum value is recorded as Vmin; the absolute value of the difference between the maximum and minimum values ​​is taken as the transition value in the curve segment; the length of the line segment between the frequency point corresponding to the maximum value and the frequency point corresponding to the minimum value in the curve segment is taken as the frequency difference of the curve segment; and the ratio of the transition value to the frequency difference is taken as the classification feature of the curve segment. Using i as the sequence number of the original frame, Fram i This represents the classification feature in the i-th original frame. Starting from the first original frame, the next original frame is traversed in the order of signal acquisition time. If the classification feature in the current original frame is greater than the average of the classification features in all previously traversed original frames, then the signal end point in the current original frame is marked as the segmentation point.

[0042] Furthermore, in S400, the method of dividing the electrical signal into multiple speech segments based on the merging result and inputting all speech segments into different recognition models to output consulting information specifically includes: dividing the electrical signal into multiple speech segments based on the optimal segmentation scheme, and sequentially inputting each target speech segment into a pre-trained multilingual speech recognition model to obtain the text content and language identifier corresponding to each speech segment; combining the engineering domain knowledge base and preset rules to perform semantic parsing and intent recognition on the text content; extracting key information related to engineering design parameters, construction schemes, safety risks, equipment selection, cost control, etc. from the recognition results; generating corresponding engineering consulting information and outputting it.

[0043] Furthermore, the multilingual speech recognition model is a multilingual acoustic model based on the self-supervised pre-trained av2vec2.0 architecture combined with a language model.

[0044] Furthermore, the engineering domain knowledge base is an engineering specification and standard library built on a relational database.

[0045] Furthermore, the preset rule is an IF-THEN form expert rule defined in advance by engineering experts. Example

[0046] This embodiment 2 replaces the method of calculating the classification features in each signal frame and merging the signal frames based on the classification features in embodiment 1. Specifically, it iterates through the classification features, fundamental frequency, fundamental frequency change rate, pitch span and resonance amplitude of all signal frames within the range of i.

[0047] The fundamental frequency of the signal frame is obtained using the pitch function in MATLAB.

[0048] Furthermore, the fundamental frequency change rate is the absolute value of the difference between the fundamental frequencies of adjacent signal frames.

[0049] Specifically, for any given signal frame, the fundamental frequency change rate is the absolute value of the difference between the fundamental frequency of the current signal frame and the fundamental frequency of the most recently traversed signal frame.

[0050] Furthermore, the pitch span is the absolute value of the difference between the highest and lowest fundamental frequencies of the signal frame within the time window, and the time window length is 8 frames.

[0051] Furthermore, the resonance amplitude is calculated by taking the polynomial roots of the results after LPC linear prediction of all signal frames into a sequence and arranging them in order of magnitude. The positive frequencies of the paired conjugate complex roots are then obtained.

[0052] For the i-th signal frame, a feature vector Xi is constructed. This feature vector contains the classification features of the signal frame, the fundamental frequency, the rate of change of the fundamental frequency, the pitch span, and the resonance amplitude. The same method is used to construct the feature vectors of all signal frames, forming a one-dimensional feature vector sequence {Xi}, i∈[0,N]. Using the end point of any signal frame as a segmentation point, the sequence {Xi} is divided into K segments. Each segment contains... One signal frame, This invention introduces a segmentation scheme S, which is used to represent the boundary positions of each speech segment. Specifically, the segmentation scheme S is represented as an ordered set of boundary indices: ,in =0, And for any satisfy ,and , Indicates the first division The first speech segment, also representing the sequence number of the end point of the signal frame, is formed by the sequence of frames between adjacent boundary indices in all signal frames. The frame index range corresponding to each audio segment is: to ,Include Given 10 signal frames, assuming that the signal frames of the same language follow a Gaussian distribution, calculate the mean of each segment as 1. The covariance matrix is ; Segmentation scheme S is used to divide all signal frames into K consecutive speech segments.

[0053] Furthermore, this invention determines the optimal segmentation scheme S by calculating the complexity of all segmentation schemes, specifically expressed as: in As an interval [ The cost of internal speech segments, and These represent the index values ​​of the two endpoints of the speech segment. Used to characterize the stability of acoustic features within the speech segment. To correspond with the number of speech segments Related redundancy assessment values.

[0054] Furthermore, The expression is as follows:

[0055] in Indicates the first The t-th signal frame in a speech segment Indicates the first The number of signal frames contained in each speech segment, where T is the transpose symbol; Furthermore, The specific derivation process is as follows: ; Where d is the dimension of the feature vector, and r is the index of the dimension in the feature vector. Indicates the first The number of speech frames contained in a speech segment. Indicates the first Dimensional features in The sequence of values ​​on a signal frame, Var(·) is the variance calculation function, used to calculate the variance of the eigenvalues ​​of each dimension in the eigenvector of the signal frame.

[0056] All possible segmentation schemes are identified by combination, and the model complexity of each segmentation scheme is calculated. The segmentation scheme with the lowest complexity is then selected as the optimal segmentation scheme.

[0057] Furthermore, this invention also provides an embodiment of a machine learning-based end-to-end engineering consulting system, such as... Figure 2 The diagram shows a structural diagram of a machine learning-based full-process engineering consulting system according to the present invention. This embodiment of the machine learning-based full-process engineering consulting system includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps described in the above embodiment of the machine learning-based full-process engineering consulting system.

[0058] The system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program in a unit of the following system: A signal conversion unit is used to acquire input speech via a sensor on the microphone and convert it into a digital signal; The frequency domain conversion unit is used to convert digital signals into frequency domain signals through Fourier transform and divide them into signal frames; The signal segmentation unit is used to calculate the classification features in the converted frequency domain signal frame and divide the digital signal into multiple speech segments based on the classification features; The information extraction unit is used to extract consultation information from the sealed speech segments through the model.

[0059] The machine learning-based end-to-end engineering consulting system described above can run on computing devices such as desktop minicomputers, laptops, handheld computers, and cloud servers. The system that can run on this machine learning-based end-to-end engineering consulting system may include, but is not limited to, processors and memory. Those skilled in the art will understand that the examples described are merely illustrations of a machine learning-based end-to-end engineering consulting system and do not constitute a limitation on such a system. It may include more or fewer components, combinations of certain components, or different components. For example, the machine learning-based end-to-end engineering consulting system may also include input / output devices, network access devices, buses, etc.

[0060] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the machine learning-based full-process engineering consulting system, connecting various parts of the system through various interfaces and lines.

[0061] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the machine learning-based full-process engineering consulting system by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0062] Although the invention has been described in considerable detail and particularly with regard to several of the described embodiments, it is not intended to limit itself to any of these details or embodiments or any particular embodiment, thereby effectively covering the intended scope of the invention. Furthermore, the invention has been described above with respect to embodiments foreseeable by the inventors in order to provide a useful description, and non-substantial modifications to the invention that have not yet been foreseen may still represent equivalent modifications.

Claims

1. A machine learning-based full-process engineering consulting method, characterized in that, The method includes the following steps: S100: When a user initiates a consultation request, the input consultation voice is acquired through a microphone device and converted into an electrical signal. S200 uses short-time Fourier transform to locate segmentation points in the electrical signal and divides the electrical signal into multiple signal frames based on these segmentation points; S300, calculate the classification features in each signal frame, and merge the signal frames based on the classification features; S400 divides the electrical signal into multiple speech segments based on the merging result, inputs all speech segments into different recognition models, and outputs consultation information. In S300, the merging of signal frames based on classification features includes: Within the range of values ​​of i, iterate through all the classification features, fundamental frequency, fundamental frequency change rate, pitch span, and resonance amplitude of all signal frames; For the i-th signal frame, a feature vector Xi is constructed. This feature vector includes the classification features of the signal frame, the fundamental frequency, the rate of change of the fundamental frequency, the pitch span, and the resonance amplitude. The same method is used to construct the feature vectors of all signal frames, forming a one-dimensional feature vector sequence {Xi}, i∈[0,N]. Using the end point of any signal frame as a segmentation point, the sequence {Xi} is divided into K segments. Each segment contains... One signal frame; A segmentation scheme S is introduced to represent the boundary positions of each speech segment. The segmentation scheme S is represented as an ordered set of boundary indices: ; in =0, And for any ,satisfy ,and , Indicates the first division The first speech segment, also representing the sequence number of the end point of the signal frame, is formed by the sequence of frames between adjacent boundary indices in all signal frames. The frame index range corresponding to each speech segment is: to ,Include Given 10 signal frames, assuming that the signal frames of the same language follow a Gaussian distribution, calculate the mean of each segment as 1. The covariance matrix is ; The optimal segmentation scheme S is determined by calculating the complexity of all segmentation schemes. The specific expression is: ； in Indicates interval [ The cost of internal speech segments, and These represent the index values ​​of the two endpoints of the speech segment. Used to characterize the stability of acoustic features within the speech segment. To correspond with the number of speech segments Related redundant assessment values.

2. The whole-process engineering consulting method based on machine learning according to claim 1, characterized in that, In S100, when the user initiates a consultation request, the method of acquiring the input consultation voice through the microphone device and converting it into an electrical signal is as follows: After receiving the consultation request initiated by the user, the user controls the sensor with conversion function on the microphone to acquire the input consultation voice, and at the same time converts it into a digital signal through the analog-to-digital converter installed on the sensor. The converted digital signal is then processed by noise reduction filtering to obtain an electrical signal.

3. The whole-process engineering consulting method based on machine learning according to claim 1, characterized in that, In S200, the method of locating segmentation points for electrical signals through short-time Fourier transform and dividing electrical signals into multiple signal frames based on segmentation points specifically includes: first, passing the electrical signal through a first-order high-pass filter to enhance the high-frequency components in the signal; dividing the signal after passing through the filter into multiple original frames with a length of 20ms, each original frame corresponding to a segment of electrical signal; sequentially applying a window function to weight each original frame; performing a Fourier transform on each weighted electrical signal to obtain the corresponding spectral information; constructing a spectrogram from all spectral information; drawing the envelope of the spectrogram; using the envelope as the speech curve of the input speech; marking the start and end times of each word's pronunciation on the speech curve using an endpoint detection algorithm as segmentation points; and dividing the speech curve into multiple signal frames based on the segmentation points.

4. The whole-process engineering consulting method based on machine learning according to claim 1, characterized in that, In S300, the method for calculating the classification features in each signal frame and merging the signal frames based on the classification features is as follows: on the speech curve, starting from the curve start point, traverse all the curve segments contained in the signal frames sequentially along the signal acquisition time. For each traversed curve segment, take the maximum and minimum values ​​of the amplitude in the curve, and take the absolute value of the difference between the maximum and minimum values ​​as the transition value in the curve segment. Take the length of the line segment between the frequency point corresponding to the maximum value and the frequency point corresponding to the minimum value in the curve segment as the frequency difference of the curve segment. Take the ratio of the transition value to the frequency difference as the classification feature of the curve segment. Using i as the sequence number of the original frame, Fram i Let i represent the classification feature in the i-th original frame. Starting from the first original frame, the next original frame is traversed in the order of signal acquisition time. If the classification feature in the current original frame is greater than the average of the classification features in all previously traversed original frames, the signal end point in the current original frame is marked as the segmentation point. The electrical signal is divided into multiple speech segments according to the position of the marked point.

5. The whole-process engineering consulting method based on machine learning according to claim 4, characterized in that, The The expression is as follows: ; in Indicates the first The t-th signal frame in a speech segment Indicates the first The number of speech frames contained in a speech segment, where T is the transpose symbol.

6. The whole-process engineering consulting method based on machine learning according to claim 5, characterized in that, The specific derivation process is as follows: ; Where d is the dimension of the feature vector, and r is the index of the dimension in the feature vector. Indicates the first The number of speech frames contained in a speech segment. Indicates the first Dimensional features in The sequence of values ​​on each signal frame; Var(·) is the variance calculation function, used to calculate the variance of the eigenvalues ​​in each dimension of the eigenvector of the signal frame; All possible segmentation schemes are identified through combination, and the model complexity of each segmentation scheme is calculated. The segmentation scheme with the lowest complexity is selected as the optimal segmentation scheme.

7. The whole-process engineering consulting method based on machine learning according to claim 1, characterized in that, In S400, the method for dividing the electrical signal into multiple speech segments based on the merging result and inputting all speech segments into different recognition models to output consulting information is as follows: Based on the optimal segmentation scheme, the electrical signal is divided into multiple speech segments, and each target speech segment is sequentially input into a pre-trained multilingual speech recognition model to obtain the text content and language identifier corresponding to each speech segment. Combining the engineering domain knowledge base and preset rules, semantic parsing and intent recognition are performed on the text content. Key information related to engineering design parameters, construction schemes, safety risks, equipment selection, and cost control is extracted from the recognition results to generate corresponding engineering consulting information and output it.

8. A machine learning-based end-to-end engineering consulting system, characterized in that, The machine learning-based full-process engineering consulting system includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the machine learning-based full-process engineering consulting method according to any one of claims 1-7.

Images