Automatic voice ticket generation system and method based on multi-modal processing

Abstract

The application provides an automatic voice work order generation system and method based on multi-modal processing, relates to the technical field of voice work order generation, and comprises a multi-modal input layer for collecting voice data, device metadata and auxiliary modal data; a voice processing layer for performing audio preprocessing on the voice data to extract audio feature information and transcribing the audio feature information into text information; a semantic analysis layer for receiving the device metadata and the auxiliary modal data, the audio feature information and the transcribed text information, and performing multi-modal field adaptive analysis in combination with historical work order data; and a work order output layer for converting a structured work order element into a standardized work order after receiving the structured work order element. The application can solve the technical problem that in the prior art, due to high dependence on manual intervention and fragmentation of multi-modal data processing, the voice recognition accuracy is low, which further affects the work order generation efficiency, and through end-to-end automatic conversion of voice data to a structured work order, the work order generation efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of voice work order generation technology, and in particular to an automatic voice work order generation system and method based on multimodal processing. Background Technology

[0002] Traditional work order generation methods typically rely on manual operation. Maintenance personnel must manually listen to voice messages, organize information, and fill out work orders. This is not only time-consuming and tedious, but also prone to high error rates due to human error, especially in complex environments where speech recognition accuracy is significantly affected, resulting in work orders lacking necessary details. Furthermore, there is a fragmentation problem when processing multimodal data; voice, text, and device metadata are often processed independently without an effective collaborative optimization mechanism, leading to incomplete or lost work order information.

[0003] In summary, existing technologies suffer from low speech recognition accuracy due to high reliance on human intervention and fragmented multimodal data processing, which further impacts work order generation efficiency. Summary of the Invention

[0004] The purpose of this application is to provide an automatic voice work order generation system and method based on multimodal processing, in order to solve the technical problems in the prior art, which are low voice recognition accuracy due to high dependence on manual intervention and fragmented multimodal data processing, which further affects the efficiency of work order generation.

[0005] In view of the above problems, this application provides an automatic voice work order generation system and method based on multimodal processing.

[0006] In a first aspect, this application provides an automatic voice work order generation system based on multimodal processing, wherein the automatic voice work order generation system based on multimodal processing includes: a multimodal input layer for collecting voice data, device metadata, and auxiliary modal data; a voice processing layer for performing audio preprocessing on the voice data transmitted by the multimodal input layer to extract audio feature information, and then transcribing the voice data into transcribed text information; a semantic parsing layer for receiving device metadata and auxiliary modal data transmitted by the multimodal input layer, and audio feature information and transcribed text information input by the voice processing layer, and performing multimodal domain adaptive parsing in conjunction with historical work order data to output structured work order elements; and a work order output layer for receiving the structured work order elements transmitted by the semantic parsing layer, and converting the structured work order elements into standardized work orders based on standardized work order templates, wherein the standardized work orders are automatically populated into the maintenance work order system; wherein the multimodal input layer, voice processing layer, semantic parsing layer, and work order output layer are cascaded.

[0007] Optionally, the auxiliary modal data is received through a preset LCD screen input interface; when the voice data is a real-time call voice stream, the real-time call voice stream is captured by listening to the RTP protocol through a softphone client, wherein the real-time call voice stream is stored in a circular buffer; when the voice data is a SIP phone recording file, the SIP phone recording file is imported through a SIP protocol parsing interface; when collecting the voice data, the device metadata is synchronously associated.

[0008] Optionally, when the voice data is a real-time call voice stream: Step a: Read voice segments from the circular buffer based on a preset time window, and drive the FunASR engine streaming recognition module to perform speech transcription of the voice segments, outputting incremental recognition text; Step b: Call the telephone scenario adaptation model to correct speech transcription errors in the incremental recognition text, and output the transcribed text information.

[0009] Optionally, when the voice data is a SIP phone recording file: Step c: Drive the FunASR engine offline recognition module to perform parallel speech transcription through an asynchronous queue scheduling mechanism to obtain the initial transcribed text; Step d: Call the recording scenario adaptation model to correct speech transcription errors in the initial transcribed text and output the transcribed text information.

[0010] Optionally, during the process of driving the FunASR engine streaming recognition module to perform speech transcription of speech segments, automatic transcription WER statistics are performed; when the transcription WER > 3%, the retraining of the telephone scenario adaptation model is triggered.

[0011] Optionally, telephone recording segment features, accent variation features, speech rate variation features, and environmental noise features are extracted from the speech data transmitted from the multimodal input layer; based on the telephone recording segment features, accent variation features, and speech rate variation features, the parameters of the telephone scene adaptation model or the recording scene adaptation model are tuned and optimized; based on the environmental noise features, spectral subtraction noise reduction and deep learning noise reduction are collaboratively performed on the speech data to output the audio feature information.

[0012] Optionally, the device metadata, auxiliary modal data, audio feature information, and transcribed text information are concatenated to obtain multimodal fusion features; the historical work order data is used for context supplementation to obtain multimodal supplementary features; the Tongyi Qianwen Big Language Model is used, and LoRA fine-tuning is used to inject operation and maintenance domain terms into the multimodal supplementary features to extract key entities of the work order; the key entities of the work order are subjected to context correction and verification based on the regular rule base, and the structured work order elements are output.

[0013] Optionally, in the scenario of obtaining the transcribed text information by transcribing the real-time call voice stream: after the voice processing layer reads the voice segments of the real-time call voice stream from the circular buffer based on a preset time window, it drives the FunASR engine streaming recognition module to perform voice transcription and output incremental recognition text; the voice processing layer also stores the incremental recognition text in a temporary buffer of a double-buffering mechanism; after the semantic parsing layer extracts the incremental recognition text from the temporary buffer, it performs multimodal semantic parsing in combination with the device metadata and real-time audio features, extracts the key entities of the work order to generate the structured work order elements, and synchronously updates the structured work order elements to the formal buffer of the double-buffering mechanism.

[0014] Secondly, this application also provides an automatic voice work order generation method based on multimodal processing, wherein the automatic voice work order generation method based on multimodal processing includes: receiving voice data, device metadata, and auxiliary modal data; performing audio preprocessing on the voice data to extract audio feature information, and then transcribing the voice data into transcribed text information; performing multimodal fusion on the device metadata, auxiliary modal data, audio feature information, and transcribed text information, and then performing multimodal domain adaptive parsing in conjunction with historical work order data to output structured work order elements; and converting the structured work order elements into standardized work orders based on standardized work order templates, wherein the standardized work orders are automatically populated into the operation and maintenance work order system.

[0015] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0016] The system employs a multimodal input layer to collect voice data, device metadata, and auxiliary modal data. A voice processing layer preprocesses the voice data from the multimodal input layer to extract audio features, then transcribes the voice data into transcribed text. A semantic parsing layer receives device metadata and auxiliary modal data from the multimodal input layer, along with audio features and transcribed text from the voice processing layer. It then performs multimodal domain-adaptive parsing based on historical work order data to output structured work order elements. A work order output layer receives these structured work order elements from the semantic parsing layer and converts them into standardized work orders based on standardized work order templates. These standardized work orders are automatically populated into the maintenance work order system. The multimodal input layer, voice processing layer, semantic parsing layer, and work order output layer are cascaded. In other words, by integrating multimodal feature collaboration, batch processing, and domain knowledge enhancement into an automatic voice work order generation scheme, end-to-end automated conversion from voice data to structured work orders is achieved, improving voice processing accuracy and significantly increasing work order generation efficiency.

[0017] The above description is merely an overview of the technical solution of this application. To better understand the technical means of this application and to facilitate its implementation according to the description, and to make the above and other objects, features, and advantages of this application more apparent, specific embodiments of this application are described below. It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent through the following description. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the automatic voice work order generation system based on multimodal processing in this application.

[0020] Figure 2 This is a flowchart illustrating the automatic voice work order generation method based on multimodal processing in this application.

[0021] Figure labeling: Multimodal input layer 11, speech processing layer 12, semantic parsing layer 13, work order output layer 14. Detailed Implementation

[0022] This application provides an automatic voice work order generation system and method based on multimodal processing, solving the technical problems in existing technologies where high reliance on manual intervention and fragmented multimodal data processing lead to low speech recognition accuracy, further impacting work order generation efficiency. By integrating multimodal feature collaboration, batch processing, and domain knowledge enhancement into an automatic voice work order generation scheme, end-to-end automated conversion of voice data into structured work orders is achieved, improving voice processing accuracy and thus significantly increasing work order generation efficiency.

[0023] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. It should be understood that this application is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. It should also be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all of them.

[0024] Example 1, please refer to the appendix. Figure 1 This application provides an automatic voice work order generation system based on multimodal processing, wherein the automatic voice work order generation system based on multimodal processing is used to implement the steps of the automatic voice work order generation method based on multimodal processing, and the automatic voice work order generation system based on multimodal processing includes:

[0025] The multimodal input layer 11 is used to collect voice data, device metadata, and auxiliary modal data.

[0026] Furthermore, the multimodal input layer 11 in the automatic voice work order generation system based on multimodal processing is also used for: receiving the auxiliary modal data through a preset LCD screen input interface; when the voice data is a real-time call voice stream, capturing the real-time call voice stream through a softphone client listening to the RTP protocol, wherein the real-time call voice stream is stored in a circular buffer; when the voice data is a SIP phone recording file, importing the SIP phone recording file through a SIP protocol parsing interface; and synchronously associating the device metadata when collecting the voice data.

[0027] Specifically, the multimodal input layer is the first layer of the system architecture, responsible for receiving and aggregating data from different sources and of different types. Multimodal means that the data comes in multiple forms, not just single voice. It supports SIP phone recording (WAV), real-time RTP voice stream capture, is compatible with IP phones, control room phones, and other devices, and can be expanded with LCD screen text input.

[0028] Voice data refers to sound signals processed through speech recognition technology, typically the content spoken by maintenance personnel or customers when reporting faults, including real-time call audio streams and SIP phone recordings. Device metadata is non-voice information related to the device, usually including device identification (such as device ID), device operating status, device type, installation location, etc., providing background information when a fault occurs and helping to improve the efficiency of fault identification and handling. Auxiliary modal data refers to other information that supplements voice data, which may include text input, environmental data, etc., helping to comprehensively understand the voice content and enhance the accuracy of speech recognition.

[0029] If the voice data is a real-time call audio stream, the softphone client listens to the real-time audio stream via the RTP protocol, capturing the call data in real time. The captured audio stream is stored in a circular buffer to ensure that no data is lost during processing. If some segments of the audio stream are not processed in time, these segments are stored at the beginning of the buffer to ensure that data is not lost due to streaming input. If the voice data is a SIP phone recording file, these recording files are imported through the SIP protocol parsing interface. These SIP phone recording files are usually stored in WAV or MP3 format. The audio signals in the files are extracted and converted into digital signals for subsequent processing. Real-time call audio streams refer to audio data transmitted in real time during a call, usually using the RTP protocol. The RTP protocol is widely used in VoIP communication to provide low-latency, real-time audio streams. A circular buffer is a special data buffer that stores data in a circular structure. Data is cyclically overwritten at the end of the buffer, ensuring continuous storage of the real-time audio stream. Using a circular buffer can prevent data loss due to data delays or untimely processing. SIP is a communication protocol used to initiate, maintain, and terminate real-time sessions, widely used in Internet voice communication. The SIP protocol parsing interface is used to parse the voice data in SIP telephone recordings and convert it into digital data that can be processed later.

[0030] While collecting voice data, auxiliary modal data from equipment or maintenance personnel is received via a pre-set LCD screen input interface. Descriptions of equipment malfunctions and manually entered fault diagnosis results from operators help further improve the accuracy of voice recognition and the quality of work order generation. During voice data collection, equipment metadata, including equipment ID, equipment type, equipment status, and location, is also synchronously associated. This metadata helps provide further contextual information for the voice content. For example, if the voice data mentions a malfunction in a specific classroom, the equipment metadata can provide more contextual information about the malfunction, improving the accuracy and completeness of the work order.

[0031] For offline recordings, WAV format call recordings are obtained through the SIP protocol parsing interface, and recording metadata (call duration, device ID) is collected synchronously. Noise reduction and sampling rate standardization are performed, and audio features (such as Mel-frequency cepstral coefficients) are extracted for audio preprocessing. The FunASR engine is used for batch speech-to-text transcription, and a telephone scenario adaptation model is used to improve accuracy, achieving a word error rate (WER) of ≤3%. A large language model, combined with audio features and historical work order data, performs semantic understanding to extract key entities and intents. Structured work orders are generated according to preset templates and automatically populated into the operations and maintenance system, with a work order generation time of ≤30 seconds.

[0032] For real-time voice, a softphone client is deployed to monitor RTP voice streams, employing a circular buffer for continuous caching to capture real-time voice. The FunASR streaming interface is integrated, outputting incremental recognition results every 500ms and dynamically adjusting Voice Endpoint Detection (VAD) parameters to recognize streaming voice. Semantic parsing is performed by combining real-time voice features (such as speech rate changes and stress positions) with the context, using a double-buffering mechanism to achieve parallel processing of the current segment and background inference for multimodal parsing. The work order fields are progressively updated via the WebSocket protocol, synchronously completing the structured storage of multimodal data (voice + real-time text) for work orders and generating real-time work orders.

[0033] By collecting and processing voice data, device metadata, and auxiliary modal data in real time, the accuracy and efficiency of work order generation are improved. Particularly during real-time voice stream processing, the use of a circular buffer ensures data integrity and prevents data loss due to delays or untimely processing. Simultaneously, the synchronous association of device metadata and auxiliary modal data generates more complete and accurate work orders, significantly reducing the probability of manual intervention and errors.

[0034] The speech processing layer 12 is used to perform audio preprocessing on the speech data transmitted by the multimodal input layer 11 to extract audio feature information, and then transcribe the speech data into transcribed text information.

[0035] Furthermore, the voice processing layer 12 in the automatic voice work order generation system based on multimodal processing is also used for: when the voice data is a real-time call voice stream: step a: read voice segments from the circular buffer based on a preset time window, and drive the FunASR engine streaming recognition module to perform speech transcription of the voice segments, and output incremental recognition text; step b: call the telephone scenario adaptation model to correct speech transcription errors in the incremental recognition text, and output the transcribed text information.

[0036] Furthermore, the voice processing layer 12 in the automatic voice work order generation system based on multimodal processing is also used for: when the voice data is a SIP telephone recording file: step c: drive the FunASR engine offline recognition module to perform parallel speech transcription through an asynchronous queue scheduling mechanism to obtain the initial transcribed text; step d: call the recording scenario adaptation model to correct speech transcription errors in the initial transcribed text and output the transcribed text information.

[0037] Furthermore, the speech processing layer 12 in the automatic speech work order generation system based on multimodal processing is also used to: automatically count the transcription WER during the process of driving the FunASR engine streaming recognition module to perform speech transcription of speech segments; and trigger the retraining of the telephone scenario adaptation model when the transcription WER > 3%.

[0038] Furthermore, the speech processing layer 12 in the automatic speech work order generation system based on multimodal processing is also used to: extract telephone recording audio segment features, accent change features, speech rate change features, and environmental noise features from the speech data transmitted from the multimodal input layer 11; optimize the parameters of the telephone scene adaptation model or the recording scene adaptation model based on the telephone recording audio segment features, accent change features, and speech rate change features; and collaboratively perform spectral subtraction noise reduction and deep learning noise reduction on the speech data based on the environmental noise features, and output the audio feature information.

[0039] Specifically, real-time call audio streams refer to real-time audio data transmitted using the RTP protocol during a call. This is typically used in VoIP systems to provide low-latency, high-real-time audio data streams. When capturing a real-time call audio stream, audio segments are first read from a circular buffer based on a preset time window. The preset time window refers to the duration of each audio data segment extracted from the circular buffer. For example, assuming the time window is set to 500 milliseconds (0.5 seconds), the audio data from the most recent 500 milliseconds is extracted from the circular buffer, ensuring real-time processing of streaming audio data. The FunASR engine's streaming recognition module then performs speech-to-text transcription on the audio segments. This process is real-time, allowing transcription to begin as soon as an audio segment is read, and outputting incrementally recognized text—that is, the text content is gradually updated with each new audio segment. For example, within 500 milliseconds, the FunASR engine might recognize words like "projector" or "cannot start" in a sentence and output them as recognized text.

[0040] In the incremental text recognition generated in the first step, word errors or semantic inaccuracies may occur due to poor speech quality, noise interference, accents, and other factors. The telephone scenario adaptation model, specifically optimized for the challenges of telephone calls, corrects the errors of ordinary speech recognition systems in telephone scenarios. By using the telephone scenario adaptation model, word errors in the recognition results are corrected, resulting in more accurate transcribed text information. For example, in a noisy telephone environment, an item might be identified as information. The adaptation model corrects this error based on context and domain-specific vocabulary, outputting the item as the correct word.

[0041] Telephone scenario-adapted models are speech recognition models specific to telephone call scenarios. Due to noise, speech quality issues, and accent differences in the telephone environment, ordinary speech recognition models may not be able to effectively process speech data during telephone calls. Telephone scenario-adapted models, through optimization for telephone scenarios, can reduce the impact of noise and improve the accuracy of speech recognition. Speech-to-text error correction refers to using algorithms to correct recognition errors based on the initial transcribed text. Errors may be caused by accents, noise, or imperfections in the speech recognition model; the correction model can correct these errors through context or domain knowledge. Incremental text recognition refers to gradually outputting recognition results during streaming speech recognition. Unlike traditional batch recognition, incremental recognition allows for the output of partial text before the complete input of speech data, and the text content is gradually updated as the speech stream continues to input.

[0042] SIP is a protocol for enabling real-time voice and video communication. SIP call recordings are audio recordings generated during SIP calls, typically stored in WAV or MP3 formats, containing the voice content of the call and serving as the foundation for subsequent speech recognition and transcription. When the voice data is a SIP call recording, an asynchronous queue scheduling mechanism efficiently processes a large number of SIP call recordings. First, the call recordings to be processed are added to an asynchronous queue in sequence. The asynchronous queue can distribute tasks to multiple parallel processing units, thus achieving batch processing. The FunASR engine's offline recognition module is responsible for executing these tasks, with each recording file processed in parallel within a different processing unit. Each recording file is then transcribed to generate preliminary transcribed text. For example, assuming there are 1000 SIP call recordings, the system uses an asynchronous queue to distribute them to 10 processing units for simultaneous transcription, greatly improving processing efficiency and reducing the total transcription time. If the processing time for each recording file is 10 seconds, then through parallel processing, the total processing time for 1000 recordings is 100 seconds, significantly improving efficiency compared to serial processing.

[0043] Asynchronous queue scheduling is a data processing method that allows the system to process multiple tasks in parallel without blocking the main thread by adding tasks sequentially to a queue and executing them asynchronously. This improves data processing efficiency and is particularly suitable for processing large-scale data, such as scenarios in speech recognition that require processing a large number of audio files. The FunASR engine's offline recognition module is part of the FunASR engine and is specifically designed for handling offline speech recognition tasks. It can batch process audio files and perform speech-to-text transcription without real-time constraints, outputting the initial transcribed text. The FunASR engine's offline recognition module is typically suitable for non-real-time audio data such as SIP phone recordings and conference recordings.

[0044] The recording scenario adaptation model is a speech recognition model designed for specific scenarios (such as telephone calls or meeting recordings). It optimizes the accuracy of speech transcription based on factors such as noise, accents, and device quality within that scenario. Once the initial transcribed text is obtained, the recording scenario adaptation model is invoked to correct word errors. Because telephone calls typically involve accent differences, background noise, or call quality issues, the initial transcribed text may contain some errors. The recording scenario adaptation model analyzes this text and corrects it using contextual information. For example, if "projector" is mistakenly transcribed as "projection" in a telephone recording, the adaptation model will use contextual information to correct it, outputting "projector" as the correct word.

[0045] Speech-to-text error correction refers to correcting errors in the initially generated transcribed text to improve its accuracy. This includes correcting spelling errors and misunderstandings in speech recognition (e.g., errors caused by accents, background noise, or equipment problems). The model's error correction mechanism automatically corrects these errors, outputting accurate transcribed text.

[0046] During the speech transcription process of speech segments driven by the FunASR engine's streaming recognition module, the transcription WER (Word Error Rate) is calculated in real time to evaluate the accuracy of the transcription results. The FunASR engine is a speech recognition engine that uses deep learning technology to process audio data, converting speech signals into text. The FunASR engine is used for real-time streaming speech recognition and progressively outputs incrementally recognized text. Streaming recognition is one of the modules in a speech recognition system, specifically designed for processing continuously flowing streams of speech data in real time. Unlike traditional batch recognition methods, streaming recognition can progressively output recognition results, making it suitable for real-time or near-real-time applications such as telephone calls, live streaming, and conferences.

[0047] Word Error Rate (WER) is a commonly used metric for evaluating the accuracy of speech recognition systems. It represents the difference between the recognized text and the actual text, and is calculated using the formula WER = (S + D + I) / N, where S is the number of substitution errors, D is the number of deletion errors, I is the number of insertion errors, and N is the number of words in the actual text. A lower WER indicates more accurate recognition. In practical applications, WER typically needs to be kept below 3% to ensure high recognition accuracy.

[0048] During speech-to-text transcription, the WER (Word Error Rate) is calculated in real time. Each time the FunASR engine completes the transcription of a speech segment, it automatically compares the transcribed text with the original text (or manually proofread text) to calculate the current error rate. For example, suppose the transcribed text of a speech segment is "The projector cannot start," while the original text is "The projector cannot start." If they match, the WER is 0. If there are word errors or omissions, the WER is calculated based on the error type (such as substitution, deletion, or insertion errors).

[0049] A threshold (e.g., 3%) is set. When the WER (Warning Error Rate) of the transcribed text exceeds this threshold, it indicates a significant deviation in the current speech recognition result. This may be due to background noise, accents, signal problems, or other factors leading to reduced recognition accuracy. In this case, the retraining mechanism of the telephone scene adaptation model is triggered. The performance of the telephone scene adaptation model is improved using the current speech data and error feedback. Audio data containing errors and corresponding correct transcriptions are collected. By analyzing the erroneous transcriptions, the most common types of errors are identified (e.g., vocabulary errors, accent problems, noise interference, etc.). The telephone scene adaptation model is adjusted according to the error type to optimize its adaptability to noise, accents, and call quality issues. The optimized model can better handle similar problems, improving the accuracy of speech recognition. After retraining, the new telephone scene adaptation model is loaded and put into use, effectively recognizing speech content in the telephone environment, reducing the error rate, and improving the accuracy of subsequent transcriptions.

[0050] A telephone scenario-adapted model is a speech recognition model optimized for telephone call environments. Due to background noise, accent differences, and equipment limitations often present in telephone communication, ordinary speech recognition models may perform poorly in such environments. The telephone scenario-adapted model improves recognition accuracy by training and adjusting for telephone call features. Retraining refers to retraining an existing model using new data or feedback information to improve its performance. In speech recognition systems, if the current model's recognition accuracy is insufficient, it can be optimized through retraining to improve the recognition accuracy.

[0051] From the speech data transmitted from the multimodal input layer, features such as telephone recording segment characteristics, accent variation characteristics, speech rate variation characteristics, and environmental noise characteristics are extracted. Telephone recording segment characteristics are mainly extracted from the speech spectrum analysis, including frequency, pitch, and duration, used to identify the content and context of the speech. Accent variation characteristics, by analyzing the pitch, stress, and rhythm of pronunciation, can identify speaker accent differences and influence the adjustment of the recognition model. Speech rate variation characteristics, by analyzing the speaker's speaking speed, identify changes in the speaker's speaking rate, thus affecting the accuracy of speech transcription and recognition. Environmental noise characteristics, through the analysis of background noise, determine its impact on speech; this noise may originate from the external environment or poor telephone signal quality.

[0052] Based on the characteristics of telephone recording audio segments, accent variations, and speech rate variations, the parameter tuning and optimization of either the telephone scene adaptation model or the recording scene adaptation model are performed. According to accent variation and speech rate variation characteristics, the speech signal during the recognition process is dynamically tuned to optimize the model's recognition accuracy. For example, when a fast speech rate is detected, the model will speed up processing and segment the speech. For telephone recording audio segment features, the adaptation model will adjust its audio processing algorithm to adapt to audio distortion and sound quality variations during telephone calls.

[0053] Based on the extracted environmental noise features, two noise reduction methods—spectral subtraction and deep learning—are applied to the speech data. Spectral subtraction is used to analyze the noise frequency bands in the speech signal and subtract the noise portion, preserving the main components of the speech and significantly reducing interference from low-frequency noise, wind noise, and other external noise. Deep learning noise reduction technology is applied, using a trained neural network to finely remove noise. Especially in complex noise environments, deep learning noise reduction can more accurately remove interference and enhance the clarity of the speech signal. After noise reduction, audio feature information is output, including the denoised spectral features, Mel-frequency cepstral coefficients (MFCC), pitch, and tone, for subsequent speech recognition and transcription.

[0054] Google's spectral subtraction denoising is a common noise reduction method that analyzes and removes noise frequency bands from audio signals while preserving the main speech components. Spectral subtraction is often used in speech recognition to reduce background noise interference. Deep learning denoising is a technique that uses deep neural networks (such as convolutional neural networks) to identify and remove noise from audio signals. Compared to traditional spectral subtraction, deep learning denoising learns a large number of noise features, enabling more accurate noise removal while preserving the speech signal. Audio feature information is various features extracted from audio signals to characterize speech, including the speech spectrum, Mel-frequency cepstral coefficients (MFCC), pitch, tone, and temporal features. These features are crucial to the accuracy of speech recognition systems.

[0055] The semantic parsing layer 13 is used to perform multimodal domain adaptive parsing in combination with historical work order data after receiving device metadata and auxiliary modal data transmitted by the multimodal input layer 11, audio feature information and transcribed text information input by the speech processing layer 12, and output structured work order elements.

[0056] Furthermore, the semantic parsing layer 13 in the automatic voice work order generation system based on multimodal processing is also used to: concatenate the device metadata, auxiliary modal data, audio feature information, and transcribed text information to obtain multimodal fusion features; use the historical work order data for context supplementation to obtain multimodal supplementary features; use the Tongyi Qianwen Big Language Model and inject operation and maintenance domain terms into the multimodal supplementary features through LoRA fine-tuning to extract key entities of the work order; perform context correction and verification on the key entities of the work order based on the regular rule base, and output the structured work order elements.

[0057] Furthermore, the semantic parsing layer 13 in the automatic voice work order generation system based on multimodal processing is also used for: in the scenario of obtaining the transcribed text information by transcribing the real-time call voice stream: after the voice processing layer 12 reads the voice segment of the real-time call voice stream from the circular buffer based on a preset time window, it drives the FunASR engine streaming recognition module to perform voice transcription and output incremental recognition text; the voice processing layer 12 stores the incremental recognition text into a temporary buffer of the double buffering mechanism; after the semantic parsing layer 13 extracts the incremental recognition text from the temporary buffer, it performs multimodal semantic parsing in combination with the device metadata and real-time audio features, extracts the key entities of the work order to generate the structured work order elements, and synchronously updates the structured work order elements to the formal buffer of the double buffering mechanism.

[0058] Specifically, device metadata, auxiliary modal data, audio feature information, and transcribed text information are concatenated to construct a multimodal fusion feature vector. These four different types of information (device metadata, auxiliary modal data, audio feature information, and transcribed text information) are combined in a specific way to form a comprehensive feature representation. This is equivalent to merging a photograph (device metadata), a text description (auxiliary modal data), some color and texture information (audio feature information), and the photograph's text description (transcribed text information) to form a more comprehensive understanding of the scene.

[0059] Find historical work orders related to the current device ID or location (e.g., 2024-05-01, projector bulb malfunction in classroom A205), and integrate their contextual information (such as common problems and device lifespan) into the current multimodal features to obtain supplementary multimodal features. Historical work order data consists of previously processed and completed work order records, containing past fault types, processing procedures, and device information. Utilize historical work order data to enrich the multimodal fusion features.

[0060] The Tongyi Qianwen large language model was used to perform semantic understanding and intent classification on the above features. Through LoRA fine-tuning, the model was pre-injected with professional terms such as projector color cast, central control unit crash, and bulb malfunction, and its ability to extract the triplet of device location + device type + fault description was optimized. For example, given the transcribed text: "The teacher said the projector in A205 cannot be turned on," the model can accurately extract: Location: A205, Device Type: Projector, Fault Phenomenon: Cannot be turned on. Instead of directly using the original large model, LoRA fine-tuning technology was employed. Specifically, a large corpus of language related to the operation and maintenance of multimedia classrooms in universities (containing the aforementioned operation and maintenance terminology, such as projector color cast and central control unit crash) was used to fine-tune the Tongyi Qianwen model using LoRA. This fine-tuning process is like implanting a specialized domain cerebellum (LoRA adapter) into the general brain (large model), enabling it to better understand and recognize professional terms and extract key information when processing operation and maintenance-related tasks.

[0061] To ensure the accuracy of the extracted key entities, a regular expression rule library is used for contextual correction and validation. The initially extracted key entities for work orders undergo syntax and contextual validation using the regular expression rule library. For example, if classroom 205 is identified, but the rule requires the format A205, it will be corrected by combining the equipment metadata; if the fault description is a blank screen, the rule library can determine that the corresponding projector has no image, thus standardizing the field. The field information extracted by the model and validated by the rules is organized into a unified format, such as a JSON object, and the output is data that can be directly written into the work order system. If the fault type extracted by the model is not in the list, or the format is not standardized, the rule library will attempt to correct it or mark it as an anomaly, requiring manual review. After this validation step, the output is a structured, standardized, and highly accurate key element of the work order.

[0062] By fusing multimodal information (not just speech and text, but also device status, historical data, and voice features), the model can more comprehensively understand user intent and fault conditions. LoRA fine-tuning enables the general-purpose large language model to quickly adapt to specific operational domains, accurately identifying and understanding industry terminology rather than treating it as ordinary vocabulary, thus avoiding erroneous extraction or omissions caused by the model's inability to understand technical terms. The introduction of a regular expression rule base ensures that the extracted key entities conform to predefined formats and standards.

[0063] In scenarios involving transcribing real-time call audio streams to obtain transcribed text information: the speech processing layer reads audio segments from a circular buffer based on a preset time window (e.g., 500ms). The circular buffer ensures continuous processing of the latest audio data and avoids data loss due to unprocessed audio streams. The read audio segments are passed to the FunASR engine's streaming recognition module, which transcribes the speech content step by step through streaming recognition. As audio segments arrive, the FunASR engine generates incremental recognition text in real time. This text is based on partial recognition results output in real time for the current audio segment. The generated incremental recognition text is stored in a temporary buffer with a double-buffering mechanism to decouple text generation from subsequent processing, ensuring no data conflicts or delays occur during recognition.

[0064] The progressive filling of work order fields naturally supports the gradual improvement of these fields. As the voice stream is continuously input and parsed, the key elements of the work order can be gradually filled and updated, rather than being generated all at once, which is more in line with the characteristics of real-time interaction. Through a double buffering mechanism and streaming recognition, incrementally recognized text is processed and output quickly, ensuring that work orders can be generated rapidly.

[0065] The semantic parsing layer extracts incrementally recognized text from the temporary buffer and performs multimodal semantic parsing by combining device metadata (such as device ID, device type, etc.) and real-time audio features (such as speech rate, pitch, background noise, etc.). Through multimodal fusion, the context of the written text is accurately understood. After semantic parsing, key entities of the work order, such as device location, faulty device, and fault description, are extracted to generate structured work order elements. These structured work order elements are then synchronously updated to the formal buffer with a double-buffering mechanism. The formal buffer stores the final output structured data, ready to be passed to the subsequent maintenance work order system.

[0066] For offline work order generation, maintenance personnel download WAV format call recordings from the voice server via the SIP protocol. The system automatically obtains data such as time and device identifier from the recording file name; the noise reduction algorithm eliminates fan noise, the sampling rate is standardized, and 13-dimensional MFCC features are extracted; the FunASR offline transcription interface is called, the preprocessed audio data is input, the text content is output, and the transcription WER is automatically counted. If it exceeds 3%, the model is retrained; the entity that the projector in classroom 205 cannot be turned on is identified, and the classroom number format is validated using regular expressions; based on the parsing results, a JSON work order is generated according to the template and automatically filled into the maintenance system through the API interface, while the work order generation time (≤30 seconds) is recorded.

[0067] For real-time work order generation, the softphone client captures RTP voice streams in real time and stores them in a circular buffer (capacity set to 5 seconds of voice data). Every 500ms, it reads voice segments from the buffer, calls the FunASR streaming interface, outputs incrementally recognized text, and dynamically adjusts the VAD parameter to adapt to silence intervals during the call. The semantic parsing engine combines incremental text, real-time audio features (such as sudden volume changes in the current segment), and historical dialogue context to gradually improve entity extraction (such as inferring the specific location from the classroom mentioned earlier). Under the double buffering mechanism, the parsing result of the current segment is stored in a temporary buffer, and the background model inference result is synchronously updated to the formal buffer and pushed to the operation and maintenance system via WebSocket, so that the projection fault field is filled first, and the light bulb burnout details are added later.

[0068] The work order output layer 14 is used to convert the structured work order elements transmitted by the semantic parsing layer 13 into standardized work orders based on the standardized work order template, wherein the standardized work orders are automatically filled into the operation and maintenance work order system.

[0069] The multimodal input layer 11, speech processing layer 12, semantic parsing layer 13, and work order output layer 14 are cascaded.

[0070] Specifically, the work order output layer is the final module of the entire speech-to-text and work order generation system. It is responsible for converting the structured work order elements extracted by the semantic parsing layer into a standardized work order format, ready to be automatically populated into the operation and maintenance management system. The work order output layer typically interfaces with the operation and maintenance work order system, outputting work order information that conforms to the standards. A standardized work order template is a preset format used to define the structure of the work order, including field names, field types, and data formats. It is usually preset according to the needs of the operation and maintenance system to ensure that the generated work order can be directly integrated with the operation and maintenance work order system. For example, the template may include fields such as device ID, fault description, and priority. Structured work order elements refer to the key field data extracted from the work order, such as device location, fault type, and fault description. These fields have a clear format and structure, facilitating subsequent processing and management.

[0071] The maintenance work order system is a system used to manage and track equipment failures, problems, and maintenance work. It records detailed information about equipment failures and helps maintenance personnel manage repair, scheduling, and track repair work.

[0072] The work order output layer receives structured work order elements from the semantic parsing layer and fills them into a pre-defined standardized work order template. The standardized work order template specifies the format and requirements for each field of the work order, ensuring the uniformity and standardization of work order data. Once a standardized work order is generated, it is automatically populated into the maintenance work order system. The population process is implemented via API or data interface, transferring the standardized work order content to the existing work order management system.

[0073] The system comprises multiple cascaded modules, including a multimodal input layer, a speech processing layer, a semantic parsing layer, and a work order output layer. Data originates from the multimodal input layer, is transcribed by the speech processing layer, undergoes deep understanding by the semantic parsing layer, and is finally standardized and output through the work order output layer, forming a complete work order generation chain. Each layer ensures accurate and efficient work orders through precise data transmission and processing.

[0074] Standardized work order templates automatically transform multimodal data and parsing results into compliant work orders, significantly reducing the need for manual intervention and improving work order processing efficiency. Through the collaborative work of the multimodal input layer and semantic parsing layer, work order elements are comprehensively extracted from voice, text, and device data, ensuring the completeness and accuracy of the generated work order data. A double-buffering mechanism and multi-layer cascading design enable real-time processing and output of work orders, guaranteeing efficient and rapid response to meet the real-time needs of operations and maintenance.

[0075] In summary, the automatic voice work order generation system based on multimodal processing provided in this application has the following technical effects:

[0076] The system employs a multimodal input layer to collect voice data, device metadata, and auxiliary modal data. A voice processing layer preprocesses the voice data from the multimodal input layer to extract audio features, then transcribes the voice data into transcribed text. A semantic parsing layer receives device metadata and auxiliary modal data from the multimodal input layer, along with audio features and transcribed text from the voice processing layer. It then performs multimodal domain-adaptive parsing based on historical work order data to output structured work order elements. A work order output layer receives these structured work order elements from the semantic parsing layer and converts them into standardized work orders based on standardized work order templates. These standardized work orders are automatically populated into the maintenance work order system. The multimodal input layer, voice processing layer, semantic parsing layer, and work order output layer are cascaded. In other words, by integrating multimodal feature collaboration, batch processing, and domain knowledge enhancement into an automatic voice work order generation scheme, end-to-end automated conversion from voice data to structured work orders is achieved, improving voice processing accuracy and significantly increasing work order generation efficiency.

[0077] Example 2: Based on the same inventive concept as the automatic voice work order generation system based on multimodal processing in Example 1, this application also provides an automatic voice work order generation method based on multimodal processing. Please refer to the appendix. Figure 2 The automatic voice work order generation method based on multimodal processing includes:

[0078] S100: Receive voice data, device metadata, and auxiliary modal data; S200: Perform audio preprocessing on the voice data to extract audio feature information, and then transcribe the voice data into transcribed text information; S300: Perform multimodal fusion on the device metadata, auxiliary modal data, audio feature information, and transcribed text information, and then perform multimodal domain adaptive parsing in conjunction with historical work order data to output structured work order elements; S400: Convert the structured work order elements into a standardized work order based on a standardized work order template, wherein the standardized work order is automatically populated into the operation and maintenance work order system.

[0079] Furthermore, the automatic voice work order generation method based on multimodal processing also includes: receiving the auxiliary modal data through a preset LCD screen input interface; when the voice data is a real-time call voice stream, capturing the real-time call voice stream through a softphone client listening to the RTP protocol, wherein the real-time call voice stream is stored in a circular buffer; when the voice data is a SIP phone recording file, importing the SIP phone recording file through a SIP protocol parsing interface; and synchronously associating the device metadata when collecting the voice data.

[0080] Furthermore, the automatic voice work order generation method based on multimodal processing also includes: when the voice data is a real-time call voice stream: step a: read voice segments from the circular buffer based on a preset time window, and drive the FunASR engine streaming recognition module to perform speech transcription of the voice segments, and output incremental recognition text; step b: call the telephone scenario adaptation model to correct speech transcription errors in the incremental recognition text, and output the transcribed text information.

[0081] Furthermore, the automatic voice work order generation method based on multimodal processing also includes: when the voice data is a SIP telephone recording file: step c: drive the FunASR engine offline recognition module to perform parallel speech transcription through an asynchronous queue scheduling mechanism to obtain the initial transcribed text; step d: call the recording scenario adaptation model to correct speech transcription errors in the initial transcribed text and output the transcribed text information.

[0082] Furthermore, the automatic voice work order generation method based on multimodal processing also includes: automatically calculating the transcription WER during the process of driving the FunASR engine streaming recognition module to perform speech transcription of speech segments; when the transcription WER > 3%, triggering the retraining of the telephone scenario adaptation model.

[0083] Furthermore, the automatic voice work order generation method based on multimodal processing also includes: extracting features of telephone recording audio segments, accent variation features, speech rate variation features, and environmental noise features from the voice data; optimizing the parameters of the telephone scene adaptation model or the recording scene adaptation model based on the telephone recording audio segment features, accent variation features, and speech rate variation features; and collaboratively performing spectral subtraction noise reduction and deep learning noise reduction on the voice data based on the environmental noise features, and outputting the audio feature information.

[0084] Furthermore, the automatic voice work order generation method based on multimodal processing also includes: concatenating the device metadata, auxiliary modal data, audio feature information, and transcribed text information to obtain multimodal fusion features; using the historical work order data for context supplementation to obtain multimodal supplementary features; using the Tongyi Qianwen Big Language Model and fine-tuning it with LoRA to inject operation and maintenance domain terms into the multimodal supplementary features to extract key entities of the work order; performing context correction and verification on the key entities of the work order based on a regular rule base, and outputting the structured work order elements.

[0085] Furthermore, the automatic voice work order generation method based on multimodal processing also includes: after reading voice segments from the real-time call voice stream from the circular buffer based on a preset time window, driving the FunASR engine streaming recognition module to perform speech transcription and output incremental recognition text; storing the incremental recognition text in a temporary buffer of a double-buffering mechanism; after extracting the incremental recognition text from the temporary buffer, performing multimodal semantic parsing in combination with the device metadata and real-time audio features, extracting key entities of the work order to generate the structured work order elements, and synchronously updating the structured work order elements to the formal buffer of the double-buffering mechanism.

[0086] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Figure 1 The automatic voice work order generation system and specific examples based on multimodal processing in Embodiment 1 are also applicable to the automatic voice work order generation method based on multimodal processing in this embodiment. Through the foregoing detailed description of the automatic voice work order generation system based on multimodal processing, those skilled in the art can clearly understand the automatic voice work order generation method based on multimodal processing in this embodiment. Therefore, for the sake of brevity, it will not be described in detail here.

[0087] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0088] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application also intends to include such modifications and variations.

Claims

1. An automatic voice work order generation system based on multimodal processing, characterized in that, include: A multimodal input layer is used to collect voice data, device metadata, and auxiliary modal data; The speech processing layer is used to perform audio preprocessing on the speech data transmitted by the multimodal input layer to extract audio feature information, and then transcribe the speech data into transcribed text information. The semantic parsing layer is used to perform multimodal domain adaptive parsing in combination with historical work order data after receiving device metadata and auxiliary modal data transmitted by the multimodal input layer, audio feature information and transcribed text information input by the speech processing layer, and output structured work order elements. The work order output layer is used to convert the structured work order elements transmitted by the semantic parsing layer into standardized work orders based on the standardized work order template after receiving them. The standardized work orders are automatically populated into the operation and maintenance work order system. The multimodal input layer, speech processing layer, semantic parsing layer, and work order output layer are cascaded. The multimodal input layer is also used for: The auxiliary modal data is received through a preset LCD screen input interface; When the voice data is a real-time call voice stream, the real-time call voice stream is captured by listening to the RTP protocol through a softphone client, wherein the real-time call voice stream is stored in a circular buffer. When the voice data is a SIP phone recording file, the SIP phone recording file is imported through the SIP protocol parsing interface; When collecting the voice data, the device metadata is simultaneously associated; The speech processing layer is also used for: When the voice data is a real-time call voice stream: Step a: Read speech segments from the circular buffer based on a preset time window, and drive the FunASR engine streaming recognition module to perform speech transcription of the speech segments and output incremental recognized text; Step b: Call the telephone scenario adaptation model to correct word errors in the incremental recognized text and output the transcribed text information; When the voice data is a SIP phone recording file: Step c: Drive the FunASR engine offline recognition module to perform parallel speech transcription through an asynchronous queue scheduling mechanism to obtain the initial transcribed text; Step d: Call the recording scene adaptation model to correct speech-to-text errors in the initial transcribed text and output the transcribed text information; Extract telephone recording segment features, accent variation features, speech rate variation features, and environmental noise features from the speech data transmitted from the multimodal input layer; Based on the characteristics of the telephone recording audio segments, accent variation characteristics, and speech rate variation characteristics, the parameters of the telephone scene adaptation model or the recording scene adaptation model are adjusted and optimized. Based on the environmental noise characteristics, spectral subtraction noise reduction and deep learning noise reduction are performed collaboratively on the speech data to output the audio feature information.

2. The automatic voice work order generation system based on multimodal processing as described in claim 1, characterized in that, The speech processing layer is also used for: During the process of driving the FunASR engine streaming recognition module to perform speech transcription of speech segments, automatic WER statistics are performed. When the WER of the transcription is greater than 3%, the retraining of the telephone scenario adaptation model is triggered.

3. The automatic voice work order generation system based on multimodal processing as described in claim 1, characterized in that, The semantic parsing layer is also used for: By concatenating the device metadata, auxiliary modal data, audio feature information, and transcribed text information, multimodal fusion features are obtained; The historical work order data is used for context supplementation to obtain multimodal supplementary features; Using the Tongyi Qianwen Big Language Model, LoRA fine-tuning is used to inject operation and maintenance domain terms into the multimodal supplementary features to extract key entities of work orders; The key entities of the work order are subjected to context correction and validation based on the regular expression rule base, and the structured work order elements are output.

4. The automatic voice work order generation system based on multimodal processing as described in claim 3, characterized in that, In the scenario of obtaining the transcribed text information by transcribing the real-time call audio stream: After the speech processing layer reads speech segments from the real-time call speech stream from the circular buffer based on a preset time window, it drives the FunASR engine streaming recognition module to perform speech transcription and output incremental recognized text. The speech processing layer stores the incrementally recognized text in a temporary buffer with a double buffering mechanism; After the semantic parsing layer extracts the incremental recognition text from the temporary buffer, it performs multimodal semantic parsing by combining the device metadata and real-time audio features to extract the key entities of the work order and generate the structured work order elements. The structured work order elements are then synchronously updated to the formal buffer of the double buffer mechanism.

5. An automatic voice work order generation method based on multimodal processing, characterized in that, The automatic voice work order generation method based on multimodal processing, as described in any one of claims 1 to 4, is executed by the automatic voice work order generation system based on multimodal processing, wherein the automatic voice work order generation method based on multimodal processing includes: Receives voice data, device metadata, and auxiliary modal data; After performing audio preprocessing on the speech data to extract audio feature information, the speech data is transcribed into transcribed text information; After performing multimodal fusion on the device metadata, auxiliary modal data, audio feature information and transcribed text information, multimodal domain adaptive parsing is performed in combination with historical work order data to output structured work order elements; Based on the standardized work order template, the structural work order elements are converted into standardized work orders, which are then automatically populated into the operation and maintenance work order system.

Images