Speech audio correction using machine learning models

Abstract

The invention discloses speech audio correction using a machine learning model. Systems and techniques for correcting speech audio using a machine learning model are disclosed. The technique includes comparing a first transcription of the input audio to a second transcription of the input audio to identify a modified set of words, where the second transcription is a modified version of the first transcription. The technique includes generating synthetic audio using a text-to-speech machine learning model based on the modified word set, and generating modified synthetic audio using a speech matching machine learning model based on the synthetic audio and at least a portion of the input audio. The technique includes generating an output audio by combining the input audio with the modified synthetic audio.

Description

Technical Field

[0001] At least one embodiment relates to a system for correcting or modifying speech audio using a machine learning model. Background Art

[0002] Machine learning models can be trained to receive audio and output modified audio or text, and to receive text and output modified text or audio. For example, a machine learning model can convert audio into text (e.g., automatic speech recognition) or convert text into audio (e.g., text-to-speech). In some cases, a machine learning model can detect background noise in audio and / or enhance speech quality in audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is a block diagram of an example computer system for correcting or editing speech audio using a machine learning model in accordance with at least one embodiment;

[0004] Figure 2 illustrates an example data flow during correction or editing of speech audio using a machine learning model in accordance with at least one embodiment;

[0005] Figure 3 shows an example user interface for correcting or editing speech audio using a machine learning model in accordance with at least one embodiment;

[0006] Figure 4 is a flow chart of an example method for correcting or editing speech audio using a machine learning model in accordance with at least one embodiment;

[0007] Figure 5 is a flow chart of an example method for correcting or editing speech audio using a machine learning model in accordance with at least one embodiment;

[0008] Figure 6 is a block diagram of an example computing device suitable for correcting or editing speech audio using a machine learning model, according to at least some embodiments;

[0009] Figure 7A Inference and / or training logic according to at least one embodiment is shown;

[0010] Figure 7B Inference and / or training logic according to at least one embodiment is shown;

[0011] Figure 8 illustrates the training and deployment of a neural network according to at least one embodiment;

[0012] Figure 9 is an example data flow diagram of a high-level computing pipeline according to at least one embodiment;

[0013] Figure 10 is a system diagram of an example system for training, adapting, instantiating, and deploying machine learning models in an advanced computing pipeline, according to at least one embodiment. DETAILED DESCRIPTION

[0014] Podcasts and other voice recordings often contain content that needs to be edited, modified, or otherwise changed. For example, the audio may include slips of the tongue, grammatical errors, controversial content, unwanted content, inappropriate content, etc., or you may want to change the content of at least a portion of the audio (e.g., personalize the audio for a specific company, task, person, etc.). Re-recording the entire speech may require too much time and / or resources, and / or may not be feasible (e.g., because the original speaker is not available). For similar reasons, re-recording only the segment to be corrected and splicing the new audio into the existing audio may not be feasible. Some existing solutions can use text-to-speech (TTS) machine learning models to generate synthesized speech audio for the entire recording. However, by using TTS on the entire recording, the speaking style and prosody of the original speaker may be lost.

[0015] Aspects and embodiments of the present disclosure address these and other technical challenges by using a pipeline of machine learning models to modify various segments of a voice recording while preserving the speaking style and prosody of the original speaker. The first operation in the pipeline can be to generate a transcription from a given input audio. After the transcription is generated, the transcription can be modified to obtain a modified transcription (e.g., one or more slips of the tongue can be corrected, grammar can be corrected, objectionable content can be deleted, other language or text can be changed, etc.). In order to reflect the modifications made to the transcription in the original audio, the words of the original audio to be modified (e.g., the words to be synthesized) can be determined by comparing the original transcription with the modified transcription.

[0016] A text-to-speech (TTS) machine learning model can be used to generate synthesized audio using the determined words. For example, each word (or group of words) to be synthesized can be provided to a TTS model to convert the text into audio. The synthesized audio can be provided as input to a voice matching machine learning model, which is trained to modify the audio to match the speech characteristics of a reference audio, thereby generating a modified synthesized speech that is consistent with the prosody and speaking style of the speaker of the original audio. For example, a voice matching machine learning model can be trained to recognize speech characteristics (such as patterns in pitch, pronunciation, intonation, rhythm, etc.) in the reference audio and modify the input audio to better match these speech characteristics. The original audio can be used as the reference audio, and the synthesized audio from the TTS model can be the input audio to the voice matching machine learning model.

[0017] After generation, the modified synthesized audio can be provided to the ASR model to obtain a transcription with timestamps that identify the position of each word in the modified synthesized audio. The original audio and the modified synthesized audio can be aligned based on the timestamps of the original transcription and the timestamps of the transcription of the modified synthesized audio. The modified synthesized audio can be spliced ​​into the original audio to generate the final modified audio (e.g., corrected or updated speech audio).

[0018] Advantages of the disclosed technology include, but are not limited to, correcting the quality of speech audio or editing the content of speech audio by modifying a portion of the speech audio with synthesized audio while preserving the speaking style and prosody of the original speaker. By modifying only a portion of the original speech audio, computing resources can be saved compared to re-recording the entire audio or generating the entire audio as synthesized audio.

[0019] System Architecture

[0020] Figure 1 1 is a block diagram of an example computer system 100 for correcting or modifying speech audio (e.g., its content) using a machine learning model, according to at least one embodiment. In some embodiments, the system 100 may include a speech audio corrector 110, a data store 140, and a user device 150 connected to one or more networks 130. The one or more networks 130 may be a public network (e.g., the Internet), a private network (e.g., a local area network (LAN) or a wide area network (WAN)), a wireless network, a personal area network (PAN), other network types, and / or combinations thereof.

[0021] The speech audio corrector 110 may include a desktop computer, a laptop computer, a smartphone, a tablet computer, a server, a wearable device, a virtual reality (VR) / augmented reality (AR) / mixed reality (MR) headset or head-up display, a digital avatar or chatbot, an information kiosk, an in-vehicle infotainment computing device, and / or any suitable computing device capable of executing the techniques described herein. The speech audio corrector 110 may include one or more machine learning models, including an automatic speech recognition (ASR) model 112, a text-to-speech (TTS) model 114, a background noise reduction (BNR) model 116, a voice matching model 118, and a voice enhancement model 120. The ASR 112 may include any machine learning model capable of converting input audio into corresponding text. In some embodiments, the ASR 112 includes NeMo ASR. The TTS 114 may include any machine learning model capable of converting input text into corresponding audio. In some embodiments, the TTS 114 includes Riva TTS. The BNR 116 may include any machine learning model capable of separating background noise from foreground sounds in audio (e.g., by generating a noise mask that can be subtracted from or added to another audio). In some embodiments, BNR 116 comprises NVIDIA's Maxine BNR. Voice matching 118 may comprise any machine learning model capable of identifying speech characteristics (e.g., patterns of pitch, pronunciation, intonation, rhythm, etc.) in reference audio and modifying the input audio to better match these speech characteristics. In some embodiments, voice matching 118 comprises NVIDIA's Maxine Voice Font. Voice enhancement 120 may comprise any machine learning model capable of enhancing audio by, for example, removing background noise, improving speech clarity, etc. In some embodiments, voice enhancement 120 comprises NVIDIA's Maxine Voice Studio. The speech audio corrector 110 may be configured to correct speech audio using ASR 112, TTS 114, BNR 116, voice matching 118, and voice enhancement 120, as described with respect to Figure 2 Described in more detail.

[0022] The speech audio corrector 110 may include a graphical user interface (GUI) 122. The GUI 122 may include one or more elements for receiving input audio, transcription, and / or user interaction. The speech audio corrector 110 may provide the GUI 122 to one or more users (e.g., to a user device 150) via the network 130.

[0023] The data store 140 may include a persistent storage device capable of storing input audio, synthesized audio, machine learning models, machine learning model weights, transcriptions, and the like. The data store 140 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage disks, tape or hard drives, network attached storage (NAS), storage area networks (SAN), and the like. Although depicted as separate from the speech audio corrector 110, in at least some embodiments, the data store 140 may be part of the speech audio corrector 110. In at least some embodiments, the data store 140 may be a network-attached file server, while in other embodiments, the data store 140 may be some other type of persistent storage device, such as an object-oriented database, a relational database, and the like, which may be hosted by a server machine or one or more different machines coupled to the speech audio corrector 110 via the network 130.

[0024] The user device 150 can include a desktop computer, a notebook computer, a smart phone, a tablet computer, a server, a wearable device, a virtual reality (VR) / augmented reality (AR) / mixed reality (MR) headset or a head-up display, a digital avatar or a chatbot, an information kiosk, an in-vehicle infotainment computing device and / or any suitable computing device capable of performing the technology described herein. The user device 150 can interact with the voice audio corrector 110 (e.g., via one or more networks 130), and can provide input audio, transcription and / or modified transcription to the voice audio corrector 110. The input audio can be modified based on the modified transcription to generate a corrected voice audio retaining the voice characteristics of the input audio. The user device 150 can be used to display the GUI 122 of the voice audio corrector 110 and interact with it.

[0025] Correcting speech audio using machine learning models

[0026] Figure 2 An example data flow 200 is shown during correction or modification of speech audio using a machine learning model in accordance with at least one embodiment. Input audio 210 may be received (e.g., by speech audio corrector 110, from user device 150, etc.) and provided to ASR 212 (e.g., Figure 1ASR 112 of the present invention. ASR 212 may generate a transcription 214. In some embodiments, transcription 214 includes a timestamp that can be used to align the words of the audio with the speech of the audio. A modified transcription 216 may be generated based on transcription 214. In some embodiments, the original transcription (e.g., transcription 214) may be modified using user input. For example, speech audio corrector 110 may modify the original transcription in response to receiving user input (e.g., from user device 150). In some embodiments, a natural language processing (NLP) machine learning model (or another machine learning model) may be used to modify the original transcription (e.g., correct grammar, modify language, correct facts, add text / information, remove text / information, etc.). In some embodiments, a language model (such as a large language model (LLM)) may be used to update the transcription, such as by querying the language model to "edit the grammar in this text," "change the context of this text," "make this text about [Company X] instead of [Company Y]," etc., and then receiving the updated text from the language model. In some embodiments, a combination of the machine learning model and user modifications may be used to generate the modified transcription 216.

[0027] The word of the original audio to be modified (e.g., modified word 218) can be determined by comparing the transcription 214 (e.g., the initial transcription) with the modified transcription 216. In some embodiments, to create a smoother transition between the original audio and the synthesized audio, one or more words surrounding the one or more modified words may be included in the modified word 218. For example, the two words before the one or more modified words and / or the two words after the one or more modified words may be included in the modified word 218. In some embodiments, all words after the one or more modified words up to the next punctuation mark may be included in the modified word 218. The modified word 218 may include one or more words present in the second (e.g., modified) transcription that are not present in the first transcription (e.g., transcription 214). In some embodiments, the modified word 218 may include at least some text present in the second (e.g., modified) transcription that is not present in the first transcription (e.g., transcription 214). The modified word 218 may also include one or more words present in the first transcription (e.g., surrounding words).

[0028] TTS220 (e.g. Figure 1The TTS 114 in the input audio 210 may receive the modified word 218 and may generate synthesized audio based on the modified word 218. In some embodiments, the speaking rate of the speaker of the input audio 210 may be identified and provided as additional input to the TTS 220 to more closely mimic the original speaker. For example, if the speaker of the input audio 210 speaks at a high rate (e.g., the speaker speaks quickly, the speaker speaks many words per minute, etc.), the duration of the synthesized speech (e.g., the output of the TTS 220) may be shorter than the duration of the synthesized speech of a speaker with a lower speaking rate because more words are included per minute.

[0029] Voice matching 222 (e.g., Figure 1 In some embodiments, voice matching 222 may receive synthesized audio from TTS 220 as audio to be modified. In some embodiments, voice matching 222 may receive input audio 210 (or at least a portion of input audio 210 (e.g., a portion corresponding to a modified word)) as reference audio (e.g., audio having speech characteristics to which the output of voice matching 222 should be matched). In some embodiments, voice enhancement 224 receives input audio 210 as input, and the output of voice enhancement 224 (e.g., enhanced input audio) may be used as reference audio for voice matching 222. Voice enhancement 224 may improve the quality of the reference audio by removing background noise, improving speech clarity, and the like. In some embodiments, having clearer reference audio may result in modified synthesized audio (e.g., modified synthesized speech) that better matches the speech characteristics of the input audio.

[0030] The output of voice matching 222 (e.g., modified synthesized audio) can be provided to ASR 226. In some embodiments, ASR 226 is the same as ASR 212. ASR 226 can generate a synthesized transcription based on the modified synthesized audio. In some embodiments, the synthesized transcription includes timestamps that can be used to align the words of the audio with the speech of the audio. Input audio 210 can be aligned with the modified synthesized audio based on the timestamps of transcription 214 and the timestamps of the synthesized transcription. The modified synthesized audio can be spliced ​​into input audio 210 (e.g., at spliced ​​audio 232) to generate output audio 234 (e.g., corrected speech audio).

[0031] In some embodiments, the spliced ​​audio 232 can perform acoustic normalization and / or pitch normalization when combining the input audio 210 with the modified synthesized audio to avoid abrupt changes in volume or pitch of the synthesized audio compared to the surrounding audio.

[0032] In some embodiments, BNR 228 may receive input audio 210 and generate a noise mask based on background noise included in input audio 210. At add noise 230, the noise mask may be applied to the output of voice matching 222 (e.g., modified synthesized audio) to obtain noisy synthesized audio. Thus, BNR 228 and add noise 230 may add noise to the modified synthesized audio that is similar to the noise of the original audio (e.g., input audio 210). The noisy synthesized audio may be provided to concatenated audio 232 and combined with input audio 210 to obtain output audio 234. Adding noise to the modified synthesized audio may make the resulting synthesized audio more similar to the original audio (e.g., input audio 210). For example, if output audio 234 includes a first segment of original audio with background noise, a second synthesized segment without background noise, and a third segment of original audio with background noise, the temporary absence of background noise from the audio may be irritating to a listener.

[0033] In some embodiments, it may be desirable to remove background noise from the entire audio. In some embodiments, to generate noise-free output audio, the output of voice matching 222 (e.g., without any noise added by BNR 228 and noise addition 230) can be provided to concatenated audio 232 along with the output of voice enhancement 224 (e.g., an enhanced version of input audio 210). In some embodiments, noise-free output audio can be generated by providing output audio 234 to voice enhancement 224.

[0034] Figure 3 An example user interface 300 for correcting speech audio using a machine learning model is shown in accordance with at least one embodiment. In some embodiments, the user interface 300 may be Figure 1 The GUI 122 may also be Figure 1 In some embodiments, the interface 300 is displayed on the Figure 1 150. User interface 300 may include one or more graphical elements, including an original waveform visual 310, an original transcript element 320, a modified transcript element 330, a generate modified audio button 340, and a modified waveform visual 350. In some embodiments, a user may interact with one or more elements of interface 300 (e.g., via user device 150).

[0035] In some embodiments, the raw waveform visual 310 may include a graphical representation of the input audio to be corrected. For example, the raw waveform visual 310 may include one or more bars representing attributes of the input audio at discrete times (e.g., sound frequency, volume, etc.).

[0036] The raw transcript element 320 may include a title (e.g., "Raw Transcript:") and a text area containing a text representation of the input audio to be corrected. For example, the text area of ​​the raw transcript element 320 may be populated by an ASR model that converts the input audio into text. In some embodiments, the raw transcript element 320 is read-only and cannot be modified by the user.

[0037] Modified transcript element 330 may include a title (e.g., "Modified Transcription") and a text area containing a text representation of the modified audio to be generated. In some embodiments, the text area of ​​modified transcript element 330 is initially populated with a copy of the value of the text area of ​​original transcript element 320. A user may modify the value of the text area of ​​modified transcript element 330 to correct one or more words of the original transcript. In some embodiments, the original transcript may be modified based on user input. In some embodiments, the text area of ​​modified transcript element 330 may be modified by an NLP machine learning model.

[0038] The generate modified audio button 340 can trigger the generation of corrected speech audio using the machine learning model in response to user interaction (e.g., clicking, selecting, etc.). Figure 1 The speech audio corrector 110 is provided, and in response to user interaction, the generate modified audio button 340 can cause the speech audio corrector 110 to determine modified words between the original transcription and the value of the text area of ​​the modified transcription element 330. These modified words can be provided to the TTS 114 (and / or TTS 220) to obtain synthesized audio. The synthesized audio can be modified by the voice matching 118 (and / or voice matching 222) to obtain modified synthesized audio. The modified synthesized audio can be spliced ​​with the input audio corresponding to the original waveform visual 310 to obtain a modified audio output.

[0039] The modified waveform visual 350 may include a graphical representation of the modified output audio (e.g., corrected speech audio). For example, the modified waveform visual 350 may include one or more bars representing properties of the modified output audio (e.g., sound frequency, volume, etc.) at discrete times.

[0040] Figure 4 and Figure 5Flowcharts of example methods 400 and 500 for correcting speech audio using a machine learning model, according to at least one embodiment. Methods 400 and 500 can be performed using one or more processing units (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerator, a physical processing unit (PPU), a data processing unit (DPU), other processing circuitry, etc.), which can include (or communicate with) one or more memory devices. In at least one embodiment, methods 400 and 500 can be performed using a processing device. In at least one embodiment, methods 400 and 500 can be performed using a processing unit of speech audio corrector 110 and / or user device 150. In at least one embodiment, the processing unit executing any of methods 400 and / or 500 can execute instructions stored on a non-transitory computer-readable storage medium. In at least one embodiment, any of methods 400 and / or 500 can be performed using multiple processing threads (e.g., CPU threads and / or GPU threads), individual threads that execute one or more individual functions, routines, subroutines, or operations of the method. In at least one embodiment, the processing threads implementing any of methods 400 and / or 500 may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing any of methods 400 and / or 500 may execute asynchronously with respect to each other. The various operations of any of methods 400 and / or 500 may be executed in a manner similar to Figure 4 and / or Figure 5 Some operations of any of methods 400 and / or 500 may be performed concurrently with other operations. In at least one embodiment, Figure 4 and / or Figure 5 One or more of the operations shown in may not always be performed.

[0041] Figure 4 is a flow chart of an example method 400 for correcting speech audio using a machine learning model in accordance with at least one embodiment. At block 410, a processing unit executing method 400 may compare a first transcription of input audio with a second transcription of the input audio to identify a set of modified words. The second transcription may be a modified version of the first transcription. In some embodiments, the first transcription is generated using a first automatic speech recognition machine learning model (e.g., Figure 1In some embodiments, the modified word set includes at least one word present in the second transcription but not in the first transcription and one or more words present in the first transcription. In some embodiments, the input audio is at least one of podcast audio, background audio from a video, a narration of a presentation, or an audiobook recording. In some embodiments, method 400 includes generating a second transcription by modifying one or more words in the first transcription and / or applying a natural language processing machine learning model to the first transcription to obtain the second transcription. In some embodiments, modifying the one or more words in the first transcription is performed by a user or in response to user input.

[0042] At block 420, the processing unit may use a text-to-speech machine learning model (e.g., Figure 1 At block 430, the processing unit may use a voice matching machine learning model (e.g., Figure 1 In some embodiments, to generate the modified synthesized audio, the processing unit may enhance the speech machine learning model (e.g., Figure 1 The processing unit may generate a modified synthesized audio based on the synthesized audio and the enhanced input audio using a voice matching machine learning model.

[0043] At block 440, the processing unit may generate output audio by combining the input audio with the modified synthesized audio. In some embodiments, the first transcription includes a first timestamp, and to generate the output audio, the processing unit may use a first automatic speech recognition machine learning model (e.g., Figure 1 The synthesized transcript may include a second timestamp. The processing unit may further align the input audio with the modified synthesized audio based on the first timestamp and the second timestamp. In some embodiments, method 400 further includes identifying a speaking rate of the input audio. In some embodiments, the processing unit may generate the synthesized audio based on the speaking rate.

[0044] Figure 5 is a flow chart of an example method 500 for correcting speech audio using a machine learning model in accordance with at least one embodiment. In some embodiments, the processing unit may perform the method 500 as part of the method 400 (e.g., part of block 440 of the method 400). At block 510, the processing unit may apply the noise reduction machine learning model (e.g., Figure 1The BNR 116 of FIG. 5 is applied to the input audio to obtain a noise mask. At block 530, the processing unit may apply the noise mask to the modified synthesized audio to obtain noisy synthesized audio. At block 540, the processing unit may generate output audio by combining the input audio with the noisy synthesized audio.

[0045] Figure 6 6 is a block diagram of an example computing device 600 suitable for correcting speech audio using a machine learning model, according to at least some embodiments. The computing device 600 may include an interconnect system 602 that directly or indirectly couples the following devices: memory 604, one or more central processing units (CPUs) 606, one or more graphics processing units (GPUs) 608, a communication interface 610, input / output (I / O) ports 612, input / output components 614, a power supply 616, one or more presentation components 618 (e.g., a display), and one or more logic units 620. In at least one embodiment, the computing device 600 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). For non-limiting examples, the one or more GPUs 608 may include one or more vGPUs, the one or more CPUs 606 may include one or more vCPUs, and / or the one or more logic units 620 may include one or more virtual logic units. Thus, computing device 600 may include discrete components (eg, a complete GPU dedicated to computing device 600 ), virtual components (eg, a portion of a GPU dedicated to computing device 600 ), or a combination thereof.

[0046] although Figure 6 The various blocks in FIG6 are shown as being connected with lines via interconnect system 602, but this is not intended to be limiting and is provided for clarity. For example, in some embodiments, presentation component 618 (such as a display device) may be considered to be I / O component 614 (e.g., if the display is a touch screen). As another example, CPU 606 and / or GPU 608 may include memory (e.g., memory 604 may represent a storage device in addition to the memory of GPU 608, CPU 606, and / or other components). In other words, Figure 6 The term computing device is illustrative only. Categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "gaming console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types are not distinguished, as all of these are Figure 6 within the range of computing devices.

[0047] The interconnect system 602 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 602 may include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. For example, the CPU 606 may be directly connected to the memory 604. In addition, the CPU 606 may be directly connected to the GPU 608. When there is a direct connection or a point-to-point connection between components, the interconnect system 602 may include a PCIe link to perform the connection. In these examples, the computing device 600 does not necessarily need to include a PCI bus.

[0048] Memory 604 may include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 600. Computer-readable media can include volatile and non-volatile media and removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0049] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules and / or other data types). For example, memory 604 can store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements), such as an operating system. According to one or more aspects of the present disclosure, computer-readable instructions may include executable instructions for executing method 400 and / or method 500 for correcting speech audio using a machine learning model. Computer storage media may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electronically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 600. As used herein, computer storage media itself does not include signals.

[0050] Computer storage media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal (such as a carrier wave) or other transmission mechanism including any information delivery media. The term "modulated data signal" may refer to a signal whose one or more characteristics are set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media may include wired media (such as a wired network or a direct wired connection) and wireless media (such as acoustic, radio frequency (RF), infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0051] One or more CPUs 606 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 600 to perform one or more methods and / or processes described herein. For example, according to one or more aspects of the present disclosure, one or more CPUs 606 may be configured to execute instructions that perform methods 400 and 500 of correcting speech audio using a machine learning model. One or more CPUs 606 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of processing multiple software threads simultaneously. One or more CPUs 606 may include any type of processor and may include different types of processors (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server) depending on the type of computing device 600 implemented. For example, depending on the type of computing device 600, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). Computing device 600 may include one or more CPUs 606 in addition to one or more microprocessors or complementary coprocessors (eg, math coprocessors).

[0052] In addition to or as an alternative to one or more CPUs 606, one or more GPUs 608 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 600 to perform one or more methods and / or processes described herein. One or more GPUs 608 may be integrated GPUs (e.g., with one or more CPUs 606) and / or one or more GPUs 608 may be discrete GPUs. In embodiments, one or more GPUs 608 may be coprocessors for one or more CPUs 606. One or more GPUs 608 may be used by the computing device 600 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, a GPU 608 may be used for general-purpose computing on a GPU (GPGPU). One or more GPUs 608 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. One or more GPUs 608 may generate pixel data for an output image in response to a rendering command (e.g., a rendering command received from one or more CPUs 606 via a host interface). One or more GPUs 608 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 604. One or more GPUs 608 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 608 may generate pixel data or GPGPU data for different portions of an output or different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0053] In addition to or in lieu of the one or more CPUs 606 and / or the one or more GPUs 608, the one or more logic units 620 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 600 to perform one or more methods and / or processes described herein. In embodiments, the one or more CPUs 606, the one or more GPUs 608, and / or the one or more logic units 620 may individually or jointly perform any combination of methods, processes, and / or portions thereof. The one or more logic units 620 may be part of and / or integrated within the one or more CPUs 606 and / or the one or more GPUs 608, and / or the one or more logic units 620 may be discrete components or otherwise external to the one or more CPUs 606 and / or the one or more GPUs 608. In embodiments, the one or more logic units 620 may be coprocessors for the one or more CPUs 606 and / or the one or more GPUs 608.

[0054] Examples of the one or more logic units 620 include one or more processing cores and / or components thereof, such as a data processing unit (DPU), a tensor core (TC), a tensor processing unit (TPU), a pixel vision core (PVC), a vision processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, etc.

[0055] The communication interface 610 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 600 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communications). The communication interface 610 may include components and functionality that enable communication over any of a variety of different networks, such as a wireless network (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (e.g., communicating over Ethernet or InfiniBand), a low-power wide area network (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the one or more logic units 620 and / or the communication interface 610 may include one or more data processing units (DPUs) for sending data received over the network and / or over the interconnect system 602 directly to the one or more GPUs 608 (e.g., their memory).

[0056] The I / O ports 612 may enable the computing device 600 to be logically coupled to other devices, including I / O components 614, one or more presentation components 618, and / or other components, some of which may be built into (e.g., integrated into) the computing device 600. Exemplary I / O components 614 include a microphone, a mouse, a keyboard, a joystick, a gamepad, a game controller, a satellite dish, a scanner, a printer, a wireless device, and the like. The I / O components 614 may provide a natural user interface (NUI) that processes user-generated mid-air gestures, voice, or other physiological input. In some cases, the input may be sent to an appropriate network element for further processing. The NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometrics, gesture recognition on and near the screen, mid-air gestures, head and eye tracking, and touch recognition associated with the display of the computing device 600 (as described in more detail below). The computing device 600 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof, for gesture detection and recognition. Additionally, computing device 600 may include an accelerometer or gyroscope to enable detection of motion (e.g., as part of an inertial measurement unit (IMU)). In some examples, computing device 600 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0057] The power supply 616 can include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 616 can provide power to the computing device 600 to enable the components of the computing device 600 to operate.

[0058] The one or more presentation components 618 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other display components. The one or more presentation components 618 may receive data from other components (e.g., one or more GPUs 608, one or more CPUs 606, a DPU, etc.) and output data (e.g., as images, video, sound, etc.).

[0059] A network environment suitable for implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be configured to: Figure 6 The present invention may be implemented on one or more instances of computing device 600 , for example, each device may include similar components, features and / or functionality as computing device 600 .

[0060] The components of the network environment can communicate with each other via one or more networks, which can be wired, wireless, or both. The network can include multiple networks or a network of one or more networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. When the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0061] Compatible network environments may include one or more peer-to-peer network environments (in which case, the network environment may not include a server) and one or more client-server network environments (in which case, the network environment may include one or more servers). In a peer-to-peer network environment, the functionality described herein with respect to the server may be implemented on any number of client devices.

[0062] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for supporting software at the software layer and / or one or more applications at the application layer. The software or application may include network-based service software or application programs, respectively. In an embodiment, one or more client devices may use network-based service software or application programs (e.g., by accessing the service software and / or application programs via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open source software web application framework, such as one that may use a distributed file system for large-scale data processing (e.g., "big data").

[0063] A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these various functions can be distributed across multiple locations in a central server or core server (e.g., a central server or core server in one or more data centers that can be distributed across a state, a region, a country, the world, etc.). If the connection to the user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functions to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0064] One or more client devices may include Figure 6 6. The client device 600 may be embodied as a personal computer (PC), a laptop, a mobile device, a smartphone, a tablet, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a camera, a surveillance device or system, a vehicle, a boat, a spacecraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, an in-vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination thereof, or any other suitable device.

[0065] The systems and methods described herein may be used for a variety of purposes, such as, but not limited to, for performing one or more operations with respect to systems and methods associated with: machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and surveillance, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twins, data center processing, conversational AI, chatbots, digital avatars, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation of 3D assets, cloud computing, and / or any other suitable application.

[0066] The disclosed embodiments may be included in a variety of different systems, such as automotive systems (e.g., in-vehicle infotainment systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least in part in a data center, systems for generating or presenting virtual reality content, mixed reality content, or augmented reality content, systems for performing conversational AI operations, systems for performing one or more generative AI operations, systems implementing one or more language models, systems implementing one or more large language models (LLMs), systems for performing light transport simulations, systems for performing collaborative content creation of 3D assets, systems implemented at least in part using cloud computing resources, and / or other types of systems.

[0067] Reasoning and training logic

[0068] Figure 7A Inference and / or training logic 715 is shown for performing inference and / or training operations associated with one or more embodiments.

[0069] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 701 for storing forward and / or output weights and / or input / output data, and / or other parameters for configuring neurons or layers of a neural network being trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 701 storing graph code or other software to control timing and / or sequence, wherein weights and / or other parameter information are loaded to configure processing units, which include logic units, integer and / or floating point units (collectively referred to as arithmetic logic units (ALUs) or simple circuits). In at least one embodiment, code (such as graph code) loads weights or other parameter information into a processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, the code and / or data storage 701 stores input / output data during training and / or inference using aspects of one or more embodiments and / or weight parameters during forward propagation of weight parameters for each layer of a neural network trained or used in conjunction with one or more embodiments. In at least one embodiment, any portion of code and / or data storage 701 may be included within other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory.

[0070] In at least one embodiment, any portion of code and / or data storage 701 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 701 may be cache memory, dynamic random addressable memory ("DRAM"), static random addressable memory ("SRAM"), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether code and / or data storage 701 is internal or external to a processor, for example, or composed of DRAM, SRAM, flash memory, or some other type of storage, may depend on the available storage space on or off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in inference and / or training of the neural network, or some combination of these factors.

[0071] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, code and / or data storage 705 to store backpropagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, during training and / or inference using aspects of one or more embodiments, the code and / or data storage 705 stores weight parameters and / or input / output data for each layer of the neural network trained or used in conjunction with one or more embodiments during backpropagation of input / output data and / or weight parameters. In at least one embodiment, the training logic 715 may include or be coupled to code and / or data storage 705 that stores graph code or other software to control the timing and / or sequence in which weight and / or other parameter information is loaded to configure the processing units, which include logic units, integer and / or floating point units (collectively referred to as arithmetic logic units (ALUs)).

[0072] In at least one embodiment, code (such as graph code) causes weights or other parameter information to be loaded into the processor ALU based on the architecture of the neural network to which the code corresponds. In at least one embodiment, any portion of code and / or data storage 705 can be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of code and / or data storage 705 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and / or data storage 705 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether code and / or data storage 705 is internal or external to the processor, for example, whether it is composed of DRAM, SRAM, flash memory, or some other type of storage, depends on whether the available storage is on-chip or off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of data used in inference and / or training of the neural network, or some combination of these factors.

[0073] In at least one embodiment, code and / or data store 701 and code and / or data store 705 may be separate storage structures. In at least one embodiment, code and / or data store 701 and code and / or data store 705 may be a combined storage structure. In at least one embodiment, code and / or data store 701 and code and / or data store 705 may be partially combined and partially separated. In at least one embodiment, any portion of code and / or data store 701 and code and / or data store 705 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory.

[0074] In at least one embodiment, the inference and / or training logic 715 may include, but is not limited to, one or more arithmetic logic units (“ALUs”) 710 (including integer and / or floating point units) for performing logical and / or mathematical operations based at least in part on or directed by training and / or inference code (e.g., graph code), the results of which may produce activations (e.g., output values ​​from a layer or neuron within a neural network) stored in activation storage 720, which are functions of input / output and / or weight parameter data stored in code and / or data storage 701 and / or code and / or data storage 705. In at least one embodiment, activations are performed in response to executing instructions or other code, linear algebra and / or matrix-based mathematics performed by ALU 710 to generate activations stored in activation storage 720, wherein weight values ​​stored in code and / or data storage 705 and / or code and / or data storage 701 are used as operands with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in code and / or data storage 705 or code and / or data storage 701 or in another on-chip or off-chip storage.

[0075] In at least one embodiment, one or more ALUs 710 are included in one or more processors or other hardware logic devices or circuits, while in another embodiment, one or more ALUs 710 may be external to the processor or other hardware logic devices or circuits that use them (e.g., coprocessors). In at least one embodiment, one or more ALUs 710 may be included within an execution unit of a processor or otherwise included in a group of ALUs accessible by the execution units of a processor, which may be within the same processor or distributed between different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc.). In at least one embodiment, code and / or data storage 701, code and / or data storage 705, and activation storage 720 may share a processor or other hardware logic device or circuit, while in another embodiment, they may be in different processors or other hardware logic devices or circuits or some combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 720 may be included with other on-chip or off-chip data storage, including the processor's L1, L2, or L3 cache or system memory. Additionally, inference and / or training code may be stored with other code accessible to a processor or other hardware logic or circuitry and may be retrieved and / or processed using the processor's fetch, decode, schedule, execute, exit, and / or other logic circuitry.

[0076] In at least one embodiment, activation storage 720 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, activation storage 720 can be completely or partially internal or external to one or more processors or other logic circuits. In at least one embodiment, activation storage 720 can be internal or external to the processor, for example, or comprise DRAM, SRAM, flash memory, or some other storage type, depending on the available on-chip or off-chip storage, the latency requirements for performing training and / or inference functions, the batch size of data used in inferring and / or training neural networks, or some combination of these factors.

[0077] In at least one embodiment, Figure 7A The inference and / or training logic 715 shown in FIG may be used in conjunction with an application specific integrated circuit (“ASIC”), such as the one from Google. Processing unit from Graphcore TM Inference Processing Unit (IPU) from Intel Corp. (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7A The illustrated inference and / or training logic 715 may be used in conjunction with central processing unit (“CPU”) hardware, graphics processing unit (“GPU”) hardware, or other hardware such as a field programmable gate array (“FPGA”).

[0078] Figure 7B Inference and / or training logic 715 is shown in accordance with at least one embodiment. In at least one embodiment, inference and / or training logic 715 may include, but is not limited to, hardware logic where computing resources are dedicated or otherwise uniquely used in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, Figure 7B The inference and / or training logic 715 shown in FIG can be used in conjunction with an application specific integrated circuit (ASIC), such as the one from Google. Processing unit from Graphcore TM Inference Processing Unit (IPU) or from Intel (e.g., "Lake Crest") processor. In at least one embodiment, Figure 7BThe inference and / or training logic 715 shown in can be used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware such as a field programmable gate array (FPGA). In at least one embodiment, the inference and / or training logic 715 includes, but is not limited to, code and / or data storage 701 and code and / or data storage 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including bias values, gradient information, momentum values, and / or other parameter or hyperparameter information. Figure 7B In at least one embodiment shown in FIG, code and / or data storage 701 and code and / or data storage 705 are each associated with dedicated computing resources, such as computing hardware 702 and computing hardware 706, respectively. In at least one embodiment, computing hardware 702 and computing hardware 706 each include one or more ALUs that perform mathematical functions (e.g., linear algebraic functions) solely on the information stored in code and / or data storage 701 and code and / or data storage 705, respectively, with the results of the functions being stored in activation storage 720.

[0079] In at least one embodiment, each of the code and / or data stores 701 and 705 and the corresponding computing hardware 702 and 706 corresponds to a different layer of a neural network, such that activations from one storage / computation pair 701 / 702 of the code and / or data store 701 and computing hardware 702 are provided as input to the next storage / computation pair 705 / 706 of the code and / or data store 705 and computing hardware 706, reflecting the conceptual organization of the neural network. In at least one embodiment, each storage / computation pair 701 / 702 and 705 / 706 can correspond to more than one neural network layer. In at least one embodiment, additional storage / computation pairs (not shown) can be included in the inference and / or training logic 715 after or in parallel with the storage / computation pairs 701 / 702 and 705 / 706.

[0080] Neural network training and deployment

[0081] Figure 8The training and deployment of a deep neural network according to at least one embodiment is shown. In at least one embodiment, an untrained neural network 806 is trained using a training dataset 802. In at least one embodiment, the training framework 804 is the PyTorch framework, while in other embodiments, the training framework 804 is TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training frameworks. In at least one embodiment, the training framework 804 trains the untrained neural network 806 and enables it to be trained using the processing resources described herein to generate a trained neural network 808. In at least one embodiment, the weights can be randomly selected or pre-trained using a deep belief network. In at least one embodiment, the training can be performed in a supervised, partially supervised, or unsupervised manner.

[0082] In at least one embodiment, untrained neural network 806 is trained using supervised learning, where training dataset 802 includes inputs paired with expected outputs for the inputs, or where training dataset 802 includes inputs with known outputs and neural network 806 is manually graded for outputs. In at least one embodiment, untrained neural network 806 is trained in a supervised manner, processing inputs from training dataset 802 and comparing the resulting outputs to a set of expected or desired outputs. In at least one embodiment, errors are then propagated back through untrained neural network 806. In at least one embodiment, training framework 804 adjusts the weights that control untrained neural network 806. In at least one embodiment, training framework 804 includes tools for monitoring the degree to which untrained neural network 806 converges toward a model (e.g., trained neural network 808) that is suitable for generating correct answers (e.g., results 814) based on input data (e.g., new dataset 812). In at least one embodiment, training framework 804 iteratively trains untrained neural network 806 while adjusting weights to improve the output of untrained neural network 806 using a loss function and an adjustment algorithm (e.g., stochastic gradient descent). In at least one embodiment, the training framework 804 trains the untrained neural network 806 until the untrained neural network 806 reaches a desired accuracy. In at least one embodiment, the trained neural network 808 can then be deployed to implement any number of machine learning operations.

[0083] In at least one embodiment, untrained neural network 806 is trained using unsupervised learning, wherein untrained neural network 806 attempts to train itself using unlabeled data. In at least one embodiment, unsupervised learning training dataset 802 will include input data without any associated output data or "ground truth" data. In at least one embodiment, untrained neural network 806 can learn groupings within training dataset 802 and can determine how individual inputs relate to untrained dataset 802. In at least one embodiment, unsupervised training can be used to generate a self-organizing map in trained neural network 808 that can perform operations useful for reducing the dimensionality of new dataset 812. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows identification of data points in new dataset 812 that deviate from the normal pattern of new dataset 812.

[0084] In at least one embodiment, semi-supervised learning can be used, which is a technique in which the training data set 802 includes a mixture of labeled and unlabeled data. In at least one embodiment, the training framework 804 can be used to perform incremental learning, for example, through a transfer learning technique. In at least one embodiment, incremental learning enables the trained neural network 808 to adapt to the new data set 812 without forgetting the knowledge that was instilled into the trained neural network 808 during the initial training.

[0085] Reference Figure 9 , Figure 9 is an example data flow diagram of a process 900 for generating and deploying a processing and inference pipeline according to at least one embodiment. In at least one embodiment, the process 900 can be deployed to perform game name recognition analysis and inference on user feedback data at one or more facilities 902, such as a data center.

[0086] In at least one embodiment, process 900 may be performed within training system 904 and / or deployment system 906. In at least one embodiment, training system 904 may be used to perform training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use with deployment system 906. In at least one embodiment, deployment system 906 may be configured to offload processing and computing resources in a distributed computing environment to reduce infrastructure requirements at facility 902. In at least one embodiment, deployment system 906 may provide a pipeline platform for selecting, customizing, and implementing virtual instruments for use with computing devices at facility 902. In at least one embodiment, a virtual instrument may include a software-defined application for performing one or more processing operations on feedback data. In at least one embodiment, one or more applications in the pipeline may use or call services (e.g., reasoning, visualization, computation, AI, etc.) of deployment system 906 during application execution.

[0087] In at least one embodiment, some applications used in high-level processing and reasoning pipelines may use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, the machine learning model may be trained at the facility 902 using feedback data 908 (e.g., imaging data) stored at the facility 902 or feedback data 908 from another or more facilities, or a combination thereof. In at least one embodiment, the training system 904 may be used to provide applications, services, and / or other resources to generate a working, deployable machine learning model for the deployment system 906.

[0088] In at least one embodiment, the model registry 924 can be backed by an object store that can support version control and object metadata. In at least one embodiment, the model registry 924 can be accessed from within the cloud platform through, for example, a cloud store (e.g., Figure 10 The object store is accessed through an application programming interface (API) compatible with the cloud 1026. In at least one embodiment, machine learning models within the model registry 924 can be uploaded, listed, modified, or deleted by developers or partners of the system interacting with the API. In at least one embodiment, the API can provide access to methods that allow users with appropriate credentials to associate a model with an application so that the model can be executed as part of the execution of a containerized instantiation of the application.

[0089] In at least one embodiment, the training pipeline 1004 ( Figure 10) may include scenarios where a facility 902 is training its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, feedback data 908 may be received from various channels (such as forums, web forms, etc.). In at least one embodiment, once the feedback data 908 is received, AI-assisted annotation 910 may be used to help generate annotations corresponding to the feedback data 908 to serve as ground truth data for the machine learning model. In at least one embodiment, the AI-assisted annotation 910 may include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that may be trained to generate annotations corresponding to certain types of feedback data 908 (e.g., from certain devices) and / or certain types of anomalies in the feedback data 908. In at least one embodiment, the AI-assisted annotation 910 may then be used directly or may be adjusted or fine-tuned using annotation tools to generate ground truth data. In at least one embodiment, in some examples, labeled data 912 may be used as ground truth data for training the machine learning model. In at least one embodiment, AI-assisted annotations 910, labeled data 912, or a combination thereof can be used as ground truth data for training a machine learning model (e.g., via model training 914). In at least one embodiment, the trained machine learning model can be referred to as an output model 916 and can be used by the deployment system 906, as described herein.

[0090] In at least one embodiment, the training pipeline 1004 ( Figure 10) may include situations where facility 902 requires a machine learning model for performing one or more processing tasks for one or more applications in deployment system 906, but facility 902 may not currently have such a machine learning model (or may not have a model optimized, efficient, or effective for this purpose). In at least one embodiment, an existing machine learning model can be selected from model registry 924. In at least one embodiment, model registry 924 may include machine learning models that have been trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, the machine learning models in model registry 924 may be trained on imaging data from a different facility (e.g., a remotely located facility) than facility 902. In at least one embodiment, the machine learning model may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data from a specific location (e.g., which may be in the form of feedback data 908), the training may be performed at that location, or at least in a manner that protects the confidentiality of the imaging data or restricts its transfer off-site (e.g., to comply with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model is trained or partially trained at one location, the machine learning model can be added to the model registry 924. In at least one embodiment, the machine learning model can then be retrained or updated at any number of other facilities, and the retrained or updated model can be used in the model registry 924. In at least one embodiment, the machine learning model can then be selected from the model registry 924 (and referred to as the output model 916) and can be used in the deployment system 906 to perform one or more processing tasks for one or more applications of the deployment system.

[0091] In at least one embodiment, the training pipeline 1004 ( Figure 10) can be used in a scenario including a facility 902 that requires a machine learning model for performing one or more processing tasks for one or more applications in a deployment system 906, but the facility 902 may not currently have such a machine learning model (or may not have an optimized, efficient, or effective model for this purpose). In at least one embodiment, due to population differences, genetic variation, robustness of the training data used to train the machine learning model, the diversity of training data anomalies, and / or other issues with the training data, the machine learning model selected from the model registry 924 may not be fine-tuned or optimized for the feedback data 908 generated at the facility 902. In at least one embodiment, AI-assisted annotation 910 can be used to help generate annotations corresponding to the feedback data 908 to serve as ground truth data for retraining or updating the machine learning model. In at least one embodiment, labeled data 912 can be used as ground truth data for training the machine learning model. In at least one embodiment, retraining or updating the machine learning model can be referred to as model training 914. In at least one embodiment, model training 914 may include data (e.g., AI-assisted annotations 910, labeled data 912, or a combination thereof) that can be used as ground truth data for retraining or updating a machine learning model.

[0092] In at least one embodiment, deployment system 906 may include software 918, services 920, hardware 922, and / or other components, features, and functionality. In at least one embodiment, deployment system 906 may include a software "stack" such that software 918 may be built on top of services 920 and may use services 920 to perform some or all processing tasks, and services 920 and software 918 may be built on top of hardware 922 and use hardware 922 to perform processing, storage, and / or other computing tasks of deployment system 906.

[0093] In at least one embodiment, the software 918 may include any number of different containers, each of which may execute an instantiation of an application. In at least one embodiment, each application may execute one or more processing tasks (e.g., reasoning, object detection, feature detection, segmentation, image enhancement, calibration, etc.) in a high-level processing and reasoning pipeline. In at least one embodiment, for each type of computing device, there may be any number of containers that may execute data processing tasks on the feedback data 908 (or other data types, such as those described herein). In at least one embodiment, in addition to receiving and configuring imaging data for use by each container and / or for use by the facility 902 after processing through the pipeline, a high-level processing and reasoning pipeline may be defined based on the selection of different containers desired or required to process the feedback data 908 (e.g., to convert the output back into a usable data type for storage and display at the facility 902). In at least one embodiment, the combination of containers within the software 918 (e.g., that comprise the pipeline) may be referred to as a virtual instrument (as described in more detail herein), and the virtual instrument may utilize services 920 and hardware 922 to execute some or all of the processing tasks of the application instantiated in the container.

[0094] In at least one embodiment, data can be pre-processed as part of a data processing pipeline to prepare the data for processing by one or more applications. In at least one embodiment, post-processing can be performed on the output of one or more inference tasks or other processing tasks in the pipeline to prepare output data for the next application and / or prepare the output data for transmission and / or use by a user (e.g., as a response to an inference request). In at least one embodiment, the inference task can be performed by one or more machine learning models, such as trained or deployed neural networks, which can include the output model 916 of the training system 904.

[0095] In at least one embodiment, the tasks of a data processing pipeline can be encapsulated in one or more containers, each container representing a discrete, fully functional instantiation of an application and a virtualized computing environment that can reference a machine learning model. In at least one embodiment, the containers or applications can be published to a private (e.g., limited access) area of ​​a container registry (described in more detail herein), and the trained or deployed models can be stored in the model registry 924 and associated with one or more applications. In at least one embodiment, an image of the application (e.g., a container image) can be used in the container registry, and once a user selects an image from the container registry for deployment in the pipeline, the image can be used to generate an instantiated container for the application for use by the user's system.

[0096] In at least one embodiment, a developer can develop, publish, and store applications (e.g., as containers) for performing processing and / or reasoning on provided data. In at least one embodiment, development, publishing, and / or storage can be performed using a software development kit (SDK) associated with the system (e.g., to ensure that the developed applications and / or containers conform to or are compatible with the system). In at least one embodiment, the developed applications can be tested locally (e.g., at a first facility, on data from the first facility) using the SDK, which serves as a system (e.g., Figure 10 The system 1000 in FIG. 1000 may support at least some of the services 920. In at least one embodiment, once validated by the system 1000 (e.g., for accuracy, etc.), the application is made available in the container registry for selection and / or implementation by a user (e.g., a hospital, clinic, laboratory, healthcare provider, etc.) to perform one or more processing tasks on data at the user's facility (e.g., a second facility).

[0097] In at least one embodiment, the developer can then share the application or container over a network for use by a system (e.g., Figure 10 924). In at least one embodiment, completed and validated applications or containers can be stored in a container registry, and associated machine learning models can be stored in a model registry 924. In at least one embodiment, a requesting entity (providing an inference or image processing request) can browse the container registry and / or model registry 924 for applications, containers, datasets, machine learning models, etc., select the desired combination of elements to include in a data processing pipeline, and submit a processing request. In at least one embodiment, the request can include the input data necessary to execute the request and / or can include a selection of the application and / or machine learning model to be executed when processing the request. In at least one embodiment, the request can then be passed to one or more components of the deployment system 906 (e.g., a cloud) to perform processing in the data processing pipeline. In at least one embodiment, the processing performed by the deployment system 906 can include referencing elements (e.g., applications, containers, models, etc.) selected from the container registry and / or model registry 924. In at least one embodiment, once results are generated by the pipeline, the results can be returned to the user for reference (e.g., for viewing in a viewing application suite executed locally, on a local workstation, or on a terminal).

[0098] In at least one embodiment, to assist in processing or executing applications or containers in the pipeline, services 920 may be utilized. In at least one embodiment, services 920 may include computing services, collaborative content creation services, simulation services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, services 920 may provide functionality that is common to one or more applications in software 918, and thus may abstract functionality into services that can be called or utilized by applications. In at least one embodiment, the functionality provided by services 920 may run dynamically and more efficiently, while also enabling faster processing of data by allowing applications to process data in parallel (e.g., using Figure 10 In at least one embodiment, rather than requiring each application that shares the same functionality provided by service 920 to have a corresponding instance of service 920, service 920 can be shared between and among various applications. In at least one embodiment, as non-limiting examples, the service may include an inference server or engine that can be used to perform detection or segmentation tasks. In at least one embodiment, a model training service may be included that can provide machine learning model training and / or retraining capabilities.

[0099] In at least one embodiment, where the service 920 includes an AI service (e.g., an inference service), as part of the execution of an application, one or more machine learning models associated with an application for anomaly detection (e.g., tumor, growth abnormality, scarring, etc.) can be executed by calling (e.g., as an API call) the inference service (e.g., an inference server) to execute one or more machine learning models or processing thereof. In at least one embodiment, where another application includes one or more machine learning models for a segmentation task, the application can call the inference service to execute the machine learning model for performing one or more processing operations associated with the segmentation task. In at least one embodiment, the software 918 implementing the high-level processing and inference pipeline can be pipelined in that each application can call the same inference service to perform one or more inference tasks.

[0100] In at least one embodiment, hardware 922 may include a GPU, a CPU, a graphics card, an AI / deep learning system (e.g., an AI supercomputer such as NVIDIA's DGX TMIn at least one embodiment, different types of hardware 922 can be used to provide efficient, purpose-built support for the software 918 and services 920 in the deployment system 906. In at least one embodiment, GPU processing can be used to perform local processing (e.g., at the facility 902) within the AI / deep learning system, in the cloud system, and / or in other processing components of the deployment system 906 to improve the efficiency, accuracy, and effectiveness of game name recognition.

[0101] In at least one embodiment, software 918 and / or services 920 may be optimized for GPU processing, as non-limiting examples, with respect to deep learning, machine learning, and / or high performance computing, simulation, and visual computing. In at least one embodiment, at least some of the computing environments of deployment system 906 and / or training system 904 may be run on GPUs with GPU-optimized software (e.g., NVIDIA DGX TM In at least one embodiment, the cloud platform may include GPU-optimized execution for deep learning tasks, GPU processing for machine learning tasks, or other computing tasks. In at least one embodiment, an AI / deep learning supercomputer and / or GPU-optimized software (e.g., as in NVIDIA's DGX TM system) as a hardware abstraction and scaling platform to execute cloud platforms (e.g., NVIDIA's NGC TM In at least one embodiment, the cloud platform can integrate an application container cluster system or coordination system (e.g., Kubernetes) on multiple GPUs to achieve seamless scaling and load balancing.

[0102] Figure 10 is a system diagram of an example system 1000 for generating and deploying a deployment pipeline according to at least one embodiment. In at least one embodiment, the system 1000 can be used to implement Figure 9 The system 1000 may include a training system 904 and a deployment system 906. In at least one embodiment, the training system 904 and the deployment system 906 may be implemented using software 918, services 920, and / or hardware 922, as described herein.

[0103] In at least one embodiment, system 1000 (e.g., training system 904 and / or deployment system 906) can be implemented in a cloud computing environment (e.g., using cloud 1026). In at least one embodiment, system 1000 can be implemented locally (with respect to a facility) or as a combination of cloud computing resources and local computing resources. In at least one embodiment, access to APIs in cloud 1026 can be restricted to authorized users by establishing security measures or protocols. In at least one embodiment, the security protocols can include a network token, which can be signed by an authentication (e.g., AuthN, AuthZ, Gluecon, etc.) service and can carry appropriate authorization. In at least one embodiment, the APIs of the virtual instrument (described herein) or other instances of system 1000 can be restricted to a set of public Internet Service Providers (ISPs) that have been vetted or authorized for interaction.

[0104] In at least one embodiment, the various components of system 1000 can communicate with each other and among themselves using any of a variety of different network types, including but not limited to local area networks (LANs) and / or wide area networks (WANs) via wired and / or wireless communication protocols. In at least one embodiment, communications between facilities and components of system 1000 (e.g., for sending inference requests, for receiving results of inference requests, etc.) can be transmitted via one or more data buses, wireless data protocols (e.g., Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0105] In at least one embodiment, similar to the present disclosure regarding Figure 9 As described, the training system 904 can execute a training pipeline 1004. In at least one embodiment, where the deployment system 906 will use one or more machine learning models in a deployment pipeline 1010, the training pipeline 1004 can be used to train or retrain one or more (e.g., pre-trained) models, and / or implement one or more pre-trained models 1006 (e.g., without retraining or updating). In at least one embodiment, as a result of the training pipeline 1004, an output model 916 can be generated. In at least one embodiment, the training pipeline 1004 can include any number of processing steps, AI-assisted annotation 910, labeling of feedback data 908 or annotation to generate labeled data 912, selecting a model from a model registry, model training 914, training, retraining, or updating a model, and / or other processing steps. In at least one embodiment, different training pipelines 1004 can be used for different machine learning models used by the deployment system 906. In at least one embodiment, similar to the description regarding Figure 9 The training pipeline 1004 of the first example described can be used for a first machine learning model, similar to the one described with respect to Figure 9 The second example training pipeline 1004 described can be used for a second machine learning model, similar to the one described with respect to Figure 9 The third example training pipeline 1004 is described as being usable for a third machine learning model. In at least one embodiment, any combination of tasks within the training system 904 may be used, depending on the requirements of each respective machine learning model. In at least one embodiment, one or more machine learning models may already be trained and ready for deployment, so the training system 904 may not perform any processing on the machine learning model, and the machine learning model may be implemented by the deployment system 906.

[0106] In at least one embodiment, one or more output models 916 and / or pre-trained models 1006 may include any type of machine learning model, depending on the embodiment. In at least one embodiment and without limitation, the machine learning model used by the system 1000 may include linear regression, logistic regression, decision tree, support vector machine (SVM), naive Bayes, k-nearest neighbor (Knn), k-means clustering, random forest, dimensionality reduction algorithm, gradient boosting algorithm, neural network (e.g., autoencoder, convolution, recursion, perceptron, long / short term memory (LSTM), Bi-LSTM, Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machine, etc.), and / or other types of machine learning models.

[0107] In at least one embodiment, the training pipeline 1004 may include AI-assisted annotation. In at least one embodiment, the labeled data 912 may be generated by any number of techniques (e.g., traditional annotation). In at least one embodiment, in some examples, labels or other annotations may be generated in a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, another type of application suitable for generating ground truth annotations or labels, and / or may be hand-drawn. In at least one embodiment, the ground truth data may be synthetically generated (e.g., generated from a computer model or rendering), realistically generated (e.g., designed and generated from real-world data), machine-generated (e.g., using feature analysis and learning to extract features from the data and then generate labels), manually annotated (e.g., a labeler or annotation expert defines the location of the labels), and / or a combination thereof. In at least one embodiment, for each instance of feedback data 908 (or other data type used by the machine learning model), there may be corresponding ground truth data generated by the training system 904. In at least one embodiment, AI-assisted annotation may be performed as part of the deployment pipeline 1010; in addition to or in place of the AI-assisted annotation included in the training pipeline 1004. In at least one embodiment, system 1000 may include a multi-layer platform that may include a software layer (eg, software 918 ) of a diagnostic application (or other application type) that may perform one or more medical imaging and diagnostic functions.

[0108] In at least one embodiment, the software layer can be implemented as a secure, encrypted, and / or authenticated API through which an application or container can be invoked (e.g., called) from an external environment (e.g., facility 902). In at least one embodiment, the application can then call or execute one or more services 920 to perform computational, AI, or visualization tasks associated with the respective application, and the software 918 and / or services 920 can utilize hardware 922 to perform the processing tasks in an effective and efficient manner.

[0109] In at least one embodiment, the deployment system 906 can execute a deployment pipeline 1010. In at least one embodiment, the deployment pipeline 1010 can include any number of applications that can be sequential, non-sequential, or otherwise applied to feedback data (and / or other data types) - including AI-assisted annotations, as described above. In at least one embodiment, the deployment pipeline 1010 for an individual device, as described herein, can be referred to as a virtual instrument for the device. In at least one embodiment, there can be more than one deployment pipeline 1010 for a single device, depending on the information desired from the data generated by the device.

[0110] In at least one embodiment, applications that can be used to deploy pipeline 1010 can include any application that can be used to perform processing tasks on feedback data or other data from the device. In at least one embodiment, because various applications can share common image operations, in some embodiments, a data enhancement library (e.g., as one of the services 920) can be used to accelerate these operations. In at least one embodiment, in order to avoid the bottlenecks of traditional processing methods that rely on CPU processing, parallel computing platform 1030 can be used for GPU acceleration of these processing tasks.

[0111] In at least one embodiment, the deployment system 906 may include a user interface 1014 (e.g., a graphical user interface, a web interface, etc.) that can be used to select applications to be included in the deployment pipeline 1010, to arrange applications, to modify or change applications or their parameters or configuration, to use and interact with the deployment pipeline 1010 during setup and / or deployment, and / or to otherwise interact with the deployment system 906. In at least one embodiment, although not shown with respect to the training system 904, the UI 1014 (or a different user interface) can be used to select models for use in the deployment system 906, to select models for training or retraining in the training system 904, and / or to otherwise interact with the training system 904.

[0112] In at least one embodiment, in addition to the application orchestration system 1028, a pipeline manager 1012 can be used to manage the interactions between applications or containers of the deployment pipeline 1010 and services 920 and / or hardware 922. In at least one embodiment, the pipeline manager 1012 can be configured to facilitate interactions from application to application, from application to service 920, and / or from application or service to hardware 922. In at least one embodiment, although shown as included in the software 918, this is not intended to be limiting, and in some examples, the pipeline manager 1012 can be included in the service 920. In at least one embodiment, the application orchestration system 1028 (e.g., Kubernetes, Docker, etc.) can include a container orchestration system that can group applications into containers as a logical unit for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline 1010 (e.g., rebuilding an application, partitioning an application, etc.) with individual containers, each application can execute in a self-contained environment (e.g., at the kernel level) to improve speed and efficiency.

[0113] In at least one embodiment, each application and / or container (or its image) can be developed, modified, and deployed separately (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separate from the first user or developer), which can allow for focus and attention on the tasks of a single application and / or container without being hindered by the tasks of other applications or containers. In at least one embodiment, the pipeline manager 1012 and the application coordination system 1028 can facilitate communication and collaboration between different containers or applications. In at least one embodiment, the application coordination system 1028 and / or the pipeline manager 1012 can facilitate communication and resource sharing between and among each application or container, as long as the expected inputs and / or outputs of each container or application are known to the system (e.g., based on the configuration of the application or container). In at least one embodiment, because one or more applications or containers in the deployment pipeline 1010 can share the same services and resources, the application coordination system 1028 can coordinate, load balance, and determine the sharing of services or resources between and among the various applications or containers. In at least one embodiment, the scheduler can be used to track resource requirements of applications or containers, current or planned usage of those resources, and resource availability. Thus, in at least one embodiment, the scheduler can allocate resources to different applications and distribute resources between and among applications, taking into account the needs and availability of the system. In some examples, the scheduler (and / or other components of the application coordination system 1028) can determine resource availability and distribution based on constraints imposed on the system (e.g., user constraints), such as quality of service (QoS), the urgency of data output (e.g., to determine whether to perform real-time processing or delayed processing), etc.

[0114] In at least one embodiment, the services 920 utilized and shared by applications or containers in the deployment system 906 may include computing services 1016, collaborative content creation services 1017, AI services 1018, simulation services 1019, visualization services 1020, and / or other service types. In at least one embodiment, an application may call (e.g., execute) one or more services 920 to perform processing operations for the application. In at least one embodiment, an application may utilize computing services 1016 to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, one or more computing services 1016 may be utilized to perform parallel processing (e.g., using a parallel computing platform 1030) to process data substantially simultaneously by one or more applications and / or one or more tasks of a single application. In at least one embodiment, a parallel computing platform 1030 (e.g., NVIDIA's CUDA) may implement general-purpose computing on a GPU (GPGPU) (e.g., GPU 1022). In at least one embodiment, a software layer of the parallel computing platform 1030 may provide access to a virtual instruction set and parallel computing elements of the GPU to execute computational kernels. In at least one embodiment, the parallel computing platform 1030 may include memory, and in some embodiments, the memory may be shared between and among multiple containers, and / or between and among different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls may be generated for multiple containers and / or multiple processes within a container to use the same data from a shared memory segment of the parallel computing platform 1030 (e.g., where multiple different stages of an application or multiple applications are processing the same information). In at least one embodiment, rather than copying and moving data to different locations in memory (e.g., read / write operations), the same data in the same memory location may be used by any number of processing tasks (e.g., at the same time, at different times, etc.). In at least one embodiment, since data is used to generate new data as a result of processing, this information about the new location of the data may be stored and shared between the various applications. In at least one embodiment, the location of the data, as well as the location of updated or modified data, may be part of the definition of how the payload in the container is understood.

[0115] In at least one embodiment, AI service 1018 can be utilized to perform inference services for executing machine learning models associated with an application (e.g., tasked with executing one or more processing tasks of the application). In at least one embodiment, AI service 1018 can utilize AI system 1024 to execute machine learning models (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, an application in deployment pipeline 1010 can use one or more output models 916 from training system 904 and / or other models of the application to perform inference on imaging data (e.g., DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.). In at least one embodiment, two or more instances of inference using application coordination system 1028 (e.g., a scheduler) can be available. In at least one embodiment, the first category can include high-priority / low-latency paths that can achieve higher service level agreements, such as for performing inference on urgent requests in emergency situations or for radiologists during diagnostic procedures. In at least one embodiment, the second category may include a standard priority path that may be used for requests that may not be urgent or where analysis can be performed at a later time. In at least one embodiment, the application coordination system 1028 may allocate resources (e.g., services 920 and / or hardware 922) for different reasoning tasks of the AI ​​service 1018 based on the priority path.

[0116] In at least one embodiment, shared memory can be installed into the AI ​​service 1018 in the system 1000. In at least one embodiment, the shared memory can operate as a cache (or other storage device type) and can be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, a set of API instances of the deployment system 906 can receive the request and select one or more instances (e.g., for best fit, load balancing, etc.) to process the request. In at least one embodiment, to process the request, the request can be entered into a database, and if it is not already in the cache, the machine learning model can be located from the model registry 924. A validation step can ensure that the appropriate machine learning model is loaded into the cache (e.g., shared storage), and / or a copy of the model can be saved to the cache. In at least one embodiment, if the application is not already running or there are not enough instances of the application, a scheduler (e.g., a scheduler of the pipeline manager 1012) can be used to start the application referenced in the request. In at least one embodiment, if an inference server has not yet been started to execute the model, an inference server can be started. In at least one embodiment, any number of inference servers can be started for each model. In at least one embodiment, in a pull model where the inference servers are clustered, the model can be cached whenever load balancing is beneficial. In at least one embodiment, the inference servers can be statically loaded into the corresponding distributed servers.

[0117] In at least one embodiment, inference can be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server can be associated with a model (and optionally with multiple versions of a model). In at least one embodiment, if an instance of an inference server does not exist when a request to perform inference on a model is received, a new instance can be loaded. In at least one embodiment, a model can be passed to the inference server when it is started, so that the same container can be used to serve different models as long as the inference server is running as a different instance.

[0118] In at least one embodiment, during application execution, an inference request for a given application may be received, a container (e.g., an instance hosting an inference server) may be loaded (if not already loaded), and a launcher may be called. In at least one embodiment, pre-processing logic in the container may load, decode, and / or perform any additional pre-processing on the incoming data (e.g., using the CPU and / or GPU). In at least one embodiment, once the data is ready for inference, the container may perform inference on the data as needed. In at least one embodiment, this may include a single inference call for a single image (e.g., a hand X-ray), or may request inference on hundreds of images (e.g., a chest CT scan). In at least one embodiment, the application may summarize the results before completion, which may include, but is not limited to, a single confidence score, pixel-level segmentation, voxel-level segmentation, generated visualizations, or generated text summarizing the results. In at least one embodiment, different models or applications may be assigned different priorities. For example, some models may have real-time priority (turnaround time less than 1 minute), while other models may have a lower priority (e.g., turnaround time less than 10 minutes). In at least one embodiment, model execution time may be measured from the requesting mechanism or entity and may include collaborative network traversal time as well as execution time of the inference service.

[0119] In at least one embodiment, the transmission of requests between the service 920 and the inference application can be hidden behind a software development kit (SDK) and can provide robust transport via queues. In at least one embodiment, requests are placed in a queue via an API for an individual application / tenant ID combination, and the SDK pulls the request from the queue and delivers it to the application. In at least one embodiment, the name of the queue can be provided in the context of the SDK picking up the request. In at least one embodiment, asynchronous communication via queues can be useful because it can allow any instance of the application to pick up work when it is available. In at least one embodiment, results can be transmitted back through the queue to ensure no data is lost. In at least one embodiment, queues can also provide the ability to partition work, because the highest priority work can enter a queue connected to most instances of the application, while the lowest priority work can enter a queue connected to a single instance, which processes the tasks in the order they are received. In at least one embodiment, the application can run on a GPU-accelerated instance generated in the cloud 1026, and the inference service can perform inference on the GPU.

[0120] In at least one embodiment, visualization services 1020 can be utilized to generate visualizations for viewing application and / or deployment pipeline 1010 outputs. In at least one embodiment, visualization services 1020 can utilize GPU 1022 to generate visualizations. In at least one embodiment, visualization services 1020 can implement rendering effects such as ray tracing or other light transport simulation techniques to generate higher quality visualizations. In at least one embodiment, visualizations can include, but are not limited to, 2D image rendering, 3D volume rendering, 3D volume reconstruction, 2D tomographic slices, virtual reality displays, augmented reality displays, and the like. In at least one embodiment, a virtualized environment can be used to generate virtual interactive displays or environments (e.g., virtual environments) for system users (e.g., doctors, nurses, radiologists, etc.) to interact with. In at least one embodiment, visualization services 1020 can include internal visualizers, movies, and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0121] In at least one embodiment, hardware 922 may include GPUs 1022, AI system 1024, cloud 1026, and / or any other hardware used to execute training system 904 and / or deployment system 906. In at least one embodiment, GPUs 1022 (e.g., NVIDIA's TESLA and / or QUADRO GPUs) may include any number of GPUs that can be used to perform processing tasks for any feature or functionality of compute service 1016, collaborative content creation service 1017, AI service 1018, simulation service 1019, visualization service 1020, other services, and / or software 918. For example, for AI service 1018, GPUs 1022 may be used to perform pre-processing on imaging data (or other data types used by machine learning models), post-processing on the output of machine learning models, and / or perform inference (e.g., to execute machine learning models). In at least one embodiment, cloud 1026, AI system 1024, and / or other components of system 1000 may use GPUs 1022. In at least one embodiment, cloud 1026 may include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI system 1024 can use a GPU, and cloud 1026 (or at least part of the task being deep learning or inference) can be executed using one or more AI systems 1024. Likewise, while hardware 922 is shown as discrete components, this is not intended to be limiting, and any component of hardware 922 can be combined with or utilized by any other component of hardware 922.

[0122] In at least one embodiment, AI system 1024 may include a purpose-built computing system (e.g., a supercomputer or HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, in addition to CPU, RAM, storage, and / or other components, features, or functions, AI system 1024 (e.g., NVIDIA's DGX TM ) can also include GPU-optimized software (e.g., a software stack) that can use multiple GPUs 1022 to execute. In at least one embodiment, one or more AI systems 1024 can be implemented in the cloud 1026 (e.g., in a data center) to perform some or all of the AI-based processing tasks of system 1000.

[0123] In at least one embodiment, cloud 1026 may include GPU-accelerated infrastructure (e.g., NVIDIA's NGC TM ), which can provide a GPU-optimized platform for executing processing tasks of the system 1000. In at least one embodiment, the cloud 1026 can include an AI system 1024 for executing one or more AI-based tasks of the system 1000 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, the cloud 1026 can be integrated with an application coordination system 1028 that utilizes multiple GPUs to enable seamless scaling and load balancing between and among applications and services 920. In at least one embodiment, the cloud 1026 can be responsible for executing at least some of the services 920 of the system 1000, including the compute services 1016, the AI ​​services 1018, and / or the visualization services 1020, as described herein. In at least one embodiment, the cloud 1026 can perform large and small batch inference (e.g., executing NVIDIA's TENSOR RT), provide accelerated parallel computing APIs and platforms 1030 (e.g., NVIDIA's ), execute application coordination system 1028 (e.g., Kubernetes), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher quality movie effects), and / or may provide other functionality for system 1000.

[0124] In at least one embodiment, to protect patient confidentiality (e.g., in situations where patient data or records are used off-site), the cloud 1026 may include a registry - such as a deep learning container registry. In at least one embodiment, the registry may store containers for instantiating applications that may perform pre-processing, post-processing, or other processing tasks on the patient data. In at least one embodiment, the cloud 1026 may receive data that includes patient data as well as sensor data in containers, perform the requested processing only on the sensor data in those containers, and then forward the resulting output and / or visualization to the appropriate parties and / or devices (e.g., local medical devices for visualization or diagnosis) without extracting, storing, or otherwise accessing the patient data. In at least one embodiment, the confidentiality of the patient data is preserved in accordance with HIPAA and / or other data regulations.

[0125] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described above in detail. However, it should be understood that there is no intention to limit the disclosure to one or more specific forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0126] Unless otherwise noted or clearly contradicted by the context, the use of the terms "a" and "an" and "the" and similar references in the context of describing the disclosed embodiments (particularly in the context of the appended claims) should be interpreted as covering the singular and plural, rather than as definitions of terms. Unless otherwise noted, the terms "include," "have," "include," and "contain" should be interpreted as open-ended terms (meaning "including but not limited to"), unless otherwise noted. The term "connected" (when unmodified, refers to a physical connection) should be interpreted as partially or completely contained within, attached to, or connected together, even if there is some intervention. Unless otherwise noted herein, references to numerical ranges herein are intended only to be used as a shorthand method of referring to each individual value falling within the range, and each individual value is incorporated into the specification as if it were separately recited herein. In at least one embodiment, unless otherwise noted or contradicted by the context, the use of the term "set" (e.g., "item set") or "subset" should be interpreted as a non-empty set comprising one or more members. Furthermore, unless otherwise indicated or contradicted by context, the term "subset" of a corresponding set does not necessarily mean a proper subset of the corresponding set, but rather a subset and a corresponding set may be equivalent.

[0127] Unless expressly indicated otherwise or clearly contradicted by context, conjunctions such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C" are understood in context to generally refer to an item, clause, or the like, which may be A or B or C, or any non-empty subset of the set of A, B, and C. For example, in the illustrative example of a set having three members, the conjunctions "at least one of A, B, and C" and "at least one of A, B, and C" refer to any of the following sets: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctions are not generally intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C. Additionally, unless expressly indicated otherwise or contradicted by context, the term "plurality" refers to a plurality (e.g., "a plurality of items" refers to a plurality of items). In at least one embodiment, the number of items in the plurality of items is at least two, but may be more if expressly indicated or indicated by context. Further, unless specified otherwise or clear from the context, the phrase "based on" means "based at least in part on" or "based at least on" rather than "based solely on."

[0128] Unless otherwise indicated herein or clearly contradicted by the context, the operations of the processes described herein may be performed in any suitable order. In at least one embodiment, processes such as those described herein (or variations and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that are collectively executed on one or more processors by hardware or a combination thereof. In at least one embodiment, the code is stored on a computer-readable storage medium in the form of, for example, a computer program that includes a plurality of instructions that can be executed by one or more processors. In at least one embodiment, the computer-readable storage medium is a non-transitory computer-readable storage medium that excludes transient signals (e.g., propagated transient electrical or electromagnetic transmissions) but includes non-transitory data storage circuits (e.g., buffers, caches, and queues) within a transient signal transceiver. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-transitory computer-readable storage media (or other memory for storing executable instructions) having executable instructions stored thereon, which, when executed by one or more processors of a computer system (i.e., as a result of being executed), causes the computer system to perform the operations described herein. In at least one embodiment, the set of non-transitory computer-readable storage media includes a plurality of non-transitory computer-readable storage media, and one or more of the individual non-transitory storage media in the plurality of non-transitory computer-readable storage media lacks all of the code, but rather the plurality of non-transitory computer-readable storage media collectively store all of the code. In at least one embodiment, the executable instructions are executed so that different instructions are executed by different processors, for example, a non-transitory computer-readable storage medium stores instructions, and a main central processing unit ("CPU") executes some instructions, while a graphics processing unit ("GPU") executes other instructions. In at least one embodiment, different components of the computer system have separate processors, and different processors execute different subsets of instructions.

[0129] Thus, in at least one embodiment, a computer system is configured to implement one or more services that individually or collectively perform the operations of the processes described herein, and such a computer system is configured with applicable hardware and / or software that enables the implementation of the operations. Furthermore, the computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment is a distributed computer system comprising multiple devices operating in different ways such that the distributed computer system performs the operations described herein and such that no single device performs all of the operations.

[0130] The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended merely to better illuminate embodiments of the present disclosure and does not limit the scope of the disclosure unless otherwise required. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0131] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0132] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms may not be intended as synonyms for each other. Rather, in specific examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0133] In some embodiments, unless otherwise explicitly stated, it is understood that throughout the specification, terms such as "processing," "computing," "calculating," "determining," etc., refer to the actions and / or processes of a computer or computing system or similar electronic computing device that processes and / or converts data represented as physical quantities (e.g., electronic) in the registers and / or memories of the computing system into other data similarly represented as physical quantities in the memories, registers, or other such information storage, transmission, or display devices of the computing system.

[0134] In a similar manner, the term "processor" may refer to any device or portion of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As non-limiting examples, a "processor" may be a CPU or a GPU. A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Likewise, each process may refer to multiple processes to execute instructions continuously or intermittently, sequentially, or in parallel. In at least one embodiment, the terms "system" and "method" may be used interchangeably herein, as long as a system may embody one or more methods, and a method may be considered a system.

[0135] In this document, reference may be made to obtaining, acquiring, receiving, or inputting analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog and digital data can be accomplished in a variety of ways, such as by receiving data as parameters of a function call or a call to an application programming interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data via a serial or parallel interface. In at least one embodiment, the process of obtaining, acquiring, receiving, or inputting analog or digital data can be accomplished by transmitting data from a providing entity to an acquiring entity via a computer network. In at least one embodiment, reference may also be made to providing, outputting, transmitting, sending, or presenting analog or digital data. In various examples, the process of providing, outputting, transmitting, sending, or presenting analog or digital data can be accomplished by transmitting data as input or output parameters of a function call, an application programming interface, or an interprocess communication mechanism.

[0136] Although the description herein sets forth example embodiments of the described technology, other architectures may be used to implement the described functionality and are intended to fall within the scope of this disclosure. Furthermore, although specific assignments of responsibilities are defined above for descriptive purposes, the various functions and responsibilities may be assigned and divided in different ways depending on the circumstances.

[0137] Furthermore, although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A method comprising: comparing an original transcription of the input audio with a modified transcription corresponding to the input audio to identify a modified set of words; generating synthesized audio using a text-to-speech machine learning model based at least on the modified word set; generating a modified synthesized audio using a voice matching machine learning model based at least on at least a portion of the input audio and the synthesized audio; as well as Output audio is generated by combining the input audio with the modified synthesized audio.

2. The method of claim 1 , wherein the raw transcription is generated based at least on the input audio using a first automatic speech recognition machine learning model.

3. The method according to claim 1, further comprising: The modified transcript is generated by at least one of: modifying one or more words in the original transcription based on user input; or A natural language processing machine learning model is applied to the original transcription to obtain the modified transcription.

4. The method of claim 1 , wherein the modified word set includes at least some text that is present in the modified transcription but not in the original transcription.

5. The method according to claim 1, further comprising: identifying the speaking rate of the input audio, Wherein generating the synthesized audio is further based at least on the speaking rate.

6. The method of claim 1 , wherein generating the modified synthesized audio comprises: applying a speech enhancement machine learning model to the at least a portion of the input audio to obtain enhanced input audio; as well as The modified synthesized audio is generated using the voice matching machine learning model based at least on the synthesized audio and the enhanced input audio.

7. The method of claim 1 , wherein the original transcription includes a first timestamp, and wherein generating the output audio comprises: generating a synthesized transcript using a first automatic speech recognition machine learning model based at least on the modified synthesized audio, the synthesized transcript including a second timestamp; as well as The input audio is aligned with the modified synthesized audio based on at least the first time stamp and the second time stamp.

8. The method of claim 1 , wherein generating the output audio comprises: applying a denoising machine learning model to the input audio to obtain a noise mask; applying the noise mask to the modified synthesized audio to obtain noisy synthesized audio; as well as The output audio is generated by combining the input audio with the noisy synthesized audio.

9. The method of claim 1 , wherein the input audio is at least one of: podcast audio; Background audio of the video; a narrative of a speech; or Audiobook recordings.

10. A system comprising: One or more processors configured to: comparing an initial transcription of the input audio with a modified transcription corresponding to the input audio to identify a modified set of words; generating synthesized audio using a text-to-speech machine learning model based at least on the modified word set; generating a modified synthesized audio using a voice matching machine learning model based at least on at least a portion of the input audio and the synthesized audio; as well as Output audio is generated by combining the input audio with the modified synthesized audio.

11. The system of claim 10, wherein the initial transcription is generated using a first automatic speech recognition machine learning model and based at least on the input audio.

12. The system of claim 10, wherein the one or more processors are further configured to generate the modified transcription by at least one of: In response to user input, modifying one or more words in the initial transcription; or A natural language processing machine learning model is applied to the initial transcription to obtain the modified transcription.

13. The system of claim 10, wherein the modified set of words includes at least one word that is present in the modified transcription but not in the initial transcription.

14. The system of claim 10, wherein the one or more processors are further configured to: identifying the speaking rate of the input audio, Wherein the synthesized audio is further generated based at least on the speaking rate.

15. The system of claim 10, wherein the modified synthesized audio is generated by at least: applying a speech enhancement machine learning model to the at least a portion of the input audio to obtain enhanced input audio; and The modified synthesized audio is generated using the voice matching machine learning model based on the synthesized audio and the enhanced input audio.

16. The system of claim 10, wherein the initial transcription includes a first timestamp, and the output audio is generated by at least: generating a synthesized transcript using a first automatic speech recognition machine learning model based at least on the modified synthesized audio, the synthesized transcript including a second timestamp; and The input audio is aligned with the modified synthesized audio based on at least the first time stamp and the second time stamp.

17. The system of claim 10, wherein the output audio is generated by at least: applying a denoising machine learning model to the input audio to obtain a noise mask; applying the noise mask to the modified synthesized audio to obtain noisy synthesized audio; and The output audio is generated by combining the input audio with the noisy synthesized audio.

18. The system of claim 10, wherein the input audio corresponds to at least one of: podcast audio; Background audio of the video; a narrative of a speech; or Audiobook recordings.

19. The system of claim 10, wherein the system is included in at least one of: In-vehicle infotainment systems for autonomous or semi-autonomous machines; a system for performing simulation operations; Systems for performing digital twin operations; a system for performing light transport simulations; A system for performing collaborative content creation of 3D assets; Systems for performing deep learning operations; Systems implemented using edge devices; a system for generating or presenting at least one of virtual reality content, mixed reality content, or augmented reality content; Systems implemented using robots; Systems for performing conversational AI operations; A system for performing one or more generative AI operations; A system implementing one or more language models; A system implementing one or more large language models (LLMs); Systems for generating synthetic data; A system comprising one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

20. One or more processors comprising processing circuitry configured to perform operations comprising: comparing an initial recording of input audio with a modified transcription corresponding to the input audio to identify a modified set of words; generating synthesized audio using a text-to-speech machine learning model based at least on the modified word set; generating a modified synthesized audio using a voice matching machine learning model based at least on at least a portion of the input audio and the synthesized audio; as well as Output audio is generated by combining the input audio with the modified synthesized audio.

Images