Sound source visualization control method and related equipment

Abstract

The invention relates to a sound source visualization control method and related equipment. The method includes: selecting a target two-way data link from a plurality of two-way data links, obtaining audio data from an audio source through the target two-way data link; performing audio rendering and processing on the audio data to obtain characteristic information of the audio data, the characteristic information including at least one of the following: frequency information, volume information and audio content information; preprocessing the characteristic information, the preprocessing including noise reduction and gain adjustment; displaying the characteristic information and controls for adjusting the parameters of the characteristic information through a visual interface, and transmitting the parameter information obtained by the controls to the audio source through the target two-way data link. Through this application, the problem of poor compatibility of the sound source control scheme in the related art is solved, and the compatibility and performance of the sound source control scheme are improved.

Description

Technical Field

[0001] The invention relates to the field of multimedia technology, and in particular to a sound source visualization control method and related equipment. Background Art

[0002] Transmission solutions in related technologies, for example: Websocket uses the TCP transmission protocol, which has the problem of head-of-line blocking; WebRTC is suitable for P2P scenarios and supports multi-channel transmission, but there are still problems of head-of-line blocking and long connection time of data flow; WebTransport solves the problem of head-of-line blocking and has the advantages of fast transmission, low latency, security, and good robustness in complex networks, but it is not widely used and has the problem of large workload for the development of new support; HTTP polling method realizes simulated long connection communication, has weak capabilities, and is suitable for some scenarios with weak server capabilities or low browser versions.

[0003] However, the sound source control solution of the related art adopts a fixed transmission solution between the sound source and the browser, which cannot adapt to complex operating environments and cannot take into account both compatibility and high performance. Summary of the invention

[0004] The sound source visualization control method and related devices provided by the embodiments of the present invention at least solve the problem of poor compatibility of sound source control solutions in related technologies.

[0005] A sound source visualization control method, applied to a browser client, comprising:

[0006] selecting a target bidirectional data link from a plurality of bidirectional data links, and acquiring audio data from an audio source through the target bidirectional data link;

[0007] Performing audio rendering and processing on the audio data to obtain feature information of the audio data, wherein the feature information includes at least one of the following: frequency information, volume information, and audio content information;

[0008] Preprocessing the feature information, wherein the preprocessing includes noise reduction and gain adjustment;

[0009] The characteristic information and a control for adjusting the parameters of the characteristic information are displayed through a visual interface, and the parameter information obtained by the control is transmitted to the audio source through the target bidirectional data link.

[0010] In some of the embodiments, the target bidirectional data link is selected from the multiple bidirectional data links based on the communication capability of the audio source and / or the audio transmission scenario, wherein the multiple bidirectional data links include at least one of the following: WebSocket, WebTransport, WebRTC, HTTP polling.

[0011] In some embodiments, selecting the target bidirectional data link from the plurality of bidirectional data links according to the communication capability of the audio source and / or the audio transmission scenario comprises:

[0012] When the audio source supports only one bidirectional data link, determining the bidirectional data link supported by the audio source as the target bidirectional data link;

[0013] When the audio source supports multiple bidirectional data links, if there is a designated bidirectional data link, the designated bidirectional data link is determined to be the target bidirectional data link; if there is no designated bidirectional data link, one bidirectional data link is selected as the target bidirectional data link from bidirectional data links supported by both the browser client and the audio source, and another bidirectional data link is selected as a backup degradation scheme for the target bidirectional data link;

[0014] The bidirectional data link supported by the browser client is determined based on a characteristic detection result of the browser client and / or a bidirectional data link support matrix corresponding to the browser client.

[0015] In some embodiments, performing audio rendering and processing on the audio data to obtain feature information of the audio data includes:

[0016] When the format of the audio data is a format supported by HTML5, calling the WebAudio API through the audio tag of HTML5 to perform audio rendering of the audio data;

[0017] When the format of the audio data is a format not supported by HTML5, a target audio decoding and rendering library compiled into WebAssembly is loaded to perform audio rendering of the audio data.

[0018] In some of the embodiments, the target audio decoding and rendering library is compiled based on SIMD instructions supported by different CPU architectures; when performing multi-threaded parallel audio rendering of the audio data, SharedArrayBuffer shared memory technology is used to achieve data exchange and parallel processing between multiple threads.

[0019] In some embodiments, the browser client includes multiple audio decoding and rendering libraries compiled into WebAssembly; loading the target audio decoding and rendering library compiled into WebAssembly to perform audio rendering of the audio data includes:

[0020] According to the feature detection result of the browser client and / or the SIMD instruction and SharedArrayBuffer support matrix information corresponding to the browser client, an audio decoding and rendering library is selected from multiple audio decoding and rendering libraries compiled into WebAssembly as the target audio decoding and rendering library.

[0021] In some embodiments, the control for adjusting the parameters of the characteristic information includes: an audio source parameter for adjusting the characteristic information by an audio source, and a terminal side parameter for adjusting the characteristic information by a browser client; wherein,

[0022] The audio source parameters include at least one of the following: sampling bit depth, sampling rate, audio format;

[0023] The terminal side parameters include at least one of the following: noise reduction value, gain parameter, dynamic range compression rate, spectrum range, filter mode, volume, equalizer setting, and channel balance.

[0024] In some embodiments, preprocessing the feature information includes:

[0025] The frequency information of the audio data is divided into 4 bytes, and each of the 4 bytes obtained by the division is converted into a floating point number to obtain a floating point number array corresponding to the frequency information, so that the visualization interface can draw and display a spectrum image corresponding to the frequency information based on the floating point number array.

[0026] A sound source visualization control device, comprising:

[0027] A stream acquisition module, used for selecting a target bidirectional data link from a plurality of bidirectional data links, and acquiring audio data from an audio source through the target bidirectional data link;

[0028] A rendering module, used to perform audio rendering and processing on the audio data to obtain feature information of the audio data, wherein the feature information includes at least one of the following: frequency information, volume information, and audio content information;

[0029] A preprocessing module, used for preprocessing the feature information, wherein the preprocessing includes noise reduction and gain adjustment;

[0030] A visualization module is used to display the characteristic information and a control for adjusting the parameters of the characteristic information through a visualization interface, and to transmit the parameter information obtained by the control to the audio source through the target bidirectional data link.

[0031] An electronic device includes: a processor and a memory storing a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0032] A non-transitory machine-readable medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the above method.

[0033] The sound source visualization control method and related devices provided by the embodiments of the present invention select a target two-way data link from a plurality of two-way data links, and obtain audio data from an audio source through the target two-way data link; perform audio rendering and processing on the audio data to obtain feature information of the audio data, wherein the feature information includes at least one of the following: frequency information, volume information, and audio content information; perform preprocessing on the feature information, wherein the preprocessing includes noise reduction and gain adjustment; display the feature information and a control for adjusting parameters of the feature information through a visualization interface, and transmit the parameter information obtained by the control to the audio source through the target two-way data link, thereby solving the problem of poor compatibility of sound source control schemes in related technologies and improving the compatibility and performance of the sound source control schemes. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0035] Figure 1 4 is a flow chart of the sound source visualization control method of this embodiment.

[0036] Figure 2 It is a structural block diagram of the sound source visualization control device of this embodiment.

[0037] Figure 3 Schematic diagram of the use scenario of the sound source visualization control device of this embodiment.

[0038] Figure 4 Schematic diagram of the structure of the electronic device of this embodiment. DETAILED DESCRIPTION

[0039] The embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0040] Visual control of sound sources refers to allowing users to adjust and control the characteristics of sound through a graphical interface. Visual control of sound sources is widely used in audio editing software, music production software, live sound systems, and some professional acoustic research. Through visual control, users can intuitively see the sound waveform or spectrum and make precise adjustments.

[0041] In the sound source control solution using a web browser, the audio source transmits audio data to the web browser through a data link. The browser displays the audio characteristics after streaming, audio rendering and visualization, and controls the audio playback and adjusts the audio characteristics through a visual interface. These audio characteristics include but are not limited to: volume, equalizer, dynamic range, audio effects (reverberation, delay, etc.), spectrum, waveform, etc.

[0042] The related art uses a data link implemented by a fixed transmission scheme to realize information transmission between the audio source and the web browser. Since different transmission schemes have different performances, the visual control scheme of the sound source in the related art has the problem of not being compatible with complex scenarios and not taking performance into account, resulting in large transmission delays, long connection establishment time, and susceptibility to network fluctuations.

[0043] To this end, this embodiment provides a sound source visualization control method applied to a browser client. Figure 1 is a flow chart of the sound source visualization control method of this embodiment. Figure 1 As shown, the process includes the following steps:

[0044] Step S101 , selecting a target bidirectional data link from a plurality of bidirectional data links, and acquiring audio data from an audio source through the target bidirectional data link.

[0045] The above-mentioned audio data may be, for example, data obtained after the audio device in the audio source module performs sampling, quantization, encoding and other processes on the collected sound signal.

[0046] When implementing a Web-side sound source visualization control solution, selecting a suitable data link is the key to ensuring efficient transmission of audio data. Different from related technologies, in this embodiment, multiple bidirectional data links (data links implemented by corresponding transmission schemes) are built in. In some of these embodiments, according to the communication capability of the audio source and / or the audio transmission scenario, the target bidirectional data link with the most suitable performance is selected from multiple bidirectional data links to improve transmission performance. The multiple bidirectional transmission links built in this embodiment include, but are not limited to, transmission links implemented using any of the following technologies: WebSocket, WebTransport, WebRTC, HTTP polling.

[0047] WebSocket provides a full-duplex communication channel, where the server and client can send and receive data at the same time; compared to traditional HTTP requests, the WebSocket handshake process only needs to be done once, and there is no HTTP header overhead for subsequent data transmission, so the latency is lower than that of traditional HTTP. WebSocket has the feature of maintaining a connection. Once a connection is established, it will remain open unless it is manually closed or an error occurs. WebSocket is very suitable for scenarios that require real-time communication, and its API is simple and easy to maintain. The main disadvantage of WebSocket is head-of-line blocking: since it is based on the TCP protocol, if a data packet is lost or delayed, subsequent data packets will be blocked until the lost data packet is successfully retransmitted.

[0048] WebTransport supports multiplexing, and can transmit multiple data streams simultaneously in one connection, reducing the overhead of establishing connections. Based on the QUIC protocol, it has the characteristics of fast connection establishment, low latency, and no head-of-line blocking. It uses TLS encryption by default and can provide a secure communication channel. The main disadvantages of WebTransport are that there are currently few browsers that support WebTransport, and its configuration and implementation are more complicated than those of WebSocket and WebTransport.

[0049] WebRTC supports peer-to-peer (P2P) communication, which can transmit data directly between browsers without the need for an intermediate server. It has low latency and is suitable for real-time communication. It supports the transmission of audio, video, and any data, and uses DTLS and SRTP encryption by default, similar to WebTransport, to provide a secure communication channel. The main disadvantages of WebRTC are: P2P connections require a complex negotiation process, and the connection establishment time is long; P2P communication is unstable in scenarios with complex network conditions, and consumes more network bandwidth and computing resources.

[0050] HTTP polling obtains data by sending HTTP requests to the server periodically, implementing a simple polling mechanism that is compatible with almost all browsers and can be used as a backup solution for other real-time communication technologies to ensure that it can still work properly in environments that do not support WebSocket or WebRTC. The disadvantage of HTTP polling is that it has high latency, high bandwidth consumption, and high resource usage compared to other solutions.

[0051] In this embodiment, according to the communication capability of the audio source and / or the audio transmission scenario, the most suitable performance is selected from a plurality of bidirectional data links as the target bidirectional data link, thereby taking into account the communication capability of the audio source and the audio transmission scenario:

[0052] (1) When the audio source supports only one bidirectional data link, the audio source shall prevail, that is, the bidirectional data link supported by the audio source shall be determined as the target bidirectional data link.

[0053] (2) When the audio source supports multiple modes, the audio transmission scenario shall prevail, and external specification and internal adaptation are supported.

[0054] External designation refers to the external setting mode. If a specified two-way data link exists, the specified two-way data link is determined as the target two-way data link. The priority of the externally specified two-way data link is higher than the internal adaptive scheme. For example, in the P2P scenario, you can specify the use of WebRTC; if both the audio source and the browser support WebTransport, specify the use of WebTransport.

[0055] Internal adaptation includes but is not limited to: if there is no designated two-way data link, a two-way data link is selected as the target two-way data link based on the two-way data links supported by the browser client and the audio source to achieve adaptive selection of the communication mode. The basis for the selection includes but is not limited to: the characteristic detection result of the browser client and / or the two-way data link support matrix corresponding to the browser client.

[0056] In this embodiment, a feature detection result is obtained by performing feature detection on the browser client, and the feature detection result represents the type of bidirectional data link supported by the browser client in the current environment.

[0057] When the feature detection is running, it will check whether the browser client supports specific functions. The window object is the global object of the browser, representing the browser window. It is the top-level object in the browser environment and contains all the global properties and methods of the browser. In this embodiment, it can be determined whether the browser client supports WebSockets, WebRTC, and WebTransport by detecting whether WebSocket, RTCPeerConnection, and WebTransport exist in the window object of the browser.

[0058] The above two-way data link support matrix is ​​used to pre-record the support of different browsers for various communication methods, classify them according to the browser type, version and supported features, and record the results in the data structure for adaptive selection. When searching the two-way data link support matrix, first detect the browser type and version number of the browser client, and then search the two-way data link support matrix for the communication method supported by the browser client according to the browser type and version number. The type and version of the browser client can be realized by parsing the navigator.userAgent string. navigator.userAgent contains detailed information about the browser, including the browser name, version number, operating system, etc.

[0059] In the case where there are two or more bidirectional data links supported by both the browser client and the audio source, in order to adapt to changes in audio transmission scenarios, an elegant degradation strategy can also be set in some of the embodiments, that is, another one or more bidirectional data links are specified as alternatives for the currently selected target bidirectional data link to ensure the stability of communication. The elegant degradation strategy can also be implemented through a degradation matrix, which is used to record the priority of multiple bidirectional data link selections in different audio transmission scenarios. The lower priority ones are used as alternatives to the higher priority bidirectional data links to achieve elegant degradation.

[0060] Step S102: Render and process the audio data to obtain feature information of the audio data, where the feature information includes at least one of the following: frequency information, volume information, and audio content information.

[0061] The function of audio rendering is to convert audio data into electrical signals that can be played by audio output devices such as speakers or headphones. After audio rendering, features of the electrical signals can be extracted to obtain feature information of the audio data, such as frequency information and volume information. In some embodiments, during audio rendering, content recognition can also be performed on the audio to obtain audio content information, such as identifying audio content information to obtain audio text content, so that these texts can be displayed during visualization.

[0062] In some of the embodiments, a plug-in-free solution is used in the browser client to realize the visual control of the sound source. The key to realizing the plug-in-free Web-side sound source visual control solution is to make full use of modern Web technologies, especially HTML5, Web Audio API, WebAssembly and other technologies. These technologies provide powerful audio processing capabilities and rich functions without relying on any additional plug-ins.

[0063] Among them, the audio tag of HTML5 is used to embed and play audio files, supports multiple audio formats (such as MP3, WAV, OGG), and provides basic playback control functions. The Web Audio API provides more advanced audio processing functions, such as real-time audio processing, audio effects (equalizer, reverberation, etc.), spectrum analysis, etc. Through the Web Audio API, you can create complex audio processing links and implement advanced audio processing.

[0064] WebAssembly (WASM for short) is a binary format that can run in browsers, providing performance close to native code. By compiling audio processing logic into WebAssembly, complex audio processing tasks can be performed efficiently in the browser. The audio processing logic is written as C / C++ audio processing libraries, which are compiled into WebAssembly to use these high-performance audio processing functions in the browser.

[0065] In this embodiment, during the audio rendering and processing of audio data, since different audio formats have different encoding and decoding methods, the format of the audio data is first identified before audio rendering. The format of the audio data can be identified by identifying the audio file extension, checking the file header, or identifying through an existing library.

[0066] The audio data format is identified by the audio file extension, for example: ".mp3" represents the format of MPEG-1 or MPEG-2 Audio Layer III; ".wav" represents the format of Waveform Audio File Format; ".aac" represents the format of Advanced Audio Coding; ".ogg" represents the format of Ogg Vorbis; ".flac" represents the format of FreeLossless Audio Codec; ".m4a" represents the format of MPEG-4 Part 14.

[0067] Identify the audio data format by checking the file header. Many audio formats contain specific identifiers (called "Magic Numbers") in their file headers. You can identify the audio format by reading the first few bytes of the file header. For example: WAV: The first 4 bytes of the file header are usually 52 49 46 46 (RIFF), and the 8th byte is usually 57 41 56 45 (WAVE). MP3: The file header usually starts with FF FB or FF F3. AAC: The file header usually starts with FF F1 or FF F9. OGG: The first 4 bytes of the file header are usually 4F 67 67 53 (OggS). FLAC: The first 4 bytes of the file header are usually 66 4C 61 43 (fLaC). M4A: The file header usually starts with 66 74 79 70 (ftyp), followed by some specific bytes.

[0068] Libraries that can be used to identify audio data formats, such as: FileType: A lightweight JavaScript library that can identify file types. ffprobe: A tool from FFmpeg that can be used to obtain detailed information about media files. MusicMetadata: A Node.js library that can parse metadata for audio files.

[0069] After identifying the format of the audio data, when the format of the audio data is a format supported by HTML5 (for example: MP3, WAV, OGG, etc.), the audio decoding and rendering library built into the browser client is called through the audio tag of HTML5 to perform audio rendering of the audio data, and there is no need to load the target audio decoding and rendering library at this time. When the format of the audio data is a format not supported by HTML5 (for example: G711A, G726, etc.), the target audio decoding and rendering library compiled from C++ and other languages ​​into wasm language is loaded to perform audio rendering of the audio data. The above method can achieve plug-in-free while making the application more lightweight.

[0070] In the above steps, after the audio data is rendered, other processing may be performed, including but not limited to extraction of frequency signals, acquisition of real-time volume, recording and export of recording files, and audio content analysis.

[0071] Frequency signal extraction refers to decomposing the time domain signal into the superposition of different frequency components to facilitate subsequent frequency domain analysis and processing. The received audio data is in audio frames. Each frame of data is processed by FFT to obtain frequency domain information. The spectrum information obtained at this time is stored in the ArrayBuffer type for subsequent preprocessing.

[0072] The volume is obtained by extracting the sound size in the audio data through the decibel extraction algorithm. It supports RMS, peak detection, and FFT. The specific value is subject to external settings. The peak detection algorithm can be used by default.

[0073] Audio recording refers to the storage and preservation of audio data. When the audio storage format is specified externally, the audio data is transpacked into the specified format. The storage can be set to exist locally and call back data to the application layer, which is used to save the data to the service or other storage media through the protocol or other means.

[0074] Audio content analysis has expanded capabilities such as audio-to-text conversion by introducing intelligent capabilities.

[0075] In some of the embodiments, the target audio decoding and rendering library is compiled based on SIMD instructions supported by different CPU architectures; when performing multi-threaded parallel audio rendering of audio data, SharedArrayBuffer shared memory technology is used to achieve data exchange and parallel processing between multiple threads.

[0076] When the audio rendering library is compiled to generate a WebAssembly library, SIMD instruction set optimization is introduced to achieve parallel processing at the data level. By selecting common SIMD instructions supported by CPUs of various manufacturers and abstracting and unifying the SIMD instructions, the technical difficulty of SIMD of the underlying CPU is avoided.

[0077] Among them, SIMD is a special CPU instruction. When adding two arrays of length 4, ordinary instructions need to execute 4 addition instructions, while SIMD instructions only need to execute 1 vector addition. This achieves processing of multiple data through one operation, thereby improving code execution efficiency. In this embodiment, SharedArrayBuffer shared memory technology is also introduced to create a view on the shared memory to realize shared memory areas in multiple web worker threads. When accessing data at the same time, there is no need to copy data or consider data synchronization issues, thereby realizing parallel execution of computing tasks, achieving the purpose of reducing the main thread blocking while giving full play to the advantages of multi-threading.

[0078] In some of the embodiments, the browser client includes multiple audio decoding and rendering libraries compiled into WebAssembly; loading the target audio decoding and rendering library compiled into WebAssembly to perform audio rendering of audio data includes: selecting an audio decoding and rendering library from multiple audio decoding and rendering libraries compiled into WebAssembly as the target audio decoding and rendering library based on the feature detection result of the browser client and / or the SIMD instructions and SharedArrayBuffer support matrix information corresponding to the browser client.

[0079] Adaptive loading of multi-modal libraries is implemented by detecting browser features and creating a support matrix, which includes the following three processing steps:

[0080] Step 1: Determine the support of SharedArrayBuffer and SIMD in the current browser environment through feature detection. For example, the support of SharedArrayBuffer can be directly determined by whether there is SharedArrayBuffer on the window object in the browser. The support of SIMD can be determined by the WebAssembly.validate() method to verify whether a typed array containing WebAssembly binary code is legal.

[0081] Step 2: Create a "SIMD instruction and SharedArrayBuffer support matrix" to record the support of SharedArrayBuffer and SIMD in different browsers, classify them according to browser type, version and supported features, and record the results in a data structure to facilitate adaptive selection.

[0082] Step 3: Adaptively load the library based on the feature detection results and the support matrix, and select the best library mode within the optional range. For example, the library mode can be: SIMD support + SharedArrayBuffer support, SIMD support + SharedArrayBuffer not support, SIMD not support + SharedArrayBuffer support, SIMD not support + SharedArrayBuffer not support.

[0083] The libraries of these four modes are generated during WebAssembly compilation, dynamically adapt to match the browser operating environment at runtime, enable instruction set optimization and multi-threading in scenarios that support SIMD and SharedArrayBuffer, and significantly improve performance.

[0084] Step S103: preprocess the feature information, where the preprocessing includes noise reduction and gain adjustment.

[0085] In the above steps, after obtaining the frequency information, volume information and audio content information output by the audio rendering and processing module, the audio can be denoised and the gain adjusted during preprocessing. The specific elimination rules and parameters are based on the parameter information set in the audio visualization interface. According to the parameter information set in the visualization interface, the spectrum information is filtered in high-pass, band-pass, low-pass and other modes to retain the target audio and filter out interference signals; and the audio gain is adjusted according to the gain parameters; the volume preprocessing selects mean filtering or mean filtering according to the external setting parameters, directly performs time domain filtering, removes part of the noise, and preprocesses the audio content; the results of the audio content analysis are preprocessed, and special processing can be performed on sensitive content information, etc.

[0086] Since most audio processing libraries and APIs use floating point numbers to represent spectrum data, in order to simplify the integration with these audio processing libraries and APIs, the spectrum data stored in ArrayBuffer in the feature information extraction phase can be converted to floating point numbers in the preprocessing step. The conversion method can be, for example, to divide the frequency information of the audio data into 4 bytes, and convert each 4 bytes obtained by the division into floating point numbers, so as to obtain a floating point array corresponding to the frequency information, so that the visualization interface can draw and display the spectrum image corresponding to the spectrum information based on the floating point array. Among them, every 4 bytes are used to represent a 32-bit floating point number. For example: the length of the extracted spectrum signal is 16388, and the ArrayBuffer needs to be converted into a floating point number at this time. The conversion rule is to divide it into 4 bits and convert it into a floating point number. The final output includes an array of 4097 floating point numbers. The floating point array can be used to directly draw the spectrum image corresponding to the spectrum data in the visualization phase.

[0087] Step S104: displaying the characteristic information and a control for adjusting the parameters of the characteristic information through a visual interface, and transmitting the parameter information obtained by the control to the audio source through a target bidirectional data link.

[0088] In this embodiment, the visualization interface is used to display spectrum information in real time and display controls corresponding to adjustable parameters, such as controls for adjusting the following parameters: minimum and maximum spectrum ranges, high-pass, low-pass, band-pass parameters, gain parameters, volume, filter mode, etc.

[0089] The spectrum information can be drawn on the canvas. The horizontal axis shows the spectrum range and the vertical axis shows the spectrum density. When drawing the horizontal axis, the drawing density is dynamically adjusted according to the canvas size. For example, if the spectrum data is 4097 data and the display range is 820 pixels, the maximum value or average value can be obtained for every 5 data, and then the canvas can be filled through the outer bounding box mode. The same goes for other cases. Each time a data push is received, the canvas is redrawn to achieve real-time data update. After the data is merged, the amount of data drawing can be greatly reduced and the rendering performance can be improved when the canvas is small.

[0090] The control for adjusting the parameters of the characteristic information includes: audio source parameters for adjusting the characteristic information of the audio source, and terminal side parameters for adjusting the characteristic information of the browser client; wherein the audio source parameters include at least one of the following: sampling bit depth, sampling rate, audio format. The terminal side parameters include at least one of the following: noise reduction value, gain parameter, dynamic range compression rate, spectrum range, filter mode, volume, equalizer setting, channel balance.

[0091] When external parameters are adjusted on the controls of the visualization interface, the new parameter information will be fed back to the preprocessing module used to execute step S103. The preprocessing module processes the real-time data according to the new parameters, and then transfers the processing results to the visualization module, and then updates the canvas in real time, thereby realizing the end-side adjustment process.

[0092] While adjusting the external parameters on the visual interface, the new parameter information will be fed back to the audio source through the target bidirectional data link. The audio source receives the new parameter information, updates the audio source settings, and adjusts the audio source according to the new audio source settings. In some embodiments, the parameters that can be adjusted by the audio source may include parameters that can be adjusted by the preprocessing module (i.e., the terminal side parameters mentioned above), and also include parameters that cannot be adjusted by the preprocessing module (i.e., the audio source parameters mentioned above), such as sampling bit depth, sampling rate, audio format, and other parameters. The audio source collects, quantizes, and preprocesses the audio source data according to the newly set parameters, and outputs the collected and processed data to the browser client for processing.

[0093] This embodiment also provides a sound source visualization control device. Figure 2 is a structural block diagram of the sound source visualization control device of this embodiment, such as Figure 2 As shown, the device comprises:

[0094] The stream acquisition module 21 is used to select a target bidirectional data link from a plurality of bidirectional data links, and acquire audio data from an audio source through the target bidirectional data link.

[0095] The rendering module 22 is used to perform audio rendering and processing on the audio data to obtain feature information of the audio data, where the feature information includes at least one of the following: frequency information, volume information, and audio content information.

[0096] The preprocessing module 23 is used to preprocess the feature information, and the preprocessing includes noise reduction and gain adjustment.

[0097] The visualization module 24 is used to display the characteristic information and the controls for adjusting the parameters of the characteristic information through a visualization interface, and transmit the parameter information obtained by the controls to the audio source through a target bidirectional data link.

[0098] Figure 3 is a schematic diagram of a use scenario of the sound source visualization control device of this embodiment, Figure 3 It can be seen that in some embodiments, the stream acquisition module 21 includes a stream acquisition submodule, a playback control submodule and a feedback adjustment submodule. The rendering module 23 includes an audio rendering submodule, a frequency extraction submodule, a volume extraction submodule, a recording submodule and an audio content analysis submodule. The preprocessing module 23 includes a filtering submodule, a volume preprocessing submodule and an audio content preprocessing submodule.

[0099] In some of the embodiments, the stream acquisition module 21 is used to select a target bidirectional data link from a plurality of bidirectional data links according to the communication capability of the audio source and / or the audio transmission scenario, wherein the plurality of bidirectional data links include at least one of the following: WebSocket, WebTransport, WebRTC, HTTP polling.

[0100] In some of the embodiments, the stream acquisition module 21 is used to determine that the two-way data link supported by the audio source is the target two-way data link when the audio source only supports one two-way data link; when the audio source supports multiple two-way data links, if there is a designated two-way data link, the designated two-way data link is determined as the target two-way data link; if there is no designated two-way data link, one two-way data link is selected as the target two-way data link based on the two-way data links supported by both the browser client and the audio source, and another two-way data link is selected as a backup degradation scheme for the target two-way data link; wherein the two-way data link supported by the browser client is determined based on the characteristic detection result of the browser client and / or the two-way data link support matrix corresponding to the browser client.

[0101] In some of the embodiments, the rendering module 22 is used to call the Web Audio API through the audio tag of HTML5 to perform audio rendering of the audio data when the format of the audio data is a format supported by HTML5; when the format of the audio data is a format not supported by HTML5, load the target audio decoding and rendering library compiled into WebAssembly to perform audio rendering of the audio data.

[0102] In some of the embodiments, the target audio decoding and rendering library is compiled based on SIMD instructions supported by different CPU architectures; when performing multi-threaded parallel audio rendering of audio data, SharedArrayBuffer shared memory technology is used to achieve data exchange and parallel processing between multiple threads.

[0103] In some of the embodiments, the browser client includes multiple audio decoding and rendering libraries compiled into WebAssembly; the rendering module 22 is used to select an audio decoding and rendering library from multiple audio decoding and rendering libraries compiled into WebAssembly as a target audio decoding and rendering library based on the feature detection results of the browser client and / or the SIMD instructions and SharedArrayBuffer support matrix information corresponding to the browser client.

[0104] In some of the embodiments, controls for adjusting parameters of characteristic information include: audio source parameters for adjusting characteristic information of an audio source, and terminal side parameters for adjusting characteristic information of a browser client; wherein the audio source parameters include at least one of the following: sampling bit depth, sampling rate, audio format; the terminal side parameters include at least one of the following: noise reduction value, gain parameter, dynamic range compression rate, spectrum range, filtering mode, volume, equalizer setting, channel balance.

[0105] In some of the embodiments, preprocessing the feature information includes: dividing the frequency information of the audio data into 4 bytes, converting each of the 4 bytes obtained by the division into floating-point numbers, and obtaining a floating-point number array corresponding to the frequency information, so that a visualization interface can draw and display a spectrum image corresponding to the frequency information based on the floating-point number array.

[0106] The embodiments of the present invention also provide a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiments of the present invention.

[0107] The embodiment of the invention also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the invention when executed by the at least one processor.

[0108] refer to Figure 4 , a block diagram of an electronic device that can be used as a server or client of an embodiment of the invention will now be described, which is an example of hardware devices that can be applied to various aspects of the invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or required herein.

[0109] like Figure 4 As shown, the electronic device includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 to a random access memory (RAM) 403. In RAM 403, various programs and data required for the operation of the electronic device can also be stored. The computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0110] Multiple components in the electronic device are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 can be any type of device that can input information to the electronic device, and the input unit 406 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 407 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 can include but is not limited to a disk, an optical disk. The communication unit 409 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, and / or a wireless communication transceiver, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0111] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the method embodiments created by the present invention may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).

[0112] The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the computer program, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0113] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0114] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0115] The various steps described in the method implementation methods provided by the embodiments of the present invention can be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0116] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same and similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0117] The above-described embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of protection. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A sound source visualization control method, applied to a browser client, characterized in that include: selecting a target bidirectional data link from a plurality of bidirectional data links, and acquiring audio data from an audio source through the target bidirectional data link; Performing audio rendering and processing on the audio data to obtain feature information of the audio data, wherein the feature information includes at least one of the following: frequency information, volume information, and audio content information; Preprocessing the characteristic information, wherein the preprocessing includes noise reduction and gain adjustment; Displaying the characteristic information and a control for adjusting a parameter of the characteristic information through a visual interface, and transmitting the parameter information obtained by the control to the audio source through the target bidirectional data link, so that the audio source updates the audio source settings according to the parameter information, and adjusts the audio source according to the audio source settings, wherein the parameter information includes audio source parameters, and the audio source parameters include at least one of the following: sampling bit depth, sampling rate, and audio format; Among them, selecting a target bidirectional data link from multiple bidirectional data links includes: when the audio source only supports one bidirectional data link, determining the bidirectional data link supported by the audio source as the target bidirectional data link; when the audio source supports multiple bidirectional data links, if there is a designated bidirectional data link, determining the designated bidirectional data link as the target bidirectional data link, if there is no designated bidirectional data link, selecting one bidirectional data link as the target bidirectional data link based on the bidirectional data links supported by both the browser client and the audio source, and selecting another bidirectional data link as a backup degradation scheme for the target bidirectional data link after the audio transmission scenario changes; wherein the bidirectional data link supported by the browser client is determined based on a characteristic detection result of the browser client and / or a bidirectional data link support matrix corresponding to the browser client.

2. The method according to claim 1, characterized in that The multiple bidirectional data links include at least one of the following: WebSocket, WebTransport, WebRTC, and HTTP polling.

3. The method according to claim 1, characterized in that: The audio data is rendered and processed to obtain characteristic information of the audio data, including: When the format of the audio data is a format supported by HTML5, calling the Web Audio API through the audio tag of HTML5 to perform audio rendering of the audio data; When the format of the audio data is a format not supported by HTML5, a target audio decoding and rendering library compiled into WebAssembly is loaded to perform audio rendering of the audio data.

4. The method according to claim 3, characterized in that The target audio decoding and rendering library is compiled based on SIMD instructions supported by different CPU architectures; when performing multi-threaded parallel audio rendering of the audio data, SharedArrayBuffer shared memory technology is used to achieve data exchange and parallel processing between multiple threads.

5. The method according to claim 3, characterized in that: The browser client includes a plurality of audio decoding and rendering libraries compiled into WebAssembly; loading a target audio decoding and rendering library compiled into WebAssembly to perform audio rendering of the audio data includes: According to the feature detection result of the browser client and / or the SIMD instruction and SharedArrayBuffer support matrix information corresponding to the browser client, an audio decoding and rendering library is selected from multiple audio decoding and rendering libraries compiled into WebAssembly as the target audio decoding and rendering library.

6. The method according to any one of claims 1 to 5, characterized in that The controls for adjusting the parameters of the characteristic information include: the audio source parameters used by the audio source to adjust the characteristic information, and the end-side parameters used by the browser client to adjust the characteristic information; the end-side parameters include at least one of the following: noise reduction value, gain parameter, dynamic range compression rate, spectrum range, filtering mode, volume, equalizer setting, channel balance.

7. The method according to any one of claims 1 to 5, characterized in that Preprocessing the feature information includes: The frequency information of the audio data is divided into 4 bytes, and each of the 4 bytes obtained by the division is converted into a floating point number to obtain a floating point number array corresponding to the frequency information, so that the visualization interface can draw and display a spectrum image corresponding to the frequency information based on the floating point number array.

8. A sound source visualization control device, characterized in that include: A stream acquisition module, used for selecting a target bidirectional data link from a plurality of bidirectional data links, and acquiring audio data from an audio source through the target bidirectional data link; A rendering module, used to perform audio rendering and processing on the audio data to obtain feature information of the audio data, wherein the feature information includes at least one of the following: frequency information, volume information, and audio content information; A preprocessing module, used for preprocessing the feature information, wherein the preprocessing includes noise reduction and gain adjustment; a visualization module, configured to display the feature information and a control for adjusting a parameter of the feature information through a visualization interface, and to transmit the parameter information obtained by the control to the audio source through the target bidirectional data link, so that the audio source updates the audio source settings according to the parameter information, and adjusts the audio source according to the audio source settings, wherein the parameter information includes audio source parameters, and the audio source parameters include at least one of the following: sampling bit depth, sampling rate, and audio format; Among them, selecting a target bidirectional data link from multiple bidirectional data links includes: when the audio source only supports one bidirectional data link, determining the bidirectional data link supported by the audio source as the target bidirectional data link; when the audio source supports multiple bidirectional data links, if there is a designated bidirectional data link, determining the designated bidirectional data link as the target bidirectional data link, if there is no designated bidirectional data link, selecting one bidirectional data link as the target bidirectional data link based on the bidirectional data links supported by both the browser client and the audio source, and selecting another bidirectional data link as a backup degradation scheme for the target bidirectional data link after the audio transmission scenario changes; wherein the bidirectional data link supported by the browser client is determined based on a characteristic detection result of the browser client and / or a bidirectional data link support matrix corresponding to the browser client.

9. An electronic device, comprising: A processor and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory machine-readable medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images