Sound regulation and control method and system and storage medium

Abstract

The embodiment of the invention discloses a sound regulation and control method and system, the method is executed by a processor, and the method comprises the following steps: determining sound source information, the sound source information comprising the number of sound receiving devices and sound source types; based on the sound source information and the scene information, sound receiving parameters of the sound receiving equipment are adjusted, and the sound receiving parameters comprise sound receiving intensity and sound mixing proportion; wherein the sound source type is updated according to a preset period.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to a sound control method, system and storage medium. Background Art

[0002] In multimedia scenarios such as conference rooms, there are usually multiple microphones collecting sound at the same time, and each microphone corresponds to a different conference participant. In this multi-source sound collection environment, the audio signals are complex and diverse, including the voice of the main speaker, the voices of other participants, and background music or noise.

[0003] In complex audio signals, the voice of the main speaker is often masked.

[0004] Therefore, it is necessary to provide a sound control method to make the voice of the main speaker more prominent, so as to overcome the problems of inaccurate sound source recognition, single sound effect processing and background music interference. Summary of the invention

[0005] One of the embodiments of this specification provides a sound control method, which is executed by a processor. In some embodiments, the method includes: determining sound source information, the sound source information includes the number of sound receiving devices and the type of sound source; adjusting the sound receiving parameters of the sound receiving device based on the sound source information and scene information, the sound receiving parameters include the sound receiving intensity and the mixing ratio; wherein the sound source type is updated according to a preset period.

[0006] In some embodiments, determining the sound source information includes: acquiring sound data of a sound receiving device; and determining the sound source type of the sound receiving device based on the sound data.

[0007] In some embodiments, determining the sound source type of the sound receiving device based on the sound data includes: determining an evaluation result of at least one sound receiving device based on the sound data; and determining the sound source type of the at least one sound receiving device based on the evaluation result.

[0008] In some embodiments, the evaluation result of at least one sound receiving device is related to the sound source position distribution.

[0009] In some embodiments, updating the sound source type includes: determining a first data set based on sound data to be evaluated, the first data set including a plurality of first sub-data; determining a sound source change feature corresponding to the first data set based on at least one of the plurality of first sub-data; and updating the sound source type in response to the sound source change feature satisfying an update condition.

[0010] One embodiment of the present specification provides a sound control system, the system comprising: a determination module configured to determine sound source information, the sound source information including the number of sound receiving devices and the type of sound source; a control module configured to adjust the sound receiving parameters of the sound receiving device based on the sound source information and scene information, the sound receiving parameters including the sound receiving intensity and the mixing ratio. In some embodiments, the sound source type is updated according to a preset period.

[0011] One of the embodiments of the present specification provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, a sound control method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0013] Figure 1 is a schematic diagram of an application scenario of a sound control system according to some embodiments of this specification;

[0014] Figure 2 is an exemplary schematic diagram of a sound control method according to some embodiments of this specification;

[0015] Figure 3 is an exemplary schematic diagram of determining an evaluation result of at least one sound receiving device according to some embodiments of this specification;

[0016] Figure 4 is an exemplary flow chart of updating the audio source type according to some embodiments of this specification;

[0017] Figure 5 It is an exemplary schematic diagram of determining update conditions according to some embodiments of this specification. DETAILED DESCRIPTION

[0018] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0019] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0020] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0021] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0022] Figure 1 1 is a schematic diagram of an application scenario 100 of a sound control system according to some embodiments of this specification. The sound control system can realize the control and management of sound in a variety of application scenarios. For example, the application scenarios of the sound control system include but are not limited to meetings, debates, speeches, training, art performances, and other scenarios. The following describes the application scenario 100 of the sound control system using a meeting scenario as an example.

[0023] like Figure 1 As shown, the application scenario 100 of the sound control system may include a sound source 110 , a network 120 , a storage device 130 , a processor 140 , and a speaker device 150 .

[0024] The sound source 110 refers to the source of the sound. For example, the sound source may be a sound receiving device such as a microphone that collects the sound. For example, the sound source may include a microphone 110-1, a pickup 110-2, and a recording device 110-3. The number of sound sources 110 may be set according to the needs of the scene. Figure 1 The types and quantities of sound sources shown are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0025] In a conference scenario, the sound receiving device is used to collect audio signals in the conference room. The audio signal is an unprocessed sound signal. For example, each microphone in front of each speaker can be corresponded to the sound source 110, or multiple speakers can share the same microphone and use the microphone as the sound source 110. Taking the microphone as the sound source as an example, the number of microphones is the number of sound sources 110.

[0026] The network 120 may include any suitable network capable of facilitating information and / or data exchange of the application scenario 100 of the sound control system. In some embodiments, one or more components of the application scenario 100 of the sound control system (e.g., the storage device 130, the processor 140, the sound source 110, etc.) may exchange information and / or data with one or more other components of the application scenario 100 of the sound control system via the network 120.

[0027] The storage device 130 is used to store data, instructions and / or any other information. In some embodiments, the storage device can be used to store sound data and position data of the sound source 110. The storage device 130 may include one or more storage components, each of which may be an independent device or a part of other devices. In some embodiments, the storage device 130 may include a random access memory (RAM), a read-only memory (ROM), a removable memory, etc., or any combination thereof.

[0028] In some embodiments, the storage device 130 may be connected to the network 120 to implement communication with one or more components in the application scenario 100 of the sound control system. For example, the processor 140 may receive an instruction for determining an evaluation result of at least one sound receiving device stored in the storage device 130, and determine the evaluation result of at least one sound receiving device based on the sound data. In some embodiments, the storage device 130 may be a part of the processor 140.

[0029] The processor 140 is used to process information and / or data related to the application scenario 100 of the sound control system. In some embodiments, the processor 140 can process data, information and / or processing results obtained from other devices or system components, and execute program instructions based on these data, information and / or processing results to perform one or more functions described in this specification. For example, the processor 140 can adjust the sound receiving parameters of the sound receiving device based on the sound source information and the scene information.

[0030] The speaker device 150 refers to a device that converts an electrical signal into sound. For example, the speaker device may be a speaker for playing background music. In some embodiments, the processor 140 may adjust the speaker parameters of the speaker device 150 based on the sound source information and the scene information. In some embodiments, the speaker parameters may include volume, etc.

[0031] It should be noted that the application scenario is provided only for the purpose of illustration and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various modifications or changes can be made according to the description of this specification. For example, the application scenario may also include a database. For another example, the application scenario may be implemented on other devices to achieve similar or different functions. However, changes and modifications will not deviate from the scope of this specification.

[0032] In some embodiments, the sound control system may include a determination module and a control module.

[0033] In some embodiments, the determination module is configured to determine the number of sound receiving devices and the types of sound sources.

[0034] In some embodiments, the determination module may also be configured to obtain sound data of the sound receiving device; and determine the sound source type of the sound receiving device based on the sound data.

[0035] In some embodiments, the control module can be configured to adjust the sound receiving parameters of the sound receiving device based on the sound source information and the scene information. In some embodiments, the sound receiving parameters include sound receiving intensity and mixing ratio. In some embodiments, the sound source type is updated according to a preset period.

[0036] In some embodiments, the control module can be further configured to determine a first data set based on the sound data to be evaluated, the first data set including multiple first sub-data; determine a sound source change feature corresponding to the first data set based on at least one of the multiple first sub-data; and update the sound source type in response to the sound source change feature satisfying an update condition.

[0037] In some embodiments, all or part of the modules of the sound control system may be integrated into the processor 140 to perform one or more functions described in this specification.

[0038] It should be noted that the above description of the sound control system and its modules is only for the convenience of description, and this specification cannot be limited to the scope of the embodiments. It can be understood that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the various modules, or form a subsystem to connect with other modules without deviating from this principle. In some embodiments, the determination module and the control module can be different modules in a system, or a module can implement the functions of two or more of the above-mentioned modules. For example, each module can share a storage module, and each module can also have its own storage module. In some embodiments, the determination module can also be divided into multiple sub-modules. For example, the determination module can also include a judgment module. Such variations are all within the scope of protection of this specification.

[0039] Figure 2 FIG. 2 is an exemplary schematic diagram 200 of a sound control method according to some embodiments of the present specification. In some embodiments, the sound control method may be executed by the processor 140.

[0040] In some embodiments, Figure 2 As shown, the sound control method may include: determining sound source information 250; and adjusting the sound receiving parameters 270 of the sound receiving device based on the sound source information 250 and the scene information 260.

[0041] The sound source information 250 is information used to indicate the specific situation of the sound source (ie, the sound receiving device). For example, the sound source information may include the specific model and service life of the sound receiving device.

[0042] In some embodiments, the processor 140 may determine the sound source information 250 based on the sound source 110 .

[0043] In some embodiments, the sound source information 250 includes the number of sound receiving devices 230 and the sound source type 240 .

[0044] The number 230 of the sound receiving devices may be pre-stored in the storage device 130 , or may be obtained by the processor 140 through statistics when the sound receiving devices are turned on.

[0045] The sound source type refers to the category to which the sound receiving device belongs. The processor 140 can classify the sound receiving device into different sound source types based on different classification methods. For example, based on the importance, the sound source type of the sound receiving device can include a main sound source and a slave sound source. That is, the sound source type based on the sound receiving device can reflect the importance of the sound receiving device as a sound source. For another example, based on the audio content, the sound source type can include a background sound source for collecting background music and a front-end sound source for collecting the host's voice.

[0046] As an example, based on the importance, the sound source type of the sound receiving device corresponding to the main speaker can be used as the main sound source, and the secondary sound source can be the sound source from the secondary speaker. The main speaker can be a participant or event host who is currently speaking, and the secondary speaker can be a participant or event guest who is not currently speaking.

[0047] In some embodiments, the processor 140 may also use the sound receiving device of a person with a more important identity as the main sound source, and use the sound receiving device with more noise or a smaller sound as the secondary sound source.

[0048] In some embodiments, the processor 140 can also classify the sound source type of the sound receiving device from other perspectives, such as based on factors such as the speaker's gender and age. The above embodiments are intended to be illustrative, and this specification does not limit the classification method of the sound source type.

[0049] In some embodiments, the sound source type can be represented in a variety of ways, for example, the sound source type can be represented by a Boolean value. For example, when the sound source type includes a master sound source and a slave sound source, 1 represents the master sound source and 0 represents the slave sound source. The above-mentioned representation of the sound source type is only for example description, and this specification does not limit the representation of the sound source type.

[0050] In some embodiments, the processor 140 may determine the audio source type based on a variety of methods. In some embodiments, the processor 140 may determine the audio source type based on the record of the last time the audio source was enabled. For example, in the last meeting, audio source 1 was the main audio source and audio source 2 was the secondary audio source. The current meeting may continue to use the audio source type of the last time.

[0051] In some embodiments, the processor 140 may determine the type of the audio source according to the activation time of the audio source, for example, determining the first activated audio source as the main audio source.

[0052] In some embodiments, the audio source type may change during the conference. For example, when the last speaker finishes speaking, the main audio source is changed from the audio device corresponding to the last speaker to the audio device corresponding to the next speaker. For more information on determining the change of audio source type, see Figure 3 Description.

[0053] In some embodiments, the processor 140 may also determine the sound source information based on the following method: the processor 140 may obtain sound data of the sound receiving device; and determine the sound source type of the sound receiving device based on the sound data.

[0054] Sound data refers to data that can reflect the characteristics of the sound source. For example, sound data may include information such as the amplitude, frequency, phase, and volume of an audio signal at different time points.

[0055] In some embodiments, the processor 140 can directly obtain sound data based on audio processing software. For example, the processor 140 can perform Fourier transform on the audio signal through software such as MATLAB and Python to obtain sound data such as amplitude, frequency, and phase of the audio signal. For another example, the processor 140 can directly obtain sound data such as volume from a microphone or other audio input device based on the recording function of audio processing software (such as Audacity, Adobe Audition, etc.).

[0056] In some embodiments, the sound data may also be acquired by the processor 140 based on analyzing and processing the audio signal received by the sound receiving device.

[0057] For example, the sound receiving device can receive the audio signal of the conference speaker and convert the audio signal into an electrical signal, and then convert it into digital sound data through an analog-to-digital converter (ADC). The processor 140 can directly obtain the digital sound data from the sound receiving device.

[0058] For another example, the processor 140 may use an audio processing algorithm and / or software to process the collected audio signal to obtain corresponding sound data, such as processing the audio signal by reverberation, equalization, or compression to obtain sound data corresponding to the audio signal.

[0059] The above description of obtaining sound data is only an example and does not represent all possible methods or limitations. In different application scenarios, the processor 140 may use different hardware devices, signal processing technologies, and data transmission methods to obtain sound data.

[0060] The processor 140 may determine the type of sound source based on the sound data in a variety of ways.

[0061] In some embodiments, the processor 140 may use the sound receiving device with the highest sound intensity and the most concentrated amplitude in the spectrum in the sound data as the main sound source, and the other sound receiving devices as the slave sound sources.

[0062] The method for determining the sound source type mentioned in the above embodiment is only described as an example for the purpose of explanation. Other determination methods may also be used to determine the sound source type, and this specification does not limit this.

[0063] The embodiments of this specification determine the sound source type of the sound receiving device based on sound data, and can intelligently analyze and identify the activity and usage of different sound receiving devices; by distinguishing between the main sound source and the slave sound source, it is convenient to further improve the sound clarity and user experience of the main sound source, making the sound control method more efficient and intelligent in various application scenarios.

[0064] In some embodiments, Figure 2 As shown, the processor 140 may also determine an evaluation result 220 of at least one sound receiving device based on the sound data 210 ; and determine a sound source type 240 of at least one sound receiving device based on the evaluation result 220 .

[0065] The evaluation result of the sound receiving device refers to the evaluation conclusion that can reflect the sound receiving device corresponding to one or more evaluation indicators. For example, the evaluation indicators may include speech clarity index or speaker activity. Among them, the speech clarity index can be used to measure the clarity of the audio signal collected by the sound receiving device, and the speaker activity can be used to measure the activity of the sound receiving device in the meeting.

[0066] In some embodiments, the evaluation index may also include the importance of the sound source, which may reflect the importance of the sound receiving device in the meeting.

[0067] In some embodiments, the evaluation result may be represented in a variety of ways, such as numbers, percentages, or levels, etc. For example, when the evaluation result is represented by a score, the higher the score, the greater the importance of the sound source, that is, the higher the importance of the sound receiving device.

[0068] In some embodiments, the processor 140 can implement the evaluation result of determining at least one sound receiving device based on the sound data in a variety of ways. Taking the evaluation result as the importance of the sound source as an example, the processor 140 can determine the importance of the sound source of the sound receiving device based on the average strength and spectral components of the audio signal collected by the sound receiving device, based on the positive correlation between the average strength and spectral components of the audio signal collected by the sound receiving device and the importance of the sound source. As an example only, the processor 140 can determine the importance of the sound source of the sound receiving device based on the following formula.

[0069] Sound source importance = w1*average sound intensity of the audio signal collected by the current sound receiving device / reference sound intensity + w2*spectral component of the audio signal collected by the current sound receiving device.

[0070] Among them, w1 and w2 are preset weights. The initial weight values ​​can be set based on prior knowledge. For example, w1=0.5 and w2=0.5 can be set as the initial weight values. The reference sound intensity is the average sound intensity of the audio signals collected by all the sound receiving devices, and the spectral component of the audio signal collected by the current sound receiving device is the proportion of the duration of the audio signal in the preset spectrum band range in the spectrum corresponding to the audio signal collected by the current sound receiving device. For example, when the preset spectrum band range is the human voice spectrum band range (500Hz-4kHz), the spectral component of the audio signal collected by the current sound receiving device, that is, the duration corresponding to the human voice, accounts for the duration of the entire sound data collected by the current sound receiving device.

[0071] In some embodiments, the processor 140 may determine the sound source type of the sound receiving device in a variety of ways based on the evaluation result. For example, the processor 140 may use the sound receiving device whose sound source importance is greater than the importance threshold as the main sound source.

[0072] The importance threshold may be determined in a variety of ways, for example, it may be preset based on manual experience.

[0073] In some embodiments, the processor 140 may also determine the importance threshold according to the lower n quantiles of the importance of the sound source corresponding to the main sound source in the historical sound data. Wherein, n is the number of samples into which the historical sound data is divided, such as the historical sound data may be divided into n parts to obtain n historical samples. In some embodiments, n is related to the type of meeting. If it is a meeting with multiple speakers, n may be set to be larger, and if it is a meeting with a single speaker, n may be set to be smaller. Wherein, the historical sound data are all collected for the main sound source.

[0074] In some embodiments, the processor 140 may divide the historical sound data into n parts by random allocation or the like to obtain n historical samples. In some embodiments, the processor 140 may analyze and process each historical sample and obtain the sound source importance corresponding to the historical sample. In some embodiments, if a historical sample includes multiple sound receiving devices, the processor 140 may use the average or weighted value of the sound source importance of the multiple sound receiving devices as the sound source importance of the historical sample.

[0075] The lower n quantiles are the sound source importance corresponding to the 1 / nth historical sample when the sound source importance of n historical samples is arranged in ascending order of value.

[0076] For example, there are 4 historical samples, and their corresponding historical sound source importances are 90, 85, 80, and 75, n=4, then the lower quartile is 80, and the importance threshold is 80.

[0077] The embodiments of this specification determine the evaluation result of at least one sound receiving device based on sound data; based on the evaluation result, determine the sound source type of at least one sound receiving device. For example, the importance of the sound source of the sound receiving device is first evaluated, and then the sound source type of the sound receiving device is determined based on the importance, so that the determined master-slave relationship of the sound source is accurate and reliable, thereby optimizing the audio processing in the meeting and adjusting the parameters of the sound receiving device in real time, thereby improving the overall meeting quality and user experience.

[0078] In some embodiments, the evaluation result of at least one sound receiving device is also related to the sound source position distribution.

[0079] The sound source position distribution can characterize the position distribution of each sound receiving device. The processor 140 can represent the sound source position distribution in a variety of ways. For example, the sound source position distribution can be represented by a data set, sequence or matrix composed of the position coordinates of each sound receiving device, and the sound source position distribution can describe the specific position of each sound receiving device in three-dimensional space.

[0080] The processor 140 may determine the sound source position distribution in a variety of ways. For example, the sound source position distribution may be determined by the installation position of the sound receiving device. For example, the processor 140 may obtain the installation position of the sound receiving device based on GPS positioning or sensor data to determine the sound source position distribution.

[0081] In some embodiments, the processor 140 may determine the importance of the sound source of at least one sound receiving device based on the sound data and the sound source location distribution of at least one sound receiving device. For example, the sound receiving devices may be grouped based on the sound source location distribution and the sound data, and the importance of the sound source of at least one sound receiving device may be determined by a preset algorithm.

[0082] The processor 140 may group the multiple sound receiving devices in various ways.

[0083] In some embodiments, the processor 140 may use the sound source position coordinates corresponding to the sound receiving device as a clustering vector, cluster the clustering vector to form a preset number of cluster centers, and group the sound receiving devices belonging to the same cluster center.

[0084] In some embodiments, the processor 140 may determine a sound receiving device at an important position as the center of a group, and then classify the sound receiving devices whose distances from the center of the group are within a threshold into the group.

[0085] The sound receiving device at the important position can be manually designated in advance. For example, the sound receiving device at the middle position can be determined as the sound receiving device at the important position.

[0086] For each group of sound receiving devices, if the similarity of the sound data collected by any two sound receiving devices in the group is greater than the similarity threshold, the sound source importance corresponding to the sound receiving device with the largest sound data volume collected is determined as the maximum sound source importance, and the other sound source importances are determined as preset sound source importances, such as 0. Otherwise, the sound source importance of each sound receiving device in the group is calculated, and the calculation formula of the sound source importance can be found in the previous text.

[0087] The similarity of the sound data can be determined in a variety of ways. For example, the processor 140 can determine the similarity of the two sets of sound data by using a sound cross-correlation function. For more information about sound similarity, see Figure 3 Description.

[0088] It is understandable that in a meeting, there may be a situation where multiple sound receiving devices receive sound data of the same speaker, and such sound data has a high degree of similarity; the embodiments of this specification can cluster sound receiving devices that are close to each other into the same group by considering the impact of the sound source location distribution on the evaluation results, so that the determined sound source importance of the sound receiving device is not affected by the sound receiving devices that are close to each other, thereby making the determined sound source importance more accurate and reliable.

[0089] In some embodiments, the evaluation result of at least one sound receiving device is also related to the sound source change characteristics corresponding to the sound data.

[0090] The sound source change feature refers to a feature that can reflect the change of the audio signal of the sound receiving device. For example, the sound source change feature can be the change value of the sound data corresponding to the audio signal when the sound source type is updated, such as the volume change value, the frequency change value, etc. The sound source change feature can be obtained in real time by the processor 140. For instructions on updating the sound source type, please refer to Figure 3 Description of the place.

[0091] In some embodiments, the processor 140 may determine an evaluation result of at least one sound receiving device based on sound data, sound source location distribution, and sound change characteristics of the at least one sound receiving device.

[0092] For more information on determining assessment results, see Figure 3 Description.

[0093] Scene information refers to information related to the application scenario of the sound control system. For example, scene information may include the size of the space, number of people, type of activity, etc. of the application scenario. For more information about the application scenario of the sound control system, please refer to Figure 1 Description.

[0094] Taking the application scenario of a meeting as an example, the scenario information may include the type of meeting. In some embodiments, different types of meetings may be obtained based on different classification methods. For example, based on the number of hosts or speakers, meetings may be divided into meetings with multiple speakers and meetings with a single speaker. For another example, based on the purpose of the meeting, meetings may be divided into press conferences, year-end summary meetings, etc.

[0095] The following uses the number of hosts or speakers as an example. In a meeting where multiple people speak, multiple participants can speak in sequence or simultaneously, such as group discussions, roundtable meetings, or group discussions in large meetings.

[0096] A single speaker conference can be a conference where a speaker or presenter delivers a speech or report.

[0097] In some embodiments, the sound reception parameters 270 include sound reception intensity and mixing ratio.

[0098] The sound reception parameters 270 are adjustable parameters corresponding to the sound reception device, for example, the sound reception parameters may include sound reception intensity, mixing ratio, noise suppression level, etc.

[0099] The sound intensity refers to the volume of the sound receiving device, which can be expressed in many ways. For example, the magnitude of the sound intensity can be expressed in decibels. The mixing ratio refers to the ratio of the sound intensity between different sound receiving devices. The mixing ratio can be characterized based on the ratio of the sound intensity between sound receiving devices of different sound source types. For example, the mixing ratio can be a ratio of 9:1 between the sound intensity of the sound receiving device as the main sound source and the sound receiving device as the slave sound source.

[0100] In some embodiments, when the main speaker is speaking, the processor 140 can increase the sound reception strength of the sound receiving device as the main sound source, reduce the sound reception strength of the sound receiving device as the slave sound source, and lower the volume of the background music to ensure that the voice of the main speaker is clear enough.

[0101] The above description of the sound reception strength and the mixing ratio is only for example and is not intended to limit the scope of the present invention. The processor 140 may also use other methods to represent the sound reception strength and the mixing ratio. For example, the processor 140 may also measure the sound reception strength by the audio power spectrum density, or determine the mixing ratio by the relative amplitude of the sound signal.

[0102] In some embodiments, the processor 140 can adjust the sound receiving parameters of the sound receiving device in a variety of ways in combination with the sound source information and the scene information.

[0103] As an example, in a seminar, you may encounter echo and reverberation problems. Therefore, you need to combine the sound source information and scene information to adjust the sound intensity and mixing ratio of the sound receiving device corresponding to the main sound source or turn off the sound receiving device corresponding to the slave sound source.

[0104] In some embodiments, the conference type is a conference hosted by a single person, and the processor 140 can adjust the sound of the audio device corresponding to the main audio source to the default audio parameter, and adjust the audio device corresponding to the secondary audio source to a closed state.

[0105] In some embodiments, the conference type is a conference hosted by multiple people, and the processor 140 can adjust the sound reception strength of the sound receiving device corresponding to the main sound source to the default sound reception parameter, and set the mixing ratio as needed. For example, the mixing ratio can be set to main sound source:slave sound source=2:1.

[0106] In some embodiments, the audio source type may be updated according to a preset period.

[0107] The preset period refers to the length of time for updating or adjusting the audio source type according to a predetermined plan within a specific time interval. For example, the preset period may be 1 second or 1 minute.

[0108] In some embodiments, the period length of the preset period is related to the evaluation result of at least one sound receiving device. In some embodiments, the evaluation result includes the importance of multiple sound sources of each sound receiving device in different time periods of at least one sound receiving device. In some embodiments, the closer the average importance of multiple sound sources of the sound receiving device is to the importance threshold, the longer the preset period for updating the sound source type is.

[0109] More information about the evaluation results can be found in Figure 3 Description.

[0110] The mean value of the importance of the sound source can reflect the confidence of the sound receiving device as the main sound source to a certain extent. The closer the mean value of the importance of the sound source corresponding to each sound receiving device is to the importance threshold, the less certain the main sound source result determined by the sound control system from at least one sound receiving device is, which may indicate that the sound receiving devices have a greater mutual influence or the background noise is stronger. By extending the update cycle, more data can be collected in each cycle, so that the sound source type analysis can be performed based on more sufficient intermediate data, which helps to improve the accuracy of the final sound source identification result.

[0111] In some embodiments, at the end of each preset period, the processor 140 may re-evaluate the current sound receiving device based on the sound source information collected during the period to determine whether the sound source type corresponding to the sound receiving device needs to be updated. For example, according to the change in the importance of the sound source, if a new sound source type is detected or an existing sound source type is changed, the sound source type will be adjusted.

[0112] For more information about updating the audio source type, see Figure 4 Description.

[0113] The embodiments of this specification adjust the sound receiving parameters of the sound receiving device through the sound source information and scene information, and reasonably set the preset period to update the sound source type, so as to ensure that the sound control system adapts to the dynamically changing sound source environment, thereby optimizing the audio processing effect.

[0114] Figure 3 FIG. 3 is an exemplary schematic diagram 300 of determining an evaluation result of at least one sound receiving device according to some embodiments of the present specification.

[0115] In some embodiments, determining the evaluation result of at least one sound receiving device further includes: constructing a sound source location map.

[0116] A sound source location map refers to a map that can be used to describe the spatial distribution and correlation between sound receiving devices.

[0117] In some embodiments, the process of constructing a sound source location graph may include: determining nodes, each sound source location corresponds to a node; according to a preset distance threshold, connecting two nodes whose interval distance is less than the distance threshold to form an edge of the sound source location graph.

[0118] like Figure 3 As shown, the sound source location graph may include nodes 310 and edges 320, each node 310 corresponds to a different sound source (i.e., a sound receiving device), and the edge 320 connects the nodes corresponding to the sound receiving devices that are less than a distance threshold. The node attributes include the sound data corresponding to the sound receiving device, the sound source location distribution, and the sound source change characteristics, and the edge attributes include the spatial distance and correlation between the sound receiving devices corresponding to the two connected nodes.

[0119] For more information about sound data, sound source location distribution, and sound source change characteristics, see Figure 2 Description.

[0120] It is understandable that there is a certain spatial distance between each sound receiving device, and the spatial distance refers to the distance between the sound receiving devices in physical space. In some embodiments, the processor 140 can obtain the spatial distance between the sound receiving devices based on GPS positioning or sensor data, or determine the spatial distance between the sound receiving devices from the sound source position distribution data set stored in the storage device 130.

[0121] The processor 140 may also obtain the spatial distance of the sound receiving device in other ways, which is not limited in this specification.

[0122] The correlation between the sound receiving devices describes the degree of association between the sound receiving devices, and usually refers to the similarity of the sound data of the sound receiving devices.

[0123] In the edge attributes of the sound source location graph, the determination of correlation can help evaluate whether there is a certain correlation between adjacent or similar nodes, and then determine whether the sound data from different nodes come from the same receiving device.

[0124] The processor 140 may use a variety of methods to determine the correlation between the sound receiving devices. For example, the processor 140 may use statistical methods or signal processing techniques to analyze the characteristics of the sound data to determine the correlation of the sound receiving devices. For example, the processor 140 may calculate the similarity of the sound data of the nodes through a cross-correlation function.

[0125] The cross-correlation function is a method for measuring the similarity of two signals. In some embodiments, the processor 140 can substitute the sound data corresponding to the two sound receiving devices as two signals into the cross-correlation function, and select the maximum function value of the cross-correlation function as the similarity of the two sound data, representing the matching degree of the two sound data.

[0126] For more information about sound data similarity, see Figure 2 Description.

[0127] The above-mentioned manner of determining the correlation between the sound receiving devices is for illustration purposes only and is not intended to limit the scope of the present invention. The processor 140 may also determine the correlation between the sound receiving devices in other manners.

[0128] In some embodiments, Figure 3 As shown, the processor 140 can determine an evaluation result 340 of at least one sound receiving device based on the sound source location map through an evaluation model 330, wherein the evaluation model is a machine learning model.

[0129] The evaluation model refers to a model that can predict the evaluation result of the sound receiving device. For example, the evaluation model can be used to predict the importance of the sound source of the sound receiving device. The pattern prediction model can be of various types. For example, the evaluation model can be a graph convolutional network (GCN) or a graph neural network (GNN). In some embodiments, the evaluation model is a graph neural network.

[0130] In some embodiments, the input of the evaluation model is a sound source location map. The evaluation model can output the evaluation results of the corresponding sound receiving device based on each node. In some embodiments, the output of the evaluation model can be represented based on multiple forms, for example, it can be represented based on vectors, data sets, sequences, etc. As an example only, the output of the evaluation model can be represented based on a vector, such as the output of the evaluation model can be a vector S = [80, 70, 70, 50, 50], representing that the sound source importance of sound source 1 is 80, the sound source importance of sound source 2 is 70, the sound source importance of sound source 3 is 70, the sound source importance of sound source 4 is 50, and the sound source importance of sound source 5 is 50.

[0131] In some embodiments, the evaluation model can be trained based on a training data set by the processor 140, and the training data set can include multiple training samples and labels corresponding to the training samples. Specifically, the training samples with labels are input into the evaluation model, and the parameters of the evaluation model are updated through training.

[0132] The training sample may include a sound source location map corresponding to the historical sound data. In some embodiments, the label corresponding to the training sample includes the historical sound source importance corresponding to the sound source in the sound source location map.

[0133] In some embodiments, the processor 140 may train the evaluation model through multiple rounds of iterations: in each round of iterations, the processor 140 selects one or more training samples from the training data set, inputs the prediction model and obtains the prediction output, and then calculates the loss function value based on the model output and the label corresponding to the training sample, and uses the back propagation algorithm (e.g., gradient descent method) to update the model parameters to minimize the loss. Iteration ends until the end condition is met and the model training ends. The end condition may include reaching a specified number of iterations or the loss function converges.

[0134] The embodiments of this specification are based on a sound source location map, and an evaluation model is used to determine the evaluation result of at least one sound receiving device, wherein the sound source location map can comprehensively display the location distribution of different sound receiving devices in the environment and the relationship between them; the sound source location map is analyzed and evaluated by a machine learning model, so that automatic detection, classification and positioning of sound events in a complex environment can be achieved; the model can use the graph structure and node attributes to perform efficient data mining and pattern recognition, thereby improving the effects and performance of applications such as audio monitoring, speech recognition, and environmental perception, and can improve the model's understanding of the complexity of the real world, thereby enhancing its generalization ability; in addition, by updating the model parameters through multiple rounds of iterations, the model can be gradually optimized to achieve higher accuracy and performance on the training data. Finally, according to the optimization results of the loss function, the model can be made to better fit the data, thereby improving the accuracy and reliability of its predictions.

[0135] Figure 4 4 is an exemplary flow chart of updating the audio source type according to some embodiments of this specification. Process 400 may be executed by processor 140. Figure 4 As shown, process 400 includes the following steps.

[0136] S410: Determine a first data set based on the sound data to be evaluated, where the first data set includes a plurality of first sub-data.

[0137] The sound data to be evaluated refers to the audio signal collected by the sound receiving device (hereinafter referred to as the sound source to be evaluated) for which it is necessary to evaluate whether the sound source type needs to be updated. For example, the end time point of the most recent preset cycle can be used as the segmentation point, and the audio signal collected after this time point is determined as the sound data to be evaluated.

[0138] The first data set refers to a set of multiple first sub-data in the sound data to be evaluated. The first sub-data is the sound data corresponding to the audio signal in a time interval. For example, the first sub-data may be the volume and frequency corresponding to the audio signal between [02:50-02:53]. In some embodiments, within the time interval corresponding to each preset period, the processor 140 may divide the sound data to be evaluated of the sound source to be evaluated into multiple segments of data according to a preset length, and determine the multiple segments of data as the first data set, and each segment of data corresponds to a first sub-data.

[0139] S420: Determine a sound source change feature corresponding to the first data set based on at least one of the plurality of first sub-data.

[0140] In some embodiments, the processor 140 can characterize the sound data characteristics of each segment of first sub-data in the first data set by the sound data mean of the segment of first sub-data; calculate the sound data gradient between all adjacent first sub-data in the first data set; and use the maximum gradient value in the sound data gradient as the sound source change feature.

[0141] The sound data gradient is used to describe the degree of change of the sound data in time or space. For example, the sound data gradient may include the change value of the frequency, amplitude, and volume of the sound in a unit time.

[0142] Exemplarily, the frequency, amplitude, volume and other data corresponding to the audio signal collected by the sound receiving device after 04:30 to the current time point 05:00 are the sound data to be evaluated. The following explanation is given using frequency as the sound data. The first data set can be composed of the following 6 first sub-data {[04:30-04:35], [04:35-04:40], [04:40-04:45], 04:40-04:50], 04:50-04:55], 04:55-05:00]}. If the frequency mean values ​​of the 6 first sub-data are 500Hz, 500Hz, 520Hz, 510Hz, 800Hz, and 850Hz respectively. The difference of 290Hz between the frequency mean value of 510Hz of the 4th first sub-data and the frequency mean value of 800Hz of the 5th first sub-data can be used as the sound source change feature.

[0143] S430: In response to the sound source change characteristic satisfying the update condition, the sound source type is updated. For more information about updating the sound source type, see Figure 2 Description.

[0144] The update condition is a condition that needs to be met when triggering the update of the sound source type. For example, the update condition may be that the sound source change characteristic is greater than the change threshold. The change threshold can be set manually. For example, the change threshold can be set to 100Hz.

[0145] In some embodiments, when the sound source change characteristics of the sound source to be evaluated meet the update condition, the processor 140 may update the sound source type in the sound source information of the sound source to be evaluated.

[0146] For more information on determining update conditions, see Figure 5 .

[0147] In the embodiment of the present specification, in response to the sound source change characteristics satisfying the update conditions, the sound source type is updated. The processor 140 can quantify the sound source change characteristics and determine whether to update the sound source type based on the quantized data, thereby making the sound control result more accurate.

[0148] Figure 5 FIG. 5 is an exemplary schematic diagram 500 of determining an update condition according to some embodiments of the present specification.

[0149] In some embodiments, Figure 5 As shown, determining the update condition may include: determining a second data set 520 based on the evaluated sound data 510, wherein the second data set 520 includes multiple second sub-data (for example, second sub-data-1, second sub-data-2... second sub-data-n, etc.); determining an update condition 560 through a prediction model 550 based on at least one of the multiple second sub-data, the meeting type 530, and the meeting time 540, wherein the prediction model 550 is a machine learning model.

[0150] The evaluated sound data refers to the audio signal that has been evaluated by the processor 140. For example, the end time point of the most recent preset cycle can be used as the segmentation point, and the evaluated audio signal before the time point is determined as the evaluated sound data.

[0151] The second data set refers to a set of multiple second sub-data in the evaluated sound data. The second sub-data is the sound data corresponding to the audio signal in a time interval. For example, the second sub-data may be the volume and frequency corresponding to the audio signal between [00:10-00:20]. In some embodiments, within the time interval corresponding to each preset period, the processor 140 may divide the evaluated sound data of the evaluated sound source into multiple segments according to a preset length, and determine the multiple segments as the second data set, each segment corresponding to a second sub-data.

[0152] For more information about meeting types, see Figure 2 Description.

[0153] The time of the meeting can be acquired by the processor 140. For example, the processor 140 can use the input preset time point or the time point of starting the speech as the time point of starting the meeting, and continuously record the duration of the meeting.

[0154] A prediction model refers to a model that can predict update conditions. For example, a prediction model can be used to output a trigger condition for updating the sound source type: the sound source change feature is greater than the change threshold. The prediction model can be of various types. For example, the prediction model can be a time series model, such as a support vector machine, etc.

[0155] In some embodiments, the input of the prediction model may include at least one of the second sub-data, the meeting type, and the elapsed time of the meeting.

[0156] For example, the following uses frequency and volume as sound data for explanation. The input of the prediction model can be the second sub-data-1: (00:15-00:20, 50 decibels, 550 Hz), the second sub-data-2: (00:20-00:25, 50 decibels, 540 Hz) ... the second sub-data-14: (01:25-01:30, 52 decibels, 560 Hz), the second sub-data-15: (01:30-01:35, 40 decibels, 780 Hz), T = (5-person speaking meeting), t = (1.5 minutes) represents that the input of the prediction model is the volume value and frequency value in the sound data corresponding to the 15 second sub-data, the meeting type is a meeting with 5 speakers, and the meeting has been going on for 1.5 minutes.

[0157] In some embodiments, the output of the prediction model may include an update condition. For example, the output may be a vector If=(10, 200), which represents that the update condition is that the volume gradient value of adjacent sub-data is greater than 10 decibels and the frequency gradient value is greater than 200 Hz.

[0158] In some embodiments, the prediction model may include a change node prediction layer and an update condition determination layer.

[0159] In some embodiments, the change node prediction layer can be used to determine at least one node where the audio source type has changed and its change confidence based on at least one of the multiple second sub-data, the meeting type, and the meeting duration.

[0160] In some embodiments, the input of the change node prediction layer is at least one of the plurality of second sub-data, the meeting type, and the meeting duration. The change node prediction layer can output at least one node whose audio source type has changed and its change confidence based on processing at least one of the plurality of second sub-data, the meeting type, and the meeting duration.

[0161] At least one node where the sound source type changes may include a time point corresponding to when the sound source type changes, and the change confidence refers to the degree of credibility of the change in the sound source type occurring at the determined node.

[0162] The change confidence can be expressed in many ways. For example, the change confidence can be expressed as a percentage or a value between 0 and 1. The higher the change confidence, the more likely the sound source type of the node is to change.

[0163] For example, when the change node prediction layer inputs the second sub-data-1 as (00:15-00:20, 50 decibels, 550 Hz), the second sub-data-2 as (00:20-00:25, 50 decibels, 540 Hz) ... the second sub-data-14 as (01:25-01:30, 52 decibels, 560 Hz), the second sub-data-15 as (01:30-01:35, 40 decibels, 780 Hz), T = (5-person speaking meeting), t = (1.5 minutes), the output of the change node prediction layer may be that the node where the sound source type changes is 01:30, and the change confidence is 95%, indicating that the sound source type at 01:30 is likely to have changed.

[0164] In some embodiments, the update condition determination layer may be used to determine the update condition based on at least one of the plurality of second sub-data, at least one node whose sound source type has changed and its change confidence.

[0165] In some embodiments, the processor 140 may use the nodes whose sound source types have changed and their change confidences output by the change node prediction layer, and at least one of the multiple second sub-data as inputs to the update condition determination layer. The update condition determination layer may process at least one node whose sound source type has changed and its change confidence, and output an update condition.

[0166] For example, the update condition determination layer can process the node where the sound source type changes at 01:30, the change confidence is 95%, and the second sub-data -1 is: (00:15-00:20, 50 decibels, 550Hz), the second sub-data -2 is: (00:20-00:25, 50 decibels, 540Hz)... The second sub-data -14 is: (01:25-01:30, 52 decibels, 560Hz), the second sub-data -15 is: (01:30-01:35, 40 decibels, 780Hz), and output the update condition If = (10, 200), which means that the update condition is that the volume gradient value of the adjacent sub-data is greater than 10 decibels, and the frequency gradient value is greater than 200Hz.

[0167] In some embodiments, the prediction model can be obtained by jointly training the change node prediction layer and the update condition determination layer.

[0168] In some embodiments, the training sample of the joint training may include at least one second sub-data corresponding to the historical sound data, the meeting type, and the time when the meeting has been conducted. In some embodiments, the label corresponding to the training sample of the joint training includes an update condition, which is determined based on the volume gradient value and frequency gradient value of the second sub-data before and after the time node when the sound source type actually changes.

[0169] Input at least one second sub-data corresponding to the historical sound data, the meeting type, and the time the meeting has been in progress into the initial change node prediction layer to obtain at least one node whose sound source type has changed and its change confidence; input at least one node whose sound source type has changed and its change confidence as training sample data into the initial update condition determination layer to obtain the update condition output by the initial update condition determination layer. Based on the label and the output result of the initial update condition determination layer, construct a loss function, and update the parameters of the change node prediction layer and the update condition determination layer at the same time until the preset conditions are met and the training is completed. The preset conditions can be that the loss function is less than a threshold, converges, or the training cycle reaches a threshold.

[0170] The prediction model in the embodiments of this specification can use the evaluated sound data and multiple sub-data in the second data set to predict the change nodes and their change confidence in the historical sound data, and further predict the update conditions of the sound source type based on the change rules of the historical sound data, which can improve the accuracy and reliability of tracking the changes in the sound source type. By using machine learning models, especially through the change node prediction layer and the update condition determination layer, the prediction model can not only make predictions for specific types of meetings and data sets, but also adapt to different environments and data inputs, and has a certain generalization ability. In a complex conference environment, especially in the case of multiple people and multiple sound sources, the sound source type update conditions can be provided in a timely and accurate manner, which can significantly improve the user experience and ensure that the sound control system can still provide high-quality services and results in a changing conference environment.

[0171] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0172] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0173] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0174] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0175] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0176] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0177] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A sound control method, the method being executed by a processor, characterized in that: include: Determine sound source information, wherein the sound source information includes the number of sound receiving devices and the type of sound source; Based on the sound source information and the scene information, the sound reception parameters of the sound reception device are adjusted, the sound reception parameters including the sound reception intensity and the mixing ratio; wherein the sound source type is updated according to a preset period.

2. The method according to claim 1, characterized in that The determining of the sound source information comprises: Acquiring sound data of the sound receiving device; Based on the sound data, the sound source type of the sound receiving device is determined.

3. The method according to claim 2, characterized in that The determining the sound source type of the sound receiving device based on the sound data includes: determining an evaluation result of at least one sound receiving device based on the sound data; Based on the evaluation result, the sound source type of the at least one sound receiving device is determined.

4. The method according to claim 3, characterized in that The evaluation result of the at least one sound receiving device is related to the sound source position distribution.

5. The method according to claim 1, characterized in that Update the audio source type, including: Based on the sound data to be evaluated, determining a first data set, wherein the first data set includes a plurality of first sub-data; Determine, based on at least one of the plurality of first sub-data, a sound source change feature corresponding to the first data set; In response to the sound source change characteristic satisfying an update condition, the sound source type is updated.

6. A sound control system, characterized in that: The system comprises: A determination module is configured to determine sound source information, wherein the sound source information includes the number of sound receiving devices and the type of sound source; The control module is configured to adjust the sound receiving parameters of the sound receiving device based on the sound source information and the scene information, wherein the sound receiving parameters include the sound receiving intensity and the mixing ratio; wherein the sound source type is updated according to a preset period.

7. The system according to claim 6, characterized in that The determination module is further configured to: Acquiring sound data of the sound receiving device; Based on the sound data, the sound source type of the sound receiving device is determined.

8. The system according to claim 7, characterized in that The determination module is further configured to: determining an evaluation result of at least one sound receiving device based on the sound data; Based on the evaluation result, the sound source type of the at least one sound receiving device is determined.

9. The system according to claim 6, characterized in that The control module is further configured to: Based on the sound data to be evaluated, determining a first data set, wherein the first data set includes a plurality of first sub-data; Determine, based on at least one of the plurality of first sub-data, a sound source change feature corresponding to the first data set; In response to the sound source change characteristic satisfying an update condition, the sound source type is updated.

10. A computer-readable storage medium, characterized in that: The storage medium stores computer instructions, and when the computer instructions are executed by the processor, the sound control method according to any one of claims 1 to 5 is implemented.

Images