Audio processing method and device, electronic equipment and storage medium

Abstract

The invention discloses an audio processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring courseware data and video data; identifying the courseware data, and determining a subject type corresponding to the courseware data and a subject probability corresponding to the subject type; identifying the video data to obtain semantic information and a semantic probability corresponding to the semantic information; the class type is determined based on each subject type, subject probability, semantic information and semantic probability, the audio is processed according to the class type, the problem that the audio processing process cannot be automatically adjusted is solved, the subject type and the subject probability are determined according to the courseware data, and the semantic information and semantic probability are determined according to the video data. The subject type, the subject probability, the semantic information and the semantic probability are analyzed to determine the class type, the audio is processed according to the class type, the audio processing process is automatically adjusted and controlled, manual operation of a user is not needed, and the user experience is improved while the audio effect is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to an audio processing method, device, electronic equipment and storage medium. Background Art

[0002] High-quality educational resources have always been a major problem faced in the development of education in remote areas. Due to the scarcity of educational resources and weak teaching staff in rural areas, teachers often teach across disciplines and grades. Some rural schools even lack art, music, and physical education courses, further widening the gap with urban schools and resulting in the unbalanced development of compulsory education.

[0003] The emergence of remote interactive classroom application software and hardware has made up for this problem of balanced educational resources. During the remote interactive process, different classrooms have different requirements for audio. For example, when the classroom content is mainly language chat scenes such as Chinese and mathematics, and when the classroom content is music and dance, and there are other non-system sound sources playing music on the scene, the sound effects are not the same, and the audio requirements are also different. The current technical solutions generally do not distinguish between scenes, in which case the audio effect is poor; or the adjustment control in the audio processing process is left to the user to manually switch, which is inconvenient for users to operate, affects the user experience, and it is also easy for users to forget to adjust. Therefore, how to adaptively adjust and control the audio processing process has become a problem to be solved. Summary of the invention

[0004] The present invention provides an audio processing method, device, electronic device and storage medium to solve the problem that the audio processing process cannot be automatically adjusted, and the audio processing process is adaptively adjusted according to the class type.

[0005] According to one aspect of the present invention, there is provided an audio processing method, characterized in that it comprises:

[0006] Obtain courseware data and video data;

[0007] Identify the courseware data, determine the subject type corresponding to the courseware data and the subject probability corresponding to the subject type;

[0008] Identify the video data to obtain semantic information and semantic probability corresponding to the semantic information;

[0009] The class type is determined based on the subject type, subject probability, semantic information, and semantic probability, and the audio is processed according to the class type.

[0010] According to another aspect of the present invention, there is provided an audio processing device, comprising:

[0011] Data acquisition module, used to acquire courseware data and video data;

[0012] A subject determination module, used to identify the courseware data, determine the subject type corresponding to the courseware data and the subject probability corresponding to the subject type;

[0013] A semantic determination module, used to identify the video data to obtain semantic information and semantic probability corresponding to the semantic information;

[0014] The audio processing module is used to determine the class type based on the subject type, subject probability, semantic information, and semantic probability, and process the audio according to the class type.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the audio processing method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the audio processing method described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention obtains courseware data and video data; identifies the courseware data to determine the subject type corresponding to the courseware data and the subject probability corresponding to the subject type; identifies the video data to obtain semantic information and the semantic probability corresponding to the semantic information; determines the class type based on the subject type, subject probability, semantic information, and semantic probability, and processes the audio according to the class type, thereby solving the problem that the audio processing process cannot be automatically adjusted. The courseware data is identified to determine the subject type and the subject probability corresponding to each subject type, the video data is identified to determine the semantic information and the semantic probability corresponding to each semantic information, each subject type, subject probability, semantic information, and semantic probability are analyzed to determine the class type, and finally the audio is processed according to the class type, and the audio processing process is automatically adjusted and controlled without the need for manual operation by the user, thereby ensuring the audio effect while improving the user experience.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of a data processing method provided according to Embodiment 1 of the present invention;

[0024] Figure 2 This is an example diagram of implementing audio processing and transmission of traditional audio and video remote interaction according to the first embodiment of the present invention;

[0025] Figure 3 is a flow chart of a data processing method provided according to Embodiment 2 of the present invention;

[0026] Figure 4 This is an example diagram of an implementation of audio processing and transmission for audio and video remote interaction provided according to Embodiment 2 of the present invention;

[0027] Figure 5 is a structural schematic diagram of a data processing device provided according to Embodiment 3 of the present invention;

[0028] Figure 6 It is a structural schematic diagram of an electronic device for implementing the data processing method of an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flowchart of an audio processing method provided in Example 1 of the present invention. This embodiment can be applied to the situation of automatically adjusting and controlling audio. The method can be executed by an audio processing device. The audio processing device can be implemented in the form of hardware and / or software. The audio processing device can be configured in an electronic device.

[0033] Figure 2 The present invention provides an example diagram of the implementation of audio processing and transmission for traditional audio and video remote interaction. Taking a push end and a receiving end as an example, the push end collects audio during the remote interaction process and sends data to other terminals for sharing. The receiving end receives the data shared by the push end and plays it during the remote interaction process, thereby realizing remote interaction. During the remote interaction process, there can be multiple receiving ends for data sharing at one time, that is, the push end can transmit the shared data to multiple receiving ends at the same time. An audio device that can collect and play audio can be used as both a push end and a receiving end. It sends data streams when used as a push end, and receives data streams when used as a receiving end. Figure 2As shown, the push end collects audio data and performs 3A processing on the audio data, namely AEG (Acoustic Echo Chancellor), ANS (Automatic Noise Suppression) and AGC (Automatic Gain Control). After the audio processing is completed, it is encoded, and the encoded data is processed by NACK (Negative Acknowledgement) or FEC (Forward Error Correction), and the processed data is transmitted. The receiving end receives the data, performs FEC or NACK processing on the received data, and then processes the data through NETEQ (jitter buffer), decodes the processed data, plays the decoded data, and completes the remote interaction; at the same time, the decoded data is sent to the module for 3A processing, so that the parameters of the 3A processing can be adjusted through the decoded data, and the echo can be effectively removed when the collected audio data is processed by 3A through the adjusted parameters. When the audio processing method provided in the present application acts on the push end to perform 3A processing on the audio data, the classroom type is identified by adding an intelligent scene recognition function to realize intelligent sound adjustment.

[0034] like Figure 1 As shown, the method includes:

[0035] S101. Obtain courseware data and video data.

[0036] In this embodiment, the courseware data can be specifically understood as the data included in the courseware for auxiliary teaching, which can be composed of multiple pages of courseware, and can be stored in the corresponding storage space of the terminal for remote interaction, or other devices such as multimedia in the classroom. Video data can be understood as data formed by video images in the classroom collected by the terminal for remote interaction, and the video data can be composed of one or more frames of images.

[0037] During the teaching process, teachers can assist teaching by playing courseware to facilitate students to understand knowledge points. Courseware data can be pre-stored in the remote interactive terminal or in other devices. When performing scene recognition, the courseware data is directly obtained from the storage space of the remote interactive terminal, or the courseware data is obtained from other devices by communicating with other devices. A video acquisition device is set to collect video data. The video acquisition device can be a device that can collect video, such as a video recorder, an infrared thermal imager, etc.; the camera of the remote interactive terminal can be directly used as a video acquisition device, or an additional video acquisition device can be installed. The video acquisition device can be set in the classroom to collect videos during classes in the indoor classroom, or it can be set outdoors to collect videos during outdoor activity classes. The number of video acquisition devices can be one or more. For example, video acquisition devices are set at different positions and angles to collect different video data. When there are multiple video data, the multiple video data can be spliced ​​spatially to obtain video data containing multi-dimensional spatial data. After acquisition, the video data can be stored in the corresponding storage space. When performing scene recognition, the video data collected by the video acquisition device is obtained. During the remote interaction process, courseware data and video data can be obtained at a certain periodic time. The video data obtained at this time can be the video data between two data acquisition time points. Each time the courseware data and video data are obtained, scene recognition is performed to determine the class type and further adjust and control the audio processing process. This method can realize real-time adjustment of the audio processing process to avoid the inability to automatically adjust the audio processing process in time after the teaching content changes; or after the class starts, the courseware data and video data are obtained once to perform scene recognition, determine the class type and further adjust and control the audio processing process. This method can save data processing resources.

[0038] S102: Identify the courseware data, determine the subject type corresponding to the courseware data and the subject probability corresponding to the subject type.

[0039] In this embodiment, the subject type may be Chinese, mathematics, music, physical education, etc.; the subject probability may be specifically understood as the probability that the subject currently being taught belongs to each subject type, for example, the probability of belonging to Chinese, the probability of belonging to physical education.

[0040] The courseware data can be identified by keyword matching, similarity calculation, model prediction and other identification methods to determine the subject type corresponding to the courseware data and the subject probability belonging to this subject type. In the embodiment of the present application, the type of subject type identified can be one or more. When there are multiple types, the subject probability corresponding to each subject type can be obtained at the same time.

[0041] It should be noted that, since the result obtained by identifying the courseware data is the same when the courseware has not changed, the embodiment of the present application does not need to repeatedly identify the courseware data when the courseware data has not changed. Each time the courseware data is obtained, the courseware data can be compared with the previous courseware data to determine whether the courseware data has changed; or, the courseware can be uniquely identified, and after obtaining the courseware data, its corresponding courseware identification can be determined, and the courseware identification can be compared with the previous courseware identification to determine whether they are the same. If they are the same, it is determined that the courseware has not changed, and so on.

[0042] S103: Identify the video data to obtain semantic information and semantic probability corresponding to the semantic information.

[0043] In this embodiment, the semantic information can be specifically understood as information describing the semantics of the scene, such as musical instrument performance, outdoor demonstration, etc.; the semantic probability can be specifically understood as the probability of the current scene belonging to each type of semantic information;

[0044] The video data can be identified by extracting key actions from the video data, identifying the key actions through a model or algorithm to obtain semantic information, or directly identifying the video frames in the video data through a model or algorithm to obtain semantic information, and calculating the semantic probability while identifying the semantic information to obtain the semantic probability corresponding to each semantic information. In an embodiment of the present application, the identified semantic information can be pre-set semantic information, that is, different semantic information is pre-set, and different semantic information describes different scenes. When the video data is identified, the obtained semantic information is one or more of the pre-set different semantic information; the identified semantic information can also be any kind of semantic information obtained through analysis.

[0045] S104. Determine the class type based on the subject type, subject probability, semantic information, and semantic probability, and process the audio according to the class type.

[0046] In this embodiment, the class type can be used to describe the type of the course currently being conducted in the class, for example, a voice chat class, a musical instrument performance class, an outdoor demonstration sports class, etc.

[0047] The characteristic teaching scene can be determined by the subject type and semantic information. For example, the subject type with the highest subject probability and the semantic information with the highest semantic probability are selected to determine the teaching scene, and the class type is further determined according to the teaching scene; or, each subject type and subject probability are combined respectively to obtain one or more teaching scenes. For each teaching scene, the probability of the teaching scene is calculated according to the subject probability and semantic probability corresponding to each teaching scene. For example, the average value, weighted value, etc. of the subject probability and semantic probability are calculated to obtain the probability of the teaching scene, and the probability of each teaching scene is compared. The teaching scene with the largest probability is selected, and the class type is determined according to this teaching scene. When determining the class type according to the teaching scene, the corresponding relationship between different teaching scenes and class types can be pre-set, and the class type is directly determined according to the association relationship after the teaching scene is determined, or the teaching scene is analyzed to determine the corresponding class type. The adjustment control method in the audio processing process of different class types is pre-set. After the class type is determined, the adjustment control of the audio processing is performed according to the corresponding adjustment control method. The adjustment control in the audio processing process can be the adjustment control of the audio processing method, parameters, etc.

[0048] An embodiment of the present invention provides an audio processing method, which solves the problem that the audio processing process cannot be automatically adjusted. The method identifies the courseware data to determine the subject type and the subject probability corresponding to each subject type, identifies the video data to determine the semantic information and the semantic probability corresponding to each semantic information, analyzes each subject type, subject probability, semantic information and semantic probability, determines the class type, and finally processes the audio according to the class type. The audio processing process is automatically adjusted and controlled without the need for manual operation by the user, thereby ensuring the audio effect while improving the user experience.

[0049] Embodiment 2

[0050] Figure 3 This is a flowchart of an audio processing method provided in Embodiment 2 of the present invention. This embodiment is refined on the basis of the above embodiment. Figure 3 As shown, the method includes:

[0051] S201. Obtain courseware data and video data.

[0052] S202. Obtain courseware content contained in the courseware data, where the courseware content includes at least one of the following: text, graphics, tables, charts, pictures, videos, and animations.

[0053] In this embodiment, the courseware content refers to the content specifically included in the courseware for teaching, such as text, graphics, tables, pictures, animations, etc.; wherein the text can be in Chinese, English, numbers, etc.; the graphics can be graphics edited or drawn by a graphics editor, such as a rectangle, hexagon, ellipse, etc.; the chart can be a statistical chart, a tree chart, etc., and the statistical chart can be a bar chart, a line chart, a sector chart, a line chart, a pie chart, an area chart, etc.; the picture can be a picture of any type and size; the video can be a long video or a short video, and can be mainly based on people or animals, etc.; the animation can be mainly based on animated puppets, and the animated puppets can perform actions, expressions, and speak, etc.

[0054] The courseware contents included in the courseware of different types of subjects vary greatly. For example, the courseware contents of mathematics usually include text, graphics, tables, charts, etc.; the courseware contents of Chinese usually include text, pictures, etc.; the courseware contents of music usually include text, pictures, videos, etc. Obtain the courseware data, parse the courseware data, and determine the courseware contents contained in the courseware data. Since the courseware usually contains multiple pages, in the embodiment of the present application, the courseware contents of multiple pages of courseware can be unified and organized. For example, all the texts in the courseware contents are unified and organized together, and the pictures are unified and organized together, etc., to facilitate subsequent unified recognition. It is also possible not to unify the courseware contents, and directly recognize them during subsequent recognition.

[0055] S203, identifying the courseware content based on the pre-trained courseware subject recognition model, and obtaining a preset number of subject types and the subject probability corresponding to each subject type.

[0056] In this embodiment, the courseware subject identification model can be specifically understood as a network model that identifies corresponding subject types according to the courseware.

[0057] A large amount of courseware materials are pre-labeled, that is, the courseware materials are labeled with subject types, and the model is trained based on the labeled courseware materials to obtain a courseware subject recognition model that meets the requirements. The courseware content is input into the pre-trained courseware subject recognition model. The courseware subject recognition model analyzes and recognizes the courseware content according to the model parameters, determines the subject type to which the courseware content belongs, and the subject probability corresponding to each subject type. For example, if the subject type is mathematics, the corresponding subject probability is 80%; if the subject type is Chinese, the corresponding subject probability is 8%; if the subject type is music, the corresponding subject probability is 5%; if the subject type is physical education, the corresponding subject probability is 7%. The total sum of all subject probabilities is 100%.

[0058] S204: extract video frames from the video data to obtain at least one video frame to be detected.

[0059] In this embodiment, the video frames to be detected can be specifically understood as video frames with semantic detection requirements. The number of video frames to be detected can be one frame or multiple frames. In order to ensure the recognition result, multiple frames are usually selected.

[0060] Since video data is data within a period of time, which includes multiple video frames, video frames are extracted from the video data during recognition, for example, at a certain interval; or, the number of video frames to be extracted is pre-set, the extraction interval is calculated according to the length of the video data and the number of video frames to be extracted, and then extraction is performed according to the extraction interval to obtain the video frame to be detected.

[0061] S205 . Identify each to-be-detected video frame based on a predetermined image semantic segmentation model to obtain semantic information and semantic probability corresponding to the semantic information.

[0062] In this embodiment, the image semantic segmentation model can be specifically understood as a network model for performing semantic recognition on images.

[0063] Pre-label the video frames of classes of different subjects, and train the model based on the labeled video frames to obtain an image semantic segmentation model that meets the requirements. Input each video frame to be detected into the pre-trained image semantic segmentation model, and the image semantic segmentation model analyzes and recognizes the video frame to be detected based on the model parameters, and determines the semantic information based on the output results of the image semantic segmentation model, and obtains the semantic probability corresponding to the semantic information.

[0064] It should be noted that there is no strict order in the implementation of determining the subject type and subject probability and determining the semantic information and semantic probability, and they can be performed successively or in parallel. This embodiment takes the example of first determining the subject type and subject probability and then determining the semantic information and semantic probability.

[0065] S206. Determine the probability of a specific subject teaching scenario based on the subject type, subject probability, semantic information, and semantic probability.

[0066] In this embodiment, the specific subject teaching scenario can be specifically understood as a certain teaching scenario of a certain subject; different subject types and semantic information are selected to be combined to determine the specific subject teaching scenario; the probability of the specific subject teaching scenario is calculated based on the subject probability and the semantic probability, for example, the average value of the subject probability and the semantic probability, the weighted value, etc. are calculated to obtain the probability of the specific subject teaching scenario.

[0067] As an optional embodiment of this embodiment, this optional embodiment further optimizes the probability of determining a specific subject teaching scenario based on each subject type, subject probability, semantic information, and semantic probability as follows:

[0068] A1. According to the predetermined relationship between subjects and semantics, each subject type is associated with semantic information to obtain a specific subject teaching scenario.

[0069] The subject types and semantic information that can constitute a specific subject teaching scene are pre-set to form an association relationship between the subject and the semantics. For example, music classes and musical instrument performances have an association relationship, which can constitute a specific subject teaching scene "playing musical instruments in music class". According to the predetermined association relationship between the subject and the semantics, the obtained subject types and semantic information are associated and combined. When associating and combining, a subject type can be selected first, and its associated semantic information is determined through the association relationship. The sub-subject type and semantic information can be combined to obtain a specific subject teaching scene. It should be known that in the embodiment of the present application, the subject type and semantic information can be a one-to-one relationship, a one-to-many relationship, a many-to-many relationship, etc., depending on the division requirements of the teaching scene, and the embodiment of the present application does not limit this.

[0070] A2. For each specific subject teaching scenario, a weighted sum is performed on the subject probability and semantic probability corresponding to the specific subject teaching scenario to obtain the probability of the specific subject teaching scenario.

[0071] A specific subject teaching scenario is composed of a subject type and a semantic type. For each specific subject teaching scenario, the subject type and semantic information that constitute this scenario are determined. The subject probability corresponding to this subject type and the semantic probability corresponding to the semantic information can be further determined, that is, the subject probability and semantic probability corresponding to the specific subject teaching scenario are obtained, and the obtained subject probability and semantic probability are weighted and summed. The weight can be set in advance, and can be set according to the accuracy of the two models, or according to the importance of the subject type and semantic information; for example, the weight of the subject probability is set to 0.6, the weight of the semantic probability is set to 0.4, and so on. By weighted summing the subject probability and the semantic probability, the value obtained is the probability of the specific subject teaching scenario.

[0072] S207. Compare the probabilities of each specific subject teaching scenario, and determine the class type based on the specific subject teaching scenario with the highest probability.

[0073] Compare the probabilities of each specific subject teaching scenario, determine the specific subject teaching scenario with the highest probability, and directly use this specific subject teaching scenario as the class type. Alternatively, pre-set the class type classification rules and classify this specific subject teaching scenario into its corresponding class type.

[0074] As an optional embodiment of this embodiment, this optional embodiment further determines the class type according to the specific subject teaching scenario with the highest probability, and optimizes it as follows:

[0075] B1. Query the predetermined classroom type and teaching scenario data table to determine the target teaching scenario that matches the specific subject teaching scenario with the highest probability.

[0076] In this embodiment, the classroom type and teaching scene data table can be specifically understood as a data table that stores different teaching scenes and classroom types; the target teaching scene can be specifically understood as the teaching scene in the teaching scene data table that matches the specific subject teaching scene with the highest probability.

[0077] Generate a data table of classroom types and teaching scenarios in advance, query the data table of classroom types and teaching scenarios, determine the teaching scenario in the table that matches the specific subject teaching scenario with the highest probability, and use it as the target teaching scenario.

[0078] B2. Determine the classroom type corresponding to the target teaching scenario as the classroom type corresponding to the specific subject teaching scenario with the highest probability.

[0079] Since the teaching scenarios and classroom types are stored in the classroom type and teaching scenario data table accordingly, after determining the target teaching scenario, its corresponding classroom type can be determined, and the classroom type can be directly determined as the classroom type corresponding to the specific subject teaching scenario with the highest probability.

[0080] In an embodiment of the present application, a probability threshold can also be set. After determining the specific subject teaching scenario with the highest probability, the probability is compared with the probability threshold. If it is greater than the probability threshold, the class type can be determined based on the specific teaching scenario with the highest probability; otherwise, it can be determined that the accuracy is low and the class type cannot be identified.

[0081] In the embodiment of the present application, the classroom type is divided into voice chat classroom, music instrument performance classroom and outdoor sports demonstration classroom according to the demand for audio. For example, when the specific subject teaching scene is playing a musical instrument in a music class, the classroom type is a musical instrument performance music classroom; when the specific subject teaching scene is an outdoor demonstration in a physical education class, the classroom type is an outdoor demonstration physical education class; the rest of the scenes are recognized as voice chat classrooms by default.

[0082] S208. When the class type is a voice chat class, the background noise suppression function and the automatic gain control function are turned on, and the audio is processed based on the background noise suppression function and the automatic gain control function.

[0083] When the class type is a voice chat class, ensure that the background noise suppression function and the automatic gain control function are enabled. You can determine whether the background noise suppression function and the automatic gain control function are enabled. If they are already enabled, there is no need to enable them again. If they are not enabled, enable the background noise suppression function and the automatic gain control function. After enabling the above functions, when processing the collected audio, the background noise suppression function and the automatic gain control function can be used to make the human voice in the voice chat class clear.

[0084] S209. When the class type is a music instrument performance class, turn off the background noise suppression function and the automatic gain control function, and process the audio.

[0085] When the class type is a music instrument performance class, ensure that the background noise suppression function and the automatic gain control function are turned off. It can be determined whether the background noise suppression function and the automatic gain control function are activated. If they are already turned off, there is no need to turn them off again. If they are activated, turn off the background noise suppression function and the automatic gain control function. When the background noise suppression function and the automatic gain control function are turned off, the collected audio is processed so that the sound of the instrument is not lost, and the music class content can be interacted remotely normally.

[0086] S210. When the class type is an outdoor demonstration physical education class, start the background noise suppression function and the automatic gain control function and increase the coefficients of the background noise suppression function and the automatic gain control function according to preset parameters, and process the audio based on the adjusted coefficients of the background noise suppression function and the automatic gain control function.

[0087] In this embodiment, the preset parameters may be specifically understood as pre-set parameter values, and the preset parameters may include one or more parameters for adjusting one or more coefficients in the background noise suppression function and the automatic gain control function.

[0088] Preset parameters are set in advance, and the preset parameters can be set according to the scene's audio processing requirements. When the class type is an outdoor demonstration physical education class, the background noise suppression function and the automatic gain control function are started. The implementation method can also be to first determine whether the background noise suppression function and the automatic gain control function are started. If they are already started, there is no need to start them again. If they are not started, start the background noise suppression function and the automatic gain control function. After starting, obtain the preset parameters, and increase the coefficients of the background noise suppression function and the automatic gain control function based on the preset parameters, for example, adjust the original coefficients to the preset parameters, or increase the original coefficients by the size of the preset parameters. After starting and adjusting the background noise suppression function and the automatic gain control function, when the collected audio is subsequently processed, the audio can be processed based on the adjusted background noise suppression function and the coefficients of the automatic gain control function to improve the outdoor classroom voice experience.

[0089] For example, Figure 4 This paper provides an example diagram of the implementation of audio processing and transmission for remote audio and video interaction, which applies scene recognition to 3A. The rest of the processing flow is similar to Figure 2 The audio processing in the same as the transmission process, scene recognition includes determining the class type based on the courseware data and video data. After the scene recognition of the class type acts on 3A, the ANS and AGC functions can be controlled to achieve automatic adjustment of the user's voice parameters and microphone status, thereby achieving the purpose of optimizing the audio playback effect.

[0090] In an embodiment of the present application, after the audio processing process is automatically adjusted according to the class type, since the automatic gain control function is controlled and adjusted, it is possible to avoid the problem of severe howling due to sound gain issues when multiple devices on site join a unified remote interaction process.

[0091] The embodiment of the present invention provides an audio processing method, which solves the problem that the audio cannot be adjusted automatically. The courseware content is identified by pre-training the courseware subject recognition model to determine the subject type and the subject probability corresponding to each subject type; the video data is extracted to determine the video frame to be detected, and the video frame to be detected in the video data is identified by pre-training the image semantic segmentation model to determine the semantic information and the semantic probability corresponding to each semantic information; the specific subject teaching scene is determined according to each subject type and semantic information, and the probability of the specific subject teaching scene is determined by weighted summation of the subject probability and the semantic probability, and finally the class type is determined based on the probability of the specific subject teaching scene, and whether to start the background noise suppression function and the automatic gain control function according to the class type, and the coefficients of the background noise suppression function and the automatic gain control function are adjusted to realize adaptive adjustment and control of audio processing. The embodiment of the present application realizes scene recognition by determining the class type, considers the scene when processing the audio, and performs adaptive adjustment and control, without manual operation by the user, and can switch between different types of classes, and can obtain better audio and video interactive effects without manual audio and video effect debugging, meeting the class needs of different teachers, ensuring the audio effect and improving the user experience.

[0092] Embodiment 3

[0093] Figure 5 FIG. 1 is a schematic diagram of the structure of an audio processing device provided in Embodiment 3 of the present invention. Figure 5 As shown, the device includes: a data acquisition module 31, a subject determination module 32, a semantic determination module 33 and an audio processing module 34.

[0094] Wherein, the data acquisition module 31 is used to acquire courseware data and video data;

[0095] The subject determination module 32 is used to identify the courseware data, determine the subject type corresponding to the courseware data and the subject probability corresponding to the subject type;

[0096] A semantic determination module 33, used to identify the video data to obtain semantic information and semantic probability corresponding to the semantic information;

[0097] The audio processing module 34 is used to determine the class type based on the subject type, subject probability, semantic information, and semantic probability, and process the audio according to the class type.

[0098] An embodiment of the present invention provides an audio processing device, which solves the problem that the audio processing process cannot be automatically adjusted. The device identifies the courseware data to determine the subject type and the subject probability corresponding to each subject type, identifies the video data to determine the semantic information and the semantic probability corresponding to each semantic information, analyzes each subject type, subject probability, semantic information and semantic probability, determines the class type, and finally processes the audio according to the class type. The audio processing process is automatically adjusted and controlled without the need for manual operation by the user, thereby ensuring the audio effect while improving the user experience.

[0099] Optionally, the subject determination module 32 includes:

[0100] A courseware content acquisition unit, used to acquire the courseware content contained in the courseware data, wherein the courseware content includes at least one of the following: text, graphics, tables, charts, pictures, videos and animations;

[0101] The subject determination unit is used to identify the courseware content based on a pre-trained courseware subject identification model to obtain a preset number of subject types and a subject probability corresponding to each of the subject types.

[0102] Optionally, the semantic determination module 33 includes:

[0103] A video frame acquisition unit, used to extract video frames from the video data to obtain at least one video frame to be detected;

[0104] The semantic determination unit is used to identify each of the video frames to be detected based on a predetermined image semantic segmentation model to obtain semantic information and semantic probability corresponding to the semantic information.

[0105] Optionally, the audio processing module 34 includes:

[0106] A scene probability determination unit, used to determine the probability of a specific subject teaching scene based on each of the subject types, subject probabilities, semantic information, and semantic probabilities;

[0107] The class type determination unit is used to compare the probabilities of the specific subject teaching scenarios and determine the class type according to the specific subject teaching scenario with the highest probability.

[0108] Optionally, a scene probability determination unit is specifically used to associate each subject type with semantic information according to a predetermined association relationship between the subject and semantics to obtain a specific subject teaching scene; for each specific subject teaching scene, a weighted sum is taken of the subject probability and semantic probability corresponding to the specific subject teaching scene to obtain the probability of the specific subject teaching scene.

[0109] Optionally, the classroom type determination unit is specifically used to query the predetermined classroom type and teaching scenario data table to determine the target teaching scenario that matches the specific subject teaching scenario with the highest probability; and determine the classroom type corresponding to the target teaching scenario as the classroom type corresponding to the specific subject teaching scenario with the highest probability.

[0110] Optionally, the audio processing module 34 includes:

[0111] A first processing unit, configured to enable a background noise suppression function and an automatic gain control function when the classroom type is a voice chat classroom, and process the audio based on the background noise suppression function and the automatic gain control function;

[0112] A second processing unit is used to turn off the background noise suppression function and the automatic gain control function and process the audio when the class type is a musical instrument performance music class;

[0113] The third processing unit is used to start the background noise suppression function and the automatic gain control function when the class type is an outdoor demonstration physical education class, and to increase the coefficients of the background noise suppression function and the automatic gain control function according to preset parameters, and to process the audio based on the adjusted coefficients of the background noise suppression function and the automatic gain control function.

[0114] The audio processing device provided in the embodiment of the present invention can execute the audio processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0115] Embodiment 4

[0116] Figure 6A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0117] like Figure 6 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0118] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0119] The processor 41 may be a variety of general and / or dedicated processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as an audio processing method.

[0120] In some embodiments, the audio processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the audio processing method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the audio processing method in any other suitable manner (e.g., by means of firmware).

[0121] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0123] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0125] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0126] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0127] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0128] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An audio processing method, characterized in that: include: Obtain courseware data and video data; Identify the courseware data, determine the subject type corresponding to the courseware data and the subject probability corresponding to the subject type; Identify the video data to obtain semantic information and semantic probability corresponding to the semantic information; The class type is determined based on the subject type, subject probability, semantic information, and semantic probability, and the audio is processed according to the class type.

2. The method according to claim 1, characterized in that The step of identifying the courseware data and determining the subject type corresponding to the courseware data and the subject probability corresponding to the subject type includes: Obtaining courseware content contained in the courseware data, wherein the courseware content includes at least one of the following: text, graphics, tables, charts, pictures, videos, and animations; The courseware content is identified based on a pre-trained courseware subject identification model to obtain a preset number of subject types and a subject probability corresponding to each subject type.

3. The method according to claim 1, characterized in that The identifying the video data to obtain semantic information and semantic probability corresponding to the semantic information includes: Extracting video frames from the video data to obtain at least one video frame to be detected; Based on a predetermined image semantic segmentation model, each of the video frames to be detected is identified to obtain semantic information and semantic probability corresponding to the semantic information.

4. The method according to claim 1, characterized in that: The determining of the class type based on the subject type, subject probability, semantic information, and semantic probability includes: Determine the probability of a specific subject teaching scenario based on each of the subject types, subject probabilities, semantic information, and semantic probabilities; The probabilities of the specific subject teaching scenarios are compared, and the class type is determined according to the specific subject teaching scenario with the highest probability.

5. The method according to claim 4, characterized in that The method of determining the probability of a specific subject teaching scenario based on the subject type, subject probability, semantic information, and semantic probability includes: According to the predetermined association relationship between the subject and the semantics, each subject type is associated with the semantic information to obtain a specific subject teaching scenario; For each specific subject teaching scenario, a weighted sum is performed on the subject probability and semantic probability corresponding to the specific subject teaching scenario to obtain the probability of the specific subject teaching scenario.

6. The method according to claim 4, characterized in that Determining the class type according to the most likely specific subject teaching scenario includes: Querying a predetermined class type and teaching scenario data table to determine a target teaching scenario that matches the specific subject teaching scenario with the highest probability; The classroom type corresponding to the target teaching scenario is determined as the classroom type corresponding to the specific subject teaching scenario with the highest probability.

7. The method according to claim 1, characterized in that The processing of the audio according to the class type includes: When the classroom type is a voice chat classroom, a background noise suppression function and an automatic gain control function are turned on, and the audio is processed based on the background noise suppression function and the automatic gain control function; When the class type is a musical instrument performance class, the background noise suppression function and the automatic gain control function are turned off, and the audio is processed; When the class type is an outdoor demonstration physical education class, the background noise suppression function and the automatic gain control function are started and the coefficients of the background noise suppression function and the automatic gain control function are increased according to preset parameters, and the audio is processed based on the adjusted background noise suppression function and the automatic gain control function coefficients.

8. An audio processing device, characterized in that: include: Data acquisition module, used to acquire courseware data and video data; A subject determination module, used to identify the courseware data, determine the subject type corresponding to the courseware data and the subject probability corresponding to the subject type; A semantic determination module, used to identify the video data to obtain semantic information and semantic probability corresponding to the semantic information; The audio processing module is used to determine the class type based on the subject type, subject probability, semantic information, and semantic probability, and process the audio according to the class type.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the audio processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the audio processing method according to any one of claims 1 to 7 when executed.

Images