Data processing method and device, equipment and storage medium

Abstract

The invention discloses a data processing method and device, equipment and a storage medium, and relates to the technical field of audio and video processing, and the method comprises the steps: collecting audio and video data; obtaining an initial sample set based on the audio and video data; processing the initial sample set through a semantic segmentation algorithm and a direction arrival algorithm to obtain processed data; and outputting an audio and video stream based on the processed data. The method and the device can avoid interference of a noise environment.

Description

Technical Field

[0001] The present application relates to the field of audio and video processing technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In order to meet people's communication needs in different scenarios, audio and video processing technology is applied to application scenarios such as online meetings.

[0003] However, current audio and video processing technologies are easily interfered with in noisy environments.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a data processing method, device, equipment and storage medium, aiming to solve the technical problem that current audio and video processing technology is easily interfered in a noisy environment.

[0006] To achieve the above purpose, the present application proposes a data processing method, which includes:

[0007] Collect audio and video data;

[0008] Obtaining an initial sample set based on the audio and video data;

[0009] Processing the initial sample set by using a semantic segmentation algorithm and a directional arrival algorithm to obtain processed data;

[0010] Output audio and video streams based on the processed data.

[0011] In one embodiment, the step of obtaining an initial sample set based on the audio and video data includes:

[0012] Receive acquisition parameter calibration instructions and calibrate acquisition parameters;

[0013] The step of obtaining an initial sample set based on the audio and video data comprises:

[0014] Based on the acquisition parameters, the audio and video data are calibrated to obtain the initial sample set.

[0015] In one embodiment, the step of processing the initial sample set by a semantic segmentation algorithm and a directional arrival algorithm to obtain processed data includes:

[0016] Performing picture pixel set classification and / or information labeling on the initial sample set by using the semantic segmentation algorithm to obtain a second sample set;

[0017] Processing the second sample set by the directional arrival algorithm to obtain a third sample set;

[0018] The third sample set is processed for behavior recognition by using the semantic segmentation algorithm to obtain the processed data.

[0019] In one embodiment, the step of performing picture pixel set classification and / or information labeling on the initial sample set by the semantic segmentation algorithm to obtain the second sample set includes:

[0020] The second sample set is obtained by using the semantic segmentation algorithm to distinguish background images and character images for different pixel sets in the initial sample set, and / or marking at least one of individual information of different speakers, human voice content information, and human voice position information.

[0021] In one embodiment, the step of processing the second sample set by the directional arrival algorithm to obtain the third sample set includes:

[0022] The direction arrival algorithm is used to perform error data processing on at least one of the individual information of different speakers, vocal content information, and vocal position information in the second sample, and the speaker position and / or discussant position among the speakers are solidified to obtain the third sample set.

[0023] In one embodiment, the step of performing behavior recognition processing on the third sample set by using the semantic segmentation algorithm to obtain the processed data includes:

[0024] Performing speech behavior recognition on the third sample set;

[0025] When the speech behavior of the speaker in the third sample set is identified, reducing and / or filtering the audio and video data from the unexpected direction source, and / or enlarging the pixel block of the speaker;

[0026] Perform image blur processing on at least one of the site, background, and expected outward-direction personnel in the third sample set to obtain the processed data.

[0027] In one embodiment, the step of outputting the audio and video stream based on the processed data includes:

[0028] Obtaining features of different character image sets based on the initial sample set;

[0029] Based on the features of the different character image sets, the processed data is cropped and different audio and video streams are output.

[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, the data processing device comprising:

[0031] A data acquisition module, used to collect audio and video data; used to obtain an initial sample set based on the audio and video data;

[0032] A data processing module, used to process the initial sample set through a semantic segmentation algorithm and a direction arrival algorithm to obtain processed data;

[0033] A data output module is used to output audio and video streams based on the processed data.

[0034] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.

[0035] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.

[0036] One or more technical solutions proposed in this application have at least the following technical effects:

[0037] Semantic segmentation algorithms are used to classify different objects in audio and video. Semantic segmentation algorithms can accurately determine which areas contain possible sound sources. This helps reduce the focus on non-sound source areas, thereby reducing noise interference. Direction of arrival algorithms provide accurate location information of sound sources, thereby distinguishing different sound sources in a noisy environment. Therefore, by processing the initial sample set through semantic segmentation algorithms and directional arrival algorithms, the impact of noise can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided;

[0041] Figure 2 This is a flowchart of the data processing method for this application;

[0042] Figure 3 This is a functional diagram of the data processing system for this application;

[0043] Figure 4 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application;

[0044] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] Technical terms involved in the embodiments of this application:

[0049] Semantic segmentation: Semantic segmentation is a technology in image processing that can classify objects in an image into specific categories. Unlike traditional classification methods, semantic segmentation not only identifies objects in an image, but also can be accurate to each pixel, and can more efficiently and accurately divide the bounding boxes, instance individuals and other information of different objects. Commonly used processing methods include PascalVOC, ADE20K, CamVid, etc. At present, deep learning models have made significant progress in semantic segmentation tasks, but in some audio and video processing scenarios, the effect will be affected.

[0050] Direction of Arrival (DOA) algorithm is a technique for locating the direction of a sound source or signal source, and is widely used in acoustic sensor arrays, audio monitoring, and communication systems. Existing DOA algorithm technologies, such as multiple signal classification (MUSIC) and estimated signal parameters (ESPRIT), obtain audio data spectra based on array design, covariance matrix decomposition, and other methods. They perform well in scenarios with fewer sound sources, but their accuracy and robustness decrease in environments with multiple sound sources and noise.

[0051] The main solution of the embodiment of the present application is: collecting audio and video data; obtaining an initial sample set based on the audio and video data; processing the initial sample set through a semantic segmentation algorithm and a directional arrival algorithm to obtain processed data; and outputting an audio and video stream based on the processed data.

[0052] In this embodiment, for the convenience of description, the following description is made with the data processing system as the execution subject.

[0053] Because the current audio and video processing technology is easily interfered in a noisy environment.

[0054] The present application provides a solution to classify different objects in audio and video through a semantic segmentation algorithm. Through the semantic segmentation algorithm, it is possible to accurately determine which areas contain possible sound sources. This helps to reduce the focus on non-sound source areas, thereby reducing noise interference. The directional arrival algorithm provides accurate location information of the sound source, thereby distinguishing different sound sources in a noisy environment. Therefore, by processing the initial sample set through the semantic segmentation algorithm and the directional arrival algorithm, the impact of noise can be effectively reduced.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a data processing device, etc. that can realize the above functions, or an electronic system, a data processing system, etc. that can realize the above functions. The following takes the data processing system as an example to illustrate this embodiment and the following embodiments.

[0056] In the current era, data retrieval algorithms are widely used in the field of audio and video processing. Video conferencing systems have integrated many emerging algorithm technologies and functions. For example, the DOA algorithm can help video conferencing systems locate the source of human voices. It mainly relies on minimum mean square error, maximum likelihood estimation, feature decomposition and other processing methods to screen data. However, it is not easy to remove image noise with traditional algorithms, such as portraits on TVs and portraits hanging on walls. The present invention is based on the positioning of speakers by the DOA algorithm, and introduces semantic segmentation detection, thereby obtaining more accurate speaker coordinates.

[0057] Based on this, the present application embodiment provides a data processing method, referring to Figure 1 , Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided.

[0058] In this embodiment, the data processing method includes steps S10, S30-S50:

[0059] Step S10, collecting audio and video data;

[0060] It should be noted that the embodiments of the present application take online video conferencing as an example to illustrate the data processing method.

[0061] Optionally, the images in the video conferencing scene can be collected for video stream information transmission and audio information transmission. By using independently developed audio and video terminals, high-definition image data and voice information can be obtained.

[0062] Step S30, obtaining an initial sample set based on the audio and video data;

[0063] Optionally, the audio and video data may be used as an initial sample set for subsequent processing.

[0064] Optionally, the audio and video data may be pre-processed and used as an initial sample set for subsequent processing.

[0065] Step S40, processing the initial sample set by a semantic segmentation algorithm and a direction arrival algorithm to obtain processed data;

[0066] Among them, the initial sample set can be classified into picture pixel sets and / or information marked by the semantic segmentation algorithm to obtain a second sample set; the second sample set can be processed by the direction arrival algorithm to obtain a third sample set; the third sample set can be processed for behavior recognition by the semantic segmentation algorithm to obtain the processed data.

[0067] Step S50: outputting an audio and video stream based on the processed data.

[0068] Among them, the independently developed audio and video output logic can be used to optimize the audio and video data processed by the above process and output high-definition audio and video streams.

[0069] This embodiment provides a data processing method, which classifies different objects in audio and video through a semantic segmentation algorithm. The semantic segmentation algorithm can accurately determine which areas contain possible sound sources. This helps to reduce the focus on non-sound source areas, thereby reducing noise interference. The directional arrival algorithm provides accurate location information of the sound source, thereby distinguishing different sound sources in a noisy environment. Therefore, by processing the initial sample set through the semantic segmentation algorithm and the directional arrival algorithm, the impact of noise can be effectively reduced.

[0070] Before the DOA algorithm is processed, this embodiment preferentially uses the semantic segmentation algorithm to screen the sampled data, excludes invalid data outside the specified area, improves the reliability of the sample set, and achieves a certain degree of optimization of the output effect in the preprocessing stage.

[0071] Compared with the solution of processing human voice data with a single DOA algorithm, this embodiment introduces a semantic segmentation algorithm to analyze the picture pixel set, realizes the effective combination of video and audio, and improves the accuracy of array direction processing with the help of the feedback results of video information.

[0072] Compared with image noise removal which is not easy to achieve with traditional algorithms, such as portraits on televisions, portraits hanging on walls, etc., this embodiment introduces a semantic segmentation algorithm to screen the sampled data and exclude invalid data outside the specified area, thereby effectively removing image noise.

[0073] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, before obtaining the initial sample set based on the audio and video data in step S30, the data processing method further includes step S20:

[0074] Step S20, receiving an acquisition parameter calibration instruction and calibrating the acquisition parameters;

[0075] The acquisition parameters may be the acquisition range of the video conference, the position of the speaker, etc. The user may specify the acquisition range of the video conference, the position of the speaker, etc. in the data processing system.

[0076] The step S30 of obtaining an initial sample set based on the audio and video data further comprises a step S31:

[0077] Step S31, based on the acquisition parameters, the audio and video data are calibrated to obtain the initial sample set.

[0078] Among them, the user inputs the acquisition parameters in advance, performs image recalibration, and only selects the audio and video stream data within the calibration distance as the sample set.

[0079] In this embodiment, the audio and video data are calibrated based on the acquisition parameters to obtain the initial sample set, thereby improving the data quality of the initial sample set.

[0080] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S40 processes the initial sample set by a semantic segmentation algorithm and a direction arrival algorithm to obtain the processed data, including steps S41 to S43:

[0081] Step S41, performing picture pixel set classification and / or information labeling on the initial sample set by using the semantic segmentation algorithm to obtain a second sample set;

[0082] Through the semantic segmentation algorithm, each frame of the initial sample set is processed and each pixel in the image is assigned to the corresponding category, which can help identify different objects and their locations in the image.

[0083] On the basis of classification, additional information tags can be added to the identified objects, such as the object's bounding box, center point coordinates, category label, etc. This tagging information helps in subsequent processing steps.

[0084] Step S42, processing the second sample set by using the direction arrival algorithm to obtain a third sample set;

[0085] The audio data in the second sample set is processed using the DOA algorithm to determine the specific direction of each sound source. Combined with the visual information provided by semantic segmentation, the sound source can be more accurately located and associated with the object in the image.

[0086] Step S43: performing behavior recognition processing on the third sample set by using the semantic segmentation algorithm to obtain the processed data.

[0087] The image sequences in the third sample set are further processed using a semantic segmentation algorithm to identify and label the behaviors in them. Combining previous visual and auditory information, complex behavioral patterns can be more accurately identified and understood.

[0088] Action recognition usually requires analyzing multiple consecutive video frames to capture changes in actions. Semantic segmentation algorithms can provide detailed information for each frame and help identify motion patterns in consecutive frames.

[0089] This embodiment uses the semantic segmentation algorithm to classify the picture pixel set and / or mark the information of the initial sample set to obtain a second sample set, processes the second sample set through the direction arrival algorithm to obtain a third sample set, and performs behavior recognition processing on the third sample set through the semantic segmentation algorithm to obtain the processed data, which combines visual and auditory information, can more accurately identify and understand objects and behaviors in audio and video data, and is suitable for a variety of application scenarios.

[0090] Before the DOA algorithm processes the human voice data, the semantic segmentation algorithm is introduced to pre-process the sample set, and the sample data is optimized in the initial feedback step, which reduces the error caused by dirty data to a certain extent, improves processing efficiency, and optimizes output quality. In the subsequent process processing, the semantic segmentation algorithm is used again to identify the dynamic factors of the video conference scene, focusing on detecting the speech behavior of the participants, and applying derivative functions such as screen zooming and background framing.

[0091] Based on the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the third embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S41, classifying the picture pixel set and / or marking the information of the initial sample set by the semantic segmentation algorithm, and obtaining the second sample set includes step S411:

[0092] Step S411, using the semantic segmentation algorithm to distinguish the background image and the character image for different pixel sets in the initial sample set, and / or marking at least one of the individual information of different speakers, the human voice content information, and the human voice position information, to obtain the second sample set.

[0093] Among them, the semantic segmentation algorithm is used to process the returned audio and video streams. Different categories can be divided for different pixel sets, distinguishing the meeting background picture and the picture of the participants, and marking different speakers, sound coordinates and other information. The sample set with less dirty data after preprocessing is returned to the subsequent modules for use.

[0094] Specifically, you can choose a pre-trained semantic segmentation model suitable for person segmentation, such as U-Net, DeepLab, or Mask R-CNN. These models have been trained on large-scale datasets and can effectively distinguish between people and backgrounds. If higher accuracy is required, the model can be fine-tuned on a specific dataset to adapt to a specific meeting scenario.

[0095] This embodiment uses the semantic segmentation algorithm to distinguish between background images and character images for different pixel sets in the initial sample set, and / or marks at least one of the individual information of different speakers, human voice content information, and human voice position information to obtain the second sample set, which can be accurately classified.

[0096] Based on the fourth embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the fourth embodiment can be referred to the above description, and will not be described in detail later. On this basis, in step S42, before obtaining the initial sample set based on the audio and video data, the data processing method further includes step S421:

[0097] Step S421, performing error data processing on at least one of the individual information of different speakers, vocal content information, and vocal position information in the second sample through the direction arrival algorithm, and solidifying the speaker position and / or discussant position among the speakers to obtain the third sample set.

[0098] Among them, the DOA algorithm is applied to the sample set processed by the semantic segmentation algorithm to further reduce the impact of error data, obtain more accurate voice content information and voice position information, and solidify the position of the speaker and the discussant.

[0099] Furthermore, a suitable DOA algorithm may be selected, such as MUSIC (Multiple Signal Classification), ESPRIT (Estimation of Signal Parameters via Rotational Invariance Techniques) or beamforming technology.

[0100] This embodiment performs error data processing on at least one of the individual information of different speakers, voice content information, and voice position information in the second sample through the directional arrival algorithm, and solidifies the speaker position and / or discussant position among the speakers to obtain the third sample set, which can further reduce the impact of error data and obtain a sample set with higher accuracy.

[0101] Based on the fifth embodiment of the present application, in the sixth embodiment of the present application, the same or similar contents as those in the fifth embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S43, performing behavior recognition processing on the third sample set through the semantic segmentation algorithm, and obtaining the processed data includes steps S431 to S433:

[0102] Step S431, performing speech behavior recognition on the third sample set;

[0103] Specifically, you can choose a pre-trained semantic segmentation model suitable for speech act recognition, such as U-Net, DeepLab, or Mask R-CNN. These models have been trained on large-scale datasets and can effectively distinguish between speakers and listeners. If higher accuracy is required, the model can be fine-tuned on a specific dataset to adapt to a specific meeting scenario.

[0104] Step S432, when the speech behavior of the speaker in the third sample set is identified, reducing and / or filtering the audio and video data from the unexpected direction source, and / or enlarging the pixel block of the speaker;

[0105] Among them, behavior recognition processing can be performed on the established participant image set features, voice features, coordinate position and other information. When it is detected that the participant is speaking, the information collection in other coordinate directions is reduced to obtain more accurate image and voice data.

[0106] Alternatively, when a participant is detected to be speaking, the pixel block of the speaker is enlarged.

[0107] Step S433, performing image blur processing on at least one of the site, background, and expected outward-direction personnel in the third sample set to obtain the processed data.

[0108] Among them, the video conference venue and the background of the participants can also be blurred to highlight the image of the target person.

[0109] This embodiment performs audio filtering processing so that different split screens focus on collecting and outputting audio data from the current screen direction, thereby reducing the impact of noise in the scene and optimizing the user experience of video conferencing in multi-voice scenes.

[0110] Based on the sixth embodiment of the present application, in the seventh embodiment of the present application, the same or similar contents as those in the sixth embodiment can be referred to the above description, and will not be described in detail later. On this basis, step S50, outputting the audio and video stream based on the processed data includes steps S51 to S52:

[0111] Step S51, obtaining features of different character image sets based on the initial sample set;

[0112] Optionally, features of different character image sets may be obtained based on the initial sample set using a semantic segmentation algorithm.

[0113] Specifically, you can use pre-trained semantic segmentation models (such as U-Net, DeepLab, Mask R-CNN, etc.) to segment the image and separate the person area from the background and other objects. For feature extraction of the segmented person area, you can use traditional feature descriptors (such as HOG, SIFT) or deep learning algorithms (such as convolutional neural network CNN) to extract feature vectors.

[0114] Optionally, different features of the person image set may be obtained through target detection algorithms, face detection and recognition, etc.

[0115] Step S52: based on the features of the different character image sets, the processed data is cropped and different audio and video streams are output.

[0116] In scenes such as speakers and audiences, group discussions, etc., different audio and video streams can be output after reasonable cropping based on the features of different character image sets.

[0117] Furthermore, the features of the character image sets of each group can be collected, and the processed data can be cropped based on the features of the character image sets of each group, such as differentially displaying the discussion screen of each group, and when displaying the group, the sound of other groups can be weakened or blocked according to parameters such as the sound source coordinates, thereby improving the video conferencing effect in multi-speaker scenarios.

[0118] This embodiment outputs different audio and video streams by cropping the processed data based on the features of the different character image sets, and can differentially display different audio and video, thereby improving the video conferencing effect in a multi-speaker scenario.

[0119] For example, to help understand the implementation process of the data processing method of the above embodiment, please refer to Figure 2 , Figure 2 A flow chart of the data processing method of the present application is provided, wherein the data processing system may include a video conferencing platform module, a high-definition audio and video acquisition module, a semantic segmentation acquisition module, a semantic segmentation preprocessing module, a DOA algorithm processing module, a semantic segmentation reprocessing module, and a differential audio and video stream output module. Specifically, the flow chart includes the following steps (1) to (9):

[0120] (1) The video conferencing platform module specifies the video conferencing collection scope, speaker location and other information, and coordinates the overall platform software and hardware process.

[0121] (2) The high-definition audio and video acquisition module is used to capture the images in the video conference scene and transmit the video stream information and audio information back.

[0122] (3) The semantic segmentation acquisition module performs image calibration according to the parameter range given by the video conferencing platform, and only selects the video stream data within the calibration as the sample set. Figure 3 As shown, Figure 3 This is a functional diagram of the data processing system of this application. Figure 3 Indicates the native conference scenario and the scenario with a specified calibration range or speaker position.

[0123] (4) The semantic segmentation preprocessing module uses a semantic segmentation algorithm to process the returned audio and video streams, classify different pixel sets into different categories, distinguish between the conference background image and the participant image, and mark different speakers, sound coordinates and other information. The sample set with less dirty data after preprocessing is returned to the subsequent modules for use. Figure 3 It can be seen that the present invention distinguishes different participants according to the characteristics of the character pixel set through the semantic segmentation algorithm.

[0124] (5) The DOA algorithm processing module applies the DOA algorithm to the sample set processed by the semantic segmentation algorithm to further reduce the impact of error data, obtain more accurate human voice content information and human voice position information, and solidify the speaker position and discussant position. Figure 3 It can be seen that the present invention processes the returned sample set through the DOA algorithm, thereby fixing the identity information, location information, etc. of the participants.

[0125] (6) The semantic segmentation reprocessing module performs behavior recognition processing on the given participant image set features, voice features, coordinate positions and other information. When it is detected that the participant is speaking, the information collection in other coordinate directions is reduced to obtain more accurate image and voice data.

[0126] (7) The semantic segmentation and reprocessing module is used to blur the background of the video conference venue and participants to highlight the image of the characters.

[0127] (8) The differential audio and video stream output module uses independently developed audio and video output logic to optimize the audio and video data processed by the above process and output high-definition audio and video streams.

[0128] (9) The differential audio and video stream output module supports differential screen output. In scenes such as speakers and audiences, or group discussions, it can output different audio and video streams based on the characteristics of different character image sets, and after reasonable cropping, it can differentially display the discussion screen of each group. When displaying the group, it can reduce or block the sound of other groups based on parameters such as the sound source coordinates, thereby improving the video conferencing effect in multi-speaker scenes. Figure 3 It can be seen that the present invention can combine algorithm analysis and platform capabilities to achieve functions such as background blur, speaker image magnification, and group discussion differential display.

[0129] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0130] This application also provides a data processing device, please refer to Figure 4 , the data processing device comprises:

[0131] The data acquisition module 10 is used to collect audio and video data; and to obtain an initial sample set based on the audio and video data;

[0132] A data processing module 20 is used to process the initial sample set by using a semantic segmentation algorithm and a direction arrival algorithm to obtain processed data;

[0133] The data output module 30 is used to output audio and video streams based on the processed data.

[0134] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem that the current audio and video processing technology is easily interfered in a noisy environment. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0135] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.

[0136] Reference below Figure 5 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0137] like Figure 5As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0138] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0139] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem that the current audio and video processing technology is easily interfered in a noisy environment. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and the other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0140] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0141] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0142] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.

[0143] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0144] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.

[0145] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a data processing device, the data processing device: collects audio and video data; obtains an initial sample set based on the audio and video data; processes the initial sample set through a semantic segmentation algorithm and a directional arrival algorithm to obtain processed data; and outputs an audio and video stream based on the processed data.

[0146] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0149] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem that the current audio and video processing technology is easily interfered in a noisy environment. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the data processing method provided by the above-mentioned embodiment, and will not be repeated here.

[0150] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises the following steps: Collect audio and video data; Obtaining an initial sample set based on the audio and video data; Processing the initial sample set by using a semantic segmentation algorithm and a directional arrival algorithm to obtain processed data; Output audio and video streams based on the processed data.

2. The method according to claim 1, characterized in that The step of obtaining an initial sample set based on the audio and video data includes: Receive acquisition parameter calibration instructions and calibrate acquisition parameters; The step of obtaining an initial sample set based on the audio and video data comprises: Based on the acquisition parameters, the audio and video data are calibrated to obtain the initial sample set.

3. The method according to claim 1, characterized in that The step of processing the initial sample set by using a semantic segmentation algorithm and a directional arrival algorithm to obtain processed data comprises: Performing picture pixel set classification and / or information labeling on the initial sample set by using the semantic segmentation algorithm to obtain a second sample set; Processing the second sample set by the directional arrival algorithm to obtain a third sample set; The third sample set is processed for behavior recognition by using the semantic segmentation algorithm to obtain the processed data.

4. The method according to claim 3, characterized in that The step of performing picture pixel set classification and / or information labeling on the initial sample set by the semantic segmentation algorithm to obtain the second sample set comprises: The second sample set is obtained by using the semantic segmentation algorithm to distinguish background images and character images for different pixel sets in the initial sample set, and / or marking at least one of individual information of different speakers, human voice content information, and human voice position information.

5. The method according to claim 4, characterized in that The step of processing the second sample set by using the directional arrival algorithm to obtain a third sample set comprises: The direction arrival algorithm is used to perform error data processing on at least one of the individual information of different speakers, vocal content information, and vocal position information in the second sample, and the speaker position and / or discussant position among the speakers are solidified to obtain the third sample set.

6. The method according to claim 5, characterized in that The step of performing behavior recognition processing on the third sample set by the semantic segmentation algorithm to obtain the processed data comprises: Performing speech behavior recognition on the third sample set; When the speech behavior of the speaker in the third sample set is identified, reducing and / or filtering the audio and video data from the unexpected direction source, and / or enlarging the pixel block of the speaker; Perform image blur processing on at least one of the site, background, and expected outward-direction personnel in the third sample set to obtain the processed data.

7. The method according to claim 6, characterized in that The step of outputting the audio and video stream based on the processed data comprises: Obtaining features of different character image sets based on the initial sample set; Based on the features of the different character image sets, the processed data is cropped and different audio and video streams are output.

8. A data processing device, characterized in that: The device comprises: A data acquisition module, used to collect audio and video data; used to obtain an initial sample set based on the audio and video data; A data processing module, used to process the initial sample set through a semantic segmentation algorithm and a direction arrival algorithm to obtain processed data; A data output module is used to output audio and video streams based on the processed data.

9. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images