Video image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a video image processing method and device, electronic equipment and a storage medium. The video image processing method comprises the following steps: positioning a speaker based on a surrounding sound signal collected for a video call, and obtaining sound positioning information of a target object; based on the sound positioning information, adjusting an image acquisition device to enable a video call image acquired by the image acquisition device to contain the target object; and displaying the video call image acquired by the image acquisition device. Therefore, when a video call is carried out, a speaker can be accurately tracked and shot.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for processing video images. Background Art

[0002] With the development of Internet technology, the application of video calls is becoming more and more common. In some video call scenarios, different users usually make video calls through their own video call devices (e.g., electronic devices equipped with cameras), such as multi-person online conferences.

[0003] In actual applications, the user usually activates a camera (eg, a front camera, a rear camera, or an external camera) and captures the image of the speaker through the camera to achieve a video call.

[0004] However, when the speaker moves or multiple speakers use the same video calling device to talk with others, it is usually difficult to adjust the camera in time to shoot the speaker, and the speaker may not be in the frame. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for processing a video image, so as to accurately track and capture a speaker during a video call.

[0006] On the one hand, an embodiment of the present application provides a method for processing a video image, including:

[0007] Based on the surrounding sound signals collected for the video call, the speaker is located to obtain the sound positioning information of the target object;

[0008] Based on the sound positioning information, adjusting the image acquisition device so that the video call image acquired by the image acquisition device contains the target object;

[0009] Display the video call image captured by the image capture device.

[0010] In one implementation, based on the surrounding sound signals collected for the video call, the speaker is located to obtain the sound positioning information of the target object, including:

[0011] respectively determining initial positioning information of each frequency signal in the surrounding sound signal, wherein the surrounding sound signal includes at least one frequency signal;

[0012] Sound positioning information is determined based on at least one initial positioning information.

[0013] In one implementation, determining sound positioning information according to at least one initial positioning information includes:

[0014] Obtaining frequency coefficients corresponding to at least one frequency signal;

[0015] Adjusting corresponding initial positioning information according to frequency coefficients respectively corresponding to at least one frequency signal;

[0016] The sound positioning information is determined according to the adjusted at least one initial positioning information.

[0017] In one implementation, adjusting the image acquisition device based on the sound positioning information includes:

[0018] If there are multiple image acquisition devices, the image acquisition device currently acquiring the image is switched to the image acquisition device corresponding to the sound positioning information.

[0019] In one implementation, adjusting the image acquisition device based on the sound positioning information includes:

[0020] Rotate the image acquisition device according to the sound positioning information.

[0021] In one embodiment, after displaying the video call image captured by the image capture device, the method further includes:

[0022] Based on the video call image, locate the speaker and obtain image positioning information;

[0023] Obtain target positioning information according to the image positioning information and the sound positioning information;

[0024] Crop the video call image based on the target positioning information;

[0025] Displays a cropped video call image.

[0026] In one implementation, obtaining target positioning information according to image positioning information and sound positioning information includes:

[0027] Determine a sound localization weight corresponding to the sound localization information and an image localization weight corresponding to the image localization information;

[0028] determining a difference between the sound localization weight and the image localization weight;

[0029] If the difference meets the preset threshold condition, the sound positioning information and the image positioning information are weighted and summed based on the sound positioning weight and the image positioning weight to obtain the target positioning information;

[0030] If the difference does not meet the threshold condition, one of the sound positioning information and the image positioning information is determined as the target positioning information.

[0031] In one implementation, before cropping the video call image according to the target positioning information, the method further includes:

[0032] Determine the target object based on surrounding sound signals and video call images;

[0033] Determine the current location information of the target object;

[0034] Based on the current position information, the target positioning information is corrected.

[0035] In one implementation, determining a target object based on surrounding sound signals and a video call image includes:

[0036] Perform face detection on the video call image to obtain the face area;

[0037] If it is determined that there is only one face region, the user in the video call image is determined as the target object;

[0038] If it is determined that there are multiple face regions, speaking action detection is performed for each user in the video call image;

[0039] According to the action detection result, select candidate users with speaking actions from various users;

[0040] If it is determined that there is only one candidate user, the candidate user is determined as the target object;

[0041] Otherwise, select the target user from the candidate users.

[0042] In one implementation, selecting a target object from each candidate user includes:

[0043] According to the area of ​​the face region of each candidate user, the face weight corresponding to each candidate user is determined; the face weight is positively correlated with the area;

[0044] Performing volume detection on each candidate user to determine the volume level of each candidate user;

[0045] Get the volume weight corresponding to the volume level of each candidate user;

[0046] Determine the target weight of each candidate user according to the face weight and volume weight of each candidate user;

[0047] The candidate user corresponding to the maximum value among the target weights is determined as the target object.

[0048] In one implementation, cropping the video call image according to the target positioning information includes:

[0049] Based on the target positioning information, face detection is performed on the video call image to obtain the face area of ​​the target object;

[0050] Crop the video call image based on the target subject's face area.

[0051] In one embodiment, the method further comprises:

[0052] The currently collected surrounding sound signals and target positioning information are input into a volume enhancement model; the volume enhancement model is used to enhance the voice of the target object and suppress other sounds;

[0053] receiving a sound correction signal output by a volume enhancement model;

[0054] Sends audio correction signals to other devices in the video call.

[0055] In one embodiment, the method further comprises:

[0056] Rotate the image acquisition device according to the target positioning information.

[0057] On the one hand, an embodiment of the present application provides a video image processing device, including:

[0058] A positioning unit, used to locate the speaker based on the surrounding sound signals collected for the video call and obtain sound positioning information of the target object;

[0059] An adjustment unit, configured to adjust the image acquisition device based on the sound positioning information so that the video call image acquired by the image acquisition device contains the target object;

[0060] The display unit is used to display the video call image captured by the image capture device.

[0061] In one embodiment, the positioning unit is used to:

[0062] respectively determining initial positioning information of each frequency signal in the surrounding sound signal, wherein the surrounding sound signal includes at least one frequency signal;

[0063] Sound positioning information is determined based on at least one initial positioning information.

[0064] In one embodiment, the positioning unit is used to:

[0065] Obtaining frequency coefficients corresponding to at least one frequency signal;

[0066] Adjusting corresponding initial positioning information according to frequency coefficients respectively corresponding to at least one frequency signal;

[0067] The sound positioning information is determined according to the adjusted at least one initial positioning information.

[0068] In one implementation, the adjustment unit is used to:

[0069] If there are multiple image acquisition devices, the image acquisition device currently acquiring the image is switched to the image acquisition device corresponding to the sound positioning information.

[0070] In one implementation, the adjustment unit is used to:

[0071] Rotate the image acquisition device according to the sound positioning information.

[0072] In one embodiment, the display unit is further used for:

[0073] Based on the video call image, locate the speaker and obtain image positioning information;

[0074] Obtain target positioning information according to the image positioning information and the sound positioning information;

[0075] Crop the video call image based on the target positioning information;

[0076] Displays a cropped video call image.

[0077] In one embodiment, the display unit is further used for:

[0078] Determine a sound localization weight corresponding to the sound localization information and an image localization weight corresponding to the image localization information;

[0079] determining a difference between the sound localization weight and the image localization weight;

[0080] If the difference meets the preset threshold condition, the sound positioning information and the image positioning information are weighted and summed based on the sound positioning weight and the image positioning weight to obtain the target positioning information;

[0081] If the difference does not meet the threshold condition, one of the sound positioning information and the image positioning information is determined as the target positioning information.

[0082] In one embodiment, the display unit is further used for:

[0083] Determine the target object based on surrounding sound signals and video call images;

[0084] Determine the current location information of the target object;

[0085] Based on the current position information, the target positioning information is corrected.

[0086] In one embodiment, the display unit is further used for:

[0087] Perform face detection on the video call image to obtain the face area;

[0088] If it is determined that there is only one face region, the user in the video call image is determined as the target object;

[0089] If it is determined that there are multiple face regions, speaking action detection is performed for each user in the video call image;

[0090] According to the action detection result, select candidate users with speaking actions from various users;

[0091] If it is determined that there is only one candidate user, the candidate user is determined as the target object;

[0092] Otherwise, select the target user from the candidate users.

[0093] In one embodiment, the display unit is further used for:

[0094] According to the area of ​​the face region of each candidate user, the face weight corresponding to each candidate user is determined; the face weight is positively correlated with the area;

[0095] Performing volume detection on each candidate user to determine the volume level of each candidate user;

[0096] Get the volume weight corresponding to the volume level of each candidate user;

[0097] Determine the target weight of each candidate user according to the face weight and volume weight of each candidate user;

[0098] The candidate user corresponding to the maximum value among the target weights is determined as the target object.

[0099] In one embodiment, the display unit is further used for:

[0100] Based on the target positioning information, face detection is performed on the video call image to obtain the face area of ​​the target object;

[0101] Crop the video call image based on the target subject's face area.

[0102] In one embodiment, the display unit is further used for:

[0103] The currently collected surrounding sound signals and target positioning information are input into a volume enhancement model; the volume enhancement model is used to enhance the voice of the target object and suppress other sounds;

[0104] receiving a sound correction signal output by a volume enhancement model;

[0105] Sends audio correction signals to other devices in the video call.

[0106] In one embodiment, the display unit is further used for:

[0107] Rotate the image acquisition device according to the target positioning information.

[0108] On the one hand, an embodiment of the present application provides an electronic device, including:

[0109] Processor; and

[0110] The memory stores computer instructions, wherein the computer instructions are used to enable the processor to execute the steps of the method provided in any of the above-mentioned various optional implementations of processing video images.

[0111] On the one hand, an embodiment of the present application provides a storage medium storing computer instructions, which are used to enable a computer to execute the steps of the method provided in any of the various optional implementations of processing any of the above-mentioned video images.

[0112] The video image processing method of the embodiment of the present application includes locating the speaker based on the surrounding sound signals collected for the video call to obtain the sound positioning information of the target object; adjusting the image acquisition device based on the sound positioning information so that the video call image collected by the image acquisition device contains the target object; and displaying the video call image collected by the image acquisition device. In this way, the speaker can be accurately tracked and photographed during the video call. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0114] Figure 1 It is a flowchart of a method for processing a video image according to an embodiment of the present application.

[0115] Figure 2 It is a structural block diagram of a video image processing device according to an embodiment of the present application.

[0116] Figure 3 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0117] The technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described implementation methods are part of the implementation methods of the present application, rather than all of the implementation methods. Based on the implementation methods in the present application, all other implementation methods obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In addition, the technical features involved in the different implementation methods of the present application described below can be combined with each other as long as they do not conflict with each other.

[0118] First, some terms involved in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0119] Terminal device: can be a mobile terminal, fixed terminal or portable terminal, such as a mobile phone, station, unit, device, multimedia computer, multimedia tablet, Internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system device, personal navigation device, personal digital assistant, audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the terminal device can support any type of interface for the user (such as wearable devices, electric vehicles), etc.

[0120] Server: It can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, big data and artificial intelligence platforms, etc.

[0121] The technical idea of ​​this application is explained below.

[0122] With the development of Internet technology, the application of video calls is becoming more and more common. In some video call scenarios, different users usually make video calls through their respective video call devices.

[0123] In actual applications, users usually start a camera and use it to capture the image of the speaker to make a video call. However, when the speaker moves or multiple speakers use the same video call device to talk to others, the speaker may not be in the frame, resulting in asynchrony between audio and video (i.e., the speaker does not appear in the video).

[0124] In traditional methods, the shooting angle of the camera is usually controlled by manual adjustment to ensure that the speaker is in the picture. However, the video call operation is relatively cumbersome, and it is usually difficult to adjust the camera in time to accurately capture the speaker.

[0125] Furthermore, when the external environment of the video call is noisy or there are a large number of people, it usually causes some interference to the video call, resulting in problems such as poor clarity of the speaker's voice and low sound.

[0126] Based on the defects of the above-mentioned related technologies, the embodiments of the present application provide a method, device, electronic device and storage medium for processing video images, aiming to accurately track and shoot the speaker during a video call.

[0127] A method for processing video images is provided in an embodiment of the present application. The method can be applied to electronic devices. The present application does not limit the type of electronic device. It can be any type of device suitable for implementation, such as a terminal device and a server, etc. The present application will not go into details about this.

[0128] In an application scenario, multiple users conduct a video call through multiple electronic devices. The video image processing method in the embodiment of the present application can be applied to any electronic device in the video call.

[0129] See also Figure 1 FIG. 1 is a flowchart of a method for processing a video image in an embodiment of the present application. Figure 1 The method is described below. The specific implementation process of the method is as follows:

[0130] Step 100: Based on the surrounding sound signals collected for the video call, the speaker is located to obtain sound positioning information of the target object.

[0131] In one implementation, when executing step 100, the following steps may be used:

[0132] S1001: respectively determine initial positioning information of each frequency signal in an ambient sound signal, where the ambient sound signal includes at least one frequency signal.

[0133] In one embodiment, when making a video call, multiple audio collection devices are used to collect surrounding sound signals in real time or periodically, and beamforming is used to perform spatial spectrum estimation (Direction of Arriva, DOA) tracking and positioning of human voice to obtain initial positioning information corresponding to each frequency signal in the surrounding sound signal.

[0134] The surrounding sound signal usually includes one or more frequency signals. The unit of the frequency signal is Hertz (Hz). In practical applications, the frequency signal x is usually located in [100 Hz, 8 KHz].

[0135] Optionally, the audio collection device may be a microphone. To locate the sound through the audio collection device, at least three audio collection devices are usually deployed at different spatial locations to form an audio collection device array, and then the surrounding sound signals can be collected through the audio collection device array.

[0136] As an example, a coordinate system is established with the center position of the front of the tablet as the origin of the coordinate axis, the x-axis in the coordinate system is set with the long side of the tablet, the y-axis in the coordinate system is set with the front-back direction of the tablet (i.e., the thickness direction), and the z-axis in the coordinate system is set with the up-down direction of the tablet (i.e., the width direction). Two microphones, namely, Mic1 and Mic2, are placed in the x-axis direction of the tablet, and two microphones, namely, Mic3 and Mic4, are placed in the y-axis direction of the tablet.

[0137] The surrounding sound signal may be a digital signal obtained by converting the directly collected user voice (analog signal). It should be noted that the surrounding sound signal is the currently collected sound signal, so the surrounding sound signal may be updated in real time or periodically.

[0138] Optionally, when the beamforming method is used for sound localization, a spatial filtering algorithm may be used for sound localization. For example, the spatial filtering algorithm may use Minimum Variance Distortionless Response (MVDR).

[0139] In this way, sound can be collected through the audio collection device array, and the speaker can be preliminarily located through the currently collected sound signal.

[0140] S1002: Determine sound positioning information according to at least one initial positioning information.

[0141] In one implementation, when executing S1002, the following steps may be adopted:

[0142] S1002-1: Obtain frequency coefficients corresponding to at least one frequency signal.

[0143] In one implementation, the frequency coefficient includes at least one of a weight coefficient and a time domain coefficient. As an example, the weight coefficient and the time domain coefficient respectively corresponding to at least one frequency signal are obtained.

[0144] In one implementation, weight coefficients corresponding to each frequency signal are pre-set, weight coefficient distribution of each frequency signal is obtained, and time domain coefficients corresponding to each frequency signal in different signal time periods (e.g., 100ms) are pre-set. The time domain coefficients of different signal time periods may be the same or different.

[0145] In practical applications, the weight coefficient and the time domain coefficient can be set according to the actual application scenario and are not limited here.

[0146] S1002-2: Adjust corresponding initial positioning information according to frequency coefficients corresponding to at least one frequency signal.

[0147] In one implementation, corresponding initial positioning information is adjusted based on weight coefficients and time domain coefficients respectively corresponding to at least one frequency signal.

[0148] Among them, the adjusted initial positioning information is positively correlated with the initial positioning information, the weight coefficient and the time domain coefficient.

[0149] As an example, for the target initial positioning information in each initial positioning information, the product of the target initial positioning information, the corresponding weight coefficient and the time domain coefficient is determined to obtain the adjusted initial positioning information, wherein the target initial positioning information is any one of the initial positioning information.

[0150] Optionally, the following formula may be used to determine the adjusted initial positioning information:

[0151] Degree1=A1_Fx*Degree_Fx_Average*T1;

[0152] Among them, Degree1 is the adjusted initial positioning information, A1_Fx is the weight coefficient of the frequency signal x, Degree_Fx_Average is the initial positioning information corresponding to the frequency signal x, and T1 is the time domain coefficient.

[0153] Optionally, the initial positioning information and the above-mentioned sound positioning information are used to indicate the physical space position, which may be three-dimensional coordinate information or an angle.

[0154] As an example, the direction directly opposite to the center of the tablet is 0 degrees, and the direction directly opposite to the center of the tablet is 180 degrees. If user A is directly opposite to the center of the tablet, the initial positioning information of user A is determined to be 0 degrees. If user B is directly opposite to the center of the tablet, the initial positioning information of user B is determined to be 180 degrees.

[0155] S1002-3: Determine sound positioning information according to the adjusted at least one initial positioning information.

[0156] In an implementation manner, if there are multiple initial positioning information, the volume corresponding to each initial positioning information may be determined, and the initial positioning information corresponding to the maximum volume among the volumes may be determined as the sound positioning information.

[0157] In practical applications, any other method may be used to determine the sound positioning information, which is not limited here.

[0158] In this way, the location of the person speaking in the video call can be determined by sound.

[0159] Step 101: Based on the sound localization information, adjust the image acquisition device so that the video call image acquired by the image acquisition device contains the target object.

[0160] In one implementation, if there are multiple image acquisition devices, the image acquisition device currently acquiring the image is switched to the image acquisition device corresponding to the sound positioning information.

[0161] In one implementation, if the image acquisition device is rotatable, the image acquisition device is rotated according to the sound positioning information.

[0162] Among them, the image acquisition device is used to acquire video call images of the video call.

[0163] Step 102: Display the video call image captured by the image capture device.

[0164] Furthermore, the video call image may be cropped and displayed to achieve focus on the target object.

[0165] In one implementation, when cropping and displaying the video call image, the following steps may be used:

[0166] S1021: Based on the video call image, locate the speaker and obtain image positioning information.

[0167] In one embodiment, during a video call, an image acquisition device is used to capture images of the surrounding environment to obtain a captured video call image, and face detection is performed on the video call image. Based on the face detection result, face positioning information is obtained, and based on the face positioning information, image positioning information is obtained.

[0168] The face positioning information may be the coordinates or angle of the center position of the face. For example, the face positioning information may be the two-dimensional coordinate information of the speaker's nose in the video call image, or the three-dimensional coordinate information in the actual space. Optionally, the image positioning information may be the angle of the speaker's nose.

[0169] In this way, the video call image can be used to re-position the speaker, and a re-positioning result for the speaker can be obtained, so that multiple positioning results can be obtained through different positioning methods.

[0170] S1022: Obtain target positioning information according to the image positioning information and the sound positioning information.

[0171] S1023: Crop the video call image according to the target positioning information.

[0172] In one implementation, face detection is performed on the video call image based on the target positioning information to obtain the target object face area; and the video call image is cropped according to the target object face area.

[0173] In this way, image cropping can be used to ensure that the tracked target object is always located in the center, thereby achieving audio and video synchronization.

[0174] S1024: Display the cropped video call image.

[0175] Specifically, the cropped video call image is displayed through each video call device of the video call.

[0176] In this way, the cropped video call image can be displayed to each user of the video call through the video call device.

[0177] In one implementation, when executing S1022, the following steps may be performed:

[0178] S1022-1: Determine a sound localization weight corresponding to the sound localization information and an image localization weight corresponding to the image localization information.

[0179] In one implementation, the sound localization weight may be determined based on environmental information of the surrounding environment (eg, whether it is quiet, whether there is reverberation, the number of people on the video call, and the meeting scene, etc.).

[0180] In actual applications, the sound localization weight and the image localization weight can be set according to the actual application scenario and are not limited here.

[0181] S1022-2: Determine the difference between the sound localization weight and the image localization weight.

[0182] S1022-3: If the difference meets the preset threshold condition, the sound positioning information and the image positioning information are weightedly summed based on the sound positioning weight and the image positioning weight to obtain the target positioning information.

[0183] In one implementation, the absolute value of the difference is determined. If the absolute value is lower than a difference threshold, the sound positioning information and the image positioning information are weightedly summed based on the sound positioning weight and the image positioning weight to obtain the target positioning information.

[0184] In one implementation, the following formula may be used to determine the target location information:

[0185] DE=A1*Degree1+A2*Degree2;

[0186] Among them, DE is the target positioning information, A1 is the sound positioning weight, A2 is the image positioning weight, Degree1 is the sound positioning information, and Degree2 is the image positioning information.

[0187] In actual applications, the threshold condition and the difference threshold can be set according to the actual application scenario. For example, the difference threshold is 0, which is not limited here.

[0188] S1022-4: If the difference does not meet the threshold condition, one of the sound positioning information and the image positioning information is determined as the target positioning information.

[0189] In one implementation, when the difference does not meet a preset threshold condition, if the sound localization weight is higher than the image localization weight, the sound localization information is determined as the target localization information; otherwise, the image localization information is determined as the target localization information.

[0190] It should be noted that the sound localization weight and the image localization weight represent the reliability of positioning. Therefore, if the sound localization weight is close to the image localization weight, the speaker is located by combining the sound localization information and the image localization information. Otherwise, the localization information corresponding to the larger weight is selected as the target localization information.

[0191] In this way, multiple positioning results are obtained through different positioning methods, and comprehensive positioning is performed in combination with each positioning result, which greatly improves the accuracy of speaker positioning.

[0192] Furthermore, since there are usually multiple interference sources of noise in actual application scenarios (such as air conditioners, televisions, and sounds of non-target objects, etc.), in order to reduce the impact of interference, the target positioning information can also be corrected.

[0193] In one implementation, when correcting the target positioning information, the following steps may be used:

[0194] S1022-41: Determine the target object based on surrounding sound signals and video call images.

[0195] In one embodiment, after adjusting the image acquisition device, the current surrounding sound signals are acquired in real time or periodically, and the current video call image is acquired in real time or periodically through the adjusted image acquisition device to obtain new surrounding sound signals and video call images, and based on the new surrounding sound signals and video call images, the target object is screened out from multiple speakers.

[0196] It should be noted that in step 100, only preliminary positioning is performed through sound to estimate the location of the target object, but the target object is not determined. If there are multiple speakers, the target speaker, that is, the target object, needs to be determined from the multiple speakers.

[0197] S1022-42: Determine the current location information of the target object.

[0198] S1022-43: Based on the current position information, correct the target positioning information.

[0199] In one implementation, when executing S1022-41, the following steps may be adopted:

[0200] S1022-411: Perform face detection on the video call image to obtain the face area.

[0201] S1022-412: If it is determined that there is only one face area, the user in the video call image is determined as the target object.

[0202] It should be noted that usually one face region corresponds to one user. If there is only one user (ie, only one face region), it means that the user is the target object.

[0203] S1022-413: If it is determined that there are multiple face areas, speaking action detection is performed for each user in the video call image.

[0204] Among them, speaking action detection refers to detecting whether the user has speaking (ie lip reading) actions, that is, judging whether the user has spoken by the opening and closing and shape of the user's mouth.

[0205] S1022-414: Based on the action detection result, select candidate users who have speaking actions from among the users.

[0206] It should be noted that if there is a speaking action, it means that the user has made a speaking behavior and may be the target object. If there is no speaking action, it means that the user is definitely not the target object.

[0207] S1022-415: If it is determined that there is only one candidate user, the candidate user is determined as the target object; otherwise, the target object is screened out from the candidate users.

[0208] This is because if only one user speaks, it means that the user is the speaker. Otherwise, it is necessary to further screen the target object based on the face area and volume.

[0209] In one implementation, the face weight corresponding to each candidate user is determined according to the area of ​​the face region of each candidate user; the face weight is positively correlated with the area; the volume detection is performed on each candidate user to determine the volume level of each candidate user; the volume weight set corresponding to the volume level of each candidate user is obtained; the target weight of each candidate user is determined according to the face weight and volume weight of each candidate user; and the candidate user corresponding to the maximum value among the target weights is determined as the target object.

[0210] The face weight is positively correlated with the area. As an example, the face weights of three candidate users are B1, B2, and B3, respectively.

[0211] In one implementation, the surrounding sound signals and the video call image are input into a volume detection model to obtain the volume level of each candidate user output by the volume detection model. The volume detection model may be constructed based on a convolutional neural network (CNN).

[0212] Furthermore, model training may be performed in advance based on training sample data to obtain a trained volume detection model.

[0213] Optionally, the training sample data may include various corpora spoken by users of various types (e.g., different genders, different ages) in various languages, as well as various images of the user's speaking movements. Optionally, in order to ensure the detection performance of the audio detection model, more than 1000 hours of training sample data may be selected for model training.

[0214] In one implementation, the volume level includes: no talking, speaking quietly, and normal talking. As an example, the volume range of normal talking is 60-90 dB SPL (decibels). The volume range of speaking quietly is 40-50 dB SPL.

[0215] In actual applications, volume levels can be divided according to actual application scenarios and are not limited here.

[0216] As an example, the volume weights of three candidate users are C1, C2, and C3, respectively.

[0217] Among them, the target weight is positively correlated with the face weight and the volume weight.

[0218] In one implementation, when determining the target weight Qi of candidate user i, the following steps may be used:

[0219] Qi = Bi * Ci;

[0220] Wherein, i is the serial number of the candidate user, which is a natural number, Bi is the face weight of the candidate user i, and Ci is the volume weight of each candidate user i.

[0221] Furthermore, the volume of the target object can be enhanced and the volume of other sounds can be suppressed.

[0222] Specifically, the currently collected surrounding sound signals and target positioning information are input into a volume enhancement model; the volume enhancement model is used to enhance the voice of the target object and suppress other sounds; a sound correction signal output by the volume enhancement model is received; and the sound correction signal is sent to other devices in the video call.

[0223] As an example, the surrounding sound signal contains a first frequency signal and a second frequency signal, the frequency signal corresponding to the target object is the second frequency signal, and the angle (i.e., target positioning information) is 60 degrees, then the surrounding sound signal and the target positioning information are input into the volume enhancement model. The volume enhancement model determines that the frequency signal of the target object is the second frequency signal based on the target positioning information of 60 degrees, then enhances the second frequency signal and suppresses the first frequency signal to obtain an output sound correction signal (including the enhanced second frequency signal and the suppressed first frequency signal), and then conducts a video call based on the sound correction signal.

[0224] Furthermore, before adjusting the volume, the volume correction sample data is used to perform model training to obtain a trained volume enhancement model.

[0225] It should be noted that the volume enhancement model can be trained and tested in a variety of video call environments, for example, video call environments can include home, conference rooms, and offices. Since there are many sound sources, it will affect the positioning accuracy and thus reduce the accuracy of sound correction. Therefore, the sound sources are usually set to 1-4. The target positioning information can be an angle, and the angle range of the sound source can be 0-360 degrees, so that voice tracking and enhancement can be achieved within a 360-degree space. The angle accuracy can be 1 degree, and each angle can be marked separately.

[0226] In the embodiment of the present application, all sounds other than the target object are determined to be noise. Therefore, during the training and application of the volume enhancement model, only the sound of the target object is enhanced, and other sounds are suppressed.

[0227] In practical applications, other sounds are usually suppressed by at least 20db (decibel).

[0228] This way, the sound of the target object can be enhanced, as well as noise suppression.

[0229] Furthermore, if the image acquisition device is rotatable, the image acquisition device can be further adjusted. Specifically, the image acquisition device is rotated according to the target positioning information.

[0230] In one implementation, the shooting angle of the image acquisition device is adjusted to the angle pointed to by the corrected target positioning information, so that the target object is located in the middle of the video call image.

[0231] In this way, the accuracy of the target object positioning and the shooting angle can be further improved. In one application scenario, positioning is performed by the embodiment of the present application to obtain a first positioning result, and positioning is performed by a traditional method to obtain a second positioning result. The two are compared, and the first positioning result is 5 degrees higher than the second positioning result.

[0232] In the embodiment of the present application, multiple audio acquisition devices are reasonably arranged, and surrounding sound signals are collected through the arranged audio acquisition device array, and preliminary positioning is performed based on the surrounding sound signals to obtain sound positioning information. According to the positioning result, the image acquisition device is automatically adjusted to track and shoot the target object so that the target object can be captured. There is no need to manually start, switch and adjust the image acquisition device, which reduces the tedious operation of video image acquisition and brings convenience to users.

[0233] Furthermore, multiple positioning results are obtained through sound positioning and image positioning, and the target positioning information is comprehensively determined based on the multiple positioning results, further improving the positioning accuracy. After determining the target object by combining the surrounding sound signals and the video call image, face detection can be performed on the video call image, and the target positioning information can be corrected according to the face detection results, thereby further improving the positioning accuracy.

[0234] Furthermore, according to the target positioning information, the image is cropped so that the target object is located in the middle of the image, realizing the automatic focus tracking and audio and video synchronization functions for the target object, and ensuring the logic between the audio and video call images. In addition, the speech of the target object is enhanced through the speech enhancement model, and the noise is suppressed, reducing the interference of noisy environments and multiple people talking, realizing the speech tracking and enhancement of the target object within a 360-degree range, and improving the clarity and speech quality of the speaker.

[0235] Based on the same inventive concept, a video image processing device is also provided in the embodiment of the present application. Since the principle of solving the problem by the above device and equipment is similar to that of a video image processing method, the implementation of the above device can refer to the implementation of the method, and the repeated parts will not be repeated. The device can be applied to electronic devices. The present application does not limit the type of electronic devices. It can be any type of device suitable for implementation, such as a smart phone, a tablet computer, etc., and the present application will not repeat it.

[0236] See also Figure 2 FIG. 1 is a block diagram of a video image processing device in an embodiment of the present application. In some implementations, the video image processing device in the example of the present application includes:

[0237] A positioning unit 201 is used to locate the speaker based on the surrounding sound signals collected for the video call and obtain sound positioning information of the target object;

[0238] An adjustment unit 202 is used to adjust the image acquisition device based on the sound positioning information so that the video call image acquired by the image acquisition device contains the target object;

[0239] The display unit 203 is used to display the video call image captured by the image capture device.

[0240] In one implementation, the positioning unit 201 is used to:

[0241] respectively determining initial positioning information of each frequency signal in the surrounding sound signal, wherein the surrounding sound signal includes at least one frequency signal;

[0242] Sound positioning information is determined based on at least one initial positioning information.

[0243] In one implementation, the positioning unit 201 is used to:

[0244] Obtaining frequency coefficients corresponding to at least one frequency signal;

[0245] Adjusting corresponding initial positioning information according to frequency coefficients respectively corresponding to at least one frequency signal;

[0246] The sound positioning information is determined according to the adjusted at least one initial positioning information.

[0247] In one implementation, the adjustment unit 202 is used to:

[0248] If there are multiple image acquisition devices, the image acquisition device currently acquiring the image is switched to the image acquisition device corresponding to the sound positioning information.

[0249] In one implementation, the adjustment unit 202 is used to:

[0250] Rotate the image acquisition device according to the sound positioning information.

[0251] In one implementation, the display unit 203 is further used for:

[0252] Based on the video call image, locate the speaker and obtain image positioning information;

[0253] Obtain target positioning information according to the image positioning information and the sound positioning information;

[0254] Crop the video call image based on the target positioning information;

[0255] Displays a cropped video call image.

[0256] In one implementation, the display unit 203 is further used for:

[0257] Determine a sound localization weight corresponding to the sound localization information and an image localization weight corresponding to the image localization information;

[0258] determining a difference between the sound localization weight and the image localization weight;

[0259] If the difference meets the preset threshold condition, the sound positioning information and the image positioning information are weighted and summed based on the sound positioning weight and the image positioning weight to obtain the target positioning information;

[0260] If the difference does not meet the threshold condition, one of the sound positioning information and the image positioning information is determined as the target positioning information.

[0261] In one implementation, the display unit 203 is further used for:

[0262] Determine the target object based on surrounding sound signals and video call images;

[0263] Determine the current location information of the target object;

[0264] Based on the current position information, the target positioning information is corrected.

[0265] In one implementation, the display unit 203 is further used for:

[0266] Perform face detection on the video call image to obtain the face area;

[0267] If it is determined that there is only one face region, the user in the video call image is determined as the target object;

[0268] If it is determined that there are multiple face regions, speaking action detection is performed for each user in the video call image;

[0269] According to the action detection result, select candidate users with speaking actions from various users;

[0270] If it is determined that there is only one candidate user, the candidate user is determined as the target object;

[0271] Otherwise, select the target user from the candidate users.

[0272] In one implementation, the display unit 203 is further used for:

[0273] According to the area of ​​the face region of each candidate user, the face weight corresponding to each candidate user is determined; the face weight is positively correlated with the area;

[0274] Performing volume detection on each candidate user to determine the volume level of each candidate user;

[0275] Get the volume weight corresponding to the volume level of each candidate user;

[0276] Determine the target weight of each candidate user according to the face weight and volume weight of each candidate user;

[0277] The candidate user corresponding to the maximum value among the target weights is determined as the target object.

[0278] In one implementation, the display unit 203 is further used for:

[0279] Based on the target positioning information, face detection is performed on the video call image to obtain the face area of ​​the target object;

[0280] Crop the video call image based on the target subject's face area.

[0281] In one implementation, the display unit 203 is further used for:

[0282] The currently collected surrounding sound signals and target positioning information are input into a volume enhancement model; the volume enhancement model is used to enhance the voice of the target object and suppress other sounds;

[0283] receiving a sound correction signal output by a volume enhancement model;

[0284] Sends audio correction signals to other devices in the video call.

[0285] In one implementation, the display unit 203 is further used for:

[0286] Rotate the image acquisition device according to the target positioning information.

[0287] The video image processing method of the embodiment of the present application includes locating the speaker based on the surrounding sound signals collected for the video call to obtain the sound positioning information of the target object; adjusting the image acquisition device based on the sound positioning information so that the video call image collected by the image acquisition device contains the target object; and displaying the video call image collected by the image acquisition device. In this way, the speaker can be accurately tracked and photographed during the video call.

[0288] An embodiment of the present application provides an electronic device, including:

[0289] Processor; and

[0290] The memory stores computer instructions, where the computer instructions are used to enable the processor to execute the method of any of the above embodiments.

[0291] In an embodiment of the present application, a storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a computer to execute a method in any of the above-mentioned embodiments.

[0292] Figure 3 FIG. 3 shows a schematic diagram of the structure of an electronic device 3000. Figure 3 As shown, the electronic device 3000 includes: a processor 3010 and a memory 3020 , and optionally, may also include a power supply 3030 , a display unit 3040 , and an input unit 3050 .

[0293] The processor 3010 is the control center of the electronic device 3000. It connects various components using various interfaces and lines, and performs various functions of the electronic device 3000 by running or executing software programs and / or data stored in the memory 3020, thereby monitoring the electronic device 3000 as a whole.

[0294] In the embodiment of the present application, the processor 3010 executes the various steps in the above embodiment when calling the computer program stored in the memory 3020.

[0295] Optionally, the processor 3010 may include one or more processing units; preferably, the processor 3010 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 3010. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips.

[0296] The memory 3020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, various applications, etc.; the data storage area may store data created according to the use of the electronic device 3000, etc. In addition, the memory 3020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices, etc.

[0297] The electronic device 3000 also includes a power source 3030 (such as a battery) for supplying power to various components. The power source can be logically connected to the processor 3010 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0298] The display unit 3040 may be used to display information input by the user or information provided to the user and various menus of the electronic device 3000, and in the embodiment of the present application, is mainly used to display the display interface of each application in the electronic device 3000 and objects such as text and pictures displayed in the display interface. The display unit 3040 may include a display panel 3041. The display panel 3041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0299] The input unit 3050 may be used to receive information such as numbers or characters input by the user. The input unit 3050 may include a touch panel 3051 and other input devices 3052. The touch panel 3051, also known as a touch screen, may collect touch operations of the user on or near it (such as operations of the user using a finger, a stylus, or any other suitable object or accessory on or near the touch panel 3051).

[0300] Specifically, the touch panel 3051 can detect the user's touch operation, detect the signal brought by the touch operation, convert these signals into touch point coordinates, send them to the processor 3010, and receive and execute the command sent by the processor 3010. In addition, the touch panel 3051 can be implemented in various types such as resistive, capacitive, infrared and surface acoustic wave. Other input devices 3052 can include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0301] Of course, the touch panel 3051 can cover the display panel 3041. When the touch panel 3051 detects a touch operation on or near it, it is transmitted to the processor 3010 to determine the type of touch event. Then the processor 3010 provides corresponding visual output on the display panel 3041 according to the type of touch event. Figure 3 In the embodiment, the touch panel 3051 and the display panel 3041 are used as two independent components to realize the input and output functions of the electronic device 3000, but in some embodiments, the touch panel 3051 and the display panel 3041 can be integrated to realize the input and output functions of the electronic device 3000.

[0302] The electronic device 3000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific applications, the electronic device 3000 may also include other components such as a camera. Since these components are not the key components used in the embodiments of the present application, Figure 3 It is not shown and will not be described in detail.

[0303] Those skilled in the art will understand that Figure 3 The electronic device is merely an example and does not limit the electronic device, and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0304] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0305] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the embodiments. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this application.

Claims

1. A method for processing a video image, characterized in that: include: Based on the surrounding sound signals collected for the video call, the speaker is located to obtain the sound positioning information of the target object; Based on the sound positioning information, adjusting an image acquisition device so that the video call image acquired by the image acquisition device contains the target object; The video call image captured by the image capture device is displayed.

2. The method according to claim 1, characterized in that The method of locating the speaker based on the surrounding sound signals collected for the video call and obtaining the sound positioning information of the target object includes: respectively determining initial positioning information of each frequency signal in the ambient sound signal, wherein the ambient sound signal includes at least one frequency signal; The sound positioning information is determined according to at least one initial positioning information.

3. The method according to claim 2, characterized in that The step of determining the sound positioning information according to at least one initial positioning information comprises: Obtaining frequency coefficients respectively corresponding to the at least one frequency signal; Adjusting corresponding initial positioning information according to frequency coefficients respectively corresponding to the at least one frequency signal; The sound positioning information is determined according to the adjusted at least one initial positioning information.

4. The method according to any one of claims 1 to 3, characterized in that: The adjusting the image acquisition device based on the sound positioning information includes: If there are multiple image acquisition devices, the image acquisition device currently acquiring the image is switched to the image acquisition device corresponding to the sound positioning information.

5. The method according to any one of claims 1 to 3, characterized in that: The adjusting the image acquisition device based on the sound positioning information includes: The image acquisition device is rotated according to the sound positioning information.

6. The method according to any one of claims 1 to 3, characterized in that: After displaying the video call image captured by the image capture device, the method further includes: Based on the video call image, locate the speaker and obtain image location information; Obtaining target positioning information according to the image positioning information and the sound positioning information; Cropping the video call image according to the target positioning information; Displays a cropped video call image.

7. The method according to claim 6, characterized in that The obtaining target positioning information according to the image positioning information and the sound positioning information includes: Determine a sound positioning weight corresponding to the sound positioning information and an image positioning weight corresponding to the image positioning information; determining a difference between the sound localization weight and the image localization weight; If the difference satisfies a preset threshold condition, performing weighted summation on the sound positioning information and the image positioning information based on the sound positioning weight and the image positioning weight to obtain the target positioning information; If the difference does not meet the threshold condition, one of the sound positioning information and the image positioning information is determined as the target positioning information.

8. The method according to claim 6, characterized in that Before cropping the video call image according to the target positioning information, the method further includes: Determining the target object according to the surrounding sound signal and the video call image; Determine the current location information of the target object; Based on the current position information, the target positioning information is corrected.

9. The method according to claim 8, characterized in that The determining the target object according to the surrounding sound signal and the video call image includes: Performing face detection on the video call image to obtain a face area; If it is determined that there is only one face area, the user in the video call image is determined as the target object; If it is determined that there are multiple face regions, performing speaking action detection for each user in the video call image respectively; According to the action detection result, select candidate users with speaking actions from various users; If it is determined that there is only one candidate user, the candidate user is determined as the target object; Otherwise, the target object is screened out from the candidate users.

10. The method according to claim 9, characterized in that The step of screening out the target object from the candidate users includes: Determine the face weight corresponding to each candidate user according to the area of ​​the face region of each candidate user; the face weight is positively correlated with the area; Performing volume detection on each candidate user to determine the volume level of each candidate user; Get the volume weight corresponding to the volume level of each candidate user; Determine the target weight of each candidate user according to the face weight and volume weight of each candidate user; The candidate user corresponding to the maximum value among the target weights is determined as the target object.

11. The method according to claim 6, characterized in that The step of cropping the video call image according to the target positioning information includes: Based on the target positioning information, face detection is performed on the video call image to obtain a face area of ​​the target object; The video call image is cropped according to the face area of ​​the target object.

12. The method according to claim 6, characterized in that The method further comprises: Inputting the currently collected surrounding sound signals and the target positioning information into a volume enhancement model; the volume enhancement model is used to enhance the voice of the target object and suppress other sounds; receiving a sound correction signal output by the volume enhancement model; The sound correction signal is sent to other devices of the video call.

13. The method according to claim 6, characterized in that The method further comprises: The image acquisition device is rotated according to the target positioning information.

14. A video image processing device, characterized in that: The device comprises: A positioning unit, used to locate the speaker based on the surrounding sound signals collected for the video call and obtain sound positioning information of the target object; An adjustment unit, configured to adjust an image acquisition device based on the sound positioning information so that the video call image acquired by the image acquisition device includes the target object; A display unit is used to display the video call image captured by the image capture device.

15. An electronic device, characterized in that: include: processor; as well as A memory storing computer instructions, wherein the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 13.

16. A storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 13.

Images