Video processing method, electronic device, and storage medium

Abstract

The present disclosure provides a video processing method, an electronic device and a storage medium, which can be applied to the technical field of video communication. The video processing method is applied to a conference device, the conference device comprising a first camera and a second camera with a variable view angle. The method comprises: acquiring a first video image captured by the first camera and a second video image captured by the second camera; and performing lip movement recognition based on the first video image and the second video image to determine a recognition result. The focal length of the second camera is greater than the focal length of the first camera.

Description

Technical Field

[0001] This disclosure relates to the field of video communication technology, and more specifically to a video processing method, electronic device, and storage medium. Background Technology

[0002] With the development of internet video communication technology, it has been widely used in conference scenarios. When using a camera to capture close-up shots of speakers, it is necessary to accurately locate the speakers.

[0003] In related examples, sound source localization is typically used to locate the speaker. However, it is difficult to accurately locate the true speaker when there are multiple people in the same direction. While lip movement recognition can locate the speaker relatively accurately, its accuracy is lower for speakers who are far from the camera due to the poor image clarity captured by the camera. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a video processing method, electronic device and storage medium for improving the accuracy of lip movement recognition.

[0005] According to a first aspect of this disclosure, a video processing method is provided, which is applied to a conferencing device, the conferencing device including: a first camera and a second camera with a variable viewing angle, the method including: acquiring a first video image captured by the first camera and a second video image captured by the second camera; performing lip movement recognition based on the first video image and the second video image, and determining a recognition result; wherein the focal length of the second camera is greater than the focal length of the first camera.

[0006] A second aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0007] A third aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0008] According to embodiments of this disclosure, the method uses a first camera and a second camera with a variable viewing angle to acquire a first video image and a second video image, respectively. Then, it simultaneously uses both the first and second video images for lip movement recognition. Compared to traditional methods that use only one camera for lip movement recognition, this method utilizes the second camera to achieve long-distance lip movement recognition, significantly improving the accuracy of lip movement recognition. For example, when the speaker is far from the first camera, the image captured by the first camera may be unclear. In this case, the second camera can acquire a clear second video image for lip movement recognition, thereby improving the accuracy of lip movement recognition. Attached Figure Description

[0009] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0010] Figure 1A This illustration schematically depicts an application scenario of a video processing method, electronic device, and storage medium according to embodiments of the present disclosure.

[0011] Figure 1B This illustration schematically depicts an application scenario of a video processing method, electronic device, and storage medium according to another embodiment of the present disclosure;

[0012] Figure 2 A flowchart illustrating a video processing method according to an embodiment of the present disclosure is shown schematically.

[0013] Figure 3 This illustration schematically shows an object recognition diagram of the main camera image according to an embodiment of the present disclosure;

[0014] Figure 4 A schematic diagram illustrating the main camera view and the secondary camera view according to an embodiment of the present disclosure is shown;

[0015] Figure 5 The illustration shows a schematic diagram of lip movement recognition based on the main camera image and the secondary camera image according to an embodiment of the present disclosure;

[0016] Figure 6 The illustration shows a schematic diagram of lip movement recognition based on the main camera image, the secondary camera image, and the sound source location information according to an embodiment of the present disclosure;

[0017] Figure 7 The illustration shows a schematic diagram of determining the position of a virtual candidate object in the main camera frame based on the location of a sound source according to an embodiment of the present disclosure;

[0018] Figure 8The illustration shows a schematic diagram of lip movement recognition based on the main camera image, the secondary camera image, sound source location information, and voiceprint according to an embodiment of the present disclosure;

[0019] Figure 9 This schematic diagram illustrates the determination of the true speaker from multiple speakers based on sound source location information according to an embodiment of the present disclosure; and

[0020] Figure 10 A block diagram schematically illustrates an electronic device suitable for implementing a video processing method according to an embodiment of the present disclosure. Detailed Implementation

[0021] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0024] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0025] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0026] In conference recording scenarios, lip movement recognition can accurately locate speakers to capture close-up shots. However, since lip movement recognition is an action detection technology based on facial image sequences, it heavily relies on image clarity. In real-world conference settings, participants are positioned both close to and far from the camera. When participants are far from the camera, the image captured by the camera is less clear, affecting the accuracy of lip movement recognition.

[0027] In view of this, embodiments of the present disclosure provide a video processing method. This method utilizes a first camera and a second camera with a variable viewing angle to capture a first video image and a second video image, respectively. Then, it simultaneously uses both the first and second video images for lip movement recognition. Compared to traditional methods that use only one camera for lip movement recognition, this method utilizes the second camera to assist in achieving long-distance lip movement recognition, significantly improving the accuracy of lip movement recognition. For example, when the speaker is far from the first camera, the image captured by the first camera may be unclear. In this case, the second camera can capture a clear second video image for lip movement recognition, thereby improving the accuracy of lip movement recognition.

[0028] Figure 1A The illustration shows an application scenario of a video processing method, electronic device, and storage medium according to embodiments of the present disclosure.

[0029] like Figure 1AAs shown, application scenario 100A according to this embodiment may include multiple participants 101, a first camera 102, a second camera 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first camera 102, the second camera 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc. The first camera 102 and the second camera 103 are used to capture video images of the participants 101. The shooting angle of the second camera 103 can be determined based on the location of the sound source among the multiple participants 101, or based on the relative positional relationship between the multiple participants 101 and the first camera 102. For example, the shooting angle of the second camera 103 can be aimed at participants far from the first camera 102.

[0030] Server 105 can be a server that provides various services, such as performing lip movement recognition on video images captured by the user using the first camera 102 and the second camera 103 to obtain recognition results (for example only). The recognition results are then fed back to other terminal devices, such as displaying a close-up image of the speaker.

[0031] It should be noted that the video processing method provided in this embodiment can generally be executed by server 105. Alternatively, the video processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first camera 102, the second camera 103, and / or server 105.

[0032] It should be understood that Figure 1A The number of cameras, networks, and servers shown is merely illustrative. Any number of cameras, networks, and servers can be included depending on implementation needs.

[0033] Figure 1B The illustration schematically depicts an application scenario of a video processing method, electronic device, and storage medium according to another embodiment of the present disclosure.

[0034] like Figure 1B As shown, application scenario 100B according to this embodiment may include multiple participants 101 and a conference device 106. The conference device 106 may include a first camera 102 and a second camera 103 with a variable viewing angle. The first camera 102 can capture video images of all participants, and the second camera 103 is used to capture video images of participants 101 that are far away from the first camera 102.

[0035] The video processing method provided in this embodiment can be executed by the conferencing device 106. For example, the conferencing device 106 can acquire a first video image captured by the first camera 102 and a second video image captured by the second camera 103, and perform lip movement recognition based on the first video image and the second video image.

[0036] In some embodiments, when participant 101 is speaking, but participant 101 is far away from the conference device 106, causing the lip movement recognition to fail to identify participant 101, the conference device 106 can be used to control the shooting angle of the second camera 103 to be aimed at participant 101 to capture a second video image.

[0037] The following will be based on Figures 1A-1B The described scene, through Figures 2-9 The video processing method of the disclosed embodiments will be described in detail.

[0038] Figure 2 A flowchart illustrating a video processing method according to an embodiment of the present disclosure is shown schematically.

[0039] like Figure 2 As shown, the video processing method of this embodiment 200 includes S210 to S220.

[0040] S210, acquire the first video image captured by the first camera and the second video image captured by the second camera.

[0041] S220, perform lip movement recognition based on the first video image and the second video image, and determine the recognition result.

[0042] In embodiments of this disclosure, user consent or authorization can be obtained before acquiring video images from each participant. For example, before S210, a request to obtain user information can be sent to each participant. If the user consents or authorizes the acquisition of user information, S210 is executed.

[0043] According to embodiments of this disclosure, the method is applied to a conferencing device, which includes a first camera and a second camera with a variable viewing angle. The focal length of the second camera is greater than that of the first camera.

[0044] For example: The first camera can serve as the main camera, using a wide-angle lens, to capture the first video images of all participants. Each frame in the first video image can be used as the main camera's view. The second camera can serve as the secondary camera, using a telephoto lens, to capture the second video images of participants speaking during the meeting. Each frame in the second video image can be used as the secondary camera's view.

[0045] The second camera is a variable-angle camera. Its angle can be determined based on the speaker's position to capture close-up shots. Since the speaker's position is often random in real-world applications, the second camera's angle also adjusts accordingly. For example, if the second camera's view doesn't cover the speaker, the angle can be adjusted to capture a video image that does.

[0046] Since the first camera is a wide-angle camera, it can capture video images of all participants. However, the images of participants far from the first camera are less clear. Therefore, the viewing angle of the second camera can be determined based on the position of the participants far from the first camera, so that the second camera can capture video images of the participants far from the first camera.

[0047] Based on the cooperation of the first and second cameras, when the speaker is close to the first camera, a clear facial image sequence can be extracted from the first video image captured by the first camera for lip movement recognition, thus obtaining a recognition result. When the speaker is far from the second camera, a clear facial image sequence can be extracted from the second video image captured by the second camera for lip movement recognition, thus obtaining a recognition result.

[0048] Therefore, this method uses a first camera and a second camera with a variable viewing angle to acquire first and second video images respectively. Since the two cameras have different focal lengths, when the speaker's position is far away from the first camera, the second camera can acquire a clear second video image for lip movement recognition, thereby obtaining the recognition result and further improving the accuracy of lip movement recognition for distant objects.

[0049] In some embodiments, the shooting angle of the second camera can be adjusted before lip movement recognition is performed so that the second camera can capture objects that are far away from the position of the first camera.

[0050] For example: based on the object recognition results of each object in the first video image, the position of each object is determined; the second camera is rotated according to the position of each object so that the second camera is facing at least one object away from the first camera to capture the captured second video image.

[0051] Figure 3 The illustration shows a schematic diagram of object recognition in the main camera frame according to an embodiment of the present disclosure.

[0052] like Figure 3 As shown, by performing object recognition on the main camera image 300, detection boxes can be obtained to identify the positions of various objects in the main camera image.

[0053] In some embodiments, at least one object far from the first camera can be determined based on the size of each detection frame. For example, the size of the detection frame for an object close to the first camera is larger than the size of the detection frame for an object far from the first camera. Then, the actual position of the at least one object far from the first camera is obtained, and the second camera is rotated based on the actual position of the object far from the first camera so that the second camera faces the at least one object far from the first camera to capture a second video image.

[0054] In some embodiments, the actual position of each object can be calculated based on the size of each detection frame and the pose parameters of the first camera, and based on the mapping relationship between the physical world and the camera coordinate system. Then, the second camera is rotated based on the actual position of the object away from the first camera, so that the second camera faces at least one object away from the first camera to capture a second video image.

[0055] like Figure 3 As shown, the detection frame size of participant 304 is the smallest, which means that participant 304 is far away from the first camera. The second camera can be adjusted based on the position of participant 304 so that the second camera can shoot towards the participant 304 to obtain a second video image.

[0056] In some embodiments, based on the detection frame size, if participants 303 to 305 are all far from the first camera, the second camera can be adjusted based on the position of the area where participants 303 to 305 are located, so that the image captured by the second camera can cover the area where participants 303 to 305 are located. When any participant 303 to 305 speaks, a clear second video image that can be used for lip movement recognition can be captured.

[0057] By performing object recognition on the main camera's image to identify participants far from the first camera, the second camera can cover areas where the first camera struggles to capture a clear image. This ensures that clear images suitable for lip movement recognition are captured from participants in various positions, improving the accuracy of lip movement recognition. Simultaneously, using the object recognition results from the main camera's image as the basis for adjusting the second camera's shooting angle improves the efficiency of adjusting the secondary camera's angle.

[0058] In some embodiments, sound source location information can also be obtained, and the second camera can be rotated according to the sound source location information so that the second camera is facing the sound source location information to capture a captured second video image.

[0059] Figure 4The diagram illustrates the main camera view and the secondary camera view according to an embodiment of the present disclosure.

[0060] like Figure 4 As shown in embodiment 400, in the frame 401 captured by the first camera, participant 304 is speaking. The location of participant 304 can be determined based on the sound source location information, such as DOA (Direction of Arrival). Then, the second camera can be rotated so that it captures the location of participant 304. The shooting range of the second camera can cover the area where participant 304 is located, or participant 304 can be placed in the center of the shooting frame to obtain a close-up frame 402 of participant 304.

[0061] In other embodiments, the initial shooting angle of the second camera may only capture participants 302 and 303. At this time, participant 304 is speaking. If the second camera does not capture participant 304 within a predetermined time period, such as 5 seconds, it can rotate based on the sound source location information from participant 304, enabling the second camera to capture the speaking participant 304 and obtain a second video image.

[0062] According to embodiments of this disclosure, the second camera is rotated based on the location of the sound source. Due to the large focal length of the second camera, even if the sound source is far from the first camera, making it difficult for the first camera to capture a clear video image, the second camera can still capture a video image of the participant corresponding to the sound source location for lip movement recognition. This further improves the accuracy of lip movement recognition for participants at a distance.

[0063] According to embodiments of this disclosure, lip movement recognition based on a first video image and a second video image to determine the recognition result may include the following operations: recognizing and tracking each object in the first video image to obtain a first tracking sequence for each object; determining the objects contained in the second video frame according to the mapping relationship between the first video image and the second video image, and determining a second tracking sequence corresponding to the objects contained in the second video frame; and performing lip movement recognition based on the first tracking sequence and the second tracking sequence to obtain the recognition result.

[0064] For example, based on the biometric information of each object, a target detection model can be used to identify each object in the first video image to determine each participant.

[0065] Biometric information can be any one or more features that can be used to identify participants, such as head features, facial features, or eye features.

[0066] In some embodiments, any object detection model can be used to identify and track individual objects. The first tracking sequence may include a head tracking sequence of attendees, and may also include a face tracking sequence of attendees.

[0067] According to embodiments of this disclosure, the mapping relationship between the first video image and the second video image can be determined based on the shooting parameters of the first camera and the shooting parameters of the second camera.

[0068] For example, the first video image can capture all participants, and the first tracking sequence can be a head tracking sequence for all participants. Based on the mapping relationship, a portion of the participants within a certain viewpoint range that the second video image can capture can be determined. Therefore, the head tracking sequence for the aforementioned portion of participants can be determined based on the mapping relationship.

[0069] In some embodiments, lip movement recognition is performed based on a first tracking sequence and a second tracking sequence to obtain a recognition result. To avoid duplicate recognition, objects contained in the second video image are removed from the first video image; that is, if both the first and second tracking sequences exist for the same object, then the second tracking sequence is used for lip movement recognition.

[0070] In some embodiments, the sharpness of the first and second tracking sequences of the same object can be compared, and lip movement recognition can be performed based on the tracking sequence with higher sharpness to obtain the recognition result.

[0071] By performing target tracking in the wide-angle image and then generating the tracking result of the telephoto image based on the mapping relationship between the wide-angle and telephoto images and the tracking result of the wide-angle image, the operation of object recognition in the second video image can be omitted, thus improving recognition efficiency.

[0072] According to embodiments of this disclosure, lip movement recognition based on a first tracking sequence and a second tracking sequence to obtain a recognition result may include the following operations: performing face detection based on the first tracking sequence to obtain a first face tracking sequence; performing face detection based on the second tracking sequence to obtain a second face tracking sequence; and performing lip movement recognition based on the first face tracking sequence and the second face tracking sequence to obtain a recognition result.

[0073] When both the first tracking sequence and the second tracking sequence are head tracking sequences, a face detection algorithm can be used to identify the first tracking sequence / second tracking sequence respectively.

[0074] In some embodiments, a first face tracking sequence and a second face tracking sequence can be input into a lip movement recognition model to obtain a first recognition result and a second recognition result. The first recognition result and the second recognition result can be weighted based on predetermined weights to obtain the final recognition result.

[0075] In other embodiments, the clarity of the first and second face tracking sequences of the same participant can be compared, and the face tracking sequence with higher clarity can be input into the lip movement recognition model to obtain the recognition result.

[0076] In some other embodiments, when the same participant has both a first face tracking sequence and a second face tracking sequence, the second face tracking sequence can be input into the lip movement recognition model to obtain the recognition result. When the same participant only has the first face tracking sequence, the first face tracking sequence can be input into the lip movement recognition model to obtain the recognition result.

[0077] Face tracking sequences obtained through face detection can accurately display changes in the participants' lip movements, thereby improving the accuracy of lip movement recognition results.

[0078] In some embodiments, since there may be tracking errors during the process of obtaining the first tracking sequence by head tracking, in order to correct the possible tracking errors during head tracking, after obtaining the face results based on face detection, the detected faces are matched with the faces of each participant, and similarity calculation is performed to obtain an accurate face tracking sequence corresponding to each object.

[0079] Therefore, obtaining a first face tracking sequence by face detection based on the first tracking sequence may include the following operations: extracting face feature vectors from the first tracking sequence; and determining the first face tracking sequence based on the similarity between the face feature vectors and the face feature vectors of the predetermined participants.

[0080] According to embodiments of this disclosure, the facial feature vectors of the scheduled participants can be collected with the participants' authorization and deleted after the meeting ends, in order to protect the participants' privacy data.

[0081] For example, a similarity matrix can be constructed first based on the facial feature vectors of the participants and the facial feature vectors of the pre-selected participants. The element (i, j) in this similarity matrix represents the similarity between the i-th face and the j-th pre-selected participant's face. Then, a greedy algorithm can be used to determine the first face tracking sequence for each participant.

[0082] In some embodiments, the first face tracking sequence can also be determined based on the intersection-union ratio between the face positions of each object in the first tracking sequence and the position of the last appearance of the predetermined participant in the first tracking sequence.

[0083] For example, we can first construct an intersection-over-union (IoU) matrix based on the IoU between the face detection boxes of each object in the first tracking sequence and the last appearance of the face detection box of the scheduled participant in the first tracking sequence. The element (m, n) in this IoU matrix represents the IoU between the m-th face detection box and the last appearance of the face detection box of the n-th scheduled participant in the first tracking sequence. Then, a greedy algorithm can be used to determine the first face tracking sequence for each participant.

[0084] In some embodiments, a similarity matrix can be used to identify some participants first, and then an intersection-union matrix can be used to identify the remaining participants.

[0085] The above method can also be used to obtain a second face tracking sequence based on the second tracking sequence for face detection, which will not be elaborated here.

[0086] Figure 5 The illustration shows a schematic diagram of lip movement recognition based on the main camera image and the secondary camera image according to an embodiment of the present disclosure.

[0087] like Figure 5 As shown, in embodiment 500, firstly, head tracking is performed on the main camera image 510 to obtain the main camera head tracking result 511. Then, face detection is performed on the main camera image 510 to obtain the main camera face 512. Based on the main camera head tracking result 511 and the main camera face 512, the main camera face sequence 513 is obtained.

[0088] Then, face detection is performed on the secondary camera image 520 to obtain the secondary camera face 522. The main camera head tracking result 511 is mapped to obtain the secondary camera head tracking result 521. Based on the secondary camera head tracking result 521 and the secondary camera face 522, the secondary camera face sequence 523 is generated.

[0089] Finally, based on the main camera face sequence 513 and the secondary camera face sequence 523, lip movement recognition is performed to obtain the speaker 530.

[0090] Based on facial feature similarity and / or positional intersection-union ratio, face matching with each predetermined participant can be determined. This allows for the determination of the face tracking sequence of the same participant in the main camera image and the face tracking sequence in the secondary camera image, thereby determining a face tracking sequence suitable for lip movement recognition, enabling accurate long-distance lip movement recognition.

[0091] In some embodiments, when some participants have facial occlusions that affect lip movement recognition, long-distance lip movement recognition can be performed by combining sound source location information.

[0092] For example: acquire sound source location information, determine the speaker's location based on the sound source location information; determine candidate objects based on the speaker's location, acquire the first face tracking sequence or the second face tracking sequence corresponding to the candidate object; perform lip movement recognition based on the first face tracking sequence or the second face tracking sequence corresponding to the candidate object, and determine the recognition result.

[0093] According to embodiments of this disclosure, the sound source location information can be a DOA value. Candidates can be participants within the area corresponding to the DOA value.

[0094] Figure 6 The illustration shows a schematic diagram of lip movement recognition based on the main camera image, the secondary camera image, and the sound source location information according to an embodiment of the present disclosure.

[0095] like Figure 6 As shown, the difference between this embodiment 600 and embodiment 500 is that candidate object 611 is determined based on DOA 610.

[0096] Figure 7 The illustration shows a schematic diagram of determining the position of a virtual candidate object in the main camera frame based on the direction of the sound source according to an embodiment of the present disclosure.

[0097] like Figure 7 As shown, the region corresponding to DOA610 can be target region 701. This target region 701 can include three candidate objects, such as participant 303, participant 304, and participant 309.

[0098] Since the faces of each participant have been identified during face detection, the first face tracking sequence or the second face tracking sequence corresponding to each participant 303, participant 304 and participant 309 can be obtained based on the candidate objects.

[0099] When participant 303 has both a first face tracking sequence and a second face tracking sequence, the second face tracking sequence can be used for lip movement recognition. When a change in lip movement is detected in participant 303, it can be determined that the sound source is participant 303, that is, participant 303 is speaker 530.

[0100] By combining sound source location information, the accuracy of long-distance lip movement recognition can be further improved in complex scenarios with many participants and some participants' faces being obscured.

[0101] When participants 304, 303, and 309 all have only the first face tracking sequence, lip movement recognition can be performed using the first face tracking sequence. However, when all three participants are far from the first camera, the clarity of the first face tracking sequence is poor, therefore, the result of lip movement recognition may be that no changes in lip movements can be detected.

[0102] At this point, the position of the virtual candidate object in the first video image can be determined based on the sound source location information or the candidate object. Based on the position of the virtual candidate object, the second camera is controlled to turn to the position of the virtual candidate object to capture a new second video image. Lip movement recognition is performed based on the second video image to determine the recognition result.

[0103] In some embodiments, the position 710 of the virtual candidate object can be determined based on the mapping relationship between the sound source location information and the wide-angle image.

[0104] Then, based on the mapping relationship between the actual position in the physical world and the virtual position in the wide-angle image, the actual position corresponding to the position 710 of the virtual candidate object can be calculated, so as to control the second camera to turn to the position of the virtual candidate object to capture a new second video image.

[0105] When the sound source location area includes only one candidate object, the second camera can be controlled to turn to the position of the virtual candidate object to capture a new second video image based on the actual position of the candidate object.

[0106] By adjusting the shooting angle of the second camera in a timely manner based on the location information of the sound source, clear video images of the speaker can be captured in a timely manner for accurate lip movement recognition.

[0107] When the speaker is identified based on lip movement recognition, the second camera is controlled to turn and capture the speaker's image based on the speaker's position, extracting a close-up shot of the speaker.

[0108] According to embodiments of this disclosure, the second camera may be a gimbal camera.

[0109] When lip movement recognition identifies a speaker, the second camera can be turned to film the speaker, either by covering the speaker with the PTZ camera or by centering the speaker in the second video image, in order to extract a close-up shot of the speaker.

[0110] For example, when there is only one speaker, the second camera can be turned to film the speaker based on the speaker's position.

[0111] For example, when there are multiple speakers, the second camera can be controlled to turn and capture images of the multiple speakers based on the center position of the area where the multiple speakers are located.

[0112] In some embodiments, voiceprints can be combined to further improve the accuracy of lip movement recognition.

[0113] For example: acquire sound source location information, determine the speaker's location based on the sound source location information; determine candidate objects based on the speaker's location, acquire the first face tracking sequence or the second face tracking sequence corresponding to the candidate object; perform lip movement recognition based on the first face tracking sequence or the second face tracking sequence corresponding to the candidate object, and determine the lip movement recognition result; acquire the voiceprint information of the candidate object, determine the voiceprint recognition result based on the voiceprint information; determine the speaker based on the lip movement recognition result and the voiceprint recognition result.

[0114] Figure 8 The illustration shows a schematic diagram of lip movement recognition based on the main camera image, the secondary camera image, sound source location information, and voiceprint according to an embodiment of the present disclosure.

[0115] like Figure 8 As shown, the difference between this embodiment 800 and embodiment 600 is that the step of combining voiceprint recognition to identify the speaker is added.

[0116] Voiceprint matching is performed between the audio clip of candidate 611 and the audio clip in the participant audio database 810.

[0117] Extract the main camera face sequence and the secondary camera face sequence corresponding to the candidate object 611 from the main camera face sequence 513 and the secondary camera face sequence 523, and perform lip movement recognition based on the main camera face sequence and / or the secondary camera face sequence corresponding to the candidate object 611.

[0118] When the speaker identified by lip movement matches the speaker identified by voiceprint matching, the identification is considered valid, and speaker 530 is determined.

[0119] In complex scenarios, the second camera may fail to capture the speaker in time, resulting in lip movement failing to identify the speaker. Therefore, when lip movement fails to identify the speaker, but voiceprint matching does, the participant identified by the voiceprint can still be identified as speaker 530. Similarly, when lip movement identifies the speaker, but voiceprint matching fails, the participant identified by lip movement can still be identified as speaker 530. Simultaneously, the speaker identified by lip movement and the corresponding audio segment 820 are stored in the participant audio database 810.

[0120] According to embodiments of this disclosure, the voiceprint matching result can be used as auxiliary information for the lip movement recognition result, further improving the accuracy of lip movement recognition and adapting to complex scenarios where there are many participants, frequent changes in participant positions, and easy face occlusion.

[0121] In some embodiments, if lip movement recognition identifies multiple candidate objects, the candidate object closest to the speaker's location is determined as the speaker.

[0122] Figure 9The illustration shows a schematic diagram of determining the true speaker from multiple speakers based on sound source location information according to an embodiment of the present disclosure.

[0123] like Figure 9 As shown, the difference between this embodiment 900 and embodiment 600 is that lip movement identifies multiple speakers 910. Speaker 530 can be determined from the multiple speakers 910 based on DOA 610.

[0124] For example, the target location of the speaker in the main camera's view can be determined based on the mapping relationship between the DOA value and the main camera's view. Then, based on the distance between each candidate object and the target location, the candidate object closest to the sound source is determined as the speaker.

[0125] When multiple people are detected to have lip movements, the speaker can be accurately identified by combining the sound source location information, thus adapting to lip movement recognition in complex scenarios.

[0126] Figure 10 A block diagram schematically illustrates an electronic device suitable for implementing a video processing method according to an embodiment of the present disclosure.

[0127] like Figure 10 As shown, an electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0128] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1002 and / or RAM 1003. It should be noted that the programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0129] According to embodiments of this disclosure, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to a bus 1004. The electronic device 1000 may also include one or more of the following components connected to the input / output (I / O) interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0130] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0131] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1002 and / or RAM 1003 and / or one or more memories other than ROM 1002 and RAM 1003 described above.

[0132] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the video processing method provided in the embodiments of this disclosure.

[0133] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0134] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0135] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by processor 1001, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0136] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0138] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0139] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A video processing method, characterized in that, The method is applied to a conferencing device, the conferencing device including: a first camera and a second camera with a variable viewing angle, the method including: Acquire a first video image captured by the first camera and a second video image captured by the second camera; Lip movement recognition is performed based on the first video image and the second video image to determine the recognition result; The focal length of the second camera is greater than that of the first camera.

2. The method according to claim 1, characterized in that, The step of performing lip movement recognition based on the first video image and the second video image, and determining the recognition result, includes: Each object in the first video image is identified and tracked to obtain a first tracking sequence for each object; Based on the mapping relationship between the first video image and the second video image, determine the objects contained in the second video frame and determine the second tracking sequence corresponding to the objects contained in the second video frame; Lip movement recognition is performed based on the first tracking sequence and the second tracking sequence to obtain the recognition result.

3. The method according to claim 2, characterized in that, The lip movement recognition based on the first tracking sequence and the second tracking sequence, to obtain the recognition result, includes: A first face tracking sequence is obtained by performing face detection based on the first tracking sequence; A second face tracking sequence is obtained by performing face detection based on the second tracking sequence; Lip movement recognition is performed based on the first face tracking sequence and the second face tracking sequence to obtain the recognition result.

4. The method according to claim 3, characterized in that, The process of obtaining the first face tracking sequence by performing face detection based on the first tracking sequence includes: Extract facial feature vectors from the first tracking sequence; The first face tracking sequence is determined based on the similarity between the facial feature vector and the facial feature vector of the pre-selected attendees; and / or The first face tracking sequence is determined based on the intersection-union ratio (IoU) between the face positions of each object in the first tracking sequence and the position of the last appearance of the scheduled attendee in the first tracking sequence.

5. The method according to claim 3, characterized in that, The step of performing lip movement recognition based on the first face tracking sequence and the second face tracking sequence to obtain the recognition result includes: Obtain sound source location information, and determine the speaker's location based on the sound source location information; Candidate objects are determined based on the speaker's location, and a first face tracking sequence or a second face tracking sequence corresponding to the candidate object is obtained; Lip movement recognition is performed based on the first or second face tracking sequence corresponding to the candidate object to determine the recognition result.

6. The method according to claim 4, characterized in that, The method further includes: If the lip movement cannot identify the speaker, the position of the virtual candidate object in the first video image is determined according to the sound source location information or candidate object, and the second camera is controlled to turn to the position of the virtual candidate object to capture a new second video image based on the position of the virtual candidate object. Lip movement recognition is performed based on the second video image to determine the recognition result.

7. The method according to claim 4, characterized in that, The method further includes: If the lip movement recognition detects a speaker, the second camera is controlled to turn and capture a close-up image of the speaker based on the speaker's position.

8. The method according to claim 4, characterized in that, The method further includes: If the lip movement recognition identifies multiple candidate objects, the candidate object closest to the speaker's location is determined as the speaker.

9. The method according to claim 3, characterized in that, The step of performing lip movement recognition based on the first face tracking sequence and the second face tracking sequence to obtain the recognition result includes: Obtain sound source location information, and determine the speaker's location based on the sound source location information; Candidate objects are determined based on the speaker's location, and a first face tracking sequence or a second face tracking sequence corresponding to the candidate object is obtained; Based on the first face tracking sequence or the second face tracking sequence corresponding to the candidate object, lip movement recognition is performed to determine the lip movement recognition result; Obtain the voiceprint information of the candidate object, and determine the voiceprint recognition result based on the voiceprint information; The speaker is determined based on the lip movement recognition results and voiceprint recognition results.

10. The method according to claim 9, characterized in that, The process of determining the speaker based on the lip movement recognition results and voiceprint recognition results includes: When the lip movement recognition result is "unrecognizable", the speaker is determined based on the voiceprint recognition result; When the lip movement recognition result and the voiceprint recognition result are consistent, the speaker is identified; When the voiceprint recognition result fails to identify the speaker, the speaker is determined based on the lip movement recognition result, and the speaker and the corresponding sound segment are stored in the voiceprint database. The voiceprint database is used to perform voiceprint recognition based on the sound segment and to associate the obtained voiceprint information with the speaker.

11. The method according to claim 1, characterized in that, Before performing lip movement recognition based on the first video image and the second video image to determine the recognition result, the method further includes: Obtain the sound source location information, and rotate the second camera according to the sound source location information so that the second camera is facing the sound source location information to capture a second video image.

12. The method according to claim 1, characterized in that, Before performing lip movement recognition based on the first video image and the second video image to determine the recognition result, the method further includes: Based on the object recognition results of each object in the first video image, the position of each object is determined; The second camera is rotated according to the position of each object, so that the second camera is aimed at at least one object away from the first camera to capture a second video image.

13. An electronic device, comprising: The processor, wherein the memory stores a computer program, and when the computer program instructions are executed by the processor, the processor is configured to perform the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 12.

Images