Video call method and related device, equipment and storage medium

By jointly positioning the audio and image data in the video call device and combining the historical positioning results, the accuracy and stability of the positioning of the speaker in the video call are improved, and the problem of insufficient positioning accuracy in the prior art is solved.

CN114339113BActive Publication Date: 2024-12-03IFLYTEK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111456189.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-01
Publication Date
2024-12-03
Estimated Expiration
2041-12-01

AI Technical Summary

Technical Problem

During video calls, the existing technology is difficult to effectively improve the positioning accuracy of the speaker, affecting the quality of video calls.

Method used

By jointly positioning the audio data and image data of the video call device, combining the positioning results of historical moments, the final direction of the speaker at the current moment is calculated, and the weighted results of the sound source positioning and image positioning are used to conduct video calls.

Benefits of technology

Improve the speaker positioning accuracy and video call quality to ensure stable positioning during continuous video calls in a time series.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114339113B_ABST
    Figure CN114339113B_ABST
Patent Text Reader

Abstract

The present application discloses a video call method and related devices, call equipment and storage medium, wherein the video call method includes: first positioning the audio data of the video call device at the current moment to obtain the first position of the speaker at the current moment, and second positioning the image to be measured at the current moment of the video call device to obtain the second position of the speaker at the current moment; wherein at least one of the first positioning and the second positioning refers to the positioning results of several historical moments before the current moment during the execution process, and the positioning results include the first position and the second position of the speaker at the historical moment; combining the first position and the second position to obtain the final position of the speaker at the current moment; based on the final position, a video call is made through the video call device. The above scheme can improve the positioning accuracy of the speaker during the video call.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video communication technology, and in particular to a video calling method and related devices, equipment and storage media. Background Art

[0002] With the increasing development of electronic information technology, video calls have been widely used in many scenarios such as daily life and business office. For example, in daily life, you can chat with relatives in different places through video calls; in business office, you can achieve remote connection through video calls.

[0003] During a video call, locating the speaker is one of the key factors affecting the quality of the video call. For example, based on accurate positioning, noise suppression can be performed to improve the speaker's voice quality, which can greatly improve the video call effect. In view of this, how to improve the accuracy of speaker positioning has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a video call method and related devices, equipment and storage media, which can improve the speaker positioning accuracy.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a video call method, including: performing a first positioning on the audio data of the video call device at the current moment to obtain a first position of the speaker at the current moment, and performing a second positioning on the image to be measured at the current moment of the video call device to obtain the second position of the speaker at the current moment; wherein, at least one of the first positioning and the second positioning refers to the positioning results of several historical moments before the current moment during the execution process, and the positioning results include the first position and the second position of the speaker at the historical moments; combining the first position and the second position to obtain the final position of the speaker at the current moment; based on the final position, conducting a video call through the video call device.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a video call device, including: a first positioning module, a second positioning module, a combining module and a call module, the first positioning module is used to perform a first positioning on the audio data of the video call device at the current moment to obtain the first position of the speaker at the current moment; the second positioning module is used to perform a second positioning on the image to be measured at the current moment of the video call device to obtain the second position of the speaker at the current moment; wherein, at least one of the first positioning and the second positioning refers to the positioning results of several historical moments before the current moment during the execution process, and the positioning results include the first position and the second position of the speaker at the historical moments; the combining module is used to combine the first position and the second position to obtain the final position of the speaker at the current moment; the call module is used to make a video call through the video call device based on the final position.

[0007] In order to solve the above-mentioned technical problems, the third aspect of the present application provides a video call device, including a screen, a microphone, a camera, a communication circuit, a memory and a processor, the screen, microphone, camera, communication circuit and memory are respectively coupled to the processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the video call method in the above-mentioned first aspect.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the video call method in the above first aspect.

[0009] In the above scheme, the audio data of the video call device at the current moment is first positioned to obtain the first position of the speaker at the current moment, and the image to be tested at the current moment of the video call device is second positioned to obtain the second position of the speaker at the current moment, and at least one of the first position and the second position refers to the positioning results of several historical moments before the current moment during the execution process, and the positioning results include the first position and the second position of the speaker at the historical moment. On this basis, the first position and the second position are combined to obtain the final position of the speaker at the current moment, and based on the final position, a video call is made through the video call device. On the one hand, during the video call, the final position is obtained by combining the first position and the second position obtained by sound source positioning and image positioning respectively, which is conducive to improving the positioning accuracy compared with a single positioning method. On the other hand, during the video call, at least one of the sound source positioning and the image positioning refers to the positioning results of several previous historical moments during the positioning process, which is conducive to maintaining stable positioning during the video call process with continuous time sequence. Therefore, the positioning accuracy of the speaker can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1It is a flowchart of an embodiment of the video calling method of the present application;

[0011] Figure 2 is a schematic diagram of an embodiment of a three-dimensional coordinate system;

[0012] Figure 3 This is a process diagram of an embodiment of the video calling method of the present application;

[0013] Figure 4 It is a flowchart of another embodiment of the video calling method of the present application;

[0014] Figure 5 It is a schematic diagram of the framework of an embodiment of the video calling device of the present application;

[0015] Figure 6 It is a schematic diagram of the framework of an embodiment of the video calling device of the present application;

[0016] Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0017] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0018] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0019] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0020] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of the video calling method of the present application.

[0021] Specifically, the following steps may be included:

[0022] Step S11: performing a first positioning on the audio data of the video call device at the current moment to obtain a first position of the speaker at the current moment, and performing a second positioning on the image to be tested of the video call device at the current moment to obtain a second position of the speaker at the current moment.

[0023] In the embodiment of the present disclosure, at least one of the first positioning and the second positioning refers to the positioning results of several historical moments before the current moment during the execution process, and the positioning results include the first position and the second position of the speaker at the historical moment. It should be noted that during the video call, the speaker may move. For example, in a business office scene, the narrator may have the habit of walking while talking. Examples are not given one by one here. Therefore, at the shooting moment of each video frame, the steps in the embodiment of the present application can be executed to respectively use the shooting moment of each video frame as the current moment to locate the speaker. In addition, for the convenience of description, the current moment can be recorded as t, and several historical moments before the current moment may include: t-1 moment, t-2 moment, t-3 moment, ..., tT moment, etc., which are not limited here. Furthermore, the positioning results of each historical moment can also be obtained through the steps in the embodiment of the present application. For example, at time ti, the steps in the embodiments disclosed in the present application can be used to refer to the positioning result at time ti-1 to obtain the first and second positions at time ti. Then, at time t-i+1, the steps in the embodiments disclosed in the present application can be used to refer to the positioning result at time ti to obtain the first and second positions at time t-i+1, and so on. Examples are not given one by one here.

[0024] In one implementation scenario, the video call device in the disclosed embodiment of the present application may be integrated with a screen, a microphone, and a camera to collect audio data through the microphone, capture image data through the camera, and display image data through the screen. For details, please refer to the relevant description in the disclosed embodiment of the video call device below, which will not be repeated here. It should be noted that in order to facilitate the representation of the speaker's position, a three-dimensional coordinate system can be constructed based on the video call device, and the angle between the line connecting the speaker's position and the origin of the three-dimensional coordinate system and the coordinate axis of the three-dimensional coordinate system is regarded as the speaker's position. For example, please refer to Figure 2 , Figure 2 3D coordinate system is a schematic diagram of an embodiment. Figure 2 As shown, the origin of the three-dimensional coordinate system is the center of the lower edge of the front side of the video call device, and the video call device is also integrated with a dual microphone array near the center of the lower edge of the front side. The first orientation can be represented by the connection line between the sound source and the microphone and the connection line of the array (i.e. Figure 2 In the real scene, the distance between the sound source and any microphone is much greater than the distance between the microphones, so it can be regarded as a far-field model. The incident angle of the sound source (i.e., the angle) θ satisfies Among them, τ represents the time delay of the sound reaching the two microphones, d represents the distance between the two microphones, and c represents the speed of sound. For details, please refer to the relevant technical details of the far-field model, which will not be repeated here. Further, the second orientation can be expressed as the angle between the line connecting the speaker and the origin in the image and the x-axis. Figure 2 In the scene shown, the closer the speaker is to the edge of the image, the further the speaker is from the middle of the video call device, and the smaller the first orientation and the second orientation are. The closer the speaker is to the center of the image, the closer the speaker is to the middle of the video call device, and the larger the first orientation and the second orientation are. In other words, under the premise that both are accurately positioned, no matter how the speaker moves, the deviation between the first orientation and the second orientation should theoretically be within a specific range. The specific value of the specific range can be adjusted according to the established three-dimensional coordinate system, which will not be elaborated here.

[0025] In one implementation scenario, during the execution of the first positioning, the positioning results of several historical moments before the current moment can be referred to. For each historical moment, the fusion weight of the corresponding historical moment can be obtained based on the azimuth deviation between the first orientation and the second orientation at the historical moment, and the voice detection result of the call audio collected at the historical moment, and the fusion weight of each historical moment can be used to fuse the cross-correlation results of the call audio at each historical moment to obtain the weighted cross-correlation result at the current moment. On this basis, the first orientation at the current moment can be obtained based on the weighted cross-correlation result. The above method, by weighted fusion of the cross-correlation results of the call audio at each historical moment, obtains the weighted cross-correlation result at the current moment, and then obtains the first orientation at the current moment based on the weighted cross-correlation result, which can greatly alleviate the instability of instantaneous cross-correlation during continuous time-series video calls, and is conducive to improving the accuracy of sound source positioning.

[0026] In a specific implementation scenario, for each historical moment, a first weight coefficient can be obtained based on the azimuth deviation between the first azimuth and the second azimuth at the historical moment, and a second weight coefficient can be obtained based on the voice detection result of the call audio collected at the historical moment, and the azimuth deviation is negatively correlated with the first weight coefficient, that is, the larger the azimuth deviation, the smaller the first weight coefficient, and the smaller the azimuth deviation, the larger the first weight coefficient. On this basis, based on the first weight coefficient and the second weight coefficient, the fusion weight of the corresponding historical moment can be obtained. In the above method, the first weight coefficient is obtained based on the azimuth deviation between the first azimuth and the second azimuth, and the second weight coefficient is obtained based on the voice detection result of the call audio, and the azimuth deviation is negatively correlated with the first weight coefficient. Based on the first weight coefficient and the second weight coefficient, the fusion weight is obtained. Therefore, the fusion weight can be set from the two aspects of azimuth deviation and voice detection, so as to filter the call audio that is not beneficial to the sound source positioning, which is conducive to improving the sound source positioning accuracy.

[0027] Exemplarily, for the first weight coefficient, when the first orientation is within the first deviation range of the second orientation, the first weight coefficient can be directly set to the first value, and when the first orientation exceeds the first deviation range of the second orientation, the first weight coefficient can be directly set to the second value, and the first value is greater than the second value. For example, the first deviation range can be set to 5 degrees, 10 degrees, 15 degrees, and so on, which are not limited here. In addition, the first value can be set to 1, and the second value can be set to 0, which are not limited here. Taking the first deviation range being set to 5 degrees as an example, when the first orientation is within the deviation range of 5 degrees of the second orientation, the first weight coefficient can be set to 1, or, when the first orientation exceeds the deviation range of 5 degrees of the second orientation, the first weight coefficient can be set to 0. Other situations can be deduced by analogy, and examples are not given one by one here. In the above method, when the first position is within the first deviation range of the second position, the first weight coefficient is the first value, and when the first position exceeds the first deviation range of the second position, the first weight coefficient is the second value, and the first value is greater than the second value. In the process of sound source positioning, more references can be made to the call audio at historical moments when the deviation between image positioning and sound source positioning is smaller, which is beneficial to improving the accuracy of sound source positioning.

[0028] Exemplarily, for the second weight coefficient, when the human voice detection result includes the detection of human voice in the call audio, the second weight coefficient can be set to a third value, and when the human voice detection result includes the detection of no human voice in the call audio, the second weight coefficient can be set to a fourth value, and the third value is greater than the fourth value. For example, the third value can be set to 1, and the fourth value can be set to 0. Taking this as an example, when the human voice detection result includes the detection of human voice in the call audio, the second weight coefficient can be directly set to 1, and conversely, when the human voice detection result includes the detection of no human voice in the call audio, the second weight coefficient can be directly set to 0. Other situations can be deduced by analogy, and examples are not given one by one here. In the above manner, when the human voice detection includes the detection of human voice in the call audio, the second weight coefficient is the third value, and when the human voice detection result includes the detection of no human voice in the call audio, the second weight coefficient is the fourth value, and the third value is greater than the fourth value. In the process of sound source localization, more reference can be made to the call audio containing human voices in the call audio corresponding to several historical moments, which is conducive to improving the accuracy of sound source localization.

[0029] For example, after obtaining the first weight coefficient and the second weight coefficient corresponding to each historical moment, for each historical moment, the first weight coefficient and the second weight coefficient corresponding to the historical moment can be multiplied to obtain the fusion weight of the historical moment. For ease of description, the first weight coefficient of the i-th historical moment can be recorded as The second weight coefficient is recorded as Then the fusion weight corresponding to this historical moment can be expressed as

[0030] In a specific implementation scenario, taking the video call device including two microphones as an example, the call audio at each historical moment may include the sub-audio collected by the two microphones respectively. For the convenience of description, they can be expressed as x1(t)=a1s(t-τ1)+n1(t) and x2(t)=a2s(t-τ2)+n2(t), respectively, where n1 and n2 both represent white noise and are independent of each other, s represents the sound source, τ1 and τ2 are the time from the sound source to the two microphones respectively, and s is also independent of n1 and n2 respectively. Then, at the historical moment t, the cross-correlation result of the call audio can be expressed as:

[0031] R x1x2 (τ) = E (x1(t)x2(t-τ))

[0032] =a1a2E(s(t-τ1)*s(t-τ2-τ))

[0033] =a1a2R ss (τ-(τ1-τ2)……(1)

[0034] As shown in formula (1), when τ = τ1-τ2, the cross-correlation is the largest, that is, the cross-correlation peak is the time delay difference of the sound source reaching the microphone. On this basis, the fusion weights of each historical moment can be used to weight the corresponding cross-correlation results respectively to obtain weighted cross-correlation results, and then the cross-correlation peak can be taken to obtain the time delay difference at the current moment. Based on this, combined with the calculation formula for the time delay difference in the aforementioned far-field model, the sound source incident angle θ can be calculated as the first azimuth.

[0035] In an implementation scenario, the second positioning refers to the positioning result during the execution process, and in the second positioning process, TLD (Tracking Learning Detection) is mainly used. TLD is a long-term target tracking algorithm, which consists of a detector (i.e., Detection), a tracker (i.e., Tracking) and a learner (i.e., Learning). Among them, the tracker is used to track the motion between consecutive frames. The tracker estimates the position of the target in the current frame based on the known position of the target in the previous frame, so that an activity trajectory of the target can be generated. From this activity trajectory, positive samples can be provided for the learner, and the role of the detector is to estimate the error of the tracker. The detector performs a comprehensive scan on each frame of the image to find a position similar to the target, thereby detecting and generating positive samples and negative samples, which are handed over to the learner. The algorithm selects the most credible position from all positive samples as output, and then uses this output to update the starting position of the tracker. In addition, the learner iteratively trains the classifier contained in the detector based on the positive and negative samples generated by the tracker and the detector to improve the detection accuracy. The specific principle of TLD can refer to the technical details of TLD, which will not be repeated here. Different from the traditional TLD, the disclosed embodiment of the present application further assists in the process of performing the second positioning through TLD by combining the sound source localization result. Specifically, a number of candidate positive samples can be extracted based on the activity trajectory determined by the TLD tracker, and a number of candidate positive samples can be screened using the first position of the speaker at the historical moment to obtain the target positive sample, and the detector of the TLD can be trained until convergence using the target positive sample. On this basis, the TLD is used to perform the second positioning of the image to be tested to obtain the second position at the current moment. In the above method, the candidate positive samples are extracted through the activity trajectory extracted by the tracker, and the target positive samples are further screened out from the first position of the speaker at the historical moment, and the detector is trained until convergence using the target positive sample. On this basis, the TLD is used to perform the second positioning of the image to be tested to obtain the second position at the current moment. Therefore, the positive samples used to train the detector are not only determined according to the activity trajectory of the tracker, but also the sound source localization determination is added at the same time, so that the reliability of the positive samples can be improved, which is beneficial to improve the accuracy of the detector and reduce the computational complexity.

[0036] In a specific implementation scenario, the specific process of determining the activity trajectory through the TLD tracker can refer to the relevant technical details of the TLD, which will not be repeated here.

[0037] In a specific implementation scenario, please refer to Figure 3 , Figure 3 1 is a schematic diagram of the process of an embodiment of the video call method of the present application. Figure 3 As shown, sound source localization can provide a classification standard for the learner. Taking the current moment t as an example, for the historical moment ti (where i = 1, 2, 3, etc.), the candidate positive samples extracted at the historical moment ti using the first orientation at the historical moment ti can be further classified. If the image orientation of the candidate positive sample in the image to be tested taken at the historical moment ti is the same or similar to the first orientation at the historical moment ti, the candidate positive sample can be selected as the target positive sample, otherwise it can be directly used as the target negative sample. In other words, the sound source direction can be directly defined as a positive sample, and the non-sound source direction can be defined as a negative sample. On this basis, the detector can be trained to converge, so that the detector learns the speaker's updated and more accurate image features, thereby improving the subsequent detection accuracy.

[0038] In a specific implementation scenario, before training the TLD detector, in response to the azimuth deviation between the first orientation and the second orientation at the previous historical moment of the current moment satisfying the second condition, the TLD-based learner can assign learning weights to each target positive sample (the learning weight can be set to be greater than 1, such as 1.5, 2, 2.5, etc.), and on this basis, the detector is trained until convergence based on the target positive samples assigned the learning weights. Specifically, the second condition can be set to include the azimuth deviation between the first orientation and the second orientation at the previous historical moment being lower than a preset threshold (such as 5 degrees, 10 degrees, etc.), at which time it can be considered that positive feedback of sound source localization is obtained, and the credibility of the target positive samples can be increased by assigning learning weights to them by the learner.

[0039] In a specific implementation scenario, such as Figure 3As shown, after the training converges, the detector can be used to detect the image to be tested at the current moment. In this process, the first orientation at the current moment can be used to delineate the detection area in the image to be tested, so that in the subsequent positioning process of the image to be tested using TLD, the detector only performs target detection in the detection area to improve the detection efficiency. Specifically, in the second positioning process of the image to be tested at the current moment, the tracker of TLD can provide a positive sample, and the detector of TLD can detect several positive samples in the above detection area. Finally, a positive sample that is most similar to the speaker can be selected from these positive samples, and based on the positive sample, the second orientation at the current moment is determined. It should be noted that the sample-related terms such as "positive sample", "negative sample", "candidate positive sample" and the like described in the embodiments disclosed in this application all represent sub-images captured from the image taken by the camera. For details, please refer to the relevant technical details of TLD, which will not be repeated here.

[0040] In a specific implementation scenario, as mentioned above, after obtaining the second position of the speaker at the current moment, the tracker of TLD can be updated accordingly. For details, please refer to the relevant technical details of TLD, which will not be repeated here.

[0041] Step S12: combining the first position and the second position to obtain the final position of the speaker at the current moment.

[0042] In an implementation scenario, the first position and the second position may be directly weighted to obtain the final position of the speaker at the current moment. The weights of the first position and the second position may be set according to actual conditions. For example, when the accuracy of sound source localization is higher than that of image localization, the weight of the first position may be set higher than that of the second position; conversely, when the accuracy of sound source localization is lower than that of image localization, the weight of the first position may be set lower than that of the second position. This is not limited here.

[0043] In one implementation scenario, the reliability test may be performed on the first position to obtain the reliability test result of the first position, and the reliability test may be performed on the second position to obtain the reliability test result of the second position, and then the final position may be obtained based on the first test result and the second test result. The above method first performs reliability tests on the first position and the second position respectively, and then obtains the final position based on the reliability test result, which is conducive to improving the reliability of the final position.

[0044] In a specific implementation scenario, when the first orientation is within the second deviation range of the second orientation, it can be determined that the first detection result includes that the first orientation is reliable, and when the first orientation exceeds the second deviation range of the second orientation, it can be determined that the first detection result includes that the first orientation is unreliable. Exemplarily, the second deviation range can be set to 5 degrees, 15 degrees, 20 degrees, etc., which are not limited here. Taking the second deviation range being set to 15 degrees as an example, if the first orientation is within the deviation range of 15 degrees from the second orientation, the first orientation can be considered reliable, otherwise, the first orientation can be considered unreliable. The above method determines whether the first orientation is reliable by comparing the degree to which the first orientation deviates from the second orientation, thereby performing reliability detection from the perspective of taking the second orientation as a comparison reference.

[0045] In a specific implementation scenario, as mentioned above, the second orientation can be obtained by TLD detection, and then it can be detected whether the following two are met at the same time: the similarity between the speaker's facial image and the sample image meets the second condition, and the orientation deviation between the second orientation and the first orientation meets the third condition, wherein the sample image is a positive sample of the detector used to train the TLD, and the specific meaning can refer to the above-mentioned related description, which will not be repeated here. On this basis, if the above two are met at the same time, it can be determined that the second detection result includes that the second orientation is reliable, otherwise it can be determined that the second detection result includes that the second orientation is unreliable. Exemplarily, the speaker's facial image can be the facial image that the speaker recognizes from the captured image at the first time during this video call, and the specific details can refer to the following public embodiments, which will not be repeated here. In addition, the second condition can be set to a similarity higher than a preset similarity (e.g., 70%, 75%, etc.), and the third condition can be set to a orientation deviation lower than a preset angle (e.g., 30 degrees, 35 degrees, etc.). The above method can determine the reliability of the second position from multiple dimensions by comparing the degree of deviation of the second position from the first position and the similarity between the face image and the sample image.

[0046] In a specific implementation scenario, when the first detection result includes that the first orientation is reliable and the second detection result includes that the second orientation is reliable, the first orientation and the second orientation can be fused to obtain the final orientation, such as taking the average of the two, or weighting, which is not limited here. In addition, if either one is unreliable, the other can be taken as the final orientation, or, when both are unreliable, it can be considered that the positioning at the current moment has failed. In the above manner, when both the first orientation and the second orientation are detected reliably, the final orientation is obtained by fusing the two, which is conducive to improving the accuracy of the final orientation.

[0047] Step S13: Based on the final position, conduct a video call through a video call device.

[0048] In one implementation scenario, beamforming can be performed based on the final range to suppress the noise around the speaker, thereby improving the voice quality in the final azimuth direction. It should be noted that the specific process of beamforming can refer to the technical details of beamforming, which will not be repeated here.

[0049] In one implementation scenario, the video call device can be controlled to turn toward the speaker based on the final position satisfying the fourth condition. Exemplarily, the fourth condition can be set as follows: the final position is about to exceed the video shooting range, at which time the video call device can be controlled to turn toward the speaker so that the speaker is in the center again, thereby reducing the risk of the speaker being out of the frame during the video call. It should be noted that the video call device can be configured to also include a rotating mechanism (such as a pan / tilt, etc.), and the rotating mechanism can be used to control the video call device to turn toward the speaker.

[0050] In an implementation scenario, when multiple locations are video linked, there are multiple speakers in at least one of the locations. During the speech of one of the speakers, the steps in the embodiments of the present disclosure can be used to continuously track the speaker and maintain the video call. In other words, at each moment of the speaker's speech, the steps in the embodiments of the present disclosure can be used to implement the video call.

[0051] In an implementation scenario, as mentioned above, when multiple locations are connected via video, there are multiple speakers in at least one of the locations. After one speaker finishes speaking, the other speaker can continue to connect via the video call device without manually operating the video call device. Specifically, when another speaker needs to speak, he only needs to say a preset wake-up word (such as "Mo Fei Mo Fei", etc.). At this time, in response to detecting the target audio containing the preset wake-up word, the sound source can be located based on the target audio to obtain the target position of the speaker. Based on the target position, the video call device can be controlled to turn to the speaker, and face recognition can be performed based on the target image taken by the video call device to obtain the speaker's face image, and the face image is used as the tracking target for performing the second positioning. On this basis, the relevant steps in the disclosed embodiment can be triggered to conduct a video call. The above method, in response to detecting target audio containing a preset wake-up word, performs sound source positioning based on the target audio to obtain the target position of the speaker, and based on the target position, controls the video call device to turn toward the speaker, and performs face recognition based on the target image taken by the video call device to obtain the face image of the speaker, and the face image is used as the tracking target for performing the second positioning. Therefore, there is no need for manual operation, and target tracking and video calls can be triggered only through the preset wake-up word, which is conducive to improving operational convenience.

[0052] In a specific implementation scenario, the specific process of sound source localization can refer to the technical details of the aforementioned audio cross-correlation, which will not be repeated here.

[0053] In a specific implementation scenario, face recognition can be achieved through a neural network. For details, please refer to the technical details of related networks such as FaceNet and DeepID, which will not be repeated here.

[0054] In the above scheme, the audio data of the video call device at the current moment is first positioned to obtain the first position of the speaker at the current moment, and the image to be tested at the current moment of the video call device is second positioned to obtain the second position of the speaker at the current moment, and at least one of the first position and the second position refers to the positioning results of several historical moments before the current moment during the execution process, and the positioning results include the first position and the second position of the speaker at the historical moment. On this basis, the first position and the second position are combined to obtain the final position of the speaker at the current moment, and based on the final position, a video call is made through the video call device. On the one hand, during the video call, the final position is obtained by combining the first position and the second position obtained by sound source positioning and image positioning respectively, which is conducive to improving the positioning accuracy compared with a single positioning method. On the other hand, during the video call, at least one of the sound source positioning and the image positioning refers to the positioning results of several previous historical moments during the positioning process, which is conducive to maintaining stable positioning during the video call process with continuous time sequence. Therefore, the positioning accuracy of the speaker can be improved.

[0055] See also Figure 4 , Figure 4 It is a flowchart of another embodiment of the video calling method of the present application.

[0056] Specifically, the following steps may be included:

[0057] Step S41: In response to detecting the target audio containing the preset wake-up word, sound source localization is performed based on the target audio to obtain the target position of the speaker.

[0058] For details, please refer to the relevant description in the aforementioned disclosed embodiments, which will not be repeated here.

[0059] Step S42: Based on the target direction, control the video call device to turn toward the speaker.

[0060] For details, please refer to the relevant description in the aforementioned disclosed embodiments, which will not be repeated here.

[0061] Step S43: Perform face recognition based on the target image captured by the video call device to obtain a face image of the speaker.

[0062] In the embodiment of the present disclosure, the face image can be used as a tracking target for the subsequent second positioning. For details, please refer to the relevant description in the aforementioned disclosed embodiment, which will not be repeated here.

[0063] Step S44: Perform a first positioning on the audio data of the video call device at the current moment to obtain a first position of the speaker at the current moment, and perform a second positioning on the image to be tested of the video call device at the current moment to obtain a second position of the speaker at the current moment.

[0064] In the embodiment of the present disclosure, at least one of the first positioning and the second positioning refers to the positioning results of several historical moments before the current moment during the execution process, and the positioning results include the first position and the second position of the speaker at the historical moments. For details, please refer to the relevant description in the aforementioned public embodiment, which will not be repeated here.

[0065] Step S45: combining the first position and the second position to obtain the final position of the speaker at the current moment.

[0066] For details, please refer to the relevant description in the aforementioned disclosed embodiments, which will not be repeated here.

[0067] Step S46: Based on the final location, conduct a video call through a video call device.

[0068] For details, please refer to the relevant description in the aforementioned disclosed embodiments, which will not be repeated here.

[0069] In the above scheme, before the formal video call, the speaker uses a preset wake-up word to make the video call device automatically turn to the speaker's location. On this basis, the formal video call can be carried out, which can improve the convenience of the video call.

[0070] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a video call device 50 of the present application. The video call device 50 includes a first positioning module 51, a second positioning module 52, a combining module 53 and a calling module 54. The first positioning module 51 is used to perform a first positioning on the audio data of the video call device at the current moment to obtain the first position of the speaker at the current moment; the second positioning module 52 is used to perform a second positioning on the image to be measured at the current moment of the video call device to obtain the second position of the speaker at the current moment; wherein, at least one of the first positioning and the second positioning refers to the positioning results of several historical moments before the current moment during the execution process, and the positioning results include the first position and the second position of the speaker at the historical moments; the combining module 53 is used to combine the first position and the second position to obtain the final position of the speaker at the current moment; the calling module 54 is used to make a video call through the video call device based on the final position.

[0071] In the above scheme, on the one hand, during a video call, the final position is obtained by combining the first position and the second position obtained by sound source positioning and image positioning respectively, which is conducive to improving positioning accuracy compared with a single positioning method. On the other hand, during a video call, at least one of the sound source positioning and the image positioning refers to the positioning results of several previous historical moments during the positioning process, which is conducive to maintaining stable positioning during a video call with continuous time sequence. Therefore, the positioning accuracy of the speaker can be improved.

[0072] In some disclosed embodiments, the first positioning refers to the positioning result during execution; the first positioning module 51 includes a fusion weight submodule, which is used to obtain the fusion weight of the corresponding historical moment for each historical moment based on the position deviation between the first position and the second position at the historical moment, and the voice detection result of the call audio collected at the historical moment; the first positioning module 51 includes a result weighting submodule, which is used to use the fusion weight of each historical moment to fuse the cross-correlation results of the call audio at each historical moment to obtain the weighted cross-correlation result at the current moment; the first positioning module 51 includes a first determination submodule, which is used to obtain the first position at the current moment based on the weighted cross-correlation result.

[0073] Therefore, by weighted fusion of the cross-correlation results of the call audio at each historical moment, the weighted cross-correlation result at the current moment is obtained, and then based on the weighted cross-correlation result, the first position at the current moment is obtained, which can greatly alleviate the instability of instantaneous cross-correlation during continuous time-series video calls and is conducive to improving the accuracy of sound source positioning.

[0074] In some disclosed embodiments, the fusion weight submodule includes a weight coefficient acquisition unit, which is used to obtain a first weight coefficient based on the orientation deviation between the first orientation and the second orientation at the historical moment, and to obtain a second weight coefficient based on the human voice detection result of the call audio collected at the historical moment; wherein the orientation deviation is negatively correlated with the first weight coefficient; the fusion weight submodule includes a fusion weight acquisition unit, which is used to obtain the fusion weight corresponding to the historical moment based on the first weight coefficient and the second weight coefficient.

[0075] Therefore, based on the azimuth deviation between the first azimuth and the second azimuth, a first weight coefficient is obtained, and based on the voice detection result of the call audio, a second weight coefficient is obtained, and the azimuth deviation is negatively correlated with the first weight coefficient. Based on the first weight coefficient and the second weight coefficient, a fusion weight is obtained. Therefore, the fusion weight can be set from the two aspects of azimuth deviation and voice detection, so as to filter out the call audio that is not beneficial to sound source localization, which is conducive to improving the accuracy of sound source localization.

[0076] In some disclosed embodiments, when the first orientation is within a first deviation range of the second orientation, the first weight coefficient is a first value, and when the first orientation exceeds the first deviation range of the second orientation, the first weight coefficient is a second value, and the first value is greater than the second value.

[0077] Therefore, when the first direction is within the first deviation range of the second direction, the first weight coefficient is the first value, and when the first direction exceeds the first deviation range of the second direction, the first weight coefficient is the second value, and the first value is greater than the second value. In the process of sound source positioning, more references can be made to the call audio at historical moments when the deviation between image positioning and sound source positioning is smaller, which is conducive to improving the accuracy of sound source positioning.

[0078] In some disclosed embodiments, when the human voice detection includes detecting a human voice in the call audio, the second weight coefficient is a third value; when the human voice detection result includes not detecting a human voice in the call audio, the second weight coefficient is a fourth value, and the third value is greater than the fourth value.

[0079] Therefore, when the human voice detection includes detecting the human voice in the call audio, the second weight coefficient is the third value. When the human voice detection result includes not detecting the human voice in the call audio, the second weight coefficient is the fourth value, and the third value is greater than the fourth value. In the process of sound source localization, more references can be made to the call audio containing the human voice corresponding to several historical moments, which is conducive to improving the accuracy of sound source localization.

[0080] In some disclosed embodiments, the second positioning refers to the positioning result during execution; the second positioning module 52 includes a sample extraction submodule, which is used to extract a number of candidate positive samples based on the activity trajectory determined by the TLD tracker; the second positioning module 52 includes a sample screening submodule, which is used to screen a number of candidate positive samples using the first position of the speaker at the historical moment to obtain a target positive sample; the second positioning module 52 includes a model training submodule, which is used to train the TLD detector until convergence using the target positive sample; the second positioning module 52 includes a second determination submodule, which is used to perform a second positioning on the image to be tested using TLD to obtain a second position at the current moment.

[0081] Therefore, the activity trajectory extracted by the tracker is used to extract candidate positive samples, and the first position of the speaker at the historical moment is used to further screen out the target positive samples, and the target positive samples are used to train the detector until convergence. On this basis, TLD is used to perform a second positioning on the image to be tested to obtain the second position at the current moment. Therefore, the positive samples used to train the detector are not only determined based on the activity trajectory of the tracker, but also the sound source positioning judgment is added at the same time, thereby improving the reliability of the positive samples, which is beneficial to improving the accuracy of the detector while reducing the computational complexity.

[0082] In some disclosed embodiments, the second positioning module 52 includes a learning weight submodule for assigning learning weights to each target positive sample based on a TLD learner in response to the orientation deviation between the first orientation and the second orientation at a historical moment before the current moment satisfying a first condition; the model training submodule is specifically used to train the detector until convergence based on the target positive samples assigned learning weights.

[0083] Therefore, after obtaining the positive feedback of sound source localization, the credibility of the target positive sample can be increased by giving it a learning weight through the learner.

[0084] In some disclosed embodiments, the combined module 53 includes a reliable detection submodule, which is used to perform a reliability detection on the first orientation to obtain a first detection result of the first orientation, and to perform a reliability detection on the second orientation to obtain a second detection result of the second orientation; the combined module 53 includes a final orientation submodule, which is used to obtain a final orientation based on the first detection result and the second detection result.

[0085] Therefore, firstly performing reliability tests on the first orientation and the second orientation respectively, and then obtaining the final orientation based on the reliability test results, is conducive to improving the reliability of the final orientation.

[0086] In some disclosed embodiments, the reliable detection submodule may include a first detection unit, which is specifically used to determine that a first detection result includes that the first orientation is reliable when the first orientation is within a second deviation range of the second orientation, and to determine that the first detection result includes that the first orientation is unreliable when the first orientation exceeds the second deviation range of the second orientation.

[0087] Therefore, by comparing the degree of deviation of the first orientation from the second orientation, it is determined whether the first orientation is reliable, thereby performing reliability detection from the perspective of taking the second orientation as a comparison reference.

[0088] In some disclosed embodiments, the reliable detection submodule may include a second detection unit, which is specifically used to detect whether the following two are simultaneously met: the similarity between the speaker's facial image and the sample image satisfies the second condition, and the orientation deviation between the second orientation and the first orientation satisfies the third condition; wherein the sample image is a positive sample of the detector used to train the TLD; if so, it is determined that the second detection result includes that the second orientation is reliable; if not, it is determined that the second detection result includes that the second orientation is unreliable.

[0089] Therefore, by comparing the degree to which the second orientation deviates from the first orientation and the similarity between the face image and the sample image, the reliability of the second orientation can be determined from multiple dimensions.

[0090] In some disclosed embodiments, the final orientation submodule is specifically configured to, in response to the first detection result including that the first orientation is reliable and the second detection result including that the second orientation is reliable, fuse the first orientation and the second orientation to obtain the final orientation.

[0091] Therefore, when both the first orientation and the second orientation are detected reliably, obtaining the final orientation by fusing the two is beneficial to improving the accuracy of the final orientation.

[0092] In some disclosed embodiments, the video call device 50 also includes a sound source localization module for, in response to detecting a target audio containing a preset wake-up word, performing sound source localization based on the target audio to obtain a target position of a speaker; the video call device 50 also includes a rotation module for controlling the video call device to turn toward the speaker based on the target position; the video call device 50 also includes a face recognition module for performing face recognition based on a target image taken by the video call device to obtain a face image of the speaker; wherein the face image serves as a tracking target for performing the second positioning.

[0093] Therefore, the speaker can use a preset wake-up word to make the video call device automatically turn to the speaker's location, and then formally conduct the video call, which can improve the convenience of the video call.

[0094] See also Figure 6 , Figure 6 6 is a schematic diagram of a framework of an embodiment of a video call device 60 of the present application. The video call device 60 includes a screen 61, a microphone 62, a camera 63, a communication circuit 64, a memory 65 and a processor 66. The screen 61, the microphone 62, the camera 63, the communication circuit 64 and the memory 65 are respectively coupled to the processor 66. The memory 65 stores program instructions. The processor 66 is used to execute the program instructions to implement the steps in any of the above-mentioned video call method embodiments. Specifically, the video call device 60 may include but is not limited to: a speaker with a screen, a surveillance camera, etc., which are not limited here.

[0095] Specifically, the processor 66 is used to control itself as well as the screen 61, the microphone 62, the camera 63, the communication circuit 64, and the memory 65 to implement the steps in any of the above-mentioned video call method embodiments. The processor 66 can also be called a CPU (Central Processing Unit). The processor 66 may be an integrated circuit chip with signal processing capabilities. The processor 66 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 66 can be implemented by an integrated circuit chip.

[0096] In an implementation scenario, such as Figure 6 As shown, the video call device 60 further includes a rotating mechanism 67, which at least carries the camera 63, and is used to drive the camera 63 to rotate so as to capture the speaker during the speaker's activities. In addition, the rotating mechanism 67 can also carry at least one of the screen 61 and the microphone 62 to drive the rotation thereof. It should be noted that the rotating mechanism 67 can include but is not limited to a pan / tilt head, etc., which is not limited here.

[0097] In the above scheme, on the one hand, during a video call, the final position is obtained by combining the first position and the second position obtained by sound source positioning and image positioning respectively, which is conducive to improving positioning accuracy compared with a single positioning method. On the other hand, during a video call, at least one of the sound source positioning and the image positioning refers to the positioning results of several previous historical moments during the positioning process, which is conducive to maintaining stable positioning during a video call with continuous time sequence. Therefore, the positioning accuracy of the speaker can be improved.

[0098] See also Figure 7 , Figure 7 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 70 of the present application. The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor, and the program instructions 71 are used to implement the steps in any of the above-mentioned video call method embodiments.

[0099] In the above scheme, on the one hand, during a video call, the final position is obtained by combining the first position and the second position obtained by sound source positioning and image positioning respectively, which is conducive to improving positioning accuracy compared with a single positioning method. On the other hand, during a video call, at least one of the sound source positioning and the image positioning refers to the positioning results of several previous historical moments during the positioning process, which is conducive to maintaining stable positioning during a video call with continuous time sequence. Therefore, the positioning accuracy of the speaker can be improved.

[0100] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0101] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0102] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A video calling method, characterized in that: include: Perform a first positioning on the audio data of the video call device at the current moment to obtain the first position of the speaker at the current moment, and perform a second positioning on the image to be tested at the current moment of the video call device to obtain the second position of the speaker at the current moment; wherein at least one of the first positioning and the second positioning refers to the positioning results of several historical moments before the current moment during the execution process, and the positioning results include the first position and the second position of the speaker at the historical moments, and the second positioning includes: extracting several candidate positive samples based on the activity trajectory determined by the TLD tracker; using the first position of the speaker at the historical moment to filter the several candidate positive samples to obtain a target positive sample; using the target positive sample to train the TLD detector until convergence; using the TLD to perform a second positioning on the image to be tested to obtain the second position at the current moment; Combining the first position and the second position, obtaining a final position of the speaker at the current moment; Based on the final position, a video call is conducted through the video calling device.

2. The method according to claim 1, characterized in that The first positioning refers to the positioning result during execution; the first positioning of the audio data of the video call device at the current moment to obtain the first position of the speaker at the current moment includes: For each of the historical moments, based on the position deviation between the first position and the second position at the historical moment and the human voice detection result of the call audio collected at the historical moment, obtaining a fusion weight corresponding to the historical moment; Using the fusion weights of the historical moments, the cross-correlation results of the call audio at the historical moments are fused to obtain the weighted cross-correlation result at the current moment; Based on the weighted cross-correlation result, the first orientation at the current moment is obtained.

3. The method according to claim 2, characterized in that The obtaining of the fusion weight corresponding to the historical moment based on the position deviation between the first position and the second position at the historical moment and the human voice detection result of the call audio collected at the historical moment includes: Based on the position deviation between the first position and the second position at the historical moment, a first weight coefficient is obtained, and based on the human voice detection result of the call audio collected at the historical moment, a second weight coefficient is obtained; wherein the position deviation is negatively correlated with the first weight coefficient; Based on the first weight coefficient and the second weight coefficient, a fusion weight corresponding to the historical moment is obtained.

4. The method according to claim 3, characterized in that When the first orientation is within a first deviation range of the second orientation, the first weight coefficient is a first value; when the first orientation exceeds the first deviation range of the second orientation, the first weight coefficient is a second value, and the first value is greater than the second value.

5. The method according to claim 3, characterized in that: When the human voice detection includes detecting human voice in the call audio, the second weight coefficient is a third value; when the human voice detection result includes not detecting human voice in the call audio, the second weight coefficient is a fourth value, and the third value is greater than the fourth value.

6. The method according to claim 1, characterized in that Before the detector of the TLD is trained to convergence using the target positive sample, the method further includes: In response to the orientation deviation between the first orientation and the second orientation at the historical moment before the current moment satisfying a first condition, assigning a learning weight to each of the target positive samples based on the TLD learner; The step of training the TLD detector using the target positive sample until convergence includes: The detector is trained to convergence based on the target positive samples assigned to the learning weights.

7. The method according to claim 1, characterized in that The combining the first position and the second position to obtain the final position of the speaker at the current moment includes: Performing a reliability test on the first orientation to obtain a first test result of the first orientation, and performing a reliability test on the second orientation to obtain a second test result of the second orientation; The final position is obtained based on the first detection result and the second detection result.

8. The method according to claim 7, characterized in that The performing reliability detection on the first orientation to obtain a first detection result of the first orientation includes: In a case where the first orientation is within a second deviation range of the second orientation, determining that the first detection result includes that the first orientation is reliable; In a case where the first orientation exceeds a second deviation range of the second orientation, it is determined that the first detection result includes that the first orientation is unreliable.

9. The method according to claim 7, characterized in that: The performing reliability detection on the second orientation to obtain a second detection result of the second orientation includes: Detecting whether the following two conditions are satisfied at the same time: the similarity between the face image of the speaker and the sample image satisfies a second condition, and the orientation deviation between the second orientation and the first orientation satisfies a third condition; wherein the sample image is a positive sample for training the detector of the TLD; If so, determining that the second detection result including the second orientation is reliable; If not, it is determined that the second detection result including the second position is unreliable.

10. The method according to claim 7, characterized in that The obtaining the final position based on the first detection result and the second detection result includes: In response to the first detection result including that the first orientation is reliable and the second detection result including that the second orientation is reliable, the first orientation and the second orientation are fused to obtain the final orientation.

11. The method according to claim 1, characterized in that: Before performing a first positioning on the audio data of the video calling device at the current moment to obtain a first position of the speaker at the current moment, or after conducting a video call through the video calling device based on the final position, the method further includes: In response to detecting a target audio containing a preset wake-up word, performing sound source localization based on the target audio to obtain a target position of the speaker; Based on the target position, controlling the video call device to turn toward the speaker; Performing face recognition based on the target image captured by the video calling device to obtain a face image of the speaker; The face image is used as a tracking target for performing the second positioning.

12. The method according to claim 1, characterized in that The performing a video call through the video calling device based on the final position includes at least one of the following: Performing beamforming based on the final orientation to suppress noise around the speaker; Based on the final position satisfying the fourth condition, the video calling device is controlled to turn toward the speaker.

13. A video call device, characterized in that: include: A first positioning module, used to perform a first positioning on the audio data of the video call device at the current moment to obtain a first position of the speaker at the current moment; A second positioning module is used to perform a second positioning on the image to be tested at the current moment of the video call device to obtain a second position of the speaker at the current moment; wherein, at least one of the first positioning and the second positioning refers to the positioning results of several historical moments before the current moment during execution, and the positioning results include the first position and the second position of the speaker at the historical moment, and the second positioning includes: extracting several candidate positive samples based on the activity trajectory determined by the TLD tracker; using the first position of the speaker at the historical moment to filter the several candidate positive samples to obtain a target positive sample; using the target positive sample to train the TLD detector until convergence; using the TLD to perform a second positioning on the image to be tested to obtain the second position at the current moment; A combining module, configured to combine the first position and the second position to obtain a final position of the speaker at the current moment; A call module is used to make a video call through the video call device based on the final position.

14. A video calling device, characterized in that: The device comprises a screen, a microphone, a camera, a communication circuit, a memory and a processor, wherein the screen, the microphone, the camera, the communication circuit and the memory are respectively coupled to the processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the video call method described in any one of claims 1 to 12.

15. The device according to claim 14, characterized in that The video calling device further comprises a rotating mechanism, which at least carries the camera, and is used to drive the camera to rotate so as to photograph the speaker during the speaker's activities.

16. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the video calling method according to any one of claims 1 to 12.