Video conference method and device, electronic equipment and storage medium

Abstract

The invention discloses a video conference method and device, electronic equipment and a storage medium, and belongs to the technical field of video conferences, and the method comprises the steps: obtaining a scene image through a camera at a same fixed position, and obtaining at least different position images of a same target in a same scene at an interval preset distance; establishing a 3D coordinate system according to the scene image, and determining a scene and target parameters; correcting the scene and the 3D coordinate system corresponding to the scene according to the height value of the camera, the parameter data of the target and the horizon plane; collecting a video image and a voice signal synchronized with the video image; determining the position of the current sounder in the video image based on the 3D coordinate system according to the video image and the voice signal; and marking the identity of the current sounder according to the position of the current sounder in the video image and the corresponding voice information. The problem of inaccurate position identification possibly existing in the video conference is solved, and the overall performance of the system and the user experience are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video conferencing, and in particular relates to a video conferencing method, device, electronic equipment and storage medium. Background Art

[0002] Video conferencing methods Video conferencing refers to a meeting where people in two or more locations have face-to-face conversations through communication equipment and the Internet. Depending on the number of participating locations, video conferencing can be divided into point-to-point conferencing and multi-point conferencing.

[0003] Using the video conferencing system, participants can hear the sounds of other venues, see the images, movements and expressions of participants in other venues, and send electronic presentation content to make participants feel as if they are personally on the scene.

[0004] In current video conferencing, due to the influence of the scene environment and the uncertainty of the voice signal, the speaker's voiceprint features also have various uncertainties, which in turn affects the recognition accuracy of the participants, especially in multi-person or multi-terminal video conferencing. For example, due to the influence of the scene environment and the camera hardware itself, there may be problems such as inaccurate position recognition and camera selection, that is, for different cameras, due to their different imaging components and other hardware, the recognition accuracy may vary greatly. Therefore, the existing video conferencing methods and devices still have a lot of room for improvement. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a video conferencing method and device.

[0006] The technical solution adopted to solve the above technical problems is: a video conferencing method, comprising: Acquire, by means of a camera at a same fixed position, a scene image and images of the same target at different positions at least at a preset distance apart in the same scene; Establishing a 3D coordinate system according to the scene image, determining the horizon plane of the scene, and measuring the camera height value, and the length and / or width data of the target; Correcting the scene and its corresponding 3D coordinate system according to the height value of the camera and at least two length or width data of the target and the horizon plane; When a participant appears in the scene, a video image and a voice signal synchronized with the video image are collected; Based on the 3D coordinate system and according to the video image and the voice signal, determine the position of the current speaker in the video image; The current speaker is identified according to the position of the current speaker in the video image and the corresponding voice information.

[0007] Furthermore, the method of acquiring the scene image and the images of the same target in the same scene at least at different positions with a preset distance by using a camera at the same fixed position includes: Fix the camera at a certain height and obtain the scene image through the camera; The target is sequentially set at a first position and a second position, and images containing the complete length and / or width of the target are respectively acquired.

[0008] Furthermore, the target is sequentially set at the first position and the second position, and images containing the complete length and / or width of the target are respectively acquired, and then the method further includes: The target is positioned at the third position and the fourth position in turn, and images containing the complete length and / or width of the target are acquired respectively; wherein the first position, the second position, the third position and the fourth position are located at different orientations, and at least two positions are located at the edge of the scene.

[0009] Furthermore, the step of establishing a 3D coordinate system according to the scene image, determining the horizon plane of the scene, and measuring the camera height value and the length and / or width data of the target includes: Establishing an internal 3D coordinate system according to the preset AR system and the scene image, and mapping the ground of the scene to a horizon plane in the image; Measure the camera's height relative to the scene's ground, as well as the object's length and / or width.

[0010] Furthermore, the correction of the scene and its corresponding 3D coordinate system based on the height value of the camera and at least two length or width data of the target, and the horizon plane, includes: Determine the camera's field of view according to the camera's height value; According to the length or width data of the target and its image in the scene, determine the proportional relationship between the target object and the image; Determine, according to the field of view angle and the horizon plane, a mapping relationship of the scene relative to each orthographic projection plane of the 3D coordinate system; According to the proportional relationship, the mapping correspondence between the scene position and the 3D coordinate position is corrected.

[0011] Furthermore, the determining the position of the current speaker in the video image based on the 3D coordinate system and according to the video image and the voice signal includes: Determining, according to the voice signal, a voiceprint feature of the speaker; Determining the identity information of the speaker according to the voiceprint feature; Use directional microphones and cameras in meeting scenarios to determine the position of the speaker in the video image.

[0012] Furthermore, it also includes: According to the position of the current speaker in the video image and the corresponding voiceprint information, the current speaker is identified, the voice information is converted into text information, and the identity information for identifying the speaker is added to the text in each piece of text information.

[0013] A video conferencing device, comprising: An image acquisition module, used to acquire scene images and images of the same target in the same scene at least at different positions separated by a preset distance, through a camera at the same fixed position; A coordinate system generation module, used to establish a 3D coordinate system according to the scene image, determine the horizon plane of the scene, and measure the camera height value, and the length and / or width data of the target; A data correction module, used for correcting the scene and its corresponding 3D coordinate system according to the height value of the camera and at least two length or width data of the target, and the horizon plane; A synchronous acquisition module, used to acquire a video image and a voice signal synchronized with the video image when a participant appears in the scene; A position determination module, configured to determine the position of a current speaker in the video image based on the 3D coordinate system and according to the video image and the voice signal; The identity tagging module is used to tag the current speaker according to the position of the current speaker in the video image and the corresponding voice information.

[0014] An electronic device, comprising: One or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the electronic device to perform a video conferencing method.

[0015] A computer-readable storage medium stores a computer program that enables a processor to execute a video conferencing method.

[0016] The beneficial effects of the present invention are as follows: Through the camera at the same fixed position, the scene image and the images of at least different positions of the same target at a preset distance in the same scene are obtained; a 3D coordinate system is established according to the scene image, and the horizon plane of the scene is determined, and the camera height value and the length and / or width data of the target are measured; the scene and the corresponding 3D coordinate system are corrected according to the camera height value and at least two length or width data of the target, as well as the horizon plane; when a participant appears in the scene, a video image and a voice signal synchronized with the video image are collected; based on the 3D coordinate system, the position of the current speaker in the video image is determined according to the video image and the voice signal; the current speaker is identified according to the position of the current speaker in the video image and the corresponding voice information. Through 3D space modeling and multimodal information fusion methods, the problem of inaccurate position recognition and the problem of picking cameras in video conferencing can be effectively solved, that is, for different cameras, due to their different imaging components and other hardware, the recognition accuracy may vary greatly, thereby improving the overall performance of the system and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of the steps of a video conferencing method provided by one embodiment of the present invention; Figure 2 is a structural block diagram of a video conferencing device provided by an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] Reference Figure 1 , showing a flow chart of steps of a video conferencing method of this embodiment, including: Step S100, acquiring a scene image and images of the same target in the same scene at different positions at least at a preset distance apart through a camera at the same fixed position; Step S200, establishing a 3D coordinate system according to the scene image, determining the horizon plane of the scene, and measuring the camera height value, and the length and / or width data of the target; Step S300, calibrating the scene and its corresponding 3D coordinate system according to the height value of the camera and at least two length or width data of the target, and the horizon plane; Step S400: When a participant appears in the scene, a video image and a voice signal synchronized with the video image are collected; Step S500: determining the position of the current speaker in the video image based on the 3D coordinate system and according to the video image and the voice signal; Step S600: annotate the current speaker according to the position of the current speaker in the video image and the corresponding voice information.

[0020] In this embodiment, the scene image and the image of the target at different positions are obtained by the camera at the same fixed position, a 3D coordinate system is established, and the horizon plane of the scene is determined, and the camera height value and the length and / or width data of the target are measured. This step helps to establish an accurate three-dimensional space model for subsequent positioning and correction; the scene and its corresponding 3D coordinate system are corrected according to the height value of the camera and the length or width data of the target, as well as the horizon plane; it ensures that even under different camera viewing angles, the consistent spatial relationship and proportion can be maintained, reducing the recognition error caused by hardware differences; when a participant appears in the scene, the video image and the synchronized voice signal are collected. Through the 3D coordinate system, the position of the current speaker in the video image is determined in combination with the video image and the voice signal. The accuracy of positioning can be improved by this multimodal information fusion, especially when the voiceprint feature is unclear or the environmental noise is large. According to the position of the speaker in the video image and the corresponding voice information, the current speaker is identified. This helps participants to quickly identify and improve the interactive efficiency of the meeting.

[0021] Through 3D spatial modeling and correction, it is possible to reduce recognition errors caused by scene environments and hardware differences, and improve recognition accuracy in multi-person or multi-end video conferences. Combining visual and voice information for positioning can provide supplementary information when a single modality is insufficient for accurate recognition, thus enhancing the robustness of the system; accurate voiceprint and location recognition can help participants identify more quickly, thereby optimizing the meeting process and participant experience.

[0022] In one embodiment of the present application, the acquisition of scene images and images of at least different positions of the same target at a preset distance in the same scene by a camera at the same fixed position includes: Fix the camera at a certain height and obtain the scene image through the camera; for example, fix the camera at a height of 1.5 to 1.8 meters. Specifically, the camera can be fixed to the top of a large screen or a position on a whiteboard; obtain the scene image as the background image of the scene through the camera at a fixed position.

[0023] A measurement target can be set, which can be a ruler or a target with a fixed length, or a person; the target is set in the first position and the second position in sequence, and images containing the complete target length and / or width are obtained respectively, for example, a fixed-length measuring ruler or a certain length of pole is placed in different positions within the field of view of the camera, and images of at least two positions are obtained respectively; wherein the two positions are at different horizontal distances from the camera, for example, 3 meters apart; since the same target has a different imaging size in the same scene when it is close to the camera than when it is far away. The two images of the same target are used as the measurement target for subsequent comparison.

[0024] Furthermore, the target is sequentially set at the first position and the second position, and images containing the complete length and / or width of the target are respectively acquired, and then the method further includes: The target is positioned at the third and fourth positions in turn, and images containing the complete length and / or width of the target are obtained respectively; wherein the first, second, third and fourth positions are located at different positions, and at least two positions are located at the edge of the scene. By setting the edge position, the scene can be divided into local areas, and only the image and audio content within the divided area are rendered, and the contents outside are not responded to, which can save system resources and improve the system's data processing efficiency. Only responding to the sound in the local area, when the distance of the audio signal is not within the scene, that is, in the video image, if the sound is outside the divided scene, the system will treat the audio as noise and block it.

[0025] It should be noted that, since the proportional relationship of the image may be nonlinear, more positions can be set so that the target is located in these positions, and the corresponding images can be acquired. The scene and 3D coordinates can be corrected according to the difference between the image and the actual measurement, which will make the identified position more accurate.

[0026] It should also be noted that, in video conferencing, video and audio are different signals, and the recognition methods are different. In subsequent processing and recognition, the video and audio can be better matched. Although the current face recognition technology is quite mature, in video conferencing, participants may only show their side faces or be blocked by other people in the video, and due to the limitations of the conference room, especially in smaller conference rooms, participants may be close to each other, making recognition more difficult. Through the above correction of the scene, the position of the participants can be accurately identified and corresponded to their voiceprints and identity information. When it is detected that the sound source is not in the scene, the voice information can be shielded, so that only the audio in the scene is effective audio, which can prevent the sound from being affected by external noise, especially when the environment is not ideal and unstable, which may cause the problem of extremely poor meeting effect. Thereby improving the audio effect in video conferencing.

[0027] In one embodiment of the present application, the step of establishing a 3D coordinate system according to the scene image, determining the horizon plane of the scene, and measuring the camera height value and the length and / or width data of the target includes: An internal 3D coordinate system is established based on the preset AR system and the scene image, and the ground of the scene is mapped to the horizon plane in the image; the height value of the camera relative to the ground of the scene, as well as the length and / or width data of the target are measured. Since the video or the system itself does not know the actual ground position, by establishing 3D coordinates, the ground closest to the visual reach in the video is used as the horizon, and the ground of the scene is formed with the closest ground of the imaging.

[0028] In one embodiment of the present application, the correction of the scene and its corresponding 3D coordinate system based on the height value of the camera and at least two length or width data of the target, and the horizon plane, includes: According to the height value of the camera, determine the camera's field of view; the field of view (FOV) is divided into the horizontal field of view (HFOV) and the vertical field of view (VFOV), which determine the camera's field of view in the horizontal and vertical directions, respectively. The focal length and the size of the imaging area are the main factors affecting the field of view. The longer the focal length, the smaller the field of view, and the narrower the field of view; the shorter the focal length, the larger the field of view, and the wider the field of view. The determining factors of the horizontal field of view are mainly the focal length and the width of the imaging area, while the main determining factors of the vertical field of view are the focal length and the height of the imaging area. Calculation formula for horizontal field of view: horizontal field of view (HFOV) = 2arctan (w / 2f), vertical field of view (VFOV) = 2arctan (h / 2f), where w is the width of the field of view, h is the height of the field of view, and f is the focal length of the lens. According to the length or width data of the target and its image in the scene, determine the proportional relationship between the target object and the image; wherein, the length or width of the image of the target in the system can be measured by using the tools provided by the system, and by measuring the length or width of the target, the size correspondence between the real object and the image can be obtained; According to the viewing angle and the horizon plane, the mapping relationship of the scene relative to each orthographic projection plane of the 3D coordinate system is determined; according to the proportional relationship, the mapping correspondence between the scene position and the 3D coordinate position is corrected; after determining the correspondence according to the viewing angle and the horizon plane, the ground of the scene can be determined as the intersection of the X-axis and the Y-axis of the 3D coordinate system, and the actual scene and the 3D coordinates are mapped according to the size correspondence. Through the scene mapping relationship and the position mapping relationship, when the participant speaks, it can be more accurately located which one speaks. It solves the problem that in the target AR video conference, because the 3D coordinate system of the scene will not be corrected, and because the image of the camera may be distorted, the image and the coordinate position are offset, so that the recognized position may be offset, making the recognition prone to "mistaken" confusion, that is, the recognition of the original voice of participant A may indicate the position of participant B due to the coordinate problem. Therefore, through the above-mentioned 3D coordinate and scene correction, it is ensured that the recognized position can be more accurate.

[0029] In an embodiment of the present application, determining the position of the current speaker in the video image based on the 3D coordinate system and according to the video image and the voice signal includes: Determine the voiceprint features of the speaker based on the voice signal; determine the identity information of the speaker based on the voiceprint features; and determine the position of the speaker in the video image in the conference scene using a directional microphone and a camera.

[0030] In this embodiment, after a multi-person video conference begins, the video images of the conference site and the language signals synchronized with the video images can be collected in real time, and then the speaker can be determined based on the collected video images and voice signals, and the identity of the determined speaker can be marked in the video image.

[0031] In one embodiment of the present application, it also includes: according to the position of the current speaker in the video image and according to the corresponding voiceprint information, the current speaker is identified, the voice information is converted into text information, and the identity information for identifying the speaker is added to the text in each piece of text information.

[0032] For example, when an identity is marked for a determined speaker in a video image, relevant information of the corresponding speaker can be marked at the speaker's position determined in the video image, where the relevant information of the speaker includes adding labels to the participants, such as: the participant's name, gender, position, contact information, etc., or it can be a code, such as A001, A002, etc.

[0033] After the identity of the determined speaker is marked in the video image, the video image can be sent to the conference equipment of each participant for use by multiple participants, for example: when generating meeting minutes; or, the video image can be sent to a third party; through the generated minutes, the sound is converted into corresponding text, and the corresponding identity tag is added to the text according to the voiceprint information, and the text can be sorted in chronological order, so that when viewing it, you can know what someone said at a certain moment.

[0034] It should be noted that the approximate position of the speaker can be directly obtained through a directional microphone, and recognition can be performed based on a combination of voiceprint analysis and image analysis to further improve recognition accuracy.

[0035] In this embodiment, the voice signal can be processed by using a voice model, for example, a deep neural network DNN model, a convolutional neural network CNN model, etc. The voiceprint feature is extracted from the acquired voice signal by the voice model to obtain a voiceprint vector, and the voiceprint vector is input into the classification model to obtain the speaker's information.

[0036] In one embodiment of the present application, an image recognition model can be used to identify and distinguish speakers. That is to say, in this embodiment, the collected video image is only identified for the lips in the image, without face recognition, to distinguish whether the speaker is speaking, and combined with voiceprint recognition to locate the speaker in the image and mark the identity information. For lip analysis, a CNN-LSTM model or a 3D-ConvNet model can be used. The CNN-LSTM model is an integrated model of a convolutional neural network CNN and a long short-term memory network LSTM. The CNN part of the model processes data, and the one-dimensional result is input into the LSTM model. The 3D-ConvNet model is a convolutional neural network model with a 3D convolution kernel. The 3D convolution module in the 3D-ConvNet model can be used to extract the temporal and spatial features of the video frame in the first video image to identify whether the participant is speaking.

[0037] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0038] like Figure 2 As shown, in an embodiment of the present invention, a video conferencing device is also disclosed, including: An image acquisition module 100 is used to acquire a scene image and images of the same object in the same scene at at least different positions separated by a preset distance through a camera at the same fixed position; A coordinate system generation module 200 is used to establish a 3D coordinate system according to the scene image, determine the horizon plane of the scene, and measure the camera height value, and the length and / or width data of the target; A data correction module 300, for correcting a scene and its corresponding 3D coordinate system according to a height value of a camera and at least two length or width data of a target, and a horizon plane; A synchronous acquisition module 400, used to acquire a video image and a voice signal synchronized with the video image when a participant appears in the scene; A position determination module 500, configured to determine the position of a current speaker in the video image based on the 3D coordinate system and according to the video image and the voice signal; The identity tagging module 600 is used to tag the current speaker according to the position of the current speaker in the video image and the corresponding voice information.

[0039] Reference Figure 3 In an embodiment of the present invention, the present invention further provides a computer device. The above-mentioned computer device 12 is in the form of a general computing device. The components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0040] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor, or a local bus 18 using any of a variety of bus 18 architectures. These architectures include, by way of example, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.

[0041] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0042] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 31 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although Figure 3 Not shown, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42, which are configured to perform the functions of various embodiments of the present invention.

[0043] A program / utility 41 having a set (at least one) of program modules 42 may be stored in, for example, a memory, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0044] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, camera, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 12, and / or communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. In addition, the computer device 12 may also communicate with one or more networks (e.g., local area network (LAN)), wide area network (WAN) and / or public network (e.g., Internet) through a network adapter 20. As shown, the network adapter 21 communicates with other modules of the computer device 12 through a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device driver, redundant processing unit 16, external disk drive array, RAID system, tape drive, and data backup storage system 34, etc.

[0045] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the video conferencing method provided in the embodiment of the present invention.

[0046] That is, when the processing unit 16 executes the above program, it is achieved: through a camera at the same fixed position, a scene image and images of at least different positions of the same target at a preset distance in the same scene are obtained; a 3D coordinate system is established according to the scene image, and the horizon plane of the scene is determined, and the camera height value and the length and / or width data of the target are measured; the scene and the corresponding 3D coordinate system are corrected according to the camera height value and at least two length or width data of the target, and the horizon plane; when a participant appears in the scene, a video image and a voice signal synchronized with the video image are collected; based on the 3D coordinate system and according to the video image and the voice signal, the position of the current speaker in the video image is determined; according to the position of the current speaker in the video image and the corresponding voice information, the current speaker is identified.

[0047] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video conferencing method provided in all embodiments of the present application.

[0048] That is, when the program is executed by the processor, it is implemented as follows: through a camera at the same fixed position, a scene image and images of the same target at at least different positions at a preset distance in the same scene are obtained; a 3D coordinate system is established according to the scene image, and the horizon plane of the scene is determined, and the camera height value and the length and / or width data of the target are measured; the scene and its corresponding 3D coordinate system are corrected according to the camera height value and at least two length or width data of the target, and the horizon plane; when a participant appears in the scene, a video image and a voice signal synchronized with the video image are collected; based on the 3D coordinate system and according to the video image and the voice signal, the position of the current speaker in the video image is determined; according to the position of the current speaker in the video image and the corresponding voice information, the current speaker is identified.

[0049] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device.

[0050] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0051] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, Python, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0052] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0053] Those skilled in the art will appreciate that the embodiments of the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present invention may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0054] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0055] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0056] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0057] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0058] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0059] The above is a detailed introduction to a video conferencing method, device, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A video conferencing method, characterized in that: include: Acquire, by means of a camera at a same fixed position, a scene image and images of the same target at different positions at least at a preset distance apart in the same scene; Establishing a 3D coordinate system according to the scene image, determining the horizon plane of the scene, and measuring the camera height value, and the length and / or width data of the target; Correcting the scene and its corresponding 3D coordinate system according to the height value of the camera and at least two length or width data of the target and the horizon plane; When a participant appears in the scene, a video image and a voice signal synchronized with the video image are collected; Based on the 3D coordinate system and according to the video image and the voice signal, determine the position of the current speaker in the video image; The current speaker is identified according to the position of the current speaker in the video image and the corresponding voice information.

2. The method according to claim 1, characterized in that: The method of acquiring a scene image and images of the same target in the same scene at least at different positions with a preset distance by using a camera at the same fixed position includes: Fix the camera at a certain height and obtain the scene image through the camera; The target is sequentially set at a first position and a second position, and images containing the complete length and / or width of the target are respectively acquired.

3. The method according to claim 2, characterized in that The target is sequentially set at the first position and the second position, and images containing the complete length and / or width of the target are respectively acquired, and then the following further comprises: The target is positioned at the third position and the fourth position in turn, and images containing the complete length and / or width of the target are acquired respectively; wherein the first position, the second position, the third position and the fourth position are located at different orientations, and at least two positions are located at the edge of the scene.

4. The method according to claim 1, characterized in that: The step of establishing a 3D coordinate system according to the scene image, determining the horizon plane of the scene, and measuring the camera height value and the length and / or width data of the target includes: Establishing an internal 3D coordinate system according to the preset AR system and the scene image, and mapping the ground of the scene to a horizon plane in the image; Measure the camera's height relative to the scene's ground, as well as the object's length and / or width.

5. The method according to claim 1, characterized in that The method of correcting the scene and its corresponding 3D coordinate system according to the height value of the camera and at least two length or width data of the target and the horizon plane includes: Determine the camera's field of view according to the camera's height value; According to the length or width data of the target and its image in the scene, determine the proportional relationship between the target object and the image; Determine, according to the field of view angle and the horizon plane, a mapping relationship of the scene relative to each orthographic projection plane of the 3D coordinate system; According to the proportional relationship, the mapping correspondence between the scene position and the 3D coordinate position is corrected.

6. The method according to claim 1, characterized in that The determining the position of the current speaker in the video image based on the 3D coordinate system and according to the video image and the voice signal includes: Determining, according to the voice signal, a voiceprint feature of the speaker; Determining the identity information of the speaker according to the voiceprint feature; Use directional microphones and cameras in meeting scenarios to determine the position of the speaker in the video image.

7. The method according to claim 1, characterized in that Also includes: According to the position of the current speaker in the video image and the corresponding voiceprint information, the current speaker is identified, the voice information is converted into text information, and the identity information for identifying the speaker is added to the text in each piece of text information.

8. A video conferencing device, characterized in that: include: An image acquisition module, used to acquire scene images and images of the same target in the same scene at least at different positions separated by a preset distance, through a camera at the same fixed position; A coordinate system generation module, used to establish a 3D coordinate system according to the scene image, determine the horizon plane of the scene, and measure the camera height value, and the length and / or width data of the target; A data correction module, used for correcting the scene and its corresponding 3D coordinate system according to the height value of the camera and at least two length or width data of the target, and the horizon plane; A synchronous acquisition module, used to acquire a video image and a voice signal synchronized with the video image when a participant appears in the scene; A position determination module, configured to determine the position of a current speaker in the video image based on the 3D coordinate system and according to the video image and the voice signal; The identity tagging module is used to tag the current speaker according to the position of the current speaker in the video image and the corresponding voice information.

9. An electronic device, characterized in that: include: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the electronic device to perform the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer program stored therein enables the processor to execute the method according to any one of claims 1 to 7.

Images