Shooting control method and device, intelligent equipment and storage medium

Abstract

The embodiment of the invention discloses a shooting control method and device, intelligent equipment and a storage medium, the intelligent equipment comprises a virtual camera driving program and a target application program, the shooting control method comprises the following steps: receiving video data collected by each shooting device, the number of the shooting devices being at least three, and shooting a shooting space from different positions; selecting a face which is speaking in one path of video data as an identified speaker in the shooting space; determining three-dimensional position coordinates of the spokesman relative to each shooting device; according to each path of video data and the corresponding relative position information, determining an optimal shooting device for shooting the spokesman in each shooting device; and sending video data shot by the optimal shooting device to the virtual camera driving program, so that the virtual camera driving program sends the video data to the target application program for use. By adopting the method, the technical problem that the camera cannot shoot the object needing to be shot due to the limited shooting range can be solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a shooting control method, apparatus, smart device, and storage medium. Background Technology

[0002] Today, intelligent technologies are widely used in daily work and learning scenarios such as offices and schools. For example, intelligent devices (such as conference tablets or smart blackboards) are frequently used in business meetings or classroom teaching to achieve intelligent applications. For instance, intelligent devices installed in classrooms or meeting rooms can record classroom or meeting scenes. However, the camera angle of intelligent devices is fixed. Because the camera's field of view is limited, when the object to be filmed is not within the field of view or is obstructed by other objects, the intelligent device cannot obtain video footage showing that object, thus affecting the user experience. Summary of the Invention

[0003] This application provides a shooting control method, apparatus, smart device, and storage medium to solve the technical problem in the related art that cameras cannot capture the objects to be captured due to limited shooting range.

[0004] In a first aspect, one embodiment of this application provides a shooting control method applied to a smart device, wherein the smart device has a virtual camera driver and a target application installed, and the shooting control method includes:

[0005] Receive video data collected by each shooting device, wherein there are at least three shooting devices shooting the shooting space from different positions;

[0006] Identify each face appearing in each of the video data streams, and select the face that is speaking in one of the video data streams as the identified speaker in the shooting space;

[0007] Based on the speaker's two-dimensional coordinates in the video data, determine the speaker's three-dimensional position coordinates relative to each of the shooting devices;

[0008] Based on the video data of each stream and the corresponding three-dimensional position coordinates, determine the best shooting device among the shooting devices to shoot the speaker;

[0009] The video data captured by the optimal shooting device is sent to the virtual camera driver, so that the virtual camera driver sends the video data to the target application for use.

[0010] The above-described method, by receiving video data from at least three shooting devices, selecting the speaking face from one of the video streams as the identified speaker in the shooting space, determining the speaker's three-dimensional position coordinates relative to each shooting device based on the speaker's two-dimensional coordinates in the respective video data, determining the optimal shooting device for capturing the speaker based on the speaker's three-dimensional position coordinates relative to each shooting device and the video data from each stream, and sending the video data captured by the optimal shooting device to a virtual camera driver, which then sends the video data to the target application for use, solves the technical problem in related technologies where cameras cannot capture the desired objects due to limited shooting range. Using at least three shooting devices to capture the shooting space from different positions ensures that all objects in the shooting space can be captured without blind spots. Furthermore, it can automatically identify the speaker and acquire the speaker's image, enabling tracking of the speaker's image and improving the user experience.

[0011] Secondly, one embodiment of this application provides a shooting control method applied to a smart device, wherein the smart device has a virtual camera driver and a target application installed, and the shooting control method includes:

[0012] Receive video data collected by each shooting device, wherein there are at least three shooting devices shooting the shooting space from different positions;

[0013] Based on the video data from each stream, the speaker in the shooting space is determined;

[0014] Calculate the relative position information of the speaker with respect to each of the camera devices;

[0015] Based on the video data of each stream and the corresponding relative position information, determine the best shooting device among the shooting devices to shoot the speaker;

[0016] The video data captured by the optimal shooting device is sent to the virtual camera driver, so that the virtual camera driver sends the video data to the target application for use.

[0017] The above-described method, by receiving video data from at least three shooting devices, identifying the speaker in the shooting space based on the video data, calculating the speaker's relative position information with respect to each shooting device, determining the optimal shooting device for capturing the speaker based on the relative position information and the video data, and sending the video data captured by the optimal shooting device to a virtual camera driver, which then sends the video data to the target application for use, solves the technical problem in related technologies where cameras cannot capture the desired objects due to limited shooting range. Using at least three shooting devices to capture the shooting space from different positions ensures that all objects in the shooting space can be captured without blind spots. Furthermore, it can automatically identify the speaker and acquire their image, enabling tracking of the speaker's image and improving the user experience.

[0018] In one embodiment of this application, the method further includes:

[0019] Determine the shooting parameter information of the optimal shooting device;

[0020] The shooting parameter information is sent to the optimal shooting device so that the optimal shooting device can adjust the shooting parameters it uses based on the shooting parameter information.

[0021] In addition to determining the best shooting device and obtaining the footage captured by the best shooting device for use, the shooting parameter information for use by the best shooting device can also be determined, which can further ensure the quality of the video data captured by the best shooting device, that is, ensure the quality of the captured speaker's footage.

[0022] In one embodiment of this application, before determining the shooting parameter information of the optimal shooting device, the method further includes:

[0023] Determine whether the shooting parameters of the optimal shooting device are adjustable;

[0024] If adjustable, then the operation of determining the shooting parameter information of the optimal shooting device is performed.

[0025] As mentioned above, the shooting parameter information is determined only when the shooting parameters of the optimal shooting device are adjustable. This can avoid invalid determination of shooting parameter information when a smart device is used with a non-adjustable shooting device.

[0026] In one embodiment of this application, determining the speaker in the shooting space based on the video data from each stream includes:

[0027] Face recognition is performed on each of the video data streams, and based on the results of face recognition on each of the video data streams, the face of the person speaking in one of the video data streams is selected as the speaker in the shooting space.

[0028] The above-mentioned method uses facial recognition to automatically identify speakers, eliminating the need for users to manually identify speakers.

[0029] In one embodiment of this application, the step of performing face recognition on each of the video data streams and selecting a speaking face from one of the video data streams as the identified speaker in the shooting space based on the face recognition results of each of the video data streams includes:

[0030] Perform facial recognition on the video data from each channel to obtain the lip image of each recognized face;

[0031] Based on the movement of the lips in the lip image, the face of the person speaking in one video stream is selected as the speaker in the shooting space, and the speaker's lip movement is the most obvious.

[0032] The above-mentioned method of identifying speakers by the movement of lips can ensure the rationality of automatic speaker identification.

[0033] In one embodiment of this application, calculating the relative position information of the speaker with respect to each of the camera devices includes:

[0034] Based on the speaker's two-dimensional coordinates in the video data, the relative position information of the speaker with respect to the corresponding shooting device in the video data is determined, and the relative position information of the speaker with respect to each other shooting device is determined according to the relative position information, wherein the relative position information is the three-dimensional position coordinates of the speaker with respect to the shooting device.

[0035] The above-mentioned combination of the speaker's two-dimensional coordinates in the video data can automatically obtain the speaker's relative position information with respect to each shooting device, which is convenient for subsequent determination of the best shooting device based on the relative position information.

[0036] In one embodiment of this application, determining the optimal shooting device for filming the speaker among the various shooting devices based on each video data stream and the corresponding relative position information includes:

[0037] The speaker's face in each of the video data is determined based on each of the relative position information;

[0038] The optimal camera device for photographing the speaker is determined based on the facial image.

[0039] As described above, based on the relative position information of the speaker with respect to each shooting device, the facial images of the speaker captured by each shooting device can be obtained. Then, based on the facial images, the best shooting device can be determined, thus achieving automatic determination of the best shooting device.

[0040] In one embodiment of this application, the step of determining the optimal shooting device for capturing the speaker based on the facial image includes:

[0041] The facial images are scored, and the camera device corresponding to the highest-scoring facial image is selected as the best camera device for filming the speaker.

[0042] The above method of scoring facial images to determine the best shooting device ensures the rationality of automatically determining the best shooting device.

[0043] In one embodiment of this application, determining the shooting parameter information of the optimal shooting device includes:

[0044] Based on the speaker's relative position information with respect to the optimal shooting device, multiple shooting parameter alternatives for the optimal shooting device are obtained;

[0045] The multiple shooting parameter candidate information is scored, and the shooting parameter candidate information with the highest score is selected as the determined shooting parameter information.

[0046] As mentioned above, when determining the shooting parameters, several alternative shooting parameters with better image quality can be identified based on the speaker's relative position information. These alternative shooting parameters can be scored and selected to ensure that the speaker's image quality is better when shot using the highest shooting parameter alternative.

[0047] In one embodiment of this application, the shooting parameters include Pan parameters, Tilt parameters, and Zoom parameters.

[0048] The process of determining the shooting parameter information of the optimal shooting device includes:

[0049] Based on the speaker's relative position information with respect to the optimal shooting device, determine the Pan parameter information and Tilt parameter information of the optimal shooting device;

[0050] The Zoom parameters of the optimal shooting device are determined based on the actual size of the speaker in the video data captured by the optimal shooting device, the preset desired size, and the focal length of the optimal shooting device.

[0051] As described above, the Pan and Tilt parameters can be determined based on the speaker's relative position information, and the Zoom parameter can be determined using the speaker's actual size, desired size, and the focal length of the optimal shooting device, thereby enabling the rapid determination of shooting parameter information while ensuring shooting quality.

[0052] Thirdly, one embodiment of this application also provides a shooting control device applied to a smart device, the smart device having a virtual camera driver and a target application installed, the shooting control device comprising:

[0053] The first video receiving unit is used to receive video data collected by each shooting device, wherein there are at least three shooting devices shooting the shooting space from different positions;

[0054] The first speaker identification unit is used to identify each face appearing in each of the video data streams and select the face that is speaking in one of the video data streams as the identified speaker in the shooting space.

[0055] The first position determination unit is used to determine the three-dimensional position coordinates of the speaker relative to each of the shooting devices based on the two-dimensional coordinates of the speaker in the video data.

[0056] The first device determining unit is used to determine the best shooting device for shooting the speaker among the shooting devices based on each video data and the corresponding three-dimensional position coordinates;

[0057] The first data sending unit is used to send the video data captured by the optimal shooting device to the virtual camera driver, so that the virtual camera driver can send the video data to the target application for use.

[0058] Fourthly, one embodiment of this application also provides a shooting control device applied to a smart device, the smart device having a virtual camera driver and a target application installed, the shooting control device comprising:

[0059] The second video receiving unit is used to receive video data collected by each shooting device, wherein there are at least three shooting devices shooting the shooting space from different positions;

[0060] The second speaker identification unit is used to determine the speaker in the shooting space based on the video data from each channel;

[0061] The second position determination unit is used to calculate the relative position information of the speaker with respect to each of the shooting devices;

[0062] The second device determining unit is used to determine the best shooting device for shooting the speaker among the shooting devices based on each video data and the corresponding relative position information;

[0063] The second data sending unit is used to send the video data captured by the optimal shooting device to the virtual camera driver, so that the virtual camera driver can send the video data to the target application for use.

[0064] Fifthly, one embodiment of this application also provides an intelligent device, which includes one or more processors and a memory;

[0065] The memory is used to store one or more programs;

[0066] When the one or more programs are executed by the one or more processors, the one or more processors implement the shooting control method as described in the first or second aspect.

[0067] In a sixth aspect, one embodiment of this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed, implements the shooting control method as described in the first or second aspect.

[0068] The beneficial effects of the aforementioned shooting control device, intelligent device, and storage medium can be referenced to the beneficial effects of the shooting control method. Attached Figure Description

[0069] Figure 1 A flowchart illustrating a shooting control method provided in one embodiment of this application;

[0070] Figure 2 A flowchart illustrating a shooting control method provided in another embodiment of this application;

[0071] Figure 3 A flowchart illustrating a shooting control method provided in yet another embodiment of this application;

[0072] Figure 4 A schematic diagram of camera distribution is provided for one embodiment of this application;

[0073] Figure 5 This application provides a schematic diagram of a data stream transmission according to one embodiment.

[0074] Figure 6 This is a schematic diagram of the structure of a shooting control device provided in one embodiment of this application;

[0075] Figure 7 A schematic diagram of a shooting control device provided in another embodiment of this application;

[0076] Figure 8 This is a schematic diagram of the structure of a smart device provided in one embodiment of this application. Detailed Implementation

[0077] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the scope of the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the drawings, not the entire structure.

[0078] It should be noted that, due to space limitations, this application specification does not exhaustively list all possible implementation methods. Those skilled in the art should be able to conceive after reading this application specification that, as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0079] The embodiments are described in detail below.

[0080] In related technologies, smart devices typically have only one camera, which is used for recording. For example, in a classroom setting, the smart device is usually positioned at the front, and its camera records images from front to back. In a meeting setting, the smart device can record local participants. Among these applications, recording the speaker in the shooting space is a particularly important one.

[0081] In scenarios where a speaker is being filmed, it's crucial that the smart device captures footage of the speaker speaking. However, smart devices can only film from a fixed angle. When the speaker is obstructed, the smart device will be unable to obtain footage of them speaking.

[0082] In related technologies, an external camera can also be used in conjunction with a smart device. Taking a classroom teaching scenario as an example, when there is only one external camera, it is usually installed at the front of the classroom, meaning the camera's position is fixed, and its shooting parameters (such as rotation angle, zoom level, etc.) are also fixed. This means the camera can only shoot from a fixed angle, resulting in a fixed shooting range. Alternatively, the camera's shooting parameters can be adjusted, but due to its fixed position, the range it can capture is still limited. Since there are many students in the classroom, and each student has the potential to speak, meaning the speaker appears randomly, filming the speaker is a crucial requirement in this scenario.

[0083] However, when cameras capture speakers using related technologies, the following problems arise: Due to the limited number of cameras and their fixed shooting positions (resulting in a relatively limited shooting range), when there are many people being filmed, firstly, the speaker may be obstructed by others. In this case, the camera's image will only show other people, not the speaker, resulting in a "hear the voice but not see the person" effect, leading to a poor user experience, especially when video chatting with someone at a distance; secondly, it is easy to capture only the speaker's profile or not capture the speaker at all, making the final image unsatisfactory to the user's actual needs.

[0084] In summary, in related technologies, there is a tendency to fail to capture the speaker (i.e., the subject to be captured) due to the limited shooting range of the camera.

[0085] Based on this, one embodiment of this application provides a shooting control method. This method uses at least three shooting devices to capture images of a shooting space from different positions, aiming to achieve a comprehensive, blind-spot-free shooting experience. During the shooting process, the intelligent device can select the optimal shooting device that captures the speaker's image based on the images captured by each device, and use the image captured by the optimal device to ensure that the captured image accurately reflects the speaker's speech. Furthermore, when the shooting parameters of the optimal shooting device are adjustable, the shooting parameters used by the optimal shooting device can be adjusted to ensure that the optimal shooting device captures images with optimal parameters, maximizing the quality of the captured image.

[0086] The shooting control method provided in this application embodiment can be executed by a smart device. This smart device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. Currently, smart devices can be electronic devices such as smart blackboards used in teaching scenarios, conference tablets or interactive whiteboards used in meeting scenarios, laptops, and desktop computers.

[0087] In one embodiment, the smart device includes one or more processors. These processors may include processing units such as application processors (APs), graphics processing units (GPUs), and central processing units (CPUs). The GPU, memory chips, interface circuits, and other necessary components (all configured within the smart device) can form a graphics card. The graphics card can convert the information that the smart device needs to display to drive the corresponding display screen for display. Currently, graphics cards have data processing and logical analysis capabilities; that is, by running corresponding computer-executable programs (also known as computer programs), the graphics card can perform data processing and logical analysis functions.

[0088] The smart device also includes a memory. The memory, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the shooting control method in the embodiments of this application. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the smart device. Furthermore, the memory may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include memory remotely configured relative to the processor and graphics card, and these remote memories can be connected to the smart device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. In the embodiments, the processor can implement the shooting control method provided in the embodiments of this application by running one or more programs in the memory.

[0089] Optionally, the smart device may also include a display screen, which displays information based on instructions from the processor and driven by a graphics card. The display screen may be a Liquid Crystal Display (LCD), an LED display, an Organic Light-Emitting Diode (OLED) display, or a Flexible Light-Emitting Diode (FLED) display, etc. In one embodiment, the display screen may also integrate touch functionality; in this case, the display screen includes a display panel and a touch panel. The display panel is used to provide visual output. The touch panel may be a touch component supporting infrared touch, electromagnetic touch, capacitive touch, and / or resistive touch, etc.

[0090] The smart device may also include a camera, which is a front-facing camera. Currently, this camera is fixed in position on the back of the smart device and cannot be rotated; that is, the smart device has a built-in, non-rotatable camera. The camera can be a wide-angle fixed-focus camera, or other types of cameras; this embodiment does not limit this. The camera is generally installed on the upper edge of the smart device on the side facing the user. In practical applications, the camera can also be installed in other locations; this embodiment does not limit this.

[0091] Smart devices may also include one or more communication interfaces to enable communication with other devices. For example, a smart device can communicate with a camera device via a communication interface.

[0092] In addition, smart devices may also include components such as power supplies, speakers, physical buttons, and microphone arrays, but the embodiments are not limited in this regard.

[0093] Based on the aforementioned hardware architecture, the smart device supports at least one operating system, such as Android, Windows, or Linux. The smart device can install at least one application under this operating system. The installed application can be a built-in application of the operating system, an application downloaded from a backend server, or an application from a third-party device. By running these applications, the smart device can perform corresponding functions.

[0094] In one embodiment, in order to implement the shooting control method, the smart device is equipped with a switching module, an algorithm module, a control module, and a virtual camera driver. Each of the above modules and programs can be considered as being obtained by encapsulating a computer program that can run in the smart device. Each module and program can be stored in a memory for use by the smart device.

[0095] The switching module runs within the operating system of the smart device, meaning its functions are implemented by the device's processor (such as the CPU) through the execution of corresponding computer programs. In one embodiment, the switching module can communicate with multiple shooting devices. For example, it can receive video data captured by the shooting devices, select one or more video data streams to send to other modules, and send shooting parameters used during the shooting process to the shooting devices.

[0096] The algorithm module runs on the graphics card of the smart device; that is, the graphics card executes corresponding computer programs to implement the various functions of the algorithm module. In one embodiment, the algorithm module can receive and process video data captured by the shooting device through a switching module. For example, the algorithm module can identify the speaker in the video data. It should be noted that in practical applications, the algorithm module can also run on the operating system of the smart device.

[0097] The control module runs within the operating system of the smart device; that is, the processor (such as the CPU) of the smart device implements the various functions of the control module by running corresponding computer programs. In one embodiment, based on the processing results of the algorithm module, the control module can determine which shooting device should be used to capture video data, and can also determine the shooting parameters that the shooting device should use.

[0098] The virtual camera driver runs within the operating system of the smart device, meaning its functions are implemented by the device's processor (such as the CPU) through the execution of corresponding computer programs. In one embodiment, the virtual camera driver, in conjunction with the aforementioned modules, enables applications installed on the smart device (located at the application layer) to operate the underlying camera logic seamlessly. The application doesn't need to concern itself with the number of cameras or their control logic; it only needs to obtain and use the video data sent by the virtual camera driver. The virtual camera driver is exposed to upper-layer applications, which can then use the video data sent by it. In other words, the virtual camera driver can be considered a bridge between the underlying physical camera and the upper-layer application. Currently, the application on the smart device that needs to use the video data sent by the virtual camera driver is referred to as the target application. There is at least one target application. The target application can be an application included with the operating system or an application downloaded from a server or other device. For example, a remote conferencing program installed on the smart device (such as MyLink Meeting or Tencent Meeting) can be used as a target application.

[0099] Currently, the aforementioned smart devices can cooperate with external shooting devices. That is, by cooperating with various modules of the smart devices and multiple shooting devices (which can be multiple external shooting devices, the smart device's own shooting device, and multiple external shooting devices), shooting of the shooting space can be achieved. The type of external shooting device is not currently limited. In one embodiment, taking a PTZ camera as an example, a PTZ camera can be understood as a camera with a pan-tilt unit. The pan-tilt unit can support the camera to rotate in both horizontal and vertical directions, allowing the camera to shoot from multiple angles. The currently used PTZ camera is a PTZ camera, which can achieve omnidirectional movement (up and down, left and right) and zoom (zoom). The operation of the PTZ camera (such as zoom operation) can be controlled remotely via remote control, control software, or intelligent algorithms to adapt to the video surveillance needs of different scenarios. In this case, the shooting parameters used by the PTZ camera can be adjusted. Currently, the shooting parameters include at least: Pan parameter, Tilt parameter, and Zoom parameter. The Pan parameter determines the horizontal position of the PTZ camera (which can be represented by the angle). Specifically, the Pan parameter determines the position the PTZ camera should rotate to when turning left or right in the horizontal direction; different Pan parameters correspond to different horizontal rotation positions. The Tilt parameter determines the vertical position of the PTZ camera (which can be represented by the angle). Specifically, the Tilt parameter determines the position the PTZ camera should rotate to when turning up or down in the vertical direction; different Tilt parameters correspond to different vertical rotation positions. The Zoom parameter represents the magnification and focal length of the PTZ camera; adjusting the Zoom parameter allows the PTZ camera to zoom in and out. Adjusting the Pan and Tilt parameters changes the shooting angle of the PTZ camera. Currently, setting appropriate Pan and Tilt parameters for the PTZ camera ensures that the subject being filmed (such as a speaker) is positioned in the center of the video frame captured by the PTZ camera. Adjusting the zoom parameter of the PTZ camera can change the size of the subject in the video frame. Currently, setting an appropriate zoom parameter for the PTZ camera can ensure that the subject (such as a speaker) occupies a suitable size in the video frame, and can also ensure the clarity of the subject in the video frame, that is, ensure that the video frame is a high-definition image (because the PTZ camera is a zoomable camera, so it can still capture high-definition images after zooming).

[0100] Currently, the shooting parameters used by PTZ cameras can be controlled by smart devices.

[0101] Currently, at least three shooting devices are used in conjunction with smart devices. These may include the smart device's own shooting device or not. Each shooting device captures the shooting space from different positions to ensure as much coverage as possible. Currently, the positions of the shooting devices are fixed and do not change. However, the specific positions of each shooting device can be set according to the actual situation.

[0102] The shooting space can be understood as the physical space that needs to be shot. It can be an indoor space or an outdoor space. In this embodiment, the shooting space is described as an indoor space. Currently, there are multiple human objects in the shooting space. Human objects can also be understood as the objects that need to be shot. When the shooting device shoots the shooting space, it mainly shoots the human objects in the shooting space.

[0103] Currently, the switching module, algorithm module, control module, and virtual camera driver, when combined with multiple shooting devices, can realize the shooting control method provided in the embodiments of this application.

[0104] Figure 1 A flowchart illustrating a shooting control method provided in one embodiment of this application is shown below. Figure 1 The shooting control method includes steps 110-150:

[0105] Step 110: Receive video data collected by each shooting device. There are at least three shooting devices shooting the shooting space from different positions.

[0106] This process can be referred to in the relevant description of step 210.

[0107] Step 120: Identify each face appearing in each video stream and select the face speaking in one of the video streams as the identified speaker in the shooting space.

[0108] For example, the algorithm module can implement facial recognition using artificial intelligence algorithms, such as using deep learning methods (e.g., deep learning toolkits like Openvino) to perform facial recognition on video data. The use of deep learning for facial recognition is an already implemented technique and will not be elaborated upon here.

[0109] Through facial recognition, the algorithm module can identify each face contained in each video stream. Then, based on the facial images, it can identify the speaker in the shooting space. Currently, after the algorithm module has identified each face in each video stream, it can analyze the lip movements of the faces using a lip movement algorithm (also referred to as a lip movement recognition algorithm) to select the face currently speaking in one video stream as the identified speaker. This process can be referred to in the relevant description of step 220.

[0110] Step 130: Determine the three-dimensional position coordinates of the speaker relative to each of the shooting devices based on the two-dimensional coordinates of the speaker in the video data.

[0111] Once the algorithm module identifies the speaker, it can determine the speaker's two-dimensional coordinates (located in the image pixel coordinate system, which can be the coordinates of the center point of the smallest rectangle containing the speaker) in the video data to which the speaker belongs (i.e., the video data contains a shot of the speaker and the speaker has the highest lip movement value).

[0112] Then, based on the transformation relationships between the various coordinate systems used by the imaging devices (such as the image pixel coordinate system, camera coordinate system, and world coordinate system), the speaker's three-dimensional coordinates in the camera coordinate system can be obtained. Next, based on the speaker's three-dimensional coordinates in the camera coordinate system, the speaker's three-dimensional coordinates in the world coordinate system can be determined. Finally, based on the transformation relationships between the camera coordinate systems used by other imaging devices and the world coordinate system, the speaker's three-dimensional coordinates in the camera coordinate systems used by those other imaging devices can be obtained.

[0113] In this way, the speaker's three-dimensional coordinates relative to each imaging device can be obtained. Refer to the description of step 230 for details.

[0114] Step 140: Determine the best shooting device for shooting the speaker among all shooting devices based on the video data of each channel and the corresponding three-dimensional position coordinates.

[0115] This process can be referred to in the relevant description of step 240.

[0116] Step 150: Send the video data captured by the best shooting device to the virtual camera driver so that the virtual camera driver can send the video data to the target application for use.

[0117] This process can be referred to in the relevant description of step 250.

[0118] The above-described method, by receiving video data from at least three shooting devices, selecting the speaking face from one of the video streams as the identified speaker in the shooting space, determining the speaker's three-dimensional position coordinates relative to each shooting device based on the speaker's two-dimensional coordinates in the respective video data, determining the optimal shooting device for capturing the speaker based on the speaker's three-dimensional position coordinates relative to each shooting device and the video data from each stream, and sending the video data captured by the optimal shooting device to a virtual camera driver, which then sends the video data to the target application for use, solves the technical problem in related technologies where cameras cannot capture the desired objects due to limited shooting range. Using at least three shooting devices to capture the shooting space from different positions ensures that all objects in the shooting space can be captured without blind spots. Furthermore, it can automatically identify the speaker and acquire the speaker's image, enabling tracking of the speaker's image and improving the user experience.

[0119] Figure 2 A flowchart of a shooting control method provided in another embodiment of this application is shown below. Figure 2 The shooting control method includes steps 210-250:

[0120] Step 210: Receive video data collected by each shooting device. There are at least three shooting devices shooting the shooting space from different positions.

[0121] Video data refers to the data captured by the shooting device during the shooting process, and is presented as a video stream. Currently, each shooting device performs shooting. The shooting parameters used by each shooting device are not currently limited; they can use default shooting parameters or the most recently set shooting parameters. Currently, shooting parameters include at least one of Pan parameters, Tilt parameters, and Zoom parameters. In this embodiment, Pan parameters, Tilt parameters, and Zoom parameters are used as examples for description. When the shooting parameters are different, the content of the video data captured by the shooting device will also be slightly different.

[0122] Optionally, among the at least three shooting devices currently in use, there may be one with fixed shooting parameters. For example, the at least three shooting devices may include a shooting device installed on a smart device (such as a front-facing camera), and the shooting parameters of this shooting device are not adjustable. In this case, the shooting device can use fixed shooting parameters when taking pictures. In practical applications, the shooting parameters of the at least three shooting devices currently in use can also be all adjustable, or some shooting parameters can be adjustable. For example, one shooting device may have adjustable zoom parameters but fixed pan and tilt parameters, while another shooting device may have adjustable pan and tilt parameters but fixed zoom parameters.

[0123] For example, when there is a need to film the speaker (e.g., in a teaching scenario, filming the speaker can be considered necessary at the start of the class; similarly, in a meeting scenario, filming the speaker can be considered necessary at the start of the meeting), each filming device sends the currently filmed video data to the smart device in real time. Currently, the switching module of the smart device receives the video data sent by the filming device first. Each filming device sends video data in the same way; here, we will describe it using the example of one filming device sending video data. Optionally, the filming device can be wired to the smart device via a Universal Serial Bus (USB). In this case, the filming device can send video data to the switching module of the smart device in real time via the USB line. Alternatively, the filming device and the smart device can be connected to the same Wi-Fi network. In this case, the filming device can send the filmed video data to the switching module of the smart device in real time via Wi-Fi. In one embodiment, the communication protocol used by the shooting device to send video data is not currently limited. For example, when the shooting device is connected to the smart device via a USB cable, it can use the USB protocol to send video data; when the shooting device and the smart device are on the same Wi-Fi network, it can use the TCP / IP protocol to send video data. Optionally, the shooting device also sends identification information when sending video data. The identification information can consist of numbers, letters, and / or symbols, and its specific content is not currently limited. The identification information allows the switching module to distinguish video data sent by different shooting devices, thus enabling the switching module to associate each shooting device with a specific video data. Optionally, if the shooting device includes a shooting device installed in the smart device (such as a front-facing camera), then the captured video data can be sent to the switching module via the internal transmission lines of the smart device (such as a bus or other types of transmission lines).

[0124] For example, when the switching module receives video data from each channel, it sends all the received video data to the algorithm module in real time. It is understood that the transmission between the switching module and the algorithm module is internal transmission within the smart device, which is an already implemented technical method and will not be elaborated upon here.

[0125] Step 220: Based on the video data from each source, identify the speaker in the shooting space.

[0126] In one embodiment, an algorithm module identifies the speaker in the video data. The algorithm module can identify the speaker in various ways, such as by using facial recognition to identify faces appearing in the video data, then determining whether the face is speaking based on whether the lips are moving, thus identifying the speaker's face; or by using lip-reading and motion recognition to identify the speaker's face.

[0127] In one embodiment, the example described is performing face recognition first to identify faces appearing in the video data, and then identifying a speaker based on the identified faces. In this case, step 220 may include: performing face recognition on each video stream, and selecting a speaking face from one video stream as the identified speaker in the shooting space based on the results of the face recognition on each video stream.

[0128] For example, the algorithm module can implement facial recognition using artificial intelligence algorithms, such as using deep learning methods (e.g., deep learning toolkits like Openvino) to perform facial recognition on video data. The use of deep learning for facial recognition is an already implemented technique and will not be elaborated upon here.

[0129] Through facial recognition, the algorithm module can identify every face contained in each video stream. It's understandable that there may be video data that doesn't contain faces; in this case, the algorithm module can choose not to process that video data, meaning it won't identify the speaker from that video data.

[0130] After the algorithm module obtains the faces contained in the video data, it can identify the speaker in the shooting space based on the face images. Currently, after the algorithm module identifies each face in each video data stream, it can analyze the lip images in the faces using a lip movement algorithm (also known as a lip movement recognition algorithm) to select the speaking face in one video data stream as the identified speaker. At this point, performing face recognition on each video data stream and selecting the speaking face in one video data stream as the identified speaker in the shooting space based on the results of face recognition on each video data stream can further include: performing face recognition on each video data stream to obtain the lip images of each identified face; and selecting the speaking face in one video data stream as the identified speaker in the shooting space based on the lip movement state in the lip images, where the speaker's lip movement state is the most obvious.

[0131] For example, after the algorithm module performs face recognition on the video data to obtain each face, it can obtain the face image of each face in the corresponding video data. Then, the algorithm module identifies the lips in the face image to obtain individual lip images. The identification of lip images can also be achieved using artificial intelligence. It can be understood that the video data consists of multiple frames. Currently, determining the speaker can be done by identifying the speaker in one frame of the currently received video data (currently denoted as frame N). In this case, the identified lip images are based on frame N and the consecutive frames preceding frame N. The algorithm module cannot determine the speaker from the first few received frames.

[0132] For example, after the algorithm module obtains each lip image, it can use the lip movement algorithm (which is also an artificial intelligence algorithm) to determine whether the lips in the lip image are moving. If the lips are in motion (i.e. the mouth is opening and closing or constantly changing shape), then the face to which the lips belong is determined to be the speaker's face, that is, the speaker is identified.

[0133] Optionally, if multiple video datasets contain multiple face images (i.e., multiple lip images), and at least two of these lip images show lips in motion, then the face corresponding to the lips with the highest lip movement value can be selected as the speaker's face. In other words, the lip movement value reflects the lip movement state, and the speaker's lip movement is the most pronounced, resulting in the highest lip movement value. The lip movement value can be determined by parameters such as lip movement duration, opening amplitude, and movement frequency. By assigning appropriate weights to these parameters and using a weighted average, the lip movement value can be obtained. For example, different lip opening amplitudes correspond to different scores (larger opening amplitude, higher score), different movement frequencies also correspond to different scores (higher movement frequency, higher score), and different movement durations also correspond to different scores (longer movement duration, higher score). Furthermore, opening amplitude, movement frequency, and movement duration each have corresponding weights, and these weights can be flexibly adjusted based on the actual situation. Next, based on the lip images corresponding to the same face, determine the following scores: lip amplitude score (which can be the maximum amplitude or the average amplitude across all lip images), lip movement frequency score (determined based on the duration and number of opening and closing movements of each lip image), and lip movement duration score. Then, weight the determined scores for amplitude, frequency, and duration; the resulting value is used as the lip movement value for that face. Alternatively, after determining the scores for amplitude, frequency, and duration, first determine if all scores are valid. For example, each dimension has a corresponding threshold; if the score for any dimension (amplitude, frequency, or duration) does not reach the corresponding threshold, then that dimension's score is considered invalid. If all three dimensions are valid, then the scores for the three dimensions are weighted; the resulting value is used as the lip movement value for that face. If any dimension's score is invalid, then there is no need to weight the scores for the other two dimensions; that is, it is determined that the face corresponding to the lips is not the speaker's face. Currently, the spokesperson's lip movement value is the highest.

[0134] At this point, each face appearing in the video data captured by each camera device has a corresponding lip movement value. Then, the face with the highest lip movement value is selected as the speaker. Generally, only one face has the highest lip movement value; that is, only the face from one video stream will be identified as the speaker, and the other video streams can be considered as not having identified a speaker. In rare cases, there may be multiple faces with the highest lip movement value; in this case, one of them can be selected as the speaker's face.

[0135] In practical applications, to ensure recognition accuracy, microphone identification can be combined. The microphone's installation location is currently not limited; it can be integrated into a smart device or be a separate physical entity. Currently, the microphone captures the speaker's voice. Then, the algorithm module can locate the sound source based on the direction of the captured sound to obtain the speaker's Direction of Arrival (DOA) and depth information. This DOA and depth information determines the speaker's direction and approximate distance, achieving an accuracy of approximately 20 centimeters in near-field scenarios (such as meetings or classrooms). The algorithm module then uses the speaker's direction to identify lip movements in the video data corresponding to that direction, thus identifying the speaker. For example, after determining the speaker's direction and approximate distance based on the DOA and depth information, a region (i.e., a sub-frame) where the speaker might be present can be identified in the video data captured by various cameras. Face recognition and lip movement calculation are then performed within this region to obtain the lip movement value for each face in that region, reducing the data processing load for face recognition and lip movement calculation. In practical applications, if more accurate depth information is needed, an infrared or ultrasonic ranging sensor can be added.

[0136] Understandably, in current application scenarios, a speaker's face can usually be captured by at least one camera device when they are speaking.

[0137] Step 230: Calculate the speaker's relative position information with respect to each camera device.

[0138] The speaker's relative position information compared to the camera device can also be understood as the speaker's position coordinates relative to the camera device. These position coordinates are three-dimensional coordinates; that is, the relative position information is three-dimensional position coordinates. The speaker's relative position information can also be understood as the speaker's three-dimensional coordinates in the camera coordinate system. Each camera device has corresponding relative position information.

[0139] Specifically, each imaging device can have an image pixel coordinate system, a camera coordinate system, a world coordinate system, and an image physical coordinate system. The world coordinate system is a three-dimensional coordinate system describing the physical location in the real three-dimensional world. Each imaging device and speaker has corresponding three-dimensional coordinates in the world coordinate system. The camera coordinate system is a three-dimensional coordinate system established with the optical center of the imaging device as its origin. Its Z-axis is the optical axis of the imaging device, pointing outwards. The X-axis is parallel to the horizontal plane of the image captured by the imaging device, and the plane formed by the X-axis and Y-axis is perpendicular to the optical axis of the imaging device. The camera coordinate system is the key medium between the three-dimensional information in space and the two-dimensional image information captured by the imaging device. The image pixel coordinate system is a two-dimensional coordinate system established with the upper left corner or center coordinate of the imaging device's own CCD photosensitive device (i.e., the upper left corner or center point of the captured image) as its origin. It can be understood that the video data captured by the imaging device consists of multiple frames of two-dimensional images (i.e., the captured images). Each pixel in each frame captured by the imaging device has a corresponding two-dimensional coordinate in the image pixel coordinate system, which reflects the position of the pixel in the image. The image physical coordinate system can be understood as a two-dimensional coordinate system established on the plane of the captured two-dimensional image, with its origin at the intersection of the Z-axis in the camera coordinate system and the image plane. Transformation relationships exist between these coordinate systems to achieve coordinate transformations. For example, two-dimensional coordinates in the image coordinate system can be transformed to three-dimensional coordinates in the camera coordinate system, and then the three-dimensional coordinates in the camera coordinate system can be transformed to the world coordinate system. The determination of these transformation relationships can refer to existing technologies. Optionally, transformation relationships also exist between the camera coordinate systems used by different shooting devices. These transformation relationships can be used to transform each camera coordinate system into the same camera coordinate system. The determination of these transformation relationships can refer to existing technologies and will not be elaborated upon here. In this embodiment, the speaker's relative position information can also be understood as the speaker's three-dimensional coordinates in the camera coordinate system. In practical applications, the speaker's coordinates in other coordinate systems can also be used as the speaker's relative position information.

[0140] Specifically, the speaker's two-dimensional coordinates (located in the image pixel coordinate system) within the corresponding video data are obtained. These two-dimensional coordinates can be the coordinates of the center point of the smallest rectangular area containing the speaker. Generally, in this step, when determining the speaker by combining various video data streams, we combine all video data streams to determine the stream in which a person currently speaking is most likely the speaker. Whether the speaker appears in other video data streams is not determined at this time. At this point, the speaker's two-dimensional coordinates within the corresponding video data stream can be obtained. Subsequently, based on the aforementioned transformation relationships between the coordinate systems of the shooting device, the speaker's two-dimensional coordinates in the image physical coordinate system, the speaker's three-dimensional coordinates in the camera coordinate system, and the speaker's three-dimensional coordinates in the world coordinate system can be obtained. For example, after determining the speaker's two-dimensional coordinates in the image pixel coordinate system based on the footage of the speaker captured by a certain shooting device, the intrinsic and extrinsic parameter matrices of the shooting device can be used to transform the speaker from the image pixel coordinate system to the camera coordinate system to obtain the speaker's three-dimensional coordinates in the camera coordinate system (i.e., the speaker's relative position information with respect to the shooting device). Then, based on the transformation relationship between the camera coordinate system and the world coordinate system, the speaker's three-dimensional coordinates in the world coordinate system can be determined. Next, based on the transformation relationship between the camera coordinate system and the world coordinate system used by other shooting devices (using the extrinsic parameter matrix of the other shooting devices), the speaker's three-dimensional coordinates in the camera coordinate systems used by the other shooting devices can be obtained (i.e., the speaker's relative position information with respect to the other shooting devices). In this way, the speaker's relative position information with respect to each shooting device can be obtained.

[0141] It is understood that the intrinsic and extrinsic parameters of the imaging device are commonly used parameters in this field. The intrinsic parameters include the focal length of the imaging device, pixel coordinates (related to the image pixel coordinate system), etc. The extrinsic parameters include the rotation and translation transformation relationship of the imaging device with respect to the world coordinate system (i.e., used for transformation between the world coordinate system and the world coordinate system).

[0142] In one embodiment, the relative position information of the speaker is determined by the algorithm module. In this case, step 230 may include: determining the relative position information of the speaker relative to the corresponding shooting device of the video data based on the two-dimensional coordinates of the speaker in the video data, and determining the relative position information of the speaker relative to each other shooting device according to the relative position information. The relative position information is the three-dimensional position coordinates of the speaker relative to the shooting device.

[0143] For example, after the algorithm module identifies the speaker, it can determine the speaker's two-dimensional coordinates in the video data to which it belongs (i.e., the video data contains a shot of the speaker and the speaker has the highest lip movement value). These coordinates are located in the image pixel coordinate system and can be the coordinates of the center point of the smallest rectangle containing the speaker.

[0144] Subsequently, based on the transformation relationships between the various coordinate systems used by the shooting devices (such as the image pixel coordinate system, camera coordinate system, and world coordinate system), the speaker's three-dimensional coordinates in the camera coordinate system can be obtained. These three-dimensional coordinates can be considered as the speaker's relative position information relative to the shooting device corresponding to their video data. That is, the relative position information is three-dimensional coordinates. Then, based on the speaker's three-dimensional coordinates in the camera coordinate system, the speaker's three-dimensional coordinates in the world coordinate system can be determined. Finally, based on the transformation relationships between the camera coordinate systems used by other shooting devices and the world coordinate system, the speaker's three-dimensional coordinates in the camera coordinate systems used by those other shooting devices can be obtained (i.e., the speaker's relative position information relative to those other shooting devices).

[0145] In this way, the relative position information of the speaker relative to each shooting device can be obtained. It can be understood that the relative position information of each shooting device is under the same acquisition time or the same reception time. For example, when determining the speaker, it can be to determine the speaker of the Nth frame in the current video data. Then, the relative position information of each shooting device is the relative position information under the reception time when the control module receives the Nth frame.

[0146] Optionally, for a camera device that does not capture the speaker's face, its corresponding relative position information can be left blank.

[0147] Step 240: Determine the best shooting device for shooting the speaker among all shooting devices based on the video data of each channel and the corresponding relative position information.

[0148] For example, the control module determines the optimal shooting device. For example, the algorithm module obtains the speaker's relative position information with respect to each shooting device, and then sends the video data and the speaker's relative position information to the control module. Upon receiving each video data stream and its corresponding relative position information, the control module can determine the relevant footage in each video stream. Specifically, by using the relative position information and the transformation relationship between the pixel coordinate system of the corresponding shooting device and the camera coordinate system, the speaker's two-dimensional coordinates (located in the image pixel coordinate system) in the video data can be determined. Then, the speaker can be tracked in the video data to obtain footage relevant to the speaker.

[0149] The control module then selects the optimal shot from the various screens related to each speaker, and designates the camera corresponding to that shot as the best camera for capturing the speaker's message. The optimal shot ensures the speaker's face is as clear as possible so that other users can see the speaker's speech through the video data.

[0150] In one embodiment, step 240 may specifically include steps 241-242:

[0151] Step 241: Determine the speaker's face in each video data based on each relative position information.

[0152] The control module can determine the speaker's two-dimensional coordinates (located in the image pixel coordinate system) in the corresponding video data using relative position information, and thus obtain the speaker's face image. For example, using face recognition technology to find a face image (a rectangular area) containing those two-dimensional coordinates as the center, this can be considered the speaker's face image. If the image obtained from the corresponding video data based on the relative position information does not contain a face, it can be determined that the video data does not contain the speaker's face image, meaning the speaker was not captured in that video data stream, and in this case, the video data can be discarded. Alternatively, if the relative position information is empty, it means the corresponding video data does not contain the speaker's face image, and in this case, the video data can be discarded. This indicates that the shooting device corresponding to the video data is not the optimal shooting device.

[0153] Understandably, when the control module determines a face image, it can do so based on relative position information to identify the face in the currently received frame (currently denoted as frame M). Currently, the time interval between frame M and the frame N currently received by the algorithm module when determining the speaker is very small or almost nonexistent, to ensure that the speaker's position in these frames does not change significantly. Generally, after the same speaker begins speaking, their position will not change significantly within a certain period (which can be set according to the actual situation).

[0154] Step 242: Determine the best shooting device for the speaker based on the facial image.

[0155] For example, after obtaining the speaker's facial image, the control module can track the speaker in the video data. At this point, the control module can determine the highest quality facial image and select the corresponding camera as the optimal camera. For instance, after obtaining various facial images, the control module can determine the facial image containing a frontal view and use it as the highest quality facial image. Alternatively, the control module can score each facial image and determine the highest quality facial image based on the score. The facial image used to determine the optimal camera can be a facial image located in frame M or consecutive facial images from frame M and subsequent frames.

[0156] In one embodiment, the method of determining the best shooting device by scoring facial images is used as an example. In this case, this step specifically includes: scoring the facial images and selecting the shooting device corresponding to the facial image with the highest score as the best shooting device for shooting the speaker.

[0157] After obtaining the speaker's facial image, the control module scores the image. The scoring criteria can be set according to the actual situation. For example, if a frontal view (including lips) is detected, the score is 10 points; if a profile view (including lips) is detected, the score is 8 points; and if no lips are detected, the score is 0 points. The control module recognizes the facial images according to this scoring standard, identifying frontal views, profile views, lips, etc., and scores them based on the recognition results. Each facial image has a corresponding score. It is understood that the technical means used by the control module to recognize facial images are currently not limited; for example, artificial intelligence can be used to recognize and score facial images.

[0158] In practical applications, more scoring references can be used based on the aforementioned scoring criteria. For example, in addition to scoring based on the frontal face, profile, and lips, scores can be assigned for different brightness levels of the facial image. A lower score is given for an excessively bright or dark image, while a higher score is given for a normal brightness level. Scores can also be assigned for different facial sizes (determined by pixel count or other methods). For instance, a lower score is given for a smaller face (i.e., the farther the camera is from the speaker). Optionally, when using multiple dimensions for scoring, different weights can be assigned to each dimension. The final score is then obtained by weighting the weights and their corresponding scores.

[0159] For example, after obtaining the score corresponding to each facial image, the control module can select the facial image with the highest score. Currently, the facial image with the highest score corresponds to the highest quality, and is the optimal speaking image obtained after capturing the speaker. Then, the control module determines the capturing device corresponding to the facial image with the highest score as the best capturing device.

[0160] Optionally, when multiple face images receive the highest scores, one of the face images corresponding to a camera can be randomly selected as the optimal camera, or a camera corresponding to a face image can be selected as the optimal camera according to other rules. This embodiment does not limit this selection. Further, when multiple face images receive the highest scores, all camera devices corresponding to multiple face images can be considered as the optimal camera.

[0161] Step 250: Send the video data captured by the best shooting device to the virtual camera driver so that the virtual camera driver can send the video data to the target application for use.

[0162] Optionally, after determining the optimal shooting device, the control module acquires the device information of the optimal shooting device. This device information distinguishes different shooting devices. Currently, the identification information of each shooting device is used as the device information. Optionally, when the algorithm module sends video data to the control module, it also sends the corresponding identification information, so that the control module can distinguish between different video data streams based on the identification information. When the control module determines the optimal shooting device, it can acquire the identification information corresponding to the relevant video data as the currently acquired device information.

[0163] After the control module obtains the device information, it sends the device information to the switching module so that the switching module can determine which video data corresponds to the best shooting device based on the device information.

[0164] The transmission between the control module, the algorithm module, and the switching module is an internal transmission within the intelligent device, which is an already implemented technical means and will not be elaborated upon here.

[0165] For example, after receiving the device information, the switching module can determine which video data stream corresponds to the optimal shooting device based on the device information. Specifically, after receiving the device information, the switching module obtains the identification information contained therein. Then, based on the video data sent by the shooting device and the identification information, it can determine which video data to use based on the identification information of the optimal shooting device. Afterward, the switching module sends the video data sent in real-time by the optimal shooting device to the virtual camera driver.

[0166] The transmission between the switching module and the virtual camera driver is an internal transmission within the smart device, which is an already implemented technical method and will not be elaborated upon here.

[0167] For example, after receiving video data, the virtual camera driver can send the video data to the target application for use. For instance, when a remote conferencing program (the target application) installed on a smart device needs to use the video data, the remote conferencing program can communicate with the virtual camera driver. In this case, the virtual camera driver can send the received video data as a usable video stream to the remote conferencing program, allowing the remote conferencing program to use the video data.

[0168] Optionally, the virtual camera driver can also have video processing capabilities to further process the received video data and obtain a usable captured image. For example, when there are multiple optimal shooting devices, the virtual camera driver will receive multiple video data streams. In this case, the virtual camera driver can fuse the multiple video data streams (for example, when there are two video data streams, they can be fused using a two-screen split method) to form a single video data stream, which can then be used as the captured image. As another example, when there is only one optimal shooting device, the virtual camera driver, after receiving this single video data stream, processes it to magnify the image including the speaker, thereby obtaining a usable captured image. It should be noted that the technical means used by the virtual camera driver for video processing are not currently limited; for example, video processing can be achieved through artificial intelligence technology.

[0169] Optionally, when the virtual camera driver receives video data from the optimal shooting device, real-time video data from other shooting devices, as well as real-time video data from the optimal shooting device, are also sent to the algorithm module. The algorithm module continues to identify the speaker's relative position information, and when the speaker is identified, the control module continues to determine the optimal shooting device, so that the virtual camera driver continues to receive video data from the optimal shooting device until the algorithm module no longer identifies the speaker, at which point it stops determining the optimal shooting device. When the algorithm module identifies the speaker again, the control module continues to determine the optimal shooting device.

[0170] Alternatively, when the virtual camera driver receives video data from the optimal shooting device, it identifies the speaker in the video data. The technical means for speaker identification can refer to those used in the algorithm module. Then, if the virtual camera driver determines that it cannot identify the speaker (i.e., the speaker has finished speaking) or the speaker is located at the edge of the video data, it determines that the current optimal shooting device is no longer suitable for capturing the speaker. At this point, the virtual camera driver can notify the switching module, which then sends the received video data from each stream to the algorithm module, which re-identifies the speaker.

[0171] In practical applications, there are situations where the algorithm module cannot identify the speaker, such as when there is no speaker or when the speaker has finished speaking. In such cases, when the algorithm module fails to identify the speaker, the switching module can send all video data to the virtual camera driver. The virtual camera driver then merges the video data into a single video stream for use. For example, the virtual camera driver can arrange and combine the video data in a certain order to create a single video stream for use.

[0172] The above-described method, by receiving video data from at least three shooting devices, identifying the speaker in the shooting space based on the video data, calculating the speaker's relative position information relative to each shooting device, determining the optimal shooting device for capturing the speaker based on the relative position information of the speaker relative to each shooting device and the video data from each video device, and sending the video data captured by the optimal shooting device to the virtual camera driver, which then sends the video data to the target application for use, solves the technical problem in related technologies where cameras cannot capture the objects to be captured due to limited shooting range. Using at least three shooting devices to capture the shooting space from different positions ensures that all objects in the shooting space can be captured without blind spots. Furthermore, it can automatically identify the speaker and acquire the speaker's image, enabling tracking of the speaker's image and improving the user experience. Moreover, using facial recognition to automatically identify the speaker eliminates the need for manual speaker identification by the user. Then, by combining the speaker's two-dimensional coordinates in the video data, the relative position information of the speaker relative to each shooting device can be automatically obtained, facilitating the subsequent determination of the optimal shooting device. Finally, using lip movement to identify the speaker ensures the rationality of the automatic speaker identification. Furthermore, based on the speaker's relative position to each camera device, facial images of the speaker captured by each device can be obtained. Then, the optimal camera device can be determined automatically based on these facial images. Moreover, using a scoring method to determine the optimal camera device ensures the rationality of the automatic determination process.

[0173] Based on the above embodiments, Figure 3 This is a flowchart of a shooting control method provided in another embodiment of this application. Figure 3 The shooting control method shown is based on the aforementioned shooting control method. Furthermore, the control module can determine the optimal shooting parameters that the optimal shooting device should use when shooting the speaker, so that the optimal shooting device can use these parameters for shooting. (Reference) Figure 3 The shooting control method includes steps 310-370:

[0174] Step 310: Receive video data collected by each shooting device. There are at least three shooting devices that capture the shooting space from different positions.

[0175] Step 320: Based on the video data from each source, identify the speaker in the shooting space.

[0176] Step 330: Calculate the speaker's relative position information with respect to each camera device.

[0177] Step 340: Determine the best shooting device for shooting the speaker among all shooting devices based on the video data of each channel and the corresponding relative position information.

[0178] Step 350: Determine the shooting parameters of the optimal shooting device.

[0179] Shooting parameter information refers to information containing shooting parameters. Currently, we'll take shooting parameters including Pan, Tilt, and Zoom as an example. In this case, the shooting parameter information includes the specific values ​​corresponding to the Pan, Tilt, and Zoom parameters. In practical applications, shooting parameters can include other types of parameters.

[0180] Currently, the control module determines the shooting parameter information. That is, after determining the optimal shooting device, the control module also determines the shooting parameters to be used by the optimal shooting device and obtains the shooting parameter information. When the optimal shooting device uses the shooting parameters in this information to shoot, the resulting video data containing the speaker is of superior quality compared to footage shot with other parameters.

[0181] In one embodiment, the shooting parameters of the optimal shooting device may be non-adjustable. In this case, before determining the shooting parameter information of the optimal shooting device, the method further includes: determining whether the shooting parameters of the optimal shooting device are adjustable; if they are adjustable, then performing the operation of determining the shooting parameter information of the optimal shooting device.

[0182] For example, when a smart device uses each shooting device, it can be informed whether the shooting parameters are adjustable. For instance, if the shooting parameters of a shooting device installed in the smart device are not adjustable, then the smart device can clearly identify that the shooting device is not adjustable when using its own installed shooting device. Similarly, after a PTZ camera is connected to the smart device, the smart device can clearly identify that the shooting parameters of the PTZ camera are adjustable. Optionally, the smart device can also record whether the shooting parameters of each currently used shooting device are adjustable. Based on this, when the optimal shooting device is determined, the control module of the smart device can determine whether the shooting parameters of the optimal shooting device are adjustable. If they are adjustable, step 350 is executed. If they are not adjustable, the smart device does not need to determine the shooting parameters of the optimal shooting device. In this case, the control module of the smart device can directly send the video data captured by the optimal shooting device to the virtual camera driver. Currently, in order to obtain a higher quality speaker image from the virtual camera driver when the shooting parameters of the optimal shooting device are not adjustable, the control module can, after determining the optimal shooting device, segment and enlarge the speaker's face image in the video data it captures before sending it to the virtual camera driver. This ensures that the size of the speaker's face image in the video data acquired by the virtual camera driver is reasonable, meaning that when the video data is displayed on a smart device, the face image occupies a certain area, and the face image is located as centrally as possible in the video.

[0183] There are several ways to determine the shooting parameter information.

[0184] In one embodiment, the control module can select the optimal shooting parameters based on multiple shooting parameters. In this case, step 350 may include steps 351-352:

[0185] Step 351: Based on the speaker's relative position information with respect to the optimal shooting device, obtain multiple alternative shooting parameters for the optimal shooting device.

[0186] Shooting parameter alternative information refers to the shooting parameter information that serves as an alternative. It includes specific values ​​for various shooting parameters, and each specific value is usable by the shooting device. Currently, each shooting parameter alternative information includes a specific value corresponding to the Pan parameter (i.e., Pan parameter information), a specific value corresponding to the Tilt parameter (i.e., Tilt parameter information), and a specific value corresponding to the Zoom parameter (i.e., Zoom parameter information). The specific values ​​included in different shooting parameter alternative information are not entirely the same or may be completely different.

[0187] For example, each shooting device pre-sets multiple alternative shooting parameters corresponding to different coordinate values ​​in the camera coordinate system. Alternatively, each shooting device pre-sets multiple different values ​​for the Pan parameter, Tilt parameter, and Zoom parameter corresponding to different coordinate values ​​in the camera coordinate system. Combining these values ​​yields multiple alternative shooting parameters for each coordinate value. When the shooting device shoots based on each alternative shooting parameter, the entity corresponding to the coordinate value of the alternative shooting parameter in the captured video data is located in the central area of ​​the physical space as much as possible, although the content of the video data corresponding to different alternative shooting parameter information may vary slightly. The multiple alternative shooting parameters for each coordinate value can be determined empirically.

[0188] The smart device pre-stores multiple shooting parameter alternatives corresponding to different coordinate values ​​in the camera coordinate system of each shooting device. Once the best shooting device is determined, the corresponding multiple shooting parameter alternatives can be obtained based on the speaker's relative position information relative to the best shooting device.

[0189] Step 352: Score the multiple shooting parameter candidates and select the shooting parameter candidate with the highest score as the determined shooting parameter.

[0190] For example, after the control module obtains multiple candidate shooting parameters, it can score each candidate. The scoring criteria can be set according to the actual situation; for example, the scoring criteria for selecting the best shooting device can be used to score each candidate shooting parameter. Specifically, the control module can send the candidate information for each shooting parameter and the device information of the best shooting device to the switching module. The switching module, based on the device information, sends the candidate information for the shooting parameters to the best shooting device. After receiving the candidate information, the best shooting device can obtain the specific values ​​of the shooting parameters contained therein and use those values. Then, the switching module sends the video data captured by the best shooting device to the control module through the algorithm module. This allows the control module to score the currently obtained video data based on the scoring criteria for selecting the best shooting device. For example, if the control module identifies the speaker's face in the current frame of the video data captured by the best shooting device (at this time, the control module can record the speaker's facial features and perform feature matching in the current frame of the video data captured by the best shooting device to identify the speaker's face), then if the face image contains a frontal view (including the lips), the corresponding score is 10 points; if the face image contains a side view (including the lips), the corresponding score is 8 points; and if the face image does not contain the lips, the corresponding score is 0 points. Furthermore, based on this, more scoring references are adopted. For example, in addition to scoring based on the front view, side view, and lips, scores can be assigned for different brightness levels of the facial image. A lower score is given for an excessively bright or dark image, while a higher score is given for normal brightness. Scores can also be assigned for different facial sizes (determined by pixel count or other methods). For example, a lower score is given for a smaller face (i.e., the farther the camera is from the speaker). Optionally, when using multiple dimensions for scoring, different weights can be assigned to different dimensions. Then, the final score is obtained by weighting the weights and corresponding scores.

[0191] Next, the control module assigns the corresponding scores as scores to the corresponding shooting parameter candidate information. At this point, each shooting parameter candidate information has a corresponding score. Then, the control device selects the shooting parameter candidate information with the highest score as the currently determined shooting parameter information. Optionally, if there are at least two shooting parameter candidate information with the highest score, one of them can be randomly selected as the shooting parameter information. Afterwards, the control module sends the device information of the optimal shooting device and the shooting parameter information together to the switching module.

[0192] This method determines the quality of the video data corresponding to the shooting parameters, ensuring the speaker is positioned in the center and of appropriate size. However, it involves a larger computational load, resulting in a slower processing speed.

[0193] In one embodiment, the shooting parameters include Pan parameters, Tilt parameters, and Zoom parameters. The optimal shooting parameters can also be calculated directly using the speaker's relative position information. In this case, step 350 may include steps 353-354:

[0194] Step 353: Based on the speaker's relative position information with respect to the optimal shooting device, determine the Pan parameter information and Tilt parameter information of the optimal shooting device.

[0195] For example, substituting the X-axis and Z-axis coordinates of the speaker's relative position to the optimal shooting device into the inverse trigonometric function formula arctan(X / Z), the resulting value can be used as the Pan parameter information (i.e., the specific value of the Pan parameter). When the optimal shooting device uses the Pan parameter information to shoot, the speaker will appear in a vertical area in the middle of the video data. Similarly, substituting the X-axis, Y-axis, and Z-axis coordinates of the speaker's relative position to the optimal shooting device into the inverse trigonometric function formula arctan(Y / sqrt(X^2+Z^2)), the resulting value can be used as the Tilt parameter information (i.e., the specific value of the Tilt parameter). When the optimal shooting device uses the Tilt parameter information to shoot, the speaker will appear in a horizontal area in the middle of the video data. Here, X^2 represents the square of X, and Z^2 represents the square of Z.

[0196] When shooting using specific values ​​for the Pan and Tilt parameters, the speaker is positioned in the middle area of ​​the video data.

[0197] It is understandable that arctan(X / Z) and arctan(Y / sqrt(X^2+Z^2)) are the preset inverse trigonometric function formulas. In practical applications, other calculation formulas can also be used.

[0198] Step 354: Determine the Zoom parameters of the optimal shooting device based on the actual size in the video data captured by the speaker on the optimal shooting device, the preset desired size, and the focal length of the optimal shooting device.

[0199] For example, the size can be understood as the size of the speaker in the video data, which is equal to the size of the rectangle that identifies the speaker. The representation of the size is not currently limited. For example, the total number of pixels occupied by the rectangle can be used as the size, or the width of the rectangle in pixels multiplied by the height of the rectangle in pixels can be used as the size.

[0200] The expected size refers to the size of the speaker's area in the expected video data, which can be preset on the smart device based on actual conditions. The actual size refers to the actual size occupied by the speaker in the video data. The actual size can be calculated by identifying the rectangular area of ​​the speaker in the video data; the size of the bounding box corresponding to this rectangular area is the speaker's actual size.

[0201] Next, substitute the actual size and the desired size into the formula: f * (desired size / actual size), and the resulting value is the Zoom parameter information. Here, f represents the focal length of the shooting device. When shooting using the Zoom parameter information, the speaker's size in the video data generally meets the desired size.

[0202] In practical applications, the appropriate method for determining shooting parameters can be selected based on the actual data processing speed of the smart device. For example, when the data processing speed is fast, the first method can be chosen to ensure a better image of the speaker. When the data processing speed is slow, the second method can be chosen to ensure rapid determination of shooting parameters.

[0203] Step 360: Send the shooting parameter information to the optimal shooting device so that the optimal shooting device can adjust the shooting parameters it uses based on the shooting parameter information.

[0204] For example, after the switching module determines the optimal shooting device based on the device information, it can send the shooting parameter information to the optimal shooting device. Then, the optimal shooting device obtains the specific values ​​of the shooting parameters from the shooting parameter information and uses these values ​​to adjust its own shooting parameters. Afterward, the optimal shooting device can shoot based on the adjusted shooting parameters. Optionally, when adjusting the shooting parameters, the optimal shooting device can achieve smooth motion control to avoid shaking. Optionally, when shooting after adjusting the shooting parameters, the optimal shooting device can use image processing technology to detect the speaker's face or body in the captured footage (i.e., each frame of the video data) and make fine adjustments to ensure accurate alignment with the speaker (i.e., the speaker is centered and of appropriate size).

[0205] Step 370: Send the video data captured by the best shooting device to the virtual camera driver so that the virtual camera driver can send the video data to the target application for use.

[0206] As described above, based on determining the optimal shooting device and acquiring its footage for use, the control module can further determine the shooting parameter information for use by the optimal shooting device. This further ensures the quality of the video data captured by the optimal shooting device, i.e., the quality of the speaker's footage. Furthermore, when determining the shooting parameter information, several alternative shooting parameters with good speaker footage quality can be identified based on the speaker's relative position information and scored to select the highest-quality alternative. This ensures that the speaker footage captured with the highest-quality alternative is of better quality. Moreover, the Pan and Tilt parameters can be determined based on the speaker's relative position information, and the Zoom parameter can be determined using the speaker's actual size, desired size, and the focal length of the optimal shooting device. This allows for rapid determination of shooting parameter information while ensuring shooting quality. Furthermore, the shooting parameter information is only determined when the optimal shooting device's shooting parameters are adjustable, avoiding invalid determination of shooting parameter information when using a smart device with a non-adjustable shooting device.

[0207] The following is an exemplary description of the shooting control method provided in the embodiments of this application. In this example, the shooting device uses a PTZ camera, and a total of three PTZ cameras are used. Figure 4 This is a schematic diagram of camera distribution provided in one embodiment of this application, with reference to... Figure 4 The shooting space 41 is located indoors, with each subject 42 (i.e., the person being photographed) positioned in the center of the shooting space 41, facing the direction of the smart device 44. Three PTZ cameras 43 are located in front of and behind the subjects on both sides, allowing the three PTZ cameras to capture images of the subjects from different positions.

[0208] During the filming, Figure 5 This is a schematic diagram of a data stream transmission provided in one embodiment of this application, with reference to... Figure 5 Three PTZ cameras transmit video data to the switching module via either USB or TCP / IP protocols. The switching module then sends the video data to the algorithm module. The algorithm module identifies the speaker and determines their relative position to each camera. Subsequently, the algorithm module sends the relative position information and the video data to the control module. Based on the relative position information and the video data, the control module determines the optimal camera. Figure 5Taking PTZ camera 1 as the optimal shooting device as an example, the control module sends the device information and shooting parameter information of the optimal shooting device to the switching module. The switching module then sends the shooting parameter information to the optimal shooting device, which uses the corresponding shooting parameters. The switching module then sends the video data from the optimal shooting device to the virtual camera driver, which in turn sends the video data to the target application in the application layer for use, thereby achieving the tracking and shooting of the speaker. At this point, the shooting device can be considered to be in the data acquisition layer, responsible for acquiring video data. The switching module, algorithm module, control module, and virtual camera driver can be considered to be in the operating system layer, responsible for processing the acquired video data.

[0209] One embodiment of this application also provides a shooting control device. This shooting control device is applied to a smart device, which has a virtual camera driver and a target application installed.

[0210] Figure 6 This is a schematic diagram of a shooting control device provided in one embodiment of this application. (Reference) Figure 6 The shooting control device includes: a first video receiving unit 501, a first speaker identification unit 502, a first position determination unit 503, a first device determination unit 504, and a first data sending unit 505.

[0211] The system includes a first video receiving unit 501, used to receive video data collected by at least three shooting devices that capture the shooting space from different positions; a first speaker identification unit 502, used to identify faces appearing in each video data stream and select the face speaking in one of the video data streams as the identified speaker in the shooting space; a first position determination unit 503, used to determine the three-dimensional position coordinates of the speaker relative to each shooting device based on the two-dimensional coordinates of the speaker in the video data; a first device determination unit 504, used to determine the best shooting device for capturing the speaker among the shooting devices based on each video data stream and the corresponding three-dimensional position coordinates; and a first data sending unit 505, used to send the video data captured by the best shooting device to the virtual camera driver, so that the virtual camera driver sends the video data to the target application for use.

[0212] Figure 7 This is a schematic diagram of a shooting control device provided in another embodiment of this application, with reference to... Figure 7The shooting control device includes: a first video receiving unit 601, a first speaker identification unit 602, a first position determination unit 603, a first device determination unit 604, and a first data sending unit 605.

[0213] The system includes a second video receiving unit 601, used to receive video data collected by at least three shooting devices that capture the shooting space from different positions; a second speaker identification unit 602, used to determine the speaker in the shooting space based on the video data; a second position determination unit 603, used to calculate the relative position information of the speaker relative to each shooting device; a second device determination unit 604, used to determine the best shooting device for capturing the speaker among the shooting devices based on each video data and the corresponding relative position information; and a second data sending unit 605, used to send the video data captured by the best shooting device to the virtual camera driver, so that the virtual camera driver can send the video data to the target application for use.

[0214] In one embodiment of this application, the apparatus further includes a parameter determining unit and a parameter sending unit. The parameter determining unit is used to determine the shooting parameter information of the optimal shooting device. The parameter sending unit is used to send the shooting parameter information to the optimal shooting device, so that the optimal shooting device adjusts its own shooting parameters according to the shooting parameter information.

[0215] In one embodiment of this application, the apparatus further includes: an adjustable determination unit, configured to determine whether the shooting parameters of the optimal shooting device are adjustable before determining the shooting parameter information of the optimal shooting device; and a first execution unit, configured to perform the operation of determining the shooting parameter information of the optimal shooting device if it is adjustable.

[0216] In one embodiment of this application, the second speaker identification unit 602 is specifically used to: perform face recognition on each of the video data streams, and select the face of the person speaking in one of the video data streams as the identified speaker in the shooting space based on the face recognition results of each of the video data streams.

[0217] In one embodiment of this application, the second speaker identification unit 602 includes: a lip image acquisition subunit, used to perform face recognition on each video data stream to acquire the lip image of each identified face; and a speaker determination subunit, used to select the face that is speaking in one video data stream as the identified speaker in the shooting space based on the movement state of the lips in the lip image, wherein the lip movement state of the speaker is the most obvious.

[0218] In one embodiment of this application, the second position determination unit 603 is specifically used to: determine the relative position information of the speaker relative to the corresponding shooting device of the video data based on the two-dimensional coordinates of the speaker in the video data, and determine the relative position information of the speaker relative to each other shooting device according to the relative position information, wherein the relative position information is the three-dimensional position coordinates of the speaker relative to the shooting device.

[0219] In one embodiment of this application, the second device determining unit 604 includes: a face image determining subunit, used to determine the speaker's face image in each video data based on each relative position information; and an optimal shooting device determining subunit, used to determine the optimal shooting device for shooting the speaker based on the face image.

[0220] In one embodiment of this application, an optimal shooting device determination subunit is specifically used to score facial images and select the shooting device corresponding to the facial image with the highest score as the optimal shooting device for shooting the speaker.

[0221] In one embodiment of this application, the parameter determination unit includes: a candidate information acquisition subunit, used to acquire multiple candidate shooting parameters of the best shooting device based on the relative position information of the speaker relative to the best shooting device; and an information scoring subunit, used to score the multiple candidate shooting parameters and select the candidate shooting parameters with the highest score as the determined shooting parameter information.

[0222] In one embodiment of this application, the shooting parameters include Pan parameters, Tilt parameters, and Zoom parameters. The parameter determination unit includes: a first parameter calculation subunit, used to determine the Pan parameter information and Tilt parameter information of the optimal shooting device based on the speaker's relative position information relative to the optimal shooting device; and a second parameter calculation subunit, used to determine the Zoom parameter information of the optimal shooting device based on the actual size of the speaker in the video data captured by the optimal shooting device, the preset expected size, and the focal length of the optimal shooting device.

[0223] The aforementioned shooting control devices can be used to execute the shooting control methods provided in the corresponding embodiments, and have corresponding functions and beneficial effects.

[0224] It is worth noting that in the above embodiments of the shooting control device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0225] One embodiment of this application also provides a smart device. Figure 8This is a schematic diagram of the structure of a smart device provided in one embodiment of this application, with reference to... Figure 8 The smart device includes a processor 71 and a memory 72.

[0226] Among them, processor 71 can be one or more. Figure 8 Taking one example. In addition, smart devices may also include components such as displays. These components can be connected via a bus or other means; details about each component can be found in the preceding description.

[0227] The aforementioned smart device includes a shooting control unit that can be used to execute any shooting control method, possessing corresponding functions and beneficial effects. Specific details not described here can be found in the relevant descriptions of the aforementioned shooting control methods.

[0228] One embodiment of this application also provides a storage medium containing computer-executable instructions, which, when executed by the processor or network card of a smart device, are used to perform relevant operations in the shooting control method provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0229] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0230] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing module of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing module of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0231] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0232] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0233] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A shooting control method, applied to a smart device, characterized in that, The intelligent device is installed with a virtual camera driver and a target application program, and the photographing control method comprises: receiving video data collected by each photographing device, the photographing device being at least three and photographing a photographing space from different positions; identifying each face appearing in each piece of video data and selecting a face in one piece of video data that is speaking as a speaker in the photographing space; determining three-dimensional position coordinates of the speaker relative to each photographing device according to two-dimensional coordinates of the speaker in the video data; determining an optimal photographing device photographing the speaker in each photographing device according to each piece of video data and the corresponding three-dimensional position coordinates; sending video data photographed by the optimal photographing device to the virtual camera driver, so that the virtual camera driver sends the video data to the target application program for use.

2. A shooting control method, applied to a smart device, characterized in that, The intelligent device is installed with a virtual camera driver and a target application program, and the photographing control method comprises: receiving video data collected by each photographing device, the photographing device being at least three and photographing a photographing space from different positions; determining a speaker in the photographing space based on each piece of video data; calculating relative position information of the speaker relative to each photographing device; determining an optimal photographing device photographing the speaker in each photographing device according to each piece of video data and the corresponding relative position information; sending video data photographed by the optimal photographing device to the virtual camera driver, so that the virtual camera driver sends the video data to the target application program for use.

3. The photographing control method according to claim 2, wherein The method further comprises: determining photographing parameter information of the optimal photographing device; sending the photographing parameter information to the optimal photographing device, so that the optimal photographing device adjusts photographing parameters used by itself according to the photographing parameter information.

4. The photographing control method according to claim 3, wherein Before the determination of the photographing parameter information of the optimal photographing device, the method further comprises: judging whether the photographing parameters of the optimal photographing device are adjustable; if adjustable, performing the operation of determining the photographing parameter information of the optimal photographing device.

5. The photographing control method according to claim 2, wherein The determination of the speaker in the photographing space based on each piece of video data comprises: performing face recognition on each piece of video data and selecting a face in one piece of video data that is speaking as a speaker in the photographing space based on a face recognition result of each piece of video data.

6. The photographing control method according to claim 5, wherein The face recognition on each piece of video data and the selection of a face in one piece of video data that is speaking as a speaker in the photographing space based on a face recognition result of each piece of video data comprise: performing face recognition on each piece of video data to obtain a lip picture on each recognized face; selecting a face in one piece of video data that is speaking as a speaker in the photographing space according to a motion state of lips in the lip picture, the motion state of lips of the speaker being most obvious.

7. The photographing control method according to claim 5, wherein The calculating the relative position information of the speaker relative to each of the shooting devices comprises: determining the relative position information of the speaker relative to the corresponding shooting device of the video data based on the two-dimensional coordinates of the speaker in the video data, and determining the relative position information of the speaker relative to each of the shooting devices according to the relative position information, wherein the relative position information is the three-dimensional position coordinates of the speaker relative to the shooting device.

8. The photographing control method according to claim 2, wherein The determining the best shooting device for shooting the speaker in each of the shooting devices according to each of the video data and the corresponding relative position information comprises: determining a face picture of the speaker in each of the video data according to each of the relative position information; determining the best shooting device for shooting the speaker according to the face picture.

9. The photographing control method according to claim 8, wherein The determining the best shooting device for shooting the speaker according to the face picture comprises: scoring the face picture, and selecting the shooting device corresponding to the face picture with the highest score as the best shooting device for shooting the speaker.

10. The photographing control method according to claim 3, wherein The determining the shooting parameter information of the best shooting device comprises: obtaining a plurality of shooting parameter candidate information of the best shooting device according to the relative position information of the speaker relative to the best shooting device; scoring the plurality of shooting parameter candidate information, and selecting the shooting parameter candidate information with the highest score as the determined shooting parameter information.

11. The photographing control method according to claim 3, wherein The shooting parameter comprises a Pan parameter, a Tilt parameter and a Zoom parameter. The determining the shooting parameter information of the best shooting device comprises: determining the information of the Pan parameter and the information of the Tilt parameter of the best shooting device according to the relative position information of the speaker relative to the best shooting device; determining the information of the Zoom parameter of the best shooting device according to the actual size of the speaker in the video data shot by the best shooting device, the preset expected size and the focal length of the best shooting device.

12. A photographing control apparatus applied to a smart device, characterized in that, The intelligent device is installed with a virtual camera driver and a target application program, and the shooting control device comprises: a first video receiving unit configured to receive video data collected by each of the shooting devices, wherein the shooting devices are at least three and shoot a shooting space from different positions; a first speaker identifying unit configured to identify each face appearing in each of the video data, and select a face that is speaking in one of the video data as a speaker in the shooting space; a first position determining unit configured to determine three-dimensional position coordinates of the speaker relative to each of the shooting devices according to two-dimensional coordinates of the speaker in the video data; a first device determining unit configured to determine a best shooting device for shooting the speaker in each of the shooting devices according to each of the video data and the corresponding three-dimensional position coordinates; a first data sending unit configured to send video data shot by the best shooting device to the virtual camera driver, so that the virtual camera driver sends the video data to the target application program for use.

13. A photographing control apparatus applied to a smart device, characterized in that, The intelligent device is installed with a virtual camera driver and a target application program, and the photographing control apparatus comprises: a second video receiving unit configured to receive video data collected by each photographing device, the photographing devices being at least three and photographing a photographing space from different positions; a second speaker identifying unit configured to determine a speaker in the photographing space based on each piece of video data; a second position determining unit configured to calculate relative position information of the speaker relative to each photographing device; a second device determining unit configured to determine, according to each piece of video data and the corresponding relative position information, an optimal photographing device for photographing the speaker from among the photographing devices; a second data sending unit configured to send video data photographed by the optimal photographing device to the virtual camera driver, so that the virtual camera driver sends the video data to the target application program for use.

14. A smart device, comprising: The intelligent device comprises one or more processors and a memory; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the photographing control method according to any one of claims 1-11.

15. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed to implement the photographing control method according to any one of claims 1-11.

Images