Audio and video pickup position selection method and system, audio and video acquisition terminal and storage medium

Abstract

The application discloses an audio and video pickup position selection method and system, an audio and video acquisition terminal and a storage medium, and relates to the technical field of audio and video acquisition terminals. The method comprises the following steps: acquiring an initial audio and video picture; establishing a three-dimensional coordinate system, acquiring the coordinates of each pixel point in the initial audio and video picture in the three-dimensional coordinate system and the coordinates of the positions of each pickup in the three-dimensional coordinate system; defining a virtual camera, acquiring the position coordinates of the virtual camera in the three-dimensional coordinate system after zooming of the camera; and acquiring the pickup position coordinates adapted to the shooting picture after zooming of the camera. According to the application, the audio signals of the pickups in the three-dimensional space can be selected when the camera is zoomed, the audio signals of the appropriate pickups can be better selected and matched with the audio and video, and the audio playing effect of the audio and video is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio and video collection, and in particular to an audio and video pickup position selection method and system, an audio and video collection terminal and a storage medium. BACKGROUND

[0002] In actual audio and video collection, the live environment embodied by the audio and video picture is planar, and the position of the pickup is a two-dimensional coordinate with the audio and video picture as a reference. When the video is zoomed, the two-dimensional distance between each pickup and the center point of the audio and video picture is calculated to select the sound collected by the corresponding pickup as the audio content matched with the audio and video picture.

[0003] However, the pickups in the actual video monitoring live environment are arranged in various directions of the space where the camera is located, so if multiple pickups are distributed along the normal direction of the audio and video picture captured by the camera, the appropriate pickup cannot be obtained by this method.

[0004] Therefore, there is an urgent need for an audio and video pickup position selection method to solve the problems. SUMMARY

[0005] Therefore, it is necessary to provide an audio and video pickup position selection method and system, an audio and video collection terminal and a storage medium to solve the problems of the prior art, so that the audio signal of the appropriate pickup can be better selected to match the audio and video after the camera is zoomed, thereby improving the audio playback effect of the audio and video.

[0006] To solve the above technical problems, the technical solution adopted by the present application is as follows:

[0007] In a first aspect, an audio and video pickup position selection method is provided, which includes the following steps:

[0008] An initial audio and video picture is obtained, wherein the picture content presented in the initial audio and video picture is the scene between the camera and the plane where the preset monitoring boundary is located, which is captured by the camera at an initial focal length, and the plane where the preset monitoring boundary is located is perpendicular to the lens axis direction of the camera;

[0009] A three-dimensional coordinate system is established to obtain the coordinates of each pixel point in the initial audio and video picture in the three-dimensional coordinate system and the coordinates of the positions of each pickup in the three-dimensional coordinate system;

[0010] A virtual camera is defined to obtain the position coordinates of the virtual camera in the three-dimensional coordinate system after the camera is zoomed;

[0011] The position coordinates of the pickup adapted to the captured picture after the camera is zoomed are obtained.

[0012] In a second aspect, an audio-video sound collector position selection system is provided, comprising:

[0013] an audio-video picture acquisition module configured to acquire an initial audio-video picture;

[0014] a three-dimensional coordinate system establishment module configured to establish a three-dimensional coordinate system, and acquire coordinates of each pixel point in the initial audio-video picture in the three-dimensional coordinate system and coordinates of positions of each sound collector in the three-dimensional coordinate system;

[0015] a virtual camera module configured to define a virtual camera, and acquire position coordinates of the virtual camera in the three-dimensional coordinate system after zooming of the camera;

[0016] a sound collector position acquisition module configured to acquire sound collector position coordinates adapted to the zoomed shooting picture of the camera.

[0017] In a third aspect, an audio-video collection terminal is provided, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the audio-video sound collector position selection method described above when executing the computer program.

[0018] In a fourth aspect, a storage medium is provided, which stores a computer program, the computer program comprising program instructions, the program instructions implementing the audio-video sound collector position selection method described above when executed.

[0019] To sum up, the audio-video sound collector position selection method and system, the audio-video collection terminal and the storage medium provided by the present application can select audio signals of sound collectors in a three-dimensional space that are adapted to zooming of a camera, and can better select audio signals of sound collectors that are suitable for matching with audio-video, so as to improve the audio playing effect of audio-video. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of a first audio-video sound collector position selection method provided by an embodiment of the present application;

[0021] Figure 2 is a structural block diagram of an audio-video sound collector position selection system provided by an embodiment of the present application;

[0022] Figure 3 is an internal structural block diagram of an audio-video collection terminal provided by an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of relative positions of a camera and a sound collector in a y-axis direction provided by an embodiment of the present application;

[0024] Figure 5is a principle schematic view of relative position relationship of a camera and a sound pickup device in an x-axis direction provided by an embodiment of the present application;

[0025] Figure 6 is a principle schematic view of a position of a sound pickup device in an initial audio-video picture provided by an embodiment of the present application;

[0026] Figure 7 is a principle schematic view of relative position relationship of a virtual camera and a camera entity in a z-axis direction provided by an embodiment of the present application;

[0027] Figure 8 is a principle schematic view of a similar triangle formed by a virtual camera and a camera entity when the camera is in k times zoom provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to further understand the features, technical means and specific purposes and functions achieved by the present application, the present application is described in further detail below in combination with the drawings and specific embodiments.

[0029] The audio-video sound pickup device position selection method and system, the audio-video collecting terminal and the storage medium provided by the embodiment of the present application can be applied to the audio-video collecting terminal, and the suitable sound pickup device can be better selected to match the audio-video picture in the normal direction of the picture taken by the camera, so as to obtain the signal of the sound pickup device and form the best audio-video matching signal. The audio-video sound pickup device position selection method and system, the audio-video collecting terminal and the storage medium provided by the embodiment are described in detail below.

[0030] Figure 1 is a flow schematic view of a first audio-video sound pickup device position selection method provided by an embodiment of the present application, as shown in the figure, the audio-video sound pickup device position selection method comprises steps S110-S140, and the details are as follows: Figure 1

[0031] Step S110, obtaining an initial audio-video picture; wherein the picture content presented in the initial audio-video picture is the scene between the camera and the plane where the preset monitoring boundary is located, which is taken by the camera with an initial focal length, and the plane where the preset monitoring boundary is located is perpendicular to the lens axial direction of the camera.

[0032] Step S120, establishing a three-dimensional coordinate system, obtaining the coordinates (x0, y0, d0) of each pixel point in the initial audio-video picture under the three-dimensional coordinate system and the coordinates (x1, y1, z1), (x2, y2, z2), … (x N , N , N ​); wherein, d0 represents the distance between the preset monitoring boundary and the camera in the lens axial direction of the camera, the normal direction of the initial audio and video picture is parallel to the z axis of the three-dimensional coordinate system, the edge lines of the initial audio and video picture are respectively parallel to the x axis and the y axis of the three-dimensional coordinate system, the position of the camera is set as the origin of the three-dimensional coordinate system, that is, the initial position coordinates of the camera are (0, 0, 0), and the relative position relationship between the camera and the sound pickup device can be referred to Figures 4-6 .

[0033] In step S130, a virtual camera is defined, and the position coordinates (x0, y0, z0) of the virtual camera in the three-dimensional coordinate system after zooming of the camera are obtained.

[0034] The method of step S130 specifically includes:

[0035] The position coordinates (x0, y0, z0) of the virtual camera in the three-dimensional coordinate system after zooming of the camera are obtained through the formula ; wherein, the position of the virtual camera is obtained by moving the position of the camera along the z axis direction, in this embodiment, the position of the virtual camera is on the z axis of the three-dimensional coordinate system, d x represents the distance between the virtual camera and the preset monitoring boundary in the lens axial direction, k is the zooming multiple of the camera, x0 and y0 are respectively the coordinate values of the center point of the picture taken by the camera after zooming in the initial audio and video picture in the x axis and the y axis; d0 is a preset value, and the size of the parameter d0 is adjusted to correct the audio and video picture effect after zooming of the camera; in this embodiment, the range of d0 is 3-50 m.

[0036] Specifically, when the camera zooms, the zoomed picture is enlarged relative to the initial audio and video picture, in this embodiment, when the camera zooms k times, the area of the zoomed picture is enlarged k 2 times relative to the area of the initial audio and video picture, which is similar to the picture effect taken by the camera after moving forward along the lens axial direction at the initial focal length; a virtual camera is set, the distance between the virtual camera and the preset monitoring boundary is closer, and the distance between the virtual camera and the preset monitoring boundary is set as d x , then z0 = d0-d x , that is, the coordinates of the virtual camera in the three-dimensional coordinate system are (x0, y0, z0) = (x0, y0, d0-d x ).

[0037] As shown in Figure 7 and Figure 8 , the size of the initial audio and video picture taken by the camera is 1, and when the initial audio and video picture is enlarged k 2The display range of the scene in the zoomed audio and video picture is actually 1 / k of the display range of the scene in the initial audio and video picture 2 ; assuming that the visual angle of the camera is unchanged, when the initial audio and video picture is enlarged by k 2 times, it is equivalent to the audio and video picture captured after the camera is pushed forward, and according to the similar triangle calculation, the distance d x of the virtual camera from the preset monitoring boundary meets the formula

[0038] Step S140, acquiring the position coordinates of the microphone adapted to the zoomed picture captured by the camera; after the camera is zoomed, the audio signal of the appropriate microphone can be better selected to match the captured picture, so as to improve the audio playing effect of the audio and video.

[0039] The method of step S140 specifically includes:

[0040] Selecting d i min by the formula The position coordinates of the microphone corresponding to the minimum value are the position coordinates of the microphone adapted to the zoomed picture captured by the camera, and (x i , y i , z i ) represents the position coordinates of the i-th microphone, 1≤i≤N.

[0041] In other embodiments, the method of step S140 specifically includes:

[0042] Selecting d i min by the formula The position coordinates of the microphone corresponding to the minimum value are the position coordinates of the microphone adapted to the zoomed picture captured by the camera, and (x i , y i , z i ) represents the position coordinates of the i-th microphone, 1≤i≤N.

[0043] Specifically, the importance of the audio signal of the microphone located behind the virtual camera (i.e., z i <z0) is weaker than that of the microphone located in front of the virtual camera (i.e., z iThe importance of the audio signal of the microphone is considered, so that when the distance between the microphone and the virtual camera in the z-axis direction is calculated, the distance weight of the microphone located in front of or behind the virtual camera can be reduced, and a weighting factor gamma is introduced here to adjust the distance weight of the microphone located in front of or behind the virtual camera; in the embodiment, the front of the virtual camera is the direction in which the virtual camera faces the preset monitoring boundary, and the back of the virtual camera is the direction in which the virtual camera faces away from the preset monitoring boundary.

[0044] The audio-video microphone position selection method provided by the application can select the audio signal of the microphone in the three-dimensional space when the camera zooms, solves the problem that the existing audio-video acquisition terminal selects the microphone in two-dimensional coordinates and cannot select the microphone in the lens axis direction of the camera, and thus the appropriate audio signal of the microphone can be better selected to match the audio-video, so that the audio playing effect of the audio-video is improved.

[0045] Figure 2 The structure block diagram of the audio-video microphone position selection system provided by the application is shown in FIG. 1. Figure 2 Corresponding to the audio-video microphone position selection method, the application further provides an audio-video microphone position selection system, which comprises a module for executing the audio-video microphone position selection method.

[0046] Specifically, as shown in FIG. 1, the audio-video microphone position selection system comprises an audio-video picture acquisition module 110, a three-dimensional coordinate system establishment module 120, a virtual camera module 130 and a microphone position acquisition module 140. Figure 2

[0047] The audio-video picture acquisition module 110 is configured to acquire an initial audio-video picture; in the embodiment, the audio-video picture acquisition module 110 is a camera.

[0048] The three-dimensional coordinate system establishment module 120 is configured to establish a three-dimensional coordinate system, and acquire the coordinates of each pixel point in the initial audio-video picture in the three-dimensional coordinate system and the coordinates of the positions of each microphone in the three-dimensional coordinate system.

[0049] The virtual camera module 130 is configured to define a virtual camera, and acquire the position coordinates of the virtual camera in the three-dimensional coordinate system after the camera zooms. ​

[0050] The pickup position acquisition module 140 is configured to acquire pickup position coordinates adapted to the captured image after zooming of the camera.

[0051] In one embodiment, the virtual camera module 130 defines a virtual camera, and acquires position coordinates of the virtual camera in a three-dimensional coordinate system after zooming of the camera, and is specifically configured to perform the following steps:

[0052] The position coordinates of the virtual camera in the three-dimensional coordinate system after zooming of the camera are acquired by the formula The position coordinates of the virtual camera in the three-dimensional coordinate system after zooming of the camera are acquired by the formula x , wherein d0 represents the distance between the virtual camera and the preset monitoring boundary in the lens axial direction, k is the lens zoom multiple of the camera, x0 and y0 are respectively the coordinate values of the center point of the captured image after zooming of the camera in the x-axis and y-axis in the initial audio-video image, and z0=d0-d. x .

[0053] In one embodiment, the pickup position acquisition module 140 acquires pickup position coordinates adapted to the captured image after zooming of the camera, and is specifically configured to perform the following steps:

[0054] The pickup position coordinates adapted to the captured image after zooming of the camera are acquired by the formula The pickup position coordinates corresponding to the minimum value of d i are selected, and the pickup at the position coordinates is the pickup adapted to the captured image after zooming of the camera, wherein (x i , y i , z i ) represents the position coordinates of the i-th pickup, and 1≤i≤N.

[0055] In other embodiments, the pickup position acquisition module 140 acquires pickup position coordinates adapted to the captured image after zooming of the camera, and is specifically configured to perform the following steps:

[0056] The pickup position coordinates adapted to the captured image after zooming of the camera are acquired by the formula The pickup position coordinates corresponding to the minimum value of d i are selected, and the pickup at the position coordinates is the pickup adapted to the captured image after zooming of the camera, wherein (x i , y i , x i ) represents the position coordinates of the i-th pickup, and 1≤i≤N, wherein

[0057] It should be noted that the specific implementation process of the above-mentioned audio-video pickup position selection system and each module can be clearly understood by those skilled in the art, which can be referred to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, it will not be described here.

[0058] Figure 3 This is an internal structural block diagram of an audio / video acquisition terminal provided in an embodiment of the present invention, such as... Figure 3 As shown, the audio and video acquisition terminal provided by the present invention includes a memory and a processor connected via a system bus. The memory stores a computer program, and the processor provides computing and control capabilities to support the operation of the entire terminal. When the processor executes the computer program, it implements the aforementioned audio and video microphone position selection method. By selecting the audio signal of a microphone in a suitable three-dimensional space when the camera zooms, the method solves the drawback of existing audio and video acquisition terminals that select the appropriate microphone using two-dimensional coordinates and cannot select the microphone in the lens axis direction of the camera. This allows for better selection of the appropriate microphone's audio signal to match the audio and video, thereby improving the audio playback effect of the audio and video.

[0059] The memory may include a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When the computer program is executed by the processor, it enables the processor to implement an audio / video microphone position selection method.

[0060] The internal memory may also store a computer program, which, when executed by the processor, causes the processor to perform an audio / video microphone position selection method. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the application of the present application to other devices. Specific devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0061] In one embodiment, the audio / video microphone position selection method provided in this application can be implemented as a computer program, and the computer program can be implemented in a manner such as... Figure 3 The audio / video acquisition terminal shown is running on this system. The memory of the audio / video acquisition terminal can store the various program modules that make up the audio / video microphone location selection system, for example... Figure 2 The diagram shows an audio / video image acquisition module 110, a three-dimensional coordinate system establishment module 120, a virtual camera module 130, and a microphone position acquisition module 140. The computer program comprised of these modules causes the processor to execute the steps of the audio / video microphone position selection system described in the various embodiments of this application. For example, Figure 3 The audio and video capture terminal shown can be used as follows Figure 2The audio and video picture acquisition module 110 in the audio and video sound pickup position selection system shown acquires an initial audio and video picture; the three-dimensional coordinate system establishment module 120 establishes a three-dimensional coordinate system, acquires coordinates of each pixel point in the initial audio and video picture under the three-dimensional coordinate system and coordinates of positions of each sound pickup under the three-dimensional coordinate system; the virtual camera module 130 defines a virtual camera, acquires position coordinates of the virtual camera in the three-dimensional coordinate system after zooming of the camera; and the sound pickup position acquisition module 140 acquires sound pickup position coordinates adapted to the shooting picture after zooming of the camera.

[0062] It should be understood that, in the embodiments of the present application, the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0063] It can be understood by those skilled in the art that all or part of the processes in the method of the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments of the method.

[0064] Therefore, the present application also provides a storage medium. The storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. The program instructions are executed by the processor to make the processor perform the following steps:

[0065] Step S110, acquiring an initial audio and video picture;

[0066] Step S120, establishing a three-dimensional coordinate system, acquiring coordinates of each pixel point in the initial audio and video picture under the three-dimensional coordinate system and coordinates of positions of each sound pickup under the three-dimensional coordinate system;

[0067] Step S130, defining a virtual camera, acquiring position coordinates of the virtual camera in the three-dimensional coordinate system after zooming of the camera;

[0068] Step S140, acquiring the pickup position coordinates adapted to the zoomed shooting picture of the camera.

[0069] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer readable storage media that can store program codes.

[0070] In conclusion, the audio-video pickup position selection method and system, the audio-video acquisition terminal and the storage medium can select the audio signal of the pickup in the three-dimensional space when the camera is zoomed, and can better select the audio signal of the pickup to match the audio-video, so as to improve the audio playing effect of the audio-video.

[0071] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0072] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0073] The steps in the method of the embodiments of the present application can be adjusted in sequence, combined and deleted according to actual needs. The units in the device of the embodiments of the present application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application, essentially or in the contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing an apparatus (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application.

[0074] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for selecting a position of an audio-video pickup device, characterized by, The method comprises the following steps: An initial audio-video picture is acquired; wherein the picture content presented in the initial audio-video picture is a scene between a camera and a plane where a preset monitoring boundary is located, which is captured by the camera at an initial focal length, and the plane where the preset monitoring boundary is located is perpendicular to the lens axial direction of the camera; A three-dimensional coordinate system is established, and coordinates corresponding to each pixel point in the initial audio-video picture in the three-dimensional coordinate system and coordinates corresponding to the positions of each microphone in the three-dimensional coordinate system are acquired; A virtual camera is defined, and the position coordinates of the virtual camera in the three-dimensional coordinate system after zooming of the camera are acquired; Microphone position coordinates adaptive to the captured picture after zooming of the camera are acquired; The method for defining the virtual camera and acquiring the position coordinates of the virtual camera in the three-dimensional coordinate system after zooming of the camera comprises the following steps: The position coordinates (x0, y0, z0) of the virtual camera in the three-dimensional coordinate system after zooming of the camera are obtained by the formula The position coordinates (x0, y0, z0) of the virtual camera in the three-dimensional coordinate system after zooming of the camera are obtained by the formula x The position coordinates (x0, y0, z0) of the virtual camera in the three-dimensional coordinate system after zooming of the camera are obtained by the formula x ; The method for acquiring the microphone position coordinates adaptive to the captured picture after zooming of the camera comprises the following steps: By formula Select d i The minimum value corresponds to the pickup position coordinates, and the pickup at the position coordinates is the pickup adapted to the shooting picture after the camera zooms, (x i ,y i , z i ) represents the position coordinates of the ith pickup, 1≤i≤N, wherein, 2. An audio-video pick-up position selection system characterized by, The method comprises the following steps: An audio-video picture acquisition module is configured to acquire an initial audio-video picture; A three-dimensional coordinate system establishment module is configured to establish a three-dimensional coordinate system and acquire coordinates corresponding to each pixel point in the initial audio-video picture in the three-dimensional coordinate system and coordinates corresponding to the positions of each microphone in the three-dimensional coordinate system; A virtual camera module is configured to define a virtual camera and acquire the position coordinates of the virtual camera in the three-dimensional coordinate system after zooming of the camera; A microphone position acquisition module is configured to acquire microphone position coordinates adaptive to the captured picture after zooming of the camera; The method for defining the virtual camera and acquiring the position coordinates of the virtual camera in the three-dimensional coordinate system after zooming of the camera comprises the following steps: The position coordinates (x0, y0, z0) of the virtual camera in the three-dimensional coordinate system after zooming of the camera are obtained by the formula The position coordinates (x0, y0, z0) of the virtual camera in the three-dimensional coordinate system after zooming of the camera are obtained by the formula x The position coordinates (x0, y0, z0) of the virtual camera in the three-dimensional coordinate system after zooming of the camera are obtained by the formula x ; The microphone position acquisition module is configured to acquire the microphone position coordinates adaptive to the captured picture after zooming of the camera, and specifically configured to perform the following steps: Through formula Select d i The minimum value corresponds to the microphone position coordinates. The microphone at these coordinates is the one adapted to the zoomed-out image from the camera. (x i ,y i , z i Let N represent the position coordinates of the i-th microphone, where 1 ≤ i ≤ N.

3. An audio and video acquisition terminal, characterized by: The audio-video acquisition terminal comprises a memory and a processor, the memory stores a computer program, and the processor implements the audio-video microphone position selection method of claim 1 when executing the computer program.

4. A storage medium characterized by: The storage medium stores a computer program, and the computer program comprises program instructions which, when executed, implement the audio-video microphone position selection method of claim 1.

Images