Video-based sound source positioning angle calibration method, system, device and medium

Abstract

The application discloses a kind of video-based sound source positioning angle calibration method, system, equipment and medium, relevant method includes: by face detection algorithm to the target person in each video frame in the video that camera shoots is detected, the pixel coordinates of the center point of the detection frame obtained as a sound source point, by target tracking obtains the sound source point corresponding to target person in all video frames;The camera is calibrated, and the intrinsic matrix and distortion coefficient are obtained;Based on the intrinsic matrix and distortion coefficient, each sound source point is respectively restored, and the azimuth angle of each sound source point relative to the camera is obtained.The above scheme can realize the automatic calibration of sound source positioning angle, has higher robustness and accuracy, and has fast calibration speed and low cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video signal processing, and particularly relates to a video-based sound source positioning angle calibration method, system, device and medium. BACKGROUND

[0002] Sound source localization (SSL) technology aims to estimate the position of a single or multiple sound sources relative to a microphone array. In most cases, the direction of arrival of each sound source is of interest, so most sound source localization methods focus on azimuth and elevation angle estimation. Effective sound source localization technology is very important in many applications, including automatic speech recognition (ASR), teleconference systems, robot auditory systems, and hearing aid technology. For example, to make a camera automatically turn to the sound source to record video and audio, a sound source localization system can be used; in medical devices such as hearing aids, the use of sound source localization technology can help disabled people with hearing impairment better understand language in noisy and complex scenes; in order to make the exchange and interaction between robots and humans more efficient and accurate, sound source localization is used in robot auditory systems to determine the relative position of the speaker.

[0003] After decades of development, traditional sound source localization methods can be roughly divided into the following categories: 1) methods based on signal subspace, such as multiple signal classification (MUSIC) and estimation of signal parameters via rotational invariance techniques (ESPRIT); 2) time delay of arrival (TDOA) methods using generalized cross-correlation and least squares (LS); 3) methods based on signal synchronization, such as steering response power with phase transform (SRP-PHAT) and multi-channel cross-correlation (MCCC); 4) methods based on blind identification of impulse responses, such as adaptive eigenvalue decomposition (AED) algorithm and independent component analysis; 5) sparse signal representation methods based on L1-norm penalty; 6) model-based methods, such as maximum likelihood method (MLM) and modeling of the first-arriving echo effect. However, these traditional sound source localization techniques have low positioning accuracy and anti-interference ability, and perform poorly in noisy and reverberant environments.

[0004] In recent years, with the emergence and large increase in the application of deep neural networks in various fields of machine learning, sound source localization methods based on deep neural networks have been proposed, and due to their strong regression ability, they have been proven to be superior to traditional sound source localization methods. Compared with traditional methods, methods using deep learning can be trained to adapt to different acoustic conditions, and if there is training data from different acoustic conditions, noise and reverberation can be handled robustly.

[0005] However, the research on sound source positioning technology needs to accurately label the real angle of the sound source. Common sound source angle labeling methods include device tracking and manual labeling. For example, in the DCASE competition, the characters in the video wear optical tracking devices, and other sound-emitting objects also have corresponding trackers. The Cartesian coordinates of all tracked sound sources provided by the tracker are converted into the sound source angle (Direction Of Arrival, DOA) relative to the microphone coordinates. Finally, the labeled angle is verified by manually observing each video. The method of LOCATA challenge 2020 is more complex: 10 synchronous and calibrated infrared cameras OptiTrack Flex13 are installed along the perimeter of the recording area, and reflective markers that can be detected by infrared sensors are installed at the sound-emitting source and the microphone position. The arrangement of each set of reflective markers is specific and unique, ensuring that the OptiTrack system can identify, distinguish, and determine the angles of all sound sources and microphone arrays. Finally, the OptiTrack system uses multi-point positioning of the infrared camera to obtain the real reference angle of each marker position with an accuracy of about 1mm and a frame rate of 120Hz. It can be seen that these methods are often costly and time-consuming.

[0006] Therefore, the traditional tracking labeling method is time-consuming, laborious and costly, which is not conducive to the automatic labeling of sound source positioning angles, and it is difficult to meet the technical requirements of this field. Therefore, an automatic labeling technology is needed to solve this problem. SUMMARY

[0007] The purpose of the present application is to provide a video-based sound source positioning angle labeling method, system, device and medium, which can realize automatic labeling of sound source positioning angles, has high robustness and accuracy, fast labeling speed and low cost.

[0008] The purpose of the present application is achieved by the following technical solutions:

[0009] A video-based sound source positioning angle labeling method, comprising:

[0010] Step S1: detecting the target person in each video frame of the video captured by the camera through a face detection algorithm, taking the pixel coordinates of the center point of the obtained detection frame as a sound source point, and obtaining the corresponding sound source points of the target person in all video frames through target tracking;

[0011] Step S2: calibrating the camera to obtain an intrinsic matrix and distortion coefficients;

[0012] Step S3: based on the intrinsic matrix and distortion coefficients, restoring each sound source point respectively to obtain the azimuth angle of each sound source point relative to the camera.

[0013] A video-based sound source positioning angle calibration system comprises:

[0014] A target detection tracking and sound source point determination unit is configured to perform step S1, which comprises detecting a target person in each video frame of a video captured by a camera through a face detection algorithm, taking the pixel coordinates of the center point of the obtained detection frame as a sound source point, and obtaining the sound source points corresponding to the target person in all video frames through target tracking.

[0015] A camera calibration unit is configured to perform step S2, which comprises calibrating the camera to obtain an intrinsic matrix and distortion coefficients.

[0016] A sound source positioning angle calibration unit is configured to perform step S3, which comprises restoring each sound source point based on the intrinsic matrix and distortion coefficients to obtain the azimuth angle of each sound source point relative to the camera.

[0017] A processing device comprises one or more processors, and a memory configured to store one or more programs.

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium stores a computer program, which, when executed by a processor, implements the aforementioned method.

[0020] The technical solution provided by the present application can be seen as follows: 1) The parameters of the intrinsic matrix of the camera and the distortion coefficients are obtained through the mature Zhang Zhengyou calibration method. The technical solution has a complete theoretical basis, high operability and low implementation difficulty. Since the size of the checkerboard and the origin of the world coordinate system are known, the physical coordinate values of the corner points of the calibration board can be obtained through simple calculation; 2) Through the conversion from the pixel coordinate system to the camera coordinate system, the angle of the sound source relative to the camera is automatically obtained, and the technical solution has high robustness and implementability. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 A flowchart of a video-based sound source positioning angle calibration method provided by the embodiments of the present application is shown.

[0023] Figure 2 An imaging process schematic diagram provided for the embodiment of the present application;

[0024] Figure 3 A conversion principle diagram from pixel coordinates to camera coordinates provided for the embodiment of the present application;

[0025] Figure 4 A chessboard calibration plate schematic diagram provided for the embodiment of the present application;

[0026] Figure 5 A schematic diagram of a video-based sound source positioning angle calibration system provided for the embodiment of the present application;

[0027] Figure 6 A schematic diagram of a processing device provided for the embodiment of the present application; DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0029] Firstly, the terms possibly used in the present application are explained as follows:

[0030] The terms “include”, “contain”, “have”, “possess” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, the inclusion of a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed but known in the art.

[0031] The AAAA method provided by the present application is described in detail below. The contents not described in detail in the embodiments of the present application belong to the prior art known to those skilled in the art. The specific conditions not specified in the embodiments of the present application are carried out according to the conventional conditions in the art or the conditions recommended by the manufacturer. The reagents or instruments used in the embodiments of the present application are not specified by the manufacturer, and are all conventional products that can be obtained by market purchase.

[0032] Example 1

[0033] The embodiment of the application provides a video-based sound source positioning angle calibration method, which is a novel sound source angle labeling technology based on a pinhole camera model. The angle of a target in a video relative to a camera in a real world can be calculated by using conversion between a camera coordinate system and a pixel coordinate system. Figure 1 As shown in the figure, the method mainly comprises the following steps:

[0034] Step S1, a face detection algorithm is used to detect target persons in each video frame of a video shot by a camera, pixel coordinates of center points of obtained detection frames are taken as sound source points, and target tracking is used to obtain sound source points corresponding to the target persons in all video frames.

[0035] In the embodiment of the application, all target persons (target speakers) in each video frame are detected, and a single detection frame obtained is a rectangular frame containing a head region and a shoulder region of a single target person. For a single target person, target person tracking is realized by matching detection frames in adjacent video frames. For each video frame, the number of target persons is denoted as N, and pixel coordinates of a sound source point p c of a cth target person are At this time, the pixel coordinates of the sound source point are distorted pixel coordinates, which need to be corrected. Details are described below.

[0036] Exemplarily, a face detection algorithm based on a YoLo-v5 network and a DeepSort algorithm can be selected.

[0037] Step S2, the camera is calibrated to obtain an intrinsic matrix and distortion coefficients.

[0038] In the embodiment of the application, Zhang Zhengyou calibration is selected to calibrate the camera used for shooting, and preferred implementation modes are as follows:

[0039] Step S21, the camera is used to shoot a checkerboard calibration board from different angles to obtain a group of checkerboard images.

[0040] Step S22, n corner points are selected from each checkerboard image, and pixel coordinates (u i ,v i ) of each corner point are obtained from the image, i=1…n.

[0041] Step S23, since the world coordinate system of the checkerboard calibration board is defined in advance, that is, the size of each grid of the calibration board is known, physical coordinates (U i ,V i ,W i), and since the Zhang Zhengyou calibration method fixes the world coordinate system on the checkerboard, the physical coordinates W i =0 of any point on the checkerboard. i i i =0), i=1…n.

[0042] Step S24, using the pixel coordinates (u i ,v i ), i=1…n, and the corresponding physical coordinates (U i ,V i ,W i =0), i=1…n, of each corner point, calibrate the intrinsic matrix of the camera.

[0043] The present application represents the imaging process of the camera based on four coordinates, and can locate the coordinates of the target point in the pixel in the real three-dimensional space, and the camera imaging process is as shown in Figure 2 .

[0044] Define UVW, XYZ, xy, and uv to represent the world coordinate system, the camera coordinate system (the camera coordinate system), the image coordinate system, and the pixel coordinate system, respectively; according to the imaging process of the camera as shown in Figure 2 , the conversion relationship from the world coordinate system to the pixel coordinate system is represented as:

[0045]

[0046] In the above formula, the world coordinate system UVW is subjected to a rigid body transformation: to obtain the camera coordinate system XYZ, and the above conversion relationship represents a non-distorted camera imaging model. Since the picture taken by the camera also has a certain distortion, in addition to the intrinsic matrix, the distortion coefficient also needs to be calibrated to further perform the distortion removal processing on the taken picture.

[0047] wherein the matrix is referred to as the intrinsic matrix λ of the camera; (U, V, W) is the physical coordinates of a point in the world coordinate system, (u, v) is the pixel coordinates in the pixel coordinate system; f is the image distance of the camera, dx and dy respectively represent the physical length of one pixel in the x and y directions on the camera photosensitive plate (a component inside the camera); u0 and v0 respectively represent the coordinates of the center of the camera photosensitive plate in the pixel coordinate system, and θ represents the angle between the horizontal side and the vertical side of the camera photosensitive plate; the matrix is referred to as the extrinsic matrix of the camera, and the extrinsic matrix depends on the relative position of the camera coordinate system and the world coordinate system, R represents the rotation matrix, T represents the translation vector, and the vertical axis position Z in the camera coordinate system XYZ is the scale factor.​​

[0048] Since θ is close to 90°, let θ = 90°, the pixel coordinates (u i ,v i ), i = 1...n, of each corner point in all checkerboard images and the corresponding physical coordinates (U i ,V i ,W i = 0), i = 1...n, in the world coordinate system are used to solve the intrinsic matrix λ of the camera by matrix operation:

[0049]

[0050] Step S25, the pixel coordinates (u i ,v i ), i = 1...n, of each corner point in all checkerboard images and the corresponding physical coordinates (U i ,V i ,W i = 0), i = 1...n, in the world coordinate system are used to calibrate the camera distortion coefficients:

[0051] The second-order radial distortion formula is defined as:

[0052]

[0053] Where (x, y) represents the coordinates in the image coordinate system xy, which is the ideal non-distorted normalized image coordinates, (x 2 ,y 2 ) represents the distorted normalized image coordinates; r is the distance from the image pixel point to the image center point, i.e. r 2 = x i +y i ; the coefficient k1 and the coefficient k2 are the camera distortion coefficients k:

[0054] The conversion relationship between the image coordinate system and the pixel coordinate system is:

[0055]

[0056] Where (u i ,v i ) is the pixel coordinates of each corner point, which is the ideal non-distorted pixel coordinates, let θ = 90°, then:

[0057]

[0058] Where (x i ,y iThe corresponding coordinates in the xy coordinate system.

[0059] Correspondingly, the distorted pixel coordinates for:

[0060]

[0061] in, distorted pixel coordinates The corresponding coordinates in the xy coordinate system.

[0062] Substituting the second-order radial distortion formula, we obtain the camera distortion coefficient k.

[0063] Step S3: Based on the intrinsic parameter matrix and distortion coefficients, each sound source point is restored to obtain the azimuth angle of each sound source point relative to the camera.

[0064] In this embodiment of the invention, the angle of the sound source in the real world can be obtained simply by transforming between the camera coordinate system and the pixel coordinate system. First, the linear microphone array is placed on the horizontal x-axis of the camera coordinate system, with its center point coinciding with the origin of the camera coordinate system. Therefore, the target point in the video can be transformed into the camera coordinate system, and this point can be located as the sound source point.

[0065] Figure 3 This demonstrates the transformation process from pixel coordinates to camera coordinates, where XYZ, xy, and uv represent the camera coordinate system, image coordinate system, and pixel coordinate system, respectively. O is defined as... c O and O′ represent the origins of the camera coordinate system, image coordinate system, and pixel coordinate system, respectively.

[0066] In step S1 above, the sound source point p corresponding to the c-th target person in each video frame is obtained. c pixel coordinates N represents the number of target individuals; the sound source point p c The corresponding point P in the camera coordinate system c The coordinates are represented as (X c ,Y c Z c Correspondingly, the physical coordinates of the corresponding point in the world coordinate system are represented as (U... c V c W c ).

[0067] Referring to the description in step S2 above, the transformation relationship between the camera coordinate system and the image coordinate system is expressed as follows:

[0068]

[0069] The transformation relationship between the image coordinate system and the pixel coordinate system (θ = 90°) is expressed as follows:

[0070]

[0071] Where (x, y) are the coordinates in the image coordinate system, (X, Y) are the coordinates in the image coordinate system. c ,Y c Z c (u,v) represents the coordinates in the camera coordinate system, (u,v) represents the pixel coordinates in the pixel coordinate system; f represents the image distance of the camera; dx and dy represent the physical lengths of a pixel on the camera's image sensor in the x and y directions, respectively; and u0 and v0 represent the coordinates of the center of the camera's image sensor in the pixel coordinate system, respectively.

[0072] Using the distortion coefficient k to represent the sound source point p c pixel coordinates Perform corrections to obtain the corrected pixel coordinates (u c ,v c The coordinates of the sound source point p are ideal and distortion-free pixels. Then, using the intrinsic parameter matrix, and combining the transformation relationships between the image coordinate system and the pixel coordinate system, as well as the transformation relationships between the camera coordinate system and the image coordinate system, the sound source point p is obtained. c The corresponding midpoint P in the camera coordinate system c coordinates (X) c ,Y c Z c ), represented as:

[0073] P c =Φ(p c ,λ)

[0074] Where Φ represents the transformation function from pixel coordinates to camera coordinates. When the camera is a monocular camera, Z... c =1, then vector O c P c and axis O c The angle between X and P represents the corresponding sound source point p. c The azimuth angle relative to the camera.

[0075] Based on the above principles, the above process can be achieved using the `undistortPoints` function in the OpenCV library. Specifically, the input to the `undistortPoints` function is the camera intrinsic parameter matrix λ, the camera distortion coefficients k, and the pixel coordinates (u) obtained from the face detection algorithm. c ,v c ), c = 1…N, the output is the camera coordinates (X, N) of the reconstructed sound source point. c ,Y c Z c =1), c=1…N, by calculating vector Oc P c and axis O c The angle between X and Y gives the azimuth angle of the target person relative to the camera.

[0076] The implementation process of the above method will be introduced below with a specific example.

[0077] Step S1: In a rectangular room approximately 3-5m long, 5-10m wide, and 3m high, a 6-channel linear microphone array is placed along the long axis of the room's horizontal plane, with its center coordinates as the origin. A monocular camera is then placed there. A face detection algorithm based on the YoLo-v5 network and DeepSort algorithm is used to detect the head and shoulders of target individuals in the video captured by the camera, obtaining multiple head and shoulder bounding boxes for each frame. The pixel coordinates corresponding to the center point of each head and shoulder bounding box are used as a sound source point. Simultaneously, multiple head and shoulder bounding boxes of target individuals in adjacent frames are matched to achieve target person tracking. For each video frame, the number of target individuals is denoted as N, and the sound source point p corresponding to the c-th target individual is... c The pixel coordinates are

[0078] Step S2: Using the camera used for shooting video, film from different angles. Figure 4 The checkerboard calibration board shown is photographed to obtain a set of checkerboard images. From each checkerboard image, n corner points are selected, and the pixel coordinates (u) of each corner point are obtained from the image. i ,v i ), i = 1…n. Using the pixel coordinates (u) of each corner point i ,v i ) and the physical coordinates (U) of each corner point in the world coordinate system. i V i W i =0) The intrinsic parameter matrix λ and distortion coefficient k of the camera are calculated as follows:

[0079]

[0080] k=[-0.34,0.11,0.0,0.0,-0.01]

[0081] Step S3: Using the undistortPoints function from the OpenCV library, the camera intrinsic parameter matrix λ, the camera distortion coefficients k, and the pixel coordinates of multiple target figures obtained from the face detection algorithm. The camera coordinates (X) of the sound source point can be reconstructed. c ,Y c Z c =1), c=1…N, by calculating vector Oc P c and the angle between the axis O c X is the azimuth angle of the target person relative to the camera.

[0082] In the test phase, five marked objects were randomly placed in the room where the video was recorded as calibration points, and two groups of photos were taken.

[0083] In the room, the real angle was calculated by the relative distance between the five calibration points and the camera. In the pixel coordinates of the image, the coordinates in the pixel coordinate system were obtained by calibrating the pixel point pairs of the five calibration points. Subsequently, the camera intrinsic matrix, camera distortion coefficient obtained by calibration and pixel coordinate pairs obtained from the image were used to calculate the angle of the five objects relative to the camera as the calibration angle.

[0084] Finally, the two groups of calibration angles are shown in Table 1.

[0085] Table 1: Comparison of calibration angles and real angles

[0086]

[0087]

[0088] From Table 1, it can be seen that when the camera is far away from the object, the error between the calibration angle and the real angle is less than 0.5°, the relative angle error is less than 1%, and the average angle error is 0.34°; when the camera is close to the object, the error between the calibration angle and the real angle is less than 4°, the relative angle error is less than 6%, and the average angle error is 2.19°.

[0089] Experiments show that the angle error between the calibrated sound source angle and the real angle is stable in a very low range, so the method has high robustness and accuracy in sound source angle calibration technology.

[0090] Example Two

[0091] The application also provides a video-based sound source positioning angle calibration system, which is mainly based on the method provided in the foregoing embodiments, as shown in Figure 5 The system mainly includes:

[0092] The target detection tracking and sound source point determination unit is used to execute step S1, and step S1 includes: detecting the target person in each video frame of the video captured by the camera through a face detection algorithm, taking the pixel coordinates of the center point of the obtained detection frame as a sound source point, and obtaining the sound source points corresponding to the target person in all video frames through target tracking.

[0093] The camera calibration unit is configured to perform step S2, and step S2 comprises calibrating the camera to obtain an intrinsic matrix and distortion coefficients.

[0094] The sound source positioning angle calibration unit is configured to perform step S3, and step S3 comprises respectively restoring each sound source point based on the intrinsic matrix and distortion coefficients to obtain an azimuth angle of each sound source point relative to the camera.

[0095] In the embodiment of the present application, in step S1, all target persons in each video frame are detected, and a single detection frame obtained is a rectangular frame containing a head region and a shoulder region of a single target person, for a single target person, tracking of the target person is realized by matching detection frames in adjacent video frames, for each video frame, the number of target persons is denoted as N, and a pixel coordinate of a sound source point p c of a cth target person is

[0096] In the embodiment of the present application, the step of calibrating the camera to obtain the intrinsic matrix and distortion coefficients comprises:

[0097] Step S21, the camera is used to capture a set of checkerboard images by capturing the checkerboard calibration board from different angles.

[0098] Step S22, n corner points are selected from each checkerboard image, and pixel coordinates (u i ,v i ) of each corner point are obtained from the image, i=1…n.

[0099] Step S23, according to a defined world coordinate system of the checkerboard calibration board, physical coordinates (U i ,V i ,W i =0) of each focal point in the world coordinate system are calculated, i=1…n.

[0100] Step S24, the camera intrinsic matrix is calibrated by using pixel coordinates (u i ,v i ) of each corner point, i=1…n, and corresponding physical coordinates (U i ,V i ,W i =0) in the world coordinate system, i=1…n.

[0101] UVW, XYZ, xy and uv respectively represent a world coordinate system, a camera coordinate system, an image coordinate system and a pixel coordinate system, and a conversion relationship from the world coordinate system to the pixel coordinate system is represented as:

[0102]

[0103] where matrix is called the intrinsic matrix of the camera; (U, V, W) is the physical coordinate of a point in the world coordinate system, (u, v) is the pixel coordinate in the pixel coordinate system; f is the image distance of the camera, dx, dy represent the physical length of a pixel in the x, y direction on the camera's photosensitive plate respectively; u0, v0 represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system respectively, θ represents the angle between the horizontal side and the vertical side of the camera's photosensitive plate; matrix is called the extrinsic matrix of the camera, which depends on the relative position of the camera coordinate system and the world coordinate system, R represents the rotation matrix, T represents the translation vector, and the vertical axis position Z in the camera coordinate system XYZ is the scale factor;

[0104] Let θ = 90°, and use the pixel coordinates (u i ,v i ) of each corner point in all checkerboard images, i = 1…n, and the corresponding physical coordinates (U i ,V i ,W i = 0) in the world coordinate system, i = 1…n, to solve the intrinsic matrix λ of the camera through matrix operation:

[0105]

[0106] Step S25, use the pixel coordinates (u i ,v i ) of each corner point in all checkerboard images, i = 1…n, and the corresponding physical coordinates (U i ,V i ,W i = 0) in the world coordinate system, i = 1…n, to calibrate the camera distortion coefficient:

[0107] Define the second-order radial distortion formula:

[0108]

[0109] where (x, y) represents the coordinates in the image coordinate system xy, which is the ideal normalized image coordinate without distortion, (x', y') represents the distorted normalized image coordinate; r is the distance from the image pixel point to the image center point, i.e. r 2 = x 2 +y 2 ; the coefficient k1 and the coefficient k2 are the camera distortion coefficient k: The conversion relationship between the image coordinate system and the pixel coordinate system is:

[0110]

[0111] wherein, (u i ,v i ) is the pixel coordinate of each corner point, which is the ideal non-distorted pixel coordinate, and θ=90°, then:

[0112]

[0113] wherein, (x i ,y i ) is the pixel coordinate of the corner point (u i ,v i ) corresponding coordinate in the image coordinate system xy.

[0114] Correspondingly, the distorted pixel coordinate is:

[0115]

[0116] wherein, is the distorted pixel coordinate corresponding coordinate in the image coordinate system xy.

[0117] The second-order radial distortion formula is brought in to obtain the distortion coefficient k of the camera.

[0118] In the embodiment of the application, the step of restoring each sound source point based on the intrinsic matrix and the distortion coefficient to obtain the azimuth angle of each sound source point relative to the camera comprises:

[0119] XYZ, xy and uv are defined to represent the camera coordinate system, the image coordinate system and the pixel coordinate system respectively, O c , O and O' are defined to represent the origin of the camera coordinate system, the image coordinate system and the pixel coordinate system respectively;

[0120] The number of target persons in each video frame is defined as N, and the pixel coordinate of the sound source point p c of the cth target person is The coordinates of the corresponding point P c of the sound source point p c in the camera coordinate system are represented as (X c , Y c , Z c ), and the corresponding physical coordinates of the corresponding point in the world coordinate system are represented as (U c , V c , W c );

[0121] The conversion relationship between the camera coordinate system and the image coordinate system is represented as:

[0122]

[0123] The transformation relationship between the image coordinate system and the pixel coordinate system is expressed as:

[0124]

[0125] Where (x, y) are the coordinates in the image coordinate system, (X, Y) are the coordinates in the image coordinate system. c ,Y c Z c (u,v) represents the coordinates in the camera coordinate system, (u,v) represents the pixel coordinates in the pixel coordinate system; f represents the image distance of the camera; dx and dy represent the physical lengths of a pixel on the camera's image sensor in the x and y directions, respectively; and u0 and v0 represent the coordinates of the center of the camera's image sensor in the pixel coordinate system, respectively.

[0126] Pixel coordinates Given the distorted pixel coordinates, the distortion coefficients k are used to represent the sound source point p. c pixel coordinates Perform corrections to obtain the corrected pixel coordinates (u c ,v c ), which is the ideal, distortion-free pixel coordinate; then the sound source point p is obtained using the intrinsic parameter matrix. c The corresponding midpoint P in the camera coordinate system c coordinates (X) c ,Y c Z c When the camera is a monocular camera, Z c =1, then vector O c P c and axis O c The angle between X and X represents the corresponding sound source point p. c The azimuth angle relative to the camera.

[0127] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0128] Example 3

[0129] The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0130] Further, the processing device further comprises at least one input device and at least one output device; in the processing device, the processor, the memory, the input device and the output device are connected through a bus.

[0131] In the embodiment of the present application, the specific types of the memory, the input device and the output device are not limited; for example:

[0132] The input device can be a touch screen, an image acquisition device, a physical button or a mouse, etc.

[0133] The output device can be a display terminal.

[0134] The memory can be a random access memory (RAM), or a non-volatile memory, such as a disk memory.

[0135] Embodiment four

[0136] The present application also provides a readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method provided in the foregoing embodiments is realized.

[0137] In the embodiment of the present application, the readable storage medium as the computer readable storage medium can be arranged in the foregoing processing device, for example, as the memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc. various media which can store program codes.

[0138] The above is only the preferred specific implementation of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A video-based sound source localization angle calibration method, characterized in that, The method comprises: Step S1, detecting a target person in each video frame of a video captured by a camera through a face detection algorithm, taking the pixel coordinates of the center point of the obtained detection frame as a sound source point, and obtaining the sound source points corresponding to the target person in all video frames through target tracking; Step S2, calibrating the camera to obtain an intrinsic matrix and distortion coefficients; Step S3, restoring each sound source point based on the intrinsic matrix and distortion coefficients to obtain the azimuth angle of each sound source point relative to the camera, and the step comprises: Let XYZ, xy and uv represent the camera coordinate system, the image coordinate system and the pixel coordinate system, respectively. Let O c , O and O' represent the origins of the camera coordinate system, the image coordinate system and the pixel coordinate system, respectively. Definition of the pixel coordinate of the sound source point p corresponding to the cth target person in each video frame c c = 1 … N, N represents the number of target persons, the coordinates of the corresponding point P in the camera coordinate system are represented as (X c , Y c , Z c ) c c )​​ The conversion relationship between the camera coordinate system and the image coordinate system is represented as: The conversion relationship between the image coordinate system and the pixel coordinate system is represented as: where (x, y) is the coordinate in the image coordinate system, (X c ,Y c ,Z c ) is the coordinate in the camera coordinate system, (u, v) is the pixel coordinate in the pixel coordinate system; f is the image distance of the camera, dx and dy represent the physical length of one pixel in the x and y directions on the camera photosensitive plate respectively, and u0 and v0 represent the coordinates of the center of the camera photosensitive plate in the pixel coordinate system respectively. Pixel coordinates Given the distorted pixel coordinates, the distortion coefficients k are used to represent the sound source point p. c pixel coordinates Perform corrections to obtain the corrected pixel coordinates (u c ,v c The coordinates of the sound source point p are ideal and distortion-free pixels. Then, using the intrinsic parameter matrix, and combining the transformation relationships between the image coordinate system and the pixel coordinate system, as well as the transformation relationships between the camera coordinate system and the image coordinate system, the sound source point p is obtained. c The corresponding midpoint P in the camera coordinate system c coordinates (X) c ,Y c Z c When the camera is a monocular camera, Z c =1, then vector O c P c and axis O c The angle between X and X represents the corresponding sound source point p(u). c ,v c (Azimuth angle relative to the camera) 2. The video-based sound source localization angle calibration method according to claim 1, wherein, In the step S1, all target persons in each video frame are detected, and a single detection frame obtained is a rectangular frame containing a head region and a shoulder region of a single target person. For a single target person, tracking of the target person is realized by matching detection frames in adjacent video frames. For each video frame, the number of target persons is recorded as N, and the pixel coordinates of a sound source point p c corresponding to a cth target person are 3. The video-based sound source localization angle calibration method according to claim 1, wherein, The step of calibrating the camera to obtain the intrinsic matrix and distortion coefficients comprises: Step S21, capturing a set of checkerboard images by using the camera to capture the checkerboard calibration board from different angles; Step S22, select n corner points from each checkerboard image, and obtain the pixel coordinates (u i ,v i ) of each corner point from the image, i = 1...n; Step S23, according to the defined world coordinate system of the chessboard calibration board, the physical coordinates of each focal point in the world coordinate system (U i ,V i ,W i = 0), i = 1...n; Step S24, using the pixel coordinates (u i ,v i ), i = 1...n, and the corresponding physical coordinates in the world coordinate system (U i ,V i ,W i = 0), i = 1...n, of each corner point, calibrate the camera intrinsic matrix: Define UVW, XYZ, xy, and uv to represent the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system, respectively, and the conversion relationship from the world coordinate system to the pixel coordinate system is represented as: wherein the matrix is called the intrinsic matrix of the camera; (U, V, W) is the physical coordinate of a point in the world coordinate system, (u, v) is the pixel coordinate in the pixel coordinate system; f is the image distance of the camera, dx, dy respectively represent the physical length of a pixel in the x, y direction on the photosensitive plate of the camera; u0, v0 respectively represent the coordinates of the center of the photosensitive plate of the camera in the pixel coordinate system, and θ represents the angle between the horizontal side and the vertical side of the photosensitive plate of the camera; the matrix is called the extrinsic matrix of the camera, and the extrinsic matrix depends on the relative position of the camera coordinate system and the world coordinate system, R represents a rotation matrix, T represents a translation vector, the vertical axis position Z in the camera coordinate system XYZ is a scale factor; Let θ = 90°, using the pixel coordinates (u i ,v i ), i = 1...n, of each corner point in all checkerboard images and the corresponding physical coordinates (U i ,V i ,W i = 0), i = 1...n, in the world coordinate system, the intrinsic matrix λ of the camera is solved by matrix operation: Step S25, using the pixel coordinates (u i ,v i ), i = 1...n, of each corner point in all checkerboard images and the corresponding physical coordinates (U i ,V i ,W i = 0), i = 1...n, in the world coordinate system, calibrate the camera distortion coefficients: Define the second-order radial distortion formula as: where (x, y) represents the coordinates in the image coordinate system xy, which is the ideal non-distorted normalized image coordinates, represents the distorted normalized image coordinates; r is the distance from the image pixel point to the image center point, i.e. r 2 = x 2 +y 2 ; the coefficient k1 and the coefficient k2 are the camera distortion coefficients k: The conversion relationship between the image coordinate system and the pixel coordinate system is: where (u i ,v i ) is the pixel coordinate of each corner point, which is the ideal non-distorted pixel coordinate. Let θ = 90°, then we have: Among them, (x i ,y i ) represents the pixel coordinates of the corner point (u i ,v i The corresponding coordinates in the xy coordinate system; Accordingly, the distorted pixel coordinates are: wherein is the distorted pixel coordinate corresponding coordinate in the image coordinate system xy; Substitute the second-order radial distortion formula to obtain the camera distortion coefficient k.

4. A video-based sound source localization angle calibration system, characterized in that, The method comprises: A target detection tracking and sound source point determination unit is configured to perform step S1, and step S1 comprises detecting a target person in each video frame of a video captured by a camera through a face detection algorithm, taking the pixel coordinates of the center point of the obtained detection frame as a sound source point, and obtaining the sound source points corresponding to the target person in all video frames through target tracking; A camera calibration unit is configured to perform step S2, and step S2 comprises calibrating the camera to obtain an intrinsic matrix and distortion coefficients; A sound source positioning angle calibration unit is configured to perform step S3, and step S3 comprises restoring each sound source point based on the intrinsic matrix and distortion coefficients to obtain the azimuth angle of each sound source point relative to the camera, and the step comprises: Let XYZ, xy and uv represent the camera coordinate system, the image coordinate system and the pixel coordinate system, respectively. Let O c , O and O' represent the origins of the camera coordinate system, the image coordinate system and the pixel coordinate system, respectively. Definition of the pixel coordinate of the sound source point p corresponding to the cth target person in each video frame c N represents the number of target persons, and the coordinates of the corresponding point P of the sound source point p c in the camera coordinate system are represented as (X c , Y c , Z c ) c )​ The conversion relationship between the camera coordinate system and the image coordinate system is represented as: The conversion relationship between the image coordinate system and the pixel coordinate system is represented as: where (x, y) is the coordinate in the image coordinate system, (X c ,Y c ,Z c ) is the coordinate in the camera coordinate system, (u, v) is the pixel coordinate in the pixel coordinate system; f is the image distance of the camera, dx and dy represent the physical length of one pixel in the x and y directions on the camera photosensitive plate respectively, and u0 and v0 represent the coordinates of the center of the camera photosensitive plate in the pixel coordinate system respectively. Pixel coordinates The pixel coordinates of the sound source point p c are corrected by using the distortion coefficient k to obtain the corrected pixel coordinates (u c ,v c ), which are ideal non-distorted pixel coordinates; then, by using the intrinsic matrix, the conversion relationship between the image coordinate system and the pixel coordinate system, and the conversion relationship between the camera coordinate system and the image coordinate system, the coordinates (X c ,Y c ,Z c ) of the point P c in the camera coordinate system corresponding to the sound source point p c are obtained; when the camera is a monocular camera, Z c = 1, and the angle between the vector O c P c and the axis O c X represents the azimuth angle of the corresponding sound source point p(u c ,v c ) relative to the camera.

5. A video-based sound source localization angle calibration system according to claim 4, wherein, In the step S1, all target persons in each video frame are detected, and a single detection frame obtained is a rectangular frame containing a head region and a shoulder region of a single target person. For a single target person, tracking of the target person is realized by matching detection frames in adjacent video frames. For each video frame, the number of target persons is recorded as N, and the pixel coordinates of a sound source point p c corresponding to a cth target person are 6. A video-based sound source localization angle calibration system according to claim 4, wherein, The step of calibrating the camera to obtain the intrinsic matrix and distortion coefficients comprises: Step S21, capturing a set of checkerboard images by using the camera to capture the checkerboard calibration board from different angles; Step S22, select n corner points from each checkerboard image, and obtain the pixel coordinates (u i ,v i ) of each corner point from the image, i = 1...n; Step S23, according to the defined world coordinate system of the chessboard calibration board, the physical coordinates of each focal point in the world coordinate system (U i ,V i ,W i = 0), i = 1...n; Step S24, using the pixel coordinates (u i ,v i ), i = 1...n, and the corresponding physical coordinates in the world coordinate system (U i ,V i ,W i = 0), i = 1...n, of each corner point, calibrate the camera intrinsic matrix: Define UVW, XYZ, xy, and uv to represent the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system, respectively, and the conversion relationship from the world coordinate system to the pixel coordinate system is represented as: wherein the matrix is called the intrinsic matrix of the camera; (U, V, W) is the physical coordinate of a point in the world coordinate system, (u, v) is the pixel coordinate in the pixel coordinate system; f is the image distance of the camera, dx, dy respectively represent the physical length of a pixel in the x, y direction on the photosensitive plate of the camera; u0, v0 respectively represent the coordinates of the center of the photosensitive plate of the camera in the pixel coordinate system, and θ represents the angle between the horizontal side and the vertical side of the photosensitive plate of the camera; the matrix is called the extrinsic matrix of the camera, and the extrinsic matrix depends on the relative position of the camera coordinate system and the world coordinate system, R represents a rotation matrix, T represents a translation vector, the vertical axis position Z in the camera coordinate system XYZ is a scale factor; Let θ = 90°, using the pixel coordinates (u i ,v i ), i = 1...n, of each corner point in all checkerboard images and the corresponding physical coordinates (U i ,V i ,W i = 0), i = 1...n, in the world coordinate system, the intrinsic matrix λ of the camera is solved by matrix operation: Step S25, using the pixel coordinates (u i ,v i ), i = 1...n, of each corner point in all checkerboard images and the corresponding physical coordinates (U i ,V i ,W i = 0), i = 1...n, in the world coordinate system, calibrate the camera distortion coefficients: Define the second-order radial distortion formula as: where (x, y) represents the coordinates in the image coordinate system xy, which is the ideal non-distorted normalized image coordinates, represents the distorted normalized image coordinates; r is the distance from the image pixel point to the image center point, i.e. r 2 = x 2 +y 2 ; the coefficient k1 and the coefficient k2 are the camera distortion coefficients k: The conversion relationship between the image coordinate system and the pixel coordinate system is: where (u i ,v i ) is the pixel coordinate of each corner point, which is the ideal non-distorted pixel coordinate. Let θ = 90°, then we have: where (x i ,y i ) is the coordinate of the pixel corresponding to the pixel coordinate (u i ,v i ) in the image coordinate system xy. Accordingly, the distorted pixel coordinates are: wherein is the distorted pixel coordinate corresponding coordinate in the image coordinate system xy; Substitute the second-order radial distortion formula to obtain the camera distortion coefficient k.

7. A processing device, characterized by The method comprises: One or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-3.

8. A readable storage medium, storing a computer program, characterized in that, When the computer program is executed by the processor, the method of any one of claims 1-3 is implemented.

Images