A method, apparatus, and storage medium for calibrating sound and image in a video image.

Abstract

This application discloses a method, apparatus, and storage medium for calibrating sound and image in video images, which simplifies the calibration process and improves calibration efficiency and accuracy. The method includes: acquiring a real-time preview image and establishing a coordinate system for the real-time preview image; acquiring sound source angle information and prompting the user to move to a nearby target angle based on the sound source angle information, the target angle being determined by the maximum range of the sound source angle and a preset precision; when the error between the sound source angle information and the target angle is less than a preset error, picking at least two calibration points in the real-time preview image at positions corresponding to the sound source angle information; determining the coordinate data of the at least two calibration points in the coordinate system; determining the angle line equation corresponding to the target angle based on the coordinate data; establishing a mapping relationship between the target angle and the angle line equation, and storing the mapping relationship in a calibration table.

Description

Technical Field

[0001] This application relates to the field of video image processing, and in particular to a method, apparatus and storage medium for calibrating sound and image in a video image. Background Technology

[0002] As the world enters the information age, video conferencing is becoming increasingly popular. Some high-end intelligent conferencing devices, such as intelligent conferencing terminals or intelligent conferencing systems, have speaker close-up functions, resulting in better meeting effects. Regarding speaker close-up functions, considering cost-effectiveness and practicality, existing technologies generally use linear array microphones and human image detection to locate the speaker's position, thereby providing a close-up image of the speaker during video conferencing.

[0003] Mapping sound and image in video is essential for speaker close-ups. Using deep learning to achieve this mapping involves long development cycles, high costs, and requires support from the platform's NPU hardware modules. Other methods involve pre-calibration to achieve sound-image mapping. Existing calibration methods combine camera intrinsic and extrinsic parameters and use mathematical modeling to correlate image coordinates with sound source angles. However, this method requires camera calibration to obtain intrinsic and extrinsic parameters, which is technically demanding, and the modeling and calculation processes are cumbersome, making it unsuitable for widespread application. Summary of the Invention

[0004] This application provides a method, apparatus, and storage medium for calibrating sound and image in video images, which simplifies the calibration process and improves calibration efficiency and accuracy.

[0005] The first aspect of this application provides a method for calibrating sound and image in a video image, including:

[0006] Acquire the real-time preview screen and establish the screen coordinate system of the real-time preview screen;

[0007] Acquire sound source angle information and prompt the user to move to a nearby target angle based on the sound source angle information. The target angle is obtained by dividing the maximum range of sound source angles and a preset precision.

[0008] When the error between the sound source angle information and the target angle is less than a preset error, at least two calibration points are picked up in the real-time preview screen at the position corresponding to the sound source angle information.

[0009] Determine the coordinate data of the at least two calibration points in the screen coordinate system;

[0010] Determine the angle line equation corresponding to the target angle based on the coordinate data;

[0011] Establish a mapping relationship between the target angle and the angle line equation, and store the mapping relationship in a calibration table.

[0012] Optionally, picking at least two calibration points at positions corresponding to the sound source angle information in the real-time preview screen includes:

[0013] Target detection of the mouth position is performed in the real-time preview screen to obtain the first detection result;

[0014] Based on the first detection result, determine the location of the calibration point and pick up at least two calibration points.

[0015] Optionally, if multiple first detection results exist, the step of determining the calibration point location based on the first detection results and picking the calibration point includes:

[0016] Based on the sound source angle information, the nearest target detection result is selected from the plurality of first detection results;

[0017] or,

[0018] Calculate the degree of mouth opening of the multiple first detection results, and determine the first detection result with the largest degree of mouth opening as the target detection result;

[0019] Based on the target detection results, determine the location of the calibration points and pick at least two calibration points.

[0020] Optionally, the step of performing target detection at the mouth position in the real-time preview screen to obtain a first detection result includes:

[0021] In the real-time preview, the mouth area is detected using a face region detection algorithm to obtain a mouth detection box;

[0022] Determining the calibration point location based on the first detection result and picking at least two calibration points includes:

[0023] The center point of the mouth detection frame is determined as the calibration point location, and at least two calibration points are picked up.

[0024] Optionally, the step of performing target detection at the mouth position in the real-time preview screen to obtain a first detection result includes:

[0025] In the real-time preview, the mouth position is detected by a facial key point detection algorithm to obtain a set of mouth key points;

[0026] Determining the calibration point location based on the first detection result and picking at least two calibration points includes:

[0027] The midpoint between the upper lip key point and the lower lip key point in the set of key points of the mouth is determined as the calibration point position, and at least two calibration points are picked.

[0028] Optionally, picking at least two calibration points at positions corresponding to the sound source angle information in the real-time preview screen includes:

[0029] Pick at least two calibration points on the reference line in the real-time preview screen. One end of the reference line is located directly above the camera, and the other end of the reference line is located at the speaker's mouth. The reference line is a straight line projected by a physical device.

[0030] Optionally, after acquiring the sound source angle information, the method further includes:

[0031] The sound source angle information and the target angle are displayed on the real-time preview screen.

[0032] Optionally, the Y-axis of the screen coordinate system coincides with the vertical symmetrical central axis of the real-time preview screen, and the X-axis of the screen coordinate system coincides with the top horizontal boundary of the real-time preview screen.

[0033] or,

[0034] The origin of the screen coordinate system is the center point of the real-time preview screen, the Y-axis is the vertical direction passing through the center point, and the X-axis is the horizontal direction passing through the center point.

[0035] A second aspect of this application provides a method for mapping sound and image in a video image, comprising:

[0036] Obtain a calibration table as described in the first aspect and any one of the first aspects, wherein the calibration table pre-stores a mapping relationship between several target angles and angle line equations;

[0037] Acquire the angle of the target sound source;

[0038] The sound source angle range is determined based on the target sound source angle and the preset angle tolerance;

[0039] The target angle line equation is determined based on the sound source angle range and the calibration table.

[0040] The pixel area of ​​the sound source angle range in the video frame is determined according to the target angle line equation.

[0041] or,

[0042] Determine whether the target position is within the range of the sound source angle based on the target angle line equation.

[0043] A third aspect of this application provides a method for mapping sound and image in a video image, comprising:

[0044] Obtain a calibration table as described in the first aspect and any one of the first aspects, wherein the calibration table pre-stores a mapping relationship between several target angles and angle line equations;

[0045] Obtain target detection information and determine the target location based on the target detection information;

[0046] Determine the equations of the first and second angle lines adjacent to the target position in the calibration table;

[0047] The angular range or angle of the target position in the video frame is determined based on the first angle line equation and the second angle line equation.

[0048] A fourth aspect of this application provides a calibration device for sound and image in a video image, comprising:

[0049] The acquisition unit is used to acquire the real-time preview screen and establish the screen coordinate system of the real-time preview screen;

[0050] The prompting unit is used to acquire sound source angle information and prompt the user to move to a nearby target angle based on the sound source angle information. The target angle is obtained by dividing the maximum range of sound source angles and a preset precision.

[0051] The picking unit is used to pick up at least two calibration points in the real-time preview screen at the positions corresponding to the sound source angle information when the error between the sound source angle information and the target angle is less than a preset error.

[0052] The first determining unit is used to determine the coordinate data of the at least two calibration points in the screen coordinate system;

[0053] The second determining unit is used to determine the angle line equation corresponding to the target angle based on the coordinate data;

[0054] The calibration unit is used to establish the mapping relationship between the target angle and the angle line equation, and to store the mapping relationship in the calibration table.

[0055] Optionally, the picking unit is specifically used for:

[0056] When the error between the sound source angle information and the target angle is less than a preset error, the target detection of the mouth position is performed in the real-time preview screen to obtain the first detection result;

[0057] Based on the first detection result, determine the location of the calibration point and pick up at least two calibration points.

[0058] Optionally, if there are multiple first detection results, the picking unit is further configured to:

[0059] Based on the sound source angle information, the nearest target detection result is selected from the plurality of first detection results;

[0060] or,

[0061] Calculate the degree of mouth opening of the multiple first detection results, and determine the first detection result with the largest degree of mouth opening as the target detection result;

[0062] Based on the target detection results, determine the location of the calibration points and pick at least two calibration points.

[0063] Optionally, the picking unit is further configured to:

[0064] In the real-time preview, the mouth area is detected using a face region detection algorithm to obtain a mouth detection box;

[0065] The center point of the mouth detection frame is determined as the calibration point location, and at least two calibration points are picked up.

[0066] Optionally, the picking unit is further configured to:

[0067] In the real-time preview, the mouth position is detected by a facial key point detection algorithm to obtain a set of mouth key points;

[0068] The midpoint between the upper lip key point and the lower lip key point in the set of key points of the mouth is determined as the calibration point position, and at least two calibration points are picked.

[0069] Optionally, the picking unit is further configured to:

[0070] Pick at least two calibration points on the reference line in the real-time preview screen. One end of the reference line is located directly above the camera, and the other end of the reference line is located at the speaker's mouth. The reference line is a straight line projected by a physical device.

[0071] Optionally, the calibration device further includes:

[0072] The display unit is used to display the sound source angle information and the target angle on the real-time preview screen.

[0073] Optionally, the Y-axis of the screen coordinate system coincides with the vertical symmetrical central axis of the real-time preview screen, and the X-axis of the screen coordinate system coincides with the top horizontal boundary of the real-time preview screen.

[0074] or,

[0075] The origin of the screen coordinate system is the center point of the real-time preview screen, the Y-axis is the vertical direction passing through the center point, and the X-axis is the horizontal direction passing through the center point.

[0076] The fifth aspect of this application provides a calibration device for sound and image in a video image, the device comprising:

[0077] Processor, memory, input / output units, and bus;

[0078] The processor is connected to the memory, the input / output unit, and the bus;

[0079] The memory stores a program, which the processor calls to execute the first aspect and any optional method of the first aspect for calibrating sound and image in a video image.

[0080] The sixth aspect of this application provides a computer-readable storage medium storing a program that, when executed on a computer, performs the first aspect and any optional method of the first aspect for calibrating sound and image in a video image.

[0081] As can be seen from the above technical solutions, this application has the following advantages:

[0082] By dividing the maximum range of sound source angles supported by the conference equipment into several angle lines (target angles), the user speaks normally in the preview screen, and the sound source localization module on the conference equipment obtains the sound source angle information. The user moves left / right to find the target angle to be calibrated. When the position is determined, calibration points are picked up. For each target angle, at least two calibration points are picked up in the real-time preview screen using the above method. Based on the coordinates of the calibration points, the angle line equation corresponding to each target angle can be calculated, thereby establishing a calibration table.

[0083] The above calibration process is simple to operate and has a short process, which can shorten the product development cycle and save development costs. The calibration process can be operated without professional knowledge, and users can perform the calibration themselves during use. Furthermore, the calibration method is based on the sound source localization module installed on the device, so it has good adaptability to the sound source localization error of different devices, high calibration accuracy, and can meet the requirements of high precision. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 A schematic flowchart of an embodiment of the method for calibrating sound and image in video images provided in this application;

[0086] Figure 2 A schematic diagram of the image coordinate system in the method for calibrating sound and image in the video image provided in this application;

[0087] Figure 3 A schematic diagram of calibration points for each target angle in the calibration method for sound and image in the video image provided in this application;

[0088] Figure 4 A schematic diagram of angle lines for each target angle in the calibration method for sound and image in the video image provided in this application;

[0089] Figure 5 A schematic flowchart of another embodiment of the method for calibrating sound and image in video images provided in this application;

[0090] Figure 6 A schematic flowchart of another embodiment of the method for calibrating sound and image in video images provided in this application;

[0091] Figure 7 A schematic diagram of an embodiment of the calibration device for sound and image in the video image provided in this application;

[0092] Figure 8 A schematic diagram of the physical structure of an embodiment of the calibration device for sound and image in the video image provided in this application. Detailed Implementation

[0093] This application provides a method, apparatus, and storage medium for calibrating sound and image in video images, which simplifies the calibration process and improves calibration efficiency and accuracy.

[0094] It should be noted that the method for integrating sound and image in video images provided in this application can be applied to terminals as well as servers. For example, the terminal can be a video conferencing device, a conference computer, a conference tablet, a smartphone, computer, tablet computer, smart TV, smartwatch, portable computer terminal, or a fixed terminal such as a desktop computer. For ease of explanation, this application uses a conference device as the example subject.

[0095] Please see Figure 1 , Figure 1 An embodiment of the method for calibrating sound and image in video images provided in this application includes:

[0096] 101. Obtain the real-time preview screen and establish the screen coordinate system for the real-time preview screen;

[0097] Sound-to-image mapping, or audio-visual mapping, refers to mapping the video image captured by a camera to the audio information captured by a microphone. Specific applications include determining the target location of a speaker in a video image based on the sound source angle information obtained through sound source localization technology, or determining the corresponding sound source angle based on the target location obtained through image recognition or target detection technology. To achieve sound-to-image mapping, it is first necessary to establish a mapping relationship between the sound source angle and the video image, i.e., to calibrate the sound source angle and the video image.

[0098] Before calibration, the user turns on the camera of the conference device to be calibrated for real-time preview, and the conference device acquires the real-time preview image. In some specific embodiments, it is preferred to turn on a resolution of 1920*1080 or higher, as a resolution that is too low or the image is unclear will affect the accuracy of the calibration.

[0099] After acquiring the live preview image, the conferencing equipment also needs to establish a coordinate system for that preview image. This coordinate system can be established with the Y-axis coinciding with the vertical axis of the video image and the X-axis coinciding with the top horizontal boundary of the video image. For details, please refer to [link to relevant documentation]. Figure 2 The preferred method for establishing the screen coordinate system is this. Alternatively, the screen coordinate system can be established with the center point or other point of the video image as the origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis. It should be noted that the calibration method provided in this application can be used to calibrate different screen coordinate systems; therefore, the specific method for establishing the screen coordinate system is not limited.

[0100] 102. Obtain sound source angle information and prompt the user to move to a similar target angle based on the sound source angle information. The target angle is obtained by dividing the maximum range of sound source angles and the preset precision.

[0101] Conference equipment divides the sound source angle into several equal parts based on the maximum range, accuracy, and application requirements of the sound source angle, resulting in several target angles. Different equipment has different maximum range, accuracy, and application requirements for the sound source angle, therefore the resulting target angles will vary. Specific details are not limited here. For example, if a conference equipment's application requirement is a maximum sound source angle range of [50°, 130°] and a preset accuracy of ±10°, then to meet these requirements, the angle can be divided into 16 equal parts with an angle interval of 5°, resulting in target angles of [50°, 55°, ..., 130°].

[0102] Users speak to create a sound source, and the conference equipment uses its built-in sound source localization module to obtain the sound source angle information. Based on the acquired sound source angle information and the target angle to be calibrated, the equipment prompts the user to move left or right so that the user stands at the target angle. It should be noted that, to ensure the efficiency and accuracy of calibration, environmental noise interference should be avoided as much as possible during calibration; a relatively quiet conference room or open space should be selected.

[0103] Furthermore, in some specific embodiments, in order to facilitate users in finding the target angle, the acquired sound source angle information and the target angle (specifically, two target angles adjacent to the sound source angle information) can be displayed on the real-time preview screen.

[0104] 103. When the error between the sound source angle information and the target angle is less than the preset error, pick at least two calibration points in the real-time preview screen at the positions corresponding to the sound source angle information.

[0105] When the error between the sound source angle information and the target angle acquired by the conferencing equipment is less than the preset error (i.e., the acquired sound source angle information and the target angle are equal or close), a calibration point is picked up in the preview screen. The position of the calibration point corresponds to the position of the sound source angle information, that is, the position of the calibration point is the user's current position. It should be noted that when picking up calibration points for different target angles, they should be picked up at the same position on the user's face, preferably at the user's mouth. The smaller the difference between the sound source angle information and the target angle, the more accurate the calibration data. In practical applications, this preset error can be flexibly adjusted according to different calibration accuracy requirements.

[0106] In practical applications, to ensure calibration accuracy, users should locate the target angle both near and far from the camera and pick up calibration points. There should be a preset distance between the two picked calibration points; this preset distance needs to be determined based on the usage scenario of different conference equipment. Furthermore, the picked calibration points must be within the preview screen; the user must not leave the preview screen while searching for the target angle. The two picked calibration points are called the angle line point pair of the target angle. For details, please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of the angle line points for each target angle.

[0107] In some specific embodiments, when it is determined that the error between the sound source angle information and the target angle is less than a preset error, it can be further determined whether the sound source angle information remains stable within a certain time (2s-5s). Only when it is determined that the error between the sound source angle information and the target angle is less than the preset error and remains stable, the calibration point is picked up.

[0108] It should be noted that the calibration point can be picked up manually by the user or automatically by the conference equipment; no specific limitation is made here.

[0109] 104. Determine the coordinate data of at least two calibration points in the screen coordinate system;

[0110] The conference equipment determines the coordinate data of the two calibration points picked up in step 103 in the screen coordinate system, such as... Figure 3 As shown, the conference equipment needs to be determined. Figure 3 For each target angle to be calibrated, the coordinate data of each calibration point should be associated with at least two calibration points.

[0111] 105. Determine the equation of the angle line corresponding to the target angle based on the coordinate data;

[0112] Choose a line equation. For any target angle, substitute the coordinate data of the two calibration points obtained in step 104 into the equation to solve the system of two linear equations in two variables, and obtain the angle line equation corresponding to the target angle. Regarding the selection of the line equation: There are multiple line equations. The slope-intercept form is preferred: y = kx + b. The general form: Ax + By + C = 0 can also be used (A and B are not both 0).

[0113] like Figure 4 As shown, Figure 4 The diagram shows the angle lines for each target angle. By following steps 102 to 105 above, the angle line equations for all target angles to be calibrated can be obtained.

[0114] 106. Establish the mapping relationship between the target angle and the angle line equation, and store the mapping relationship in the calibration table.

[0115] Repeat the above steps, picking at least two calibration points for each target angle, then calculating the angle line equations corresponding to different target angles. Establish the mapping relationship between all target angles and their corresponding angle line equations, and store the target angles, angle line equations, and their mapping relationships in a calibration table to complete the calibration of the conference equipment's sound and video. After the calibration table is established, the data in the calibration table can be directly used to map the sound source angle to the video image or vice versa.

[0116] It should be noted that, based on the principle of symmetry, in the entire calibration process described above, if the longitudinal axis of the linear array microphone structure of the conference equipment coincides with the longitudinal axis of the camera, then the 90° angle line will coincide with the Y-axis of the coordinate system established by the preferred calibration tool. Theoretically, data calibration can be performed symmetrically, that is, only the data on the side less than or equal to 90° or the data on the side greater than or equal to 90° needs to be calibrated, and then symmetrical processing can be performed to complete the calibration. However, considering that the installation specifications of different conference equipment structures may be inconsistent, the corresponding data calibration is not performed symmetrically. In the specific calibration process, in order to improve calibration efficiency, it is preferable to calibrate the target angles sequentially from left to right or from right to left. After the two calibration points of the previous target angle have been picked up, the calibration points of the next target angle are picked up.

[0117] In this embodiment, the maximum range of sound source angles supported by the conference equipment is divided into several angle lines (target angles). The user speaks normally in the preview screen, and the sound source localization module on the conference equipment obtains the sound source angle information. The user moves left / right to find the target angle to be calibrated. When the position is determined, calibration points are picked up. For each target angle, at least two calibration points are picked up in the real-time preview screen using the above method. The angle line equation corresponding to each target angle can be calculated based on the coordinates of the calibration points, thereby establishing a calibration table.

[0118] The above calibration process is simple to operate and has a short process, which can shorten the product development cycle and save development costs. The calibration process can be operated without professional knowledge, and users can perform the calibration themselves during use. Furthermore, the calibration method is based on the sound source localization module installed on the device, so it has good adaptability to the sound source localization error of different devices, high calibration accuracy, and can meet the requirements of high precision.

[0119] In this application, two different picking methods are provided for picking calibration points, which are described below.

[0120] 1. The conference equipment automatically picks up calibration points through target detection:

[0121] Please see Figure 5 , Figure 5 Another embodiment of the method for calibrating sound and image in video images provided in this application includes:

[0122] 501. Obtain the real-time preview screen and establish the screen coordinate system for the real-time preview screen;

[0123] 502. Obtain sound source angle information and prompt the user to move to a similar target angle based on the sound source angle information. The target angle is obtained by dividing the maximum range of sound source angles and the preset precision.

[0124] In this embodiment, steps 501 to 502 are similar to steps 101 to 102 in the previous embodiment, and will not be described again here.

[0125] 503. When the error between the sound source angle information and the target angle is less than the preset error, target detection of the mouth position is performed in the real-time preview screen to obtain the first detection result;

[0126] When a user speaks normally and moves left or right in the live preview, and the conferencing equipment determines that the obtained sound source angle information and the target angle are equal or similar and remain stable, the conferencing equipment uses target detection technology to detect the position of the user's mouth in the live preview and obtains the first detection result.

[0127] Furthermore, if multiple people are present in the real-time preview, the conferencing equipment will simultaneously detect multiple mouth positions, meaning there are multiple first detection results in the frame at the same time. At this point, the speaker's mouth position can be selected based on the acquired sound source angle information, with the first detection result closest to the sound source angle being identified as the target detection result. In addition, the conferencing equipment can also calculate the degree of mouth opening based on the detected multiple mouth positions (first detection results) to further select the speaker's mouth position; specifically, the first detection result with the largest mouth opening can be identified as the target detection result.

[0128] It should be noted that the object detection can be performed using face region detection algorithms, such as MTCNN multi-task convolutional neural network, or using facial landmark detection algorithms, without being limited here.

[0129] 504. Determine the location of the calibration points based on the first test result, and pick at least two calibration points;

[0130] The conference equipment can determine the calibration point location based on the first detection result. That is, the conference equipment automatically identifies the speaker's mouth position and uses that position as the calibration point location to pick up the calibration point. It should be noted that the user needs to find the target angle once at near and once at far from the camera. That is, for each target angle, at least two first detection results need to be obtained before and after detection, and at least two calibration points need to be picked up.

[0131] In some specific embodiments, if face region detection is used to detect the mouth position in step 503, the first detection result is a mouth detection box. In this case, the calibration point position is directly determined based on the center point of the mouth detection box, and the calibration point is automatically picked. If face keypoint detection is used to detect the mouth position in step 503, the first detection result should be a set of mouth keypoints. In this case, the calibration point position is directly determined based on the vertical midpoint between the upper lip keypoint and the lower lip keypoint in the set of mouth keypoints, and the calibration point is automatically picked.

[0132] 505. Determine the coordinate data of at least two calibration points in the screen coordinate system;

[0133] 506. Determine the equation of the angle line corresponding to the target angle based on the coordinate data;

[0134] 507. Establish the mapping relationship between the target angle and the angle line equation, and store the mapping relationship in the calibration table.

[0135] In this embodiment, steps 505 to 507 are similar to steps 104 to 106 in the previous embodiment, and will not be described again here.

[0136] In this embodiment, during the calibration process, the user only needs to continuously stand at the target angle to create sound source conditions. The conference equipment can automatically pick up the calibration point according to the speaker's mouth position, thereby automatically completing the calibration of sound and image. The calibration process is simple and fast, and there is no operational difficulty, making it suitable for widespread application.

[0137] 2. Use physical devices to simulate angle lines and pick up calibration points;

[0138] 601. Obtain the real-time preview screen and establish the screen coordinate system for the real-time preview screen;

[0139] 602. Obtain sound source angle information and prompt the user to move to a similar target angle based on the sound source angle information. The target angle is obtained by dividing the maximum range of sound source angles and the preset precision.

[0140] In this embodiment, steps 601 to 602 are similar to steps 101 to 102 in the previous embodiment, and will not be described again here.

[0141] 603. When the error between the sound source angle information and the target angle is less than the preset error, pick at least two calibration points on the reference line in the real-time preview screen. One end of the reference line is located directly above the camera, and the other end of the reference line is located at the speaker's mouth. The reference line is a straight line projected by the physical device.

[0142] The calibration process is essentially solving the equations of the angle lines for each target angle. The calibration point is a point on that angle line. For each target angle, the angle line can be simulated using a physical device. The simulated angle line serves as the reference line in this application. This reference line is also captured by the camera, allowing calibration points to be picked up from the reference line in the real-time preview. Specifically, one end of the reference line should be directly above the camera of the conferencing equipment, and the other end should be positioned at the speaker's mouth.

[0143] In some specific embodiments, the reference line can be a solid thin string or other flexible, straight line-like object. Taking a thin string as an example, one end of the string is fixed directly above the camera, and the other end is held by the speaker and pulled to the mouth position. During the operation, the string needs to be taut; the string serves as the angle line for the simulated target angle. The user directly creates a sound source condition at a distance from the camera to find the target angle and pulls the string taut to the mouth position. When the sound source angle at the current position is equal to or close to the target angle and remains stable, the conferencing equipment picks up two calibration points on the thin string in the real-time preview screen.

[0144] In other specific embodiments, the reference line can also be a laser emitted by a laser device (such as a red laser pointer used in education, which is inexpensive and readily available). Specifically, the user creates a sound source condition at a distance from the camera to find the target angle, holds the laser device near the mouth, and emits a beam of light towards a fixed point on the longitudinal axis of the camera (this fixed point is also used for other target angles). This beam of light serves as the reference line, and correspondingly, two calibration points are picked up on this reference line in the real-time preview image. Furthermore, the aforementioned laser device can also be integrated into a conference device, which identifies the speaker's mouth position and then controls the laser device to emit a laser towards the speaker's mouth position to create the reference line.

[0145] 604. Determine the coordinate data of at least two calibration points in the screen coordinate system;

[0146] 605. Determine the equation of the angle line corresponding to the target angle based on the coordinate data;

[0147] 606. Establish the mapping relationship between the target angle and the angle line equation, and store the mapping relationship in the calibration table.

[0148] In this embodiment, steps 604 to 606 are similar to steps 104 to 106 in the previous embodiment, and will not be described again here.

[0149] In this embodiment, during the calibration process, the user can simulate the angle line of the target angle through a physical device. The user only needs to stand at the target angle repeatedly to create sound source conditions and use the physical device to create a reference line. The conference equipment can automatically pick up the calibration point on the reference line in the real-time preview screen and automatically complete the calibration of sound and image. The calibration process is simple and the calibration accuracy is higher.

[0150] The above describes the method for calibrating sound and image in video images provided in this application. The following is a brief description of the principle of mapping sound source angles to video images and mapping video images to sound source angles using this calibration table:

[0151] A. Mapping of sound source angle to video frame:

[0152] 1) Obtain the calibration table, which contains pre-stored mapping relationships between several target angles and angle line equations;

[0153] The conference equipment acquires a pre-established calibration table, the process of which is described above. Figure 1 , Figure 5 , Figure 6 The corresponding embodiments are shown, and will not be repeated here.

[0154] 2) Collect the angle of the target sound source;

[0155] The conference equipment receives audio information through a microphone and determines the sound source location information in the audio information based on sound source localization technology. The terminal then collects the target sound source angle from the sound source localization information.

[0156] 3) Determine the range of sound source angles based on the target sound source angle and the preset angle tolerance;

[0157] The conference equipment determines the sound source angle range as [α-Δα, α+Δα] based on the target sound source angle α and the preset angle tolerance Δα; the preset angle tolerance can be specifically set according to different equipment requirements and equipment precision.

[0158] 4) Determine the target angle line equation based on the sound source angle range and calibration table;

[0159] The conference equipment first selects the angle lines of the nearest target angle from the calibration table as the left and right boundary lines, denoted as L1 and L2, based on the boundary values ​​of the sound source angle range, namely α-Δα and α+Δα, and the corresponding left and right boundary angles are denoted as α1 and α2.

[0160] The terminal then looks up the target angle line equations corresponding to L1 and L2 in the calibration table based on α1 and α2.

[0161] For example, if the left and right boundary angles α1 and α2 are 85° and 75° respectively, then by consulting the calibration table, the equations of the straight lines for the 85° and 75° angle lines (i.e., L1 and L2) can be obtained as follows:

[0162] L1: y = k1x + b1;

[0163] L2: y = k2x + b2.

[0164] 5) Determine the pixel area of ​​the sound source angle range in the video frame based on the target angle line equation;

[0165] In an application scenario involving the mapping of sound source angles to video frames, the corresponding pixel region in the video frame can be determined based on the sound source angle: The terminal, using the obtained target angle line equation and applying the principle of linear programming, can determine the corresponding pixel range in the image based on these two left and right boundary lines. Specifically, the terminal determines the pixel region of the sound source angle range in the video frame jointly based on the target angle line equation and a preset image height.

[0166] Specifically, the equations of lines L1 and L2 are as follows:

[0167] L1: y = k1x + b1;

[0168] L2: y = k2x + b2.

[0169] The corresponding mathematical expression for the image region is:

[0170] y≤k1x+b1;

[0171] y≥k2x+b2;

[0172] y≥0;

[0173] y <h。

[0174] Where h is the preset image height.

[0175] Based on the principle of linear programming, the region between L1 and L2 is the pixel area of ​​the target sound source angle in the video frame.

[0176] 6) Determine whether the target position is within the range of the sound source angle based on the target angle line equation.

[0177] In another application scenario where sound source angles are mapped to video frames, it is also possible to determine whether a target in the video frame is within the corresponding sound source angle range based on the sound source angle:

[0178] Based on the obtained target angle line equation, the terminal determines whether the target position is located between the two left and right boundary lines. If the target position is located between the left and right boundary lines, it can be determined that the target position is within the sound source angle range. If the target position is located outside the left and right boundary lines, it can be determined that the target position is outside the sound source angle range.

[0179] B. Mapping of video image to sound source angle:

[0180] 1) Obtain the calibration table, which contains pre-stored mapping relationships between several target angles and angle line equations;

[0181] The conference equipment acquires a pre-established calibration table, the process of which is described above. Figure 1 , Figure 5 , Figure 6 The corresponding embodiments are shown, and will not be repeated here.

[0182] 2) Obtain target detection information and determine the target location based on the target detection information;

[0183] The conferencing equipment receives video information via a camera and determines target detection information within the video information using image recognition technology. Specifically, the target detection information includes target detection results, i.e., target detection boxes. The target detection type can be face detection and / or head detection and / or human body detection, preferably face detection. The conferencing equipment determines the target's location based on the target detection results in the target detection information. Specifically, if the target detection result is a target detection box, the terminal uses the coordinates of the center point of the target detection box as the target location; if the target detection result is specific coordinates, the terminal directly determines the target location based on those coordinates.

[0184] 3) Determine the equations of the first and second angle lines adjacent to the target position in the calibration table;

[0185] Select two target angles adjacent to the target position from the target angles in the calibration table, determine the first angle line equation and the second angle line equation of the target angle respectively, and then determine whether it is between the two angle lines based on the principle of linear programming or the positional relationship between the point and the line. If it is between the two angle lines, the angles corresponding to these two angle lines (called the left and right angle lines where the target is located) can be used as the estimated value of the angle range where the target is located.

[0186] 4) Determine the angular range or angle of the target position in the video frame based on the first angle line equation and the second angle line equation.

[0187] The conferencing equipment can directly use the angular range between the target angles corresponding to the first and second angle line equations as the angular range of the target position in the video frame.

[0188] Furthermore, to determine the specific angle of the target position in the video frame, the terminal can calculate the estimated angle of the target based on the distance from the target position to the left and right angle lines and the estimated value within the aforementioned angle range. Specifically, the first distance d1 and the second distance d2 from the target position to the left and right angle lines are calculated, and the angle of the target is estimated based on the distance ratio. The formula used for estimation is as follows:

[0189] θx = θ1 + Δθ(d2 / d1 + d2);

[0190] Where θx represents the angle value of the target position in the video frame, θ1 represents the target angle corresponding to the first angle line equation, d1 represents the first distance, d2 represents the second distance, and Δθ represents the preset angle interval. Specifically, as can be seen from the aforementioned calibration process, the preset angle interval Δθ = θ2 - θ1, which is equal to the absolute value of the angle difference between the two angle lines.

[0191] The following describes the audio and video calibration device provided in this application. Please refer to [link to relevant documentation]. Figure 7 , Figure 7 One embodiment of the audio and video calibration device provided in this application includes:

[0192] Acquisition unit 701 is used to acquire the real-time preview screen and establish the screen coordinate system of the real-time preview screen;

[0193] The prompting unit 702 is used to acquire sound source angle information and prompt the user to move to a similar target angle based on the sound source angle information. The target angle is obtained by dividing the maximum range of sound source angles and a preset precision.

[0194] The picking unit 703 is used to pick up at least two calibration points in the real-time preview screen at the positions corresponding to the sound source angle information when the error between the sound source angle information and the target angle is less than a preset error.

[0195] The first determining unit 704 is used to determine the coordinate data of at least two calibration points in the screen coordinate system;

[0196] The second determining unit 705 is used to determine the angle line equation corresponding to the target angle based on the coordinate data.

[0197] The calibration unit 706 is used to establish the mapping relationship between the target angle and the angle line equation, and to store the mapping relationship in the calibration table.

[0198] Optionally, the pickup unit 702 is specifically used for:

[0199] When the error between the sound source angle information and the target angle is less than the preset error, the target detection of the mouth position is performed in the real-time preview screen to obtain the first detection result;

[0200] Determine the location of the calibration point based on the first test result, and pick up at least two calibration points.

[0201] Optionally, if there are multiple first detection results, the picking unit 702 is further used for:

[0202] Based on the sound source angle information, the nearest target detection result is selected from multiple first detection results;

[0203] or,

[0204] Calculate the degree of mouth opening of multiple first detection results, and determine the first detection result with the largest degree of mouth opening as the target detection result;

[0205] Determine the location of the calibration point based on the target detection results, and pick at least two calibration points.

[0206] Optionally, the pickup unit 702 is also specifically used for:

[0207] In the real-time preview, the mouth area is detected using a face region detection algorithm to obtain a mouth detection box;

[0208] The center point of the mouth detection frame is determined as the calibration point location, and at least two calibration points are picked.

[0209] Optionally, the pickup unit 702 is also specifically used for:

[0210] In the real-time preview, the target detection of the mouth position is performed using a facial landmark detection algorithm to obtain a set of mouth landmarks;

[0211] The midpoint between the upper lip keypoint and the lower lip keypoint in the mouth keypoint set is determined as the calibration point position, and at least two calibration points are picked.

[0212] Optionally, the pickup unit 702 is also specifically used for:

[0213] Pick at least two calibration points on the reference line in the live preview screen. One end of the reference line is located directly above the camera, and the other end is located at the speaker's mouth. The reference line is a straight line projected by the physical device.

[0214] Optionally, the calibration device may also include:

[0215] Display unit 707 is used to display the sound source angle information and target angle on the real-time preview screen.

[0216] Optionally, the Y-axis of the screen coordinate system coincides with the vertical symmetrical central axis of the live preview screen, and the X-axis of the screen coordinate system coincides with the top horizontal boundary of the live preview screen.

[0217] or,

[0218] The origin of the screen coordinate system is the center point of the real-time preview screen, the Y-axis is the vertical direction passing through the center point, and the X-axis is the horizontal direction passing through the center point.

[0219] In this embodiment, the functions of each unit are the same as described above. Figure 1 , Figure 5 or Figure 6 The steps in the method embodiments shown correspond to those in the examples, and will not be repeated here.

[0220] This application also provides a calibration device for sound and image in video images; please refer to [link to relevant documentation]. Figure 8 , Figure 8 One embodiment of the audio and video calibration device provided in this application includes:

[0221] Processor 801, memory 802, input / output unit 803, bus 804;

[0222] The processor 801 is connected to the memory 802, the input / output unit 803, and the bus 804;

[0223] The memory 802 stores a program, and the processor 801 calls the program to execute the calibration method for sound and picture in any of the above video images.

[0224] This application also relates to a computer-readable storage medium storing a program, characterized in that, when the program is run on a computer, it causes the computer to execute the calibration method for sound and image in any of the above video images.

[0225] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0226] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0227] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0228] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0229] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for calibrating sound and image in a video image, characterized in that, The calibration method includes: Acquire the real-time preview screen and establish the screen coordinate system of the real-time preview screen; Acquire sound source angle information and prompt the user to move to a nearby target angle based on the sound source angle information. The target angle is obtained by dividing the maximum range of sound source angles and a preset precision. When the error between the sound source angle information and the target angle is less than a preset error, at least two calibration points are picked up in the real-time preview screen at the position corresponding to the sound source angle information. Determine the coordinate data of the at least two calibration points in the screen coordinate system; Determine the angle line equation corresponding to the target angle based on the coordinate data; Establish a mapping relationship between the target angle and the angle line equation, and store the mapping relationship in a calibration table.

2. The calibration method according to claim 1, characterized in that, Picking at least two calibration points at the positions corresponding to the sound source angle information in the real-time preview screen includes: Target detection of the mouth position is performed in the real-time preview screen to obtain the first detection result; Based on the first detection result, determine the location of the calibration point and pick up at least two calibration points.

3. The method according to claim 2, characterized in that, If multiple first detection results exist, then determining the calibration point location based on the first detection results and picking the calibration point includes: Based on the sound source angle information, the nearest target detection result is selected from the plurality of first detection results; or, Calculate the degree of mouth opening of the multiple first detection results, and determine the first detection result with the largest degree of mouth opening as the target detection result; Based on the target detection results, determine the location of the calibration points and pick at least two calibration points.

4. The calibration method according to claim 2, characterized in that, The step of performing target detection at the mouth position in the real-time preview screen to obtain a first detection result includes: In the real-time preview, the mouth is detected at at least two locations at different distances using a face region detection algorithm, resulting in at least two mouth detection boxes. Determining the calibration point location based on the first detection result and picking at least two calibration points includes: The center points of the at least two mouth detection frames are respectively determined as calibration point positions, and at least two calibration points are picked up.

5. The calibration method according to claim 2, characterized in that, The step of performing target detection at the mouth position in the real-time preview screen to obtain a first detection result includes: In the real-time preview, the mouth is detected at at least two locations at different distances using a facial key point detection algorithm, resulting in at least two sets of mouth key points. Determining the calibration point location based on the first detection result and picking at least two calibration points includes: The midpoints of the upper lip key points and the lower lip key points in the at least two sets of mouth key points are respectively determined as calibration point positions, and at least two calibration points are picked.

6. The calibration method according to claim 1, characterized in that, Picking at least two calibration points at the positions corresponding to the sound source angle information in the real-time preview screen includes: Pick at least two calibration points on the reference line in the real-time preview screen. One end of the reference line is located directly above the camera, and the other end is located at the speaker's mouth. The reference line is a straight line projected by a physical device.

7. The calibration method according to any one of claims 1 to 6, characterized in that, The distance between at least two calibration points is greater than a preset distance.

8. The calibration method according to any one of claims 1 to 6, characterized in that, After acquiring the sound source angle information, the method further includes: The sound source angle information and the target angle are displayed on the real-time preview screen.

9. A method for mapping sound and image in a video image, characterized in that, The mapping method includes: Obtain a calibration table as described in any one of claims 1 to 8, wherein the calibration table pre-stores a mapping relationship between several target angles and angle line equations; Acquire the angle of the target sound source; The sound source angle range is determined based on the target sound source angle and the preset angle tolerance; The target angle line equation is determined based on the sound source angle range and the calibration table. The pixel area of ​​the sound source angle range in the video frame is determined according to the target angle line equation. or, Determine whether the target position is within the range of the sound source angle based on the target angle line equation.

10. A method for mapping sound and image in a video image, characterized in that, The mapping method includes: Obtain a calibration table as described in any one of claims 1 to 8, wherein the calibration table pre-stores a mapping relationship between several target angles and angle line equations; Obtain target detection information and determine the target location based on the target detection information; Determine the equations of the first and second angle lines adjacent to the target position in the calibration table; The angular range or angle of the target position in the video frame is determined based on the first angle line equation and the second angle line equation.

11. A video conferencing device, wherein the video conferencing device performs the method for calibrating sound and image in a video image as described in any one of claims 1 to 8.

12. A calibration device for sound and image in a video image, characterized in that, The calibration device includes: The acquisition unit is used to acquire the real-time preview screen and establish the screen coordinate system of the real-time preview screen; The prompting unit is used to acquire sound source angle information and prompt the user to move to a nearby target angle based on the sound source angle information. The target angle is obtained by dividing the maximum range of sound source angles and a preset precision. The picking unit is used to pick up at least two calibration points in the real-time preview screen at the positions corresponding to the sound source angle information when the error between the sound source angle information and the target angle is less than a preset error. The first determining unit is used to determine the coordinate data of the at least two calibration points in the screen coordinate system; The second determining unit is used to determine the angle line equation corresponding to the target angle based on the coordinate data; The calibration unit is used to establish the mapping relationship between the target angle and the angle line equation, and to store the mapping relationship in the calibration table.

Images