Device calibration method, sound source tracking method, device, and storage medium

Abstract

The application relates to a device calibration method, a sound source tracking method, a device and a storage medium, and belongs to the technical field of sound source tracking. The method comprises the following steps: controlling an image acquisition device to scan a current sound source tracking environment; identifying each audio acquisition device in the scanned environment image to obtain a device image area of each audio acquisition device; determining a relative position relationship of the audio acquisition device relative to the image acquisition device by using the device image area of each audio acquisition device, an image acquisition posture corresponding to the device image area, and an image acquisition device internal parameter and an image acquisition device external parameter; and establishing a mapping relationship between each device identifier and the relative position relationship to track the sound source. The method can solve the problem of low efficiency of obtaining the relative position calibrated by the user, and can improve the calibration efficiency.

Description

Technical Field

[0001] This application relates to a device calibration method, a sound source tracking method, a device, and a storage medium, belonging to the field of sound source tracking technology. Background Technology

[0002] Currently, a sound source tracking system refers to a system that uses image acquisition equipment to track the sound source in real time. Sound source tracking systems are widely used in scenarios such as remote conferencing and online teaching to track the speaker in a given situation.

[0003] A typical sound source tracking system includes a pan-tilt-zoom (PTZ) camera and multiple microphone arrays, with different microphone arrays corresponding to different pickup areas. After installing the camera and microphone arrays, the user manually configures preset positions for the PTZ camera, with each preset position corresponding to a pickup area of ​​the microphone array. Then, during sound source tracking, when a speaker speaks within a pickup area of ​​a particular microphone array, the microphone array sends the corresponding number for that pickup area to the PTZ camera. The PTZ camera then turns to the preset position corresponding to that pickup area, thus enabling the camera to track the speaker. For example, preset position 1 is located 30 degrees to the right of the PTZ camera's front. Sending the command for preset position 1 to the camera will cause it to turn 30 degrees to the right of its front.

[0004] However, manually configuring the correspondence between preset positions and the microphone array's pickup area is inefficient and requires the relative positions of the gimbal camera and the microphone array to be fixed. If the relative positions change, manual reconfiguration is required, further reducing the efficiency of obtaining the relative positions. Summary of the Invention

[0005] This application provides a device calibration method, a sound source tracking method, a device, and a storage medium, which can solve the problem of low efficiency in acquiring the relative position when the relative position of the audio acquisition device to the image acquisition device is manually configured by the user. This application provides the following technical solution:

[0006] On one hand, a device calibration method is provided for a sound source tracking system, the sound source tracking system including an image acquisition device and at least two audio acquisition devices; the image acquisition range of the image acquisition component includes the installation location of the at least two audio acquisition devices; during sound source tracking, the audio acquisition devices are used to determine the angle data of the currently emitting target sound source relative to the audio acquisition devices, and send the device identifier of the audio acquisition devices and the angle data to the image acquisition devices for sound source tracking; wherein, different audio acquisition devices acquire audio data of target sound sources in different areas; the method includes:

[0007] Before performing sound source tracking, the image acquisition device is controlled to scan the current sound source tracking environment to obtain an environmental image;

[0008] Identify each audio acquisition device in the environmental image and obtain the device image area for each audio acquisition device;

[0009] The relative positional relationship of the audio acquisition device to the image acquisition device is determined using the device image area of ​​each audio acquisition device, the image acquisition posture corresponding to the device image area, and the intrinsic and extrinsic parameters of the image acquisition device.

[0010] A mapping relationship is established between each device identifier and the relative positional relationship, so that after the image acquisition device obtains the angle data carrying the device identifier, it uses the mapping relationship to determine the relative positional relationship of the device identifier, and uses the relative positional relationship and the angle data to track the target sound source.

[0011] Optionally, the device outline of the audio acquisition device differs from the target outline of other targets in the sound source tracking environment; the step of identifying each audio acquisition device in the environmental image and obtaining the device image region of each audio acquisition device includes:

[0012] Detect the contours of each target in the environmental image;

[0013] The image region of the target whose contour matches the contour of the device is determined to obtain the device image region of the audio acquisition device.

[0014] Optionally, the audio acquisition device is equipped with a calibration mark, and the process of identifying each audio acquisition device in the environmental image and obtaining the device image area of ​​each audio acquisition device includes:

[0015] Detect the calibration markers in the environmental image to obtain the marker image region of the calibration markers;

[0016] The device image region is determined based on the identified image region.

[0017] Optionally, establishing the mapping relationship between each device identifier and the relative positional relationship includes:

[0018] For each device identifier, a mapping instruction carrying the device identifier is sent to each audio acquisition device. The mapping instruction is used to trigger the audio acquisition device corresponding to the device identifier to perform a preset mapping action.

[0019] The image acquisition device is controlled to recognize the preset mapping action;

[0020] Upon recognizing the preset mapping action, a mapping relationship is established between the relative position of the audio acquisition device executing the preset mapping action and the device identifier.

[0021] Optionally, establishing the mapping relationship between the relative positional relationship of the audio acquisition device performing the preset mapping action and the device identifier includes:

[0022] Obtain the image acquisition posture corresponding to the recognition of the preset mapping action;

[0023] Determine the relative positional relationship that matches the image acquisition posture corresponding to the preset mapping action;

[0024] Establish a mapping relationship between the matching relative positional relationship and the device identifier.

[0025] Optionally, controlling the image acquisition device to recognize the preset mapping action includes: controlling the image acquisition device to acquire images with each unmapped relative positional relationship and recognizing the preset mapping action;

[0026] Accordingly, when the preset mapping action is identified, a mapping relationship is established between the relative position of the audio acquisition device performing the preset mapping action and the device identifier, including: when the preset mapping action is identified, establishing a mapping relationship between the currently used relative position and the device identifier.

[0027] Optionally, determining the relative positional relationship between the audio acquisition device and the image acquisition device using the device image region of each audio acquisition device, the image acquisition posture corresponding to the device image region, and the intrinsic and extrinsic parameters of the image acquisition device includes:

[0028] Identify the key point locations of key points on the audio acquisition device in the device image area, and the pixel information of the audio acquisition device in the device image area;

[0029] The attitude angle of the audio acquisition device relative to the image acquisition device is determined based on the location of the key points.

[0030] Using the pixel information, the image acquisition posture, and the intrinsic and extrinsic parameters of the image acquisition device, the orientation and distance of the audio acquisition device relative to the image acquisition device are determined, and the relative positional relationship includes the orientation, the distance, and the posture angle.

[0031] On the other hand, a device calibration method is provided for a sound source tracking system, the sound source tracking system including an image acquisition device and at least two audio acquisition devices; the image acquisition range of the image acquisition component includes the installation location of the at least two audio acquisition devices; the method includes:

[0032] The angle data sent by the audio acquisition device, which carries the device identifier of the audio acquisition device, is obtained; the angle data is determined by the audio acquisition device after acquiring the audio data of the target sound source currently emitting sound.

[0033] A pre-established mapping relationship is obtained between each device identifier and the relative positional relationship between the audio acquisition device and the image acquisition device. The relative positional relationship is obtained by controlling the image acquisition device to scan the current sound source tracking environment to obtain an environmental image, identifying each audio acquisition device in the environmental image, and obtaining the device image region of each audio acquisition device. The device image region of each audio acquisition device, the image acquisition posture corresponding to the device image region, and the intrinsic and extrinsic parameters of the image acquisition device are used to determine the relationship.

[0034] The relative positional relationship of the device identifier mapping is determined using the mapping relationship.

[0035] Using the relative positional relationship and the angle data, the image acquisition device is controlled to track the target sound source.

[0036] On the other hand, an electronic device is provided, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement the device calibration method or sound source tracking method provided in the above aspects.

[0037] On the other hand, a computer-readable storage medium is provided, wherein a program is stored in the storage medium, and when executed by a processor, the program is used to implement the device calibration method or sound source tracking method provided in the above aspects.

[0038] The beneficial effects of this application include at least the following: before performing sound source tracking, controlling the image acquisition device to scan the current sound source tracking environment to obtain an environmental image; identifying each audio acquisition device in the environmental image to obtain the device image region of each audio acquisition device; using the device image region of each audio acquisition device, the image acquisition posture corresponding to the device image region, and the intrinsic and extrinsic parameters of the image acquisition device to determine the relative positional relationship of the audio acquisition device relative to the image acquisition device; establishing a mapping relationship between each device identifier and the relative positional relationship, so that after the image acquisition device obtains the angle data carrying the device identifier, it can use the mapping relationship to determine the relative positional relationship mapped by the device identifier, and use the relative positional relationship and angle data to perform sound source tracking on the target sound source; it can solve the problem of low efficiency in obtaining the relative position when the relative positional relationship of the audio acquisition device relative to the image acquisition device is manually configured by the user; since the relative positional relationship of the audio acquisition device relative to the image acquisition device can be automatically identified and the mapping relationship between the device identifier and the relative positional relationship can be automatically established, the user does not need to manually configure the relative positional relationship, which can improve the configuration efficiency of the relative positional relationship.

[0039] At the same time, even if the relative positional relationship changes, the user does not need to manually configure it. Therefore, there is no need to restrict the relative positional relationship between the image acquisition device and the audio acquisition device, which can improve the setting flexibility of the image acquisition device and the audio acquisition device in the sound source localization system.

[0040] In addition, identifying audio acquisition devices by recognizing the outline or markings of the target can reduce the difficulty of identifying audio acquisition devices and improve their recognition efficiency.

[0041] In addition, by performing a preset mapping action on the audio acquisition device corresponding to the device identifier, a mapping relationship between the device identifier and the relative position relationship is established. When an audio acquisition device with a certain device identifier sends angle data to the image acquisition device, the image acquisition device can calculate the coordinates of the position that all audio acquisition devices are pointing to based on their relative position relationship and angle data, thereby realizing speaker tracking.

[0042] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the structure of a sound source tracking system provided in one embodiment of this application;

[0044] Figure 2 This is a flowchart of a device calibration method provided in one embodiment of this application;

[0045] Figure 3 This is a flowchart of a sound source tracking method provided in one embodiment of this application;

[0046] Figure 4 This is a schematic diagram of a sound source tracking scenario provided in one embodiment of this application;

[0047] Figure 5 This is a block diagram of a device calibration apparatus provided in one embodiment of this application;

[0048] Figure 6 This is a block diagram of a sound source tracking device provided in one embodiment of this application;

[0049] Figure 7 This is a block diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0050] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0051] Figure 1 This is a schematic diagram of a sound source tracking system provided in one embodiment of this application. This sound source tracking system is suitable for remote conferencing scenarios and / or online classroom scenarios, etc. This embodiment does not limit the application scenarios of the sound source tracking system. Figure 1 It is known that the sound source tracking system includes at least: an image acquisition device 110 and at least two audio acquisition devices 120.

[0052] The audio acquisition device 120 has an audio acquisition function. In one example, the audio acquisition device 120 has a beamforming function. In this case, the audio acquisition device 120 only acquires the audio data of the sound source within a preset pickup area and suppresses the audio data outside the preset pickup area. Each audio acquisition device 120 can be a ceiling microphone, a handheld microphone, or a neck microphone, etc. This embodiment does not limit the implementation method of the audio acquisition device 120.

[0053] Optionally, the audio acquisition devices 120 can be cascaded to establish a communication connection and transmit data between them. For example, if the audio acquisition devices 120 acquire audio data from the same sound source, the slave device in the audio acquisition device 120 sends the audio data to the master device. The master device selects one of the slave devices as the audio acquisition device that acquired the audio data this time and sends a command to that slave device. This embodiment does not limit the data content transmitted between the audio acquisition devices.

[0054] In this embodiment, each audio acquisition device 120 establishes a communication connection with the image acquisition device 110. This communication connection can be wired or wireless. This embodiment does not limit the type of communication connection between the audio acquisition device and the image acquisition device. Schematic, each audio acquisition device 120 and the image acquisition device 110 are wirelessly connected and located in the same local area network.

[0055] During sound source tracking, the audio acquisition device determines the angle data of the target sound source relative to the audio acquisition device, and sends the device identifier of the audio acquisition device and the angle data to the image acquisition device for sound source tracking; wherein, different audio acquisition devices acquire audio data of target sound sources in different areas.

[0056] The image acquisition device 110 has an image acquisition function. During sound source tracking, the image acquisition device 110 is suitable for acquiring image data of the target sound source currently emitting sound.

[0057] The image acquisition posture of the image acquisition device 110 is adjustable, including the horizontal angle and the pitch angle of the image acquisition device. In one example, the image acquisition device 110 includes a gimbal and an image acquisition component. The gimbal is used to move the image acquisition component to adjust the image acquisition posture of the image acquisition device 110. The image acquisition component can be a camera, a webcam, etc., and this embodiment does not limit the implementation of the image acquisition component.

[0058] During sound source tracking, a communication connection is established between the image acquisition device 110 and the audio acquisition device 120 to receive data sent by the audio acquisition device 120.

[0059] In this embodiment, the image acquisition device 110 is also used to scan the audio acquisition devices 120 in the current sound source tracking environment to obtain the relative positional relationship of each audio acquisition device 120 relative to the image acquisition device 110. Based on this, the image acquisition range of the image acquisition component includes the installation locations of at least two audio acquisition devices. The image acquisition range of the image acquisition component refers to the maximum acquisition range that the image acquisition component can achieve. At this time, the image acquisition component can use different image acquisition postures to scan the audio acquisition devices.

[0060] Specifically, during equipment calibration, the image acquisition device is used to: scan the current sound source tracking environment and obtain an environmental image before sound source tracking; identify each audio acquisition device in the environmental image and obtain the device image area of ​​each audio acquisition device; use the device image area of ​​each audio acquisition device, the image acquisition posture corresponding to the device image area, and the intrinsic and extrinsic parameters of the image acquisition device to determine the relative positional relationship of the audio acquisition device relative to the image acquisition device; establish a mapping relationship between each device identifier and the relative positional relationship, so that after the image acquisition device obtains the angle data carrying the device identifier, it can use the mapping relationship to determine the relative positional relationship mapped by the device identifier, and use the relative positional relationship and angle data to perform sound source tracking on the target sound source.

[0061] Accordingly, during sound source tracking, the image acquisition device is used to: acquire angle data carrying the device identifier of the audio acquisition device sent by the audio acquisition device; acquire a pre-established mapping relationship between each device identifier and the relative positional relationship between the audio acquisition device and the image acquisition device; use the mapping relationship to determine the relative positional relationship of the device identifier mapping; and use the relative positional relationship and angle data to control the image acquisition device to track the target sound source.

[0062] Optionally, the above embodiments use the device calibration method and sound source tracking method in an image acquisition device as an example for illustration. In actual implementation, the sound source tracking system may also include an electronic device that is communicatively connected to the image acquisition device, and the electronic device executes the above device calibration method and sound source tracking method.

[0063] In this embodiment, before performing sound source tracking, the image acquisition device is controlled to scan the current sound source tracking environment to obtain an environmental image; each audio acquisition device in the environmental image is identified to obtain the device image region of each audio acquisition device; using the device image region of each audio acquisition device, the image acquisition posture corresponding to the device image region, and the intrinsic and extrinsic parameters of the image acquisition device, the relative positional relationship between the audio acquisition device and the image acquisition device is determined; a mapping relationship between each device identifier and the relative positional relationship is established; this can solve the problem of low efficiency in obtaining the relative position when the relative positional relationship between the audio acquisition device and the image acquisition device is manually configured by the user; since the relative positional relationship between the audio acquisition device and the image acquisition device can be automatically identified and the mapping relationship between the device identifier and the relative positional relationship can be automatically established, the user does not need to manually configure the relative positional relationship, which can improve the configuration efficiency of the relative positional relationship.

[0064] At the same time, even if the relative positional relationship changes, the user does not need to manually configure it. Therefore, there is no need to restrict the relative positional relationship between the image acquisition device and the audio acquisition device, which can improve the setting flexibility of the image acquisition device and the audio acquisition device in the sound source localization system.

[0065] Below, the equipment calibration process and the sound source tracking process will be introduced separately based on the above system.

[0066] Generally, in a sound source localization system, the image acquisition device and the audio acquisition device are integrated. In this case, the relative positional relationship between the image acquisition device and the audio acquisition device is fixed and does not require manual configuration by the user; this relative positional relationship is pre-stored in the image acquisition device. However, when the image acquisition device and the audio acquisition device are set separately, the orientation and distance of the audio acquisition device relative to the image acquisition device are flexibly set. In this case, it is necessary to calibrate the relative positional relationship between the audio acquisition device and the image acquisition device.

[0067] Traditional calibration methods involve manual calibration by the user. For example, the image acquisition device is rotated to the pickup area of ​​an audio acquisition device, and then the current image acquisition posture (preset position) of the image acquisition device is determined as the relative position relationship. A mapping relationship is then established between the device identifier of the audio acquisition device and the relative position relationship.

[0068] At this point, on the one hand, users need to establish a mapping relationship for each audio acquisition device, which reduces the efficiency of establishing the mapping relationship. On the other hand, once the position of the audio acquisition device and / or the image acquisition device changes, the user needs to recalibrate manually. To avoid the low efficiency caused by recalibration, it may be necessary to restrict the relative positional relationship between the image acquisition device and the audio acquisition device.

[0069] Based on the above-mentioned technical problems, this application provides the following equipment calibration method. Figure 2 This is a flowchart of a device calibration method provided in one embodiment of this application. This embodiment uses this method for... Figure 1 Taking the image acquisition device in the sound source tracking system shown as an example, the method includes at least the following steps:

[0070] Step 201: Before performing sound source tracking, control the image acquisition device to scan the current sound source tracking environment to obtain an environmental image.

[0071] In one example, the image acquisition device responds to a calibration command by scanning the sound source tracking environment within a pre-configured field of view to obtain an environmental image.

[0072] Optionally, the pre-configured field of view can be the maximum range that the image acquisition device can capture, or it can be the range that the various audio acquisition devices converge on. This embodiment does not limit the implementation method of the field of view. The configuration method of the field of view can be user-defined or stored by default in the image acquisition device. This embodiment does not limit the configuration method of the field of view.

[0073] Optionally, the calibration command can be obtained in several ways, including but not limited to the following: generating the calibration command when the image acquisition device is initialized and powered on; or, the calibration command is sent by another device; or, the calibration command is generated when a trigger operation is received on the calibration control on the image acquisition device. This embodiment does not limit the method of obtaining the calibration command. The calibration control can be a physical button or a virtual button displayed on a touch screen. This embodiment does not limit the implementation method of the calibration control.

[0074] In other embodiments, the image acquisition device may also scan the current sound source tracking environment at preset intervals to obtain environmental images for calibration. This embodiment does not limit the timing of when the image acquisition device begins calibration. The preset interval can be one week, one month, etc., and the preset interval can be set by the user or stored in the image acquisition device. This embodiment does not limit the setting method or value of the preset interval.

[0075] Step 202: Identify each audio acquisition device in the environmental image and obtain the device image area of ​​each audio acquisition device.

[0076] Optionally, each audio acquisition device in the environmental image can be identified to obtain the device image area for each audio acquisition device, including but not limited to the following methods:

[0077] The first method involves detecting the contours of various targets in the environmental image; determining the image region of the target whose contour matches the device contour, and thus obtaining the device image region of the audio acquisition device.

[0078] Since the device outline of the audio acquisition device is different from the target outline of other targets in the sound source tracking environment, the device image area of ​​the audio acquisition device can be obtained by recognizing the device outline of the audio acquisition device.

[0079] Optionally, the algorithms for detecting the contours of each target include, but are not limited to: using OpenCV for contour detection, or detecting target contours based on edge detection operators, or contour detection algorithms based on deep learning. This embodiment does not limit the method of detecting the contours of the targets.

[0080] The second method involves setting calibration markers on the audio acquisition device. In this case, the calibration markers in the environmental image are detected to obtain the marker image area; the device image area is then determined based on the marker image area.

[0081] The calibration marks include, but are not limited to, checkerboard patterns or infrared light. This embodiment does not limit the implementation method of the calibration marks.

[0082] In one example, determining the device image region based on the identified image region includes: determining the identified image region as the device image region.

[0083] In another example, determining the device image region based on the identified image region includes: expanding the identified image region outward by a preset size to obtain the device image region.

[0084] Optionally, after identifying each audio acquisition device in the environmental image, the image acquisition device can also determine whether the number of identified audio acquisition devices is consistent with the target number of audio acquisition devices in the sound source tracking environment; if consistent, step 203 is executed; if inconsistent, step 201 is executed again or an error message is output to inform the user that the image acquisition device has not identified all audio acquisition devices.

[0085] Step 203: Using the device image area of ​​each audio acquisition device, the image acquisition posture corresponding to the device image area, and the intrinsic and extrinsic parameters of the image acquisition device, determine the relative positional relationship between the audio acquisition device and the image acquisition device.

[0086] In one example, the relative positional relationships include the orientation, distance, and attitude angle of the audio acquisition device relative to the image acquisition component.

[0087] The orientation of the audio acquisition device relative to the image acquisition component refers to the angle between the line connecting the reference position of the audio acquisition device and the reference position of the image acquisition component and the coordinate axes in the coordinate system established based on the image acquisition component. The attitude angle of the audio acquisition device relative to the image acquisition component refers to the direction of deflection of the audio acquisition device in the coordinate system established based on the image acquisition component. Schematically, the attitude angle includes yaw, roll, and pitch.

[0088] Optionally, the coordinate system established based on the image acquisition component can be a world coordinate system with the reference position of the image acquisition device as the origin. The z-axis of this coordinate system is perpendicular to the horizontal plane, and the x-axis and y-axis are parallel to the horizontal plane and perpendicular to each other. The reference position of the image acquisition device can be the rotation center of the image acquisition device. This embodiment does not limit the method of establishing the coordinate system based on the image acquisition device.

[0089] The reference position of the audio acquisition device can be the center point of the device or other positions on the audio acquisition device. This embodiment does not limit the implementation method of the reference position of the audio acquisition device.

[0090] At this point, using the device image area of ​​each audio acquisition device, the image acquisition posture corresponding to the device image area, and the intrinsic and extrinsic parameters of the image acquisition device, the relative positional relationship between the audio acquisition device and the image acquisition device is determined. This includes: identifying the key point positions of key points on the audio acquisition device in the device image area, and the pixel information of the audio acquisition device in the device image area; determining the posture angle of the audio acquisition device relative to the image acquisition device based on the key point positions; and using the pixel information, image acquisition posture, and intrinsic and extrinsic parameters of the image acquisition device, determining the orientation and distance of the audio acquisition device relative to the image acquisition device. The relative positional relationship includes orientation, distance, and posture angle.

[0091] Among them, there are at least two key points. The key points can be at least two positions on the calibration mark on the audio acquisition device, such as the midpoint of the chessboard and each vertex of the chessboard. This embodiment does not limit the implementation method of the key points.

[0092] Since the planes where the key points are located are not parallel to the acquisition plane of the image acquisition component, the distance between the key points in the device image area will be smaller than the distance when they are parallel. Therefore, based on the difference between the first distance between the positions of the key points when they are not parallel and the second distance between the corresponding key points when they are parallel, the deflection angle of the plane where each key point is located relative to the acquisition plane can be determined, and the attitude angle can be obtained.

[0093] Using pixel information, image acquisition posture, and intrinsic and extrinsic parameters of the image acquisition device, the orientation and distance of the audio acquisition device relative to the image acquisition device are determined. The relative positional relationship includes orientation, distance, and posture angle, including: based on the principle of triangle similarity, using the focal length of the image acquisition component and the pixel position in the pixel information, the pixel position is transformed to the camera coordinate system of the image acquisition component to obtain the first coordinate position; using the intrinsic and extrinsic parameters of the image acquisition device, the first coordinate position is transformed to the world coordinate system established based on the image acquisition device to obtain the second coordinate position; based on the second coordinate position and the image acquisition posture, the orientation and distance of the audio acquisition device relative to the image acquisition device are determined.

[0094] The intrinsic and extrinsic parameters of the image acquisition device are pre-stored in the image acquisition device.

[0095] Step 204: Establish a mapping relationship between each device identifier and its relative position, so that after the image acquisition device obtains the angle data carrying the device identifier, it can use the mapping relationship to determine the relative position relationship of the device identifier, and use the relative position relationship and angle data to track the target sound source.

[0096] Since the image acquisition device does not know which audio acquisition device each relative position relationship belongs to after obtaining the relative position relationship, it is necessary to establish a mapping relationship between each device identifier and the relative position relationship.

[0097] In one example, establishing a mapping relationship between each device identifier and its relative position includes: for each device identifier, sending a mapping instruction carrying the device identifier to each audio acquisition device; controlling the image acquisition device to recognize a preset mapping action; and, if the preset mapping action is recognized, establishing a mapping relationship between the relative position of the audio acquisition device executing the preset mapping action and the device identifier.

[0098] The mapping instruction is used to trigger the audio acquisition device corresponding to the device identifier to execute a preset mapping action. The preset mapping action may be to control the indicator light on the audio acquisition device to flash, or it may be to control the indicator light on the audio acquisition device to display a preset pattern. This embodiment does not limit the implementation method of the preset mapping action.

[0099] Optionally, the mapping relationship can be established after the image acquisition device has finished scanning the audio acquisition device, or it can be established during the scanning of the audio acquisition device.

[0100] If the mapping relationship is established after the image acquisition device scans the audio acquisition device, then the mapping relationship between the relative position of the audio acquisition device that performs the preset mapping action and the device identifier is established, including: obtaining the image acquisition posture corresponding to the preset mapping action; determining the relative position relationship that matches the image acquisition posture corresponding to the preset mapping action; and establishing the mapping relationship between the matching relative position relationship and the device identifier.

[0101] Since each relative position relationship has a corresponding image acquisition posture, when the image acquisition device scans the preset mapping action, it can compare the image acquisition posture when the preset mapping action is scanned with the image acquisition posture corresponding to each relative position relationship, and determine the relative position relationship corresponding to the image acquisition posture with the smallest difference as the relative position relationship that matches the image acquisition posture corresponding to the preset mapping action.

[0102] Alternatively, controlling the image acquisition device to identify a preset mapping action includes: controlling the image acquisition device to acquire images with various unmapped relative positional relationships and identifying the preset mapping action; correspondingly, when the preset mapping action is identified, establishing a mapping relationship between the relative positional relationship of the audio acquisition device performing the preset mapping action and the device identifier includes: when the preset mapping action is identified, establishing a mapping relationship between the currently used relative positional relationship and the device identifier.

[0103] At this point, since the relative positional relationships have been determined, the probability of scanning the preset mapping action with these relative positional relationships is relatively high. This reduces the number of rotations required for the image acquisition device to scan the preset mapping action, thereby improving mapping efficiency.

[0104] If the mapping relationship is established during the scanning of audio acquisition devices, the image acquisition component first sends a mapping instruction carrying the device identifier to each audio acquisition device for each device identifier; controls the image acquisition device to recognize the preset mapping action; if the preset mapping action is recognized, step 202 is executed, and then the mapping relationship between the relative position relationship of the audio acquisition device that performs the preset mapping action and the device identifier is established.

[0105] In summary, the device calibration method provided in this embodiment obtains an environmental image by controlling an image acquisition device to scan the current sound source tracking environment before performing sound source tracking; identifies each audio acquisition device in the environmental image to obtain the device image region of each audio acquisition device; uses the device image region of each audio acquisition device, the image acquisition posture corresponding to the device image region, and the intrinsic and extrinsic parameters of the image acquisition device to determine the relative positional relationship between the audio acquisition device and the image acquisition device; and establishes a mapping relationship between each device identifier and the relative positional relationship so that after the image acquisition device obtains the angle data carrying the device identifier, it can use the mapping relationship to determine the relative positional relationship mapped by the device identifier, and use the relative positional relationship and angle data to perform sound source tracking on the target sound source. This method can solve the problem of low efficiency in obtaining the relative position when the relative positional relationship between the audio acquisition device and the image acquisition device is manually configured by the user. Since the relative positional relationship between the audio acquisition device and the image acquisition device can be automatically identified and the mapping relationship between the device identifier and the relative positional relationship can be automatically established, the relative positional relationship does not need to be manually configured by the user, thus improving the configuration efficiency of the relative positional relationship.

[0106] At the same time, even if the relative positional relationship changes, the user does not need to manually configure it. Therefore, there is no need to restrict the relative positional relationship between the image acquisition device and the audio acquisition device, which can improve the setting flexibility of the image acquisition device and the audio acquisition device in the sound source localization system.

[0107] In addition, identifying audio acquisition devices by recognizing the outline or markings of the target can reduce the difficulty of identifying audio acquisition devices and improve their recognition efficiency.

[0108] In addition, by performing a preset mapping action on the audio acquisition device corresponding to the device identifier, a mapping relationship between the device identifier and the relative position relationship is established. When an audio acquisition device with a certain device identifier sends angle data to the image acquisition device, the image acquisition device can calculate the coordinates of the position that all audio acquisition devices are pointing to based on their relative position relationship and angle data, thereby realizing speaker tracking.

[0109] Figure 3 This is a flowchart of a sound source tracking method provided in one embodiment of this application. This embodiment uses this method for... Figure 1 Taking the image acquisition device in the sound source tracking system shown as an example, the method includes at least the following steps:

[0110] Step 301: Obtain angle data carrying the device identifier of the audio acquisition device sent by the audio acquisition device.

[0111] The angle data is determined by the audio acquisition device after it has collected the audio data from the target sound source. The angle data can be the direction of arrival (DOA) determined by the audio acquisition device.

[0112] In this embodiment, audio data emitted by the same target sound source can be acquired by at least two audio acquisition devices. That is, for audio data emitted by the same target sound source, the image acquisition device can obtain the angle data of the device identifiers sent by at least two audio acquisition devices.

[0113] Step 302: Obtain the pre-established mapping relationship between each device identifier and the relative positional relationship between the audio acquisition device and the image acquisition device.

[0114] The relative positional relationship is determined by the image acquisition device scanning the current sound source tracking environment to obtain an environmental image, identifying each audio acquisition device in the environmental image, and obtaining the device image region of each audio acquisition device. This is then determined using the device image region of each audio acquisition device, the image acquisition posture corresponding to the device image region, and the intrinsic and extrinsic parameters of the image acquisition device. The specific determination method is described in the above embodiment and will not be repeated here.

[0115] Step 303: Use the mapping relationship to determine the relative positional relationship of the device identifier mapping.

[0116] Find the relative position of the device identifier in the mapping relationship.

[0117] Step 304: Using relative positional relationships and angle data, control the image acquisition device to track the target sound source.

[0118] Specifically, the image acquisition component uses relative positional relationships to determine the position of the audio acquisition device in a coordinate system established based on the image acquisition component. Then, based on the angle data indicating the target sound source relative to the audio acquisition device, the direction of the target sound source relative to the audio acquisition device in this coordinate system is determined. The intersection of the directions corresponding to each audio acquisition device is determined as the location of the target sound source. The image acquisition component is then controlled to acquire an image of the location of the target sound source in order to perform sound source tracking.

[0119] For example: Reference Figure 4 The diagram shows a top view of the sound source tracking environment, which includes an image acquisition device 41 and two audio acquisition devices, dev1 and dev2. Assume a world coordinate system is established with the location of image acquisition device 41 as the origin, with the z-axis perpendicular to the horizontal plane, and the x and y axes parallel to the horizontal plane and perpendicular to each other. At this point, dev1 and dev2 are located on the conference table, meaning their z-axis coordinates are the same. The image acquisition component 110 is generally installed above dev1 and dev2; that is, the z-axis coordinate of image acquisition component 41 is greater than the z-axis coordinates of dev1 and dev2.

[0120] After the target sound source 42 speaks, dev1 obtains the angle data α1 of the target sound source 42 and sends dev1's device identifier 1 and α1 to the image acquisition device 110; dev2 obtains the angle data α2 of the target sound source 42 and sends dev1's device identifier 2 and α2 to the image acquisition device 41.

[0121] After the image acquisition device 41 obtains the device identifier 1 and α1, it looks up the mapping relationship based on the device identifier 1 to obtain the relative position relationship, thereby determining the coordinates (x1, y1, z1) of dev1 in the world coordinate system; then, based on α1, it determines the direction of the target sound source relative to dev1 in the coordinate system (indicated by the dashed line).

[0122] After acquiring device identifier 2 and α2, image acquisition device 41 uses device identifier 2 to look up the mapping relationship to obtain the relative position relationship, thereby determining the coordinates (x2, y2, z2) of dev2 in the world coordinate system. Then, based on α2, it determines the direction (represented by the dashed line) of the target sound source relative to dev2 in this coordinate system. The intersection of the two directions (the intersection of the dashed lines) is the location of the target sound source. Afterward, image acquisition device 41 can operate to align the center area of ​​the lens with the location of the target sound source to track the target sound source.

[0123] In summary, the sound source tracking method provided in this embodiment acquires angle data carrying the device identifier of the audio acquisition device sent by the audio acquisition device; acquires a pre-established mapping relationship between each device identifier and the relative positional relationship between the audio acquisition device and the image acquisition device. The relative positional relationship is obtained by controlling the image acquisition device to scan the current sound source tracking environment to obtain an environmental image, identifying each audio acquisition device in the environmental image, and obtaining the device image region of each audio acquisition device; it is determined using the device image region of each audio acquisition device, the image acquisition posture corresponding to the device image region, and the intrinsic and extrinsic parameters of the image acquisition device; it uses the mapping relationship to determine the relative positional relationship mapped by the device identifier; and it uses the relative positional relationship and angle data to control the image acquisition device to perform sound source tracking on the target sound source. This method can solve the problem of low efficiency in acquiring the relative position when the relative positional relationship between the audio acquisition device and the image acquisition device is manually configured by the user, resulting in low sound source tracking efficiency. Since the relative positional relationship between the audio acquisition device and the image acquisition device can be automatically identified and the mapping relationship between the device identifier and the relative positional relationship can be automatically established, the relative positional relationship can be configured without the need for manual configuration by the user, thus improving the configuration efficiency of the relative positional relationship and thereby improving the sound source tracking efficiency.

[0124] Figure 5 This is a block diagram of a device calibration apparatus provided in one embodiment of this application. The apparatus is used in a sound source tracking system, which includes an image acquisition device and at least two audio acquisition devices. The image acquisition range of the image acquisition component includes the installation location of the at least two audio acquisition devices. During sound source tracking, the audio acquisition devices determine the angle data of the currently emitting target sound source relative to the audio acquisition devices, and send the device identifier of the audio acquisition devices and the angle data to the image acquisition devices for sound source tracking. Different audio acquisition devices acquire audio data of target sound sources in different areas. The apparatus includes at least the following modules: an image acquisition module 510, a device identification module 520, a location determination module 530, and a mapping establishment module 540.

[0125] The image acquisition module 510 is used to control the image acquisition device to scan the current sound source tracking environment and obtain an environmental image before performing sound source tracking;

[0126] Device identification module 520 is used to identify each audio acquisition device in the environmental image and obtain the device image area of ​​each audio acquisition device;

[0127] The position determination module 530 is used to determine the relative positional relationship between the audio acquisition device and the image acquisition device using the device image area of ​​each audio acquisition device, the image acquisition posture corresponding to the device image area, and the intrinsic and extrinsic parameters of the image acquisition device.

[0128] The mapping establishment module 540 is used to establish a mapping relationship between each device identifier and the relative position relationship, so that after the image acquisition device obtains the angle data carrying the device identifier, it can use the mapping relationship to determine the relative position relationship of the device identifier mapping, and use the relative position relationship and the angle data to perform sound source tracking on the target sound source.

[0129] For relevant details, please refer to the above method implementation examples.

[0130] Figure 6 This is a block diagram of a sound source tracking device provided in one embodiment of this application. The device is used in a sound source tracking system, which includes an image acquisition device and at least two audio acquisition devices; the image acquisition range of the image acquisition component includes the installation location of the at least two audio acquisition devices; the device includes at least the following modules: a data acquisition module 610, a relationship acquisition module 620, a location determination module 630, and a sound source tracking module 640.

[0131] The data acquisition module 610 is used to acquire angle data carrying the device identifier of the audio acquisition device sent by the audio acquisition device; the angle data is determined by the audio acquisition device after acquiring the audio data of the target sound source currently emitting sound;

[0132] The relationship acquisition module 620 is used to acquire a pre-established mapping relationship between each device identifier and the relative positional relationship between the audio acquisition device and the image acquisition device. The relative positional relationship is obtained by controlling the image acquisition device to scan the current sound source tracking environment to obtain an environmental image, identifying each audio acquisition device in the environmental image, and obtaining the device image region of each audio acquisition device. The device image region of each audio acquisition device, the image acquisition posture corresponding to the device image region, and the intrinsic and extrinsic parameters of the image acquisition device are used to determine the relationship.

[0133] The location determination module 630 is used to determine the relative positional relationship of the device identifier mapping using the mapping relationship;

[0134] The sound source tracking module 640 is used to control the image acquisition device to track the target sound source using the relative position relationship and the angle data.

[0135] For relevant details, please refer to the above method implementation examples.

[0136] It should be noted that the device calibration and sound source tracking devices provided in the above embodiments are only illustrated by the division of the above functional modules when performing device calibration and sound source tracking. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device calibration and sound source tracking device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device calibration and sound source tracking devices and device calibration and sound source tracking method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0137] Figure 7 This is a block diagram of an electronic device provided in one embodiment of this application. The electronic device may be an image acquisition device or an electronic device communicatively connected to the image acquisition device. The device includes at least a processor 701 and a memory 702.

[0138] Processor 701 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0139] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 are used to store at least one instruction, which is executed by the processor 701 to implement the device calibration and sound source tracking methods provided in the method embodiments of this application.

[0140] In some embodiments, the electronic device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 701, memory 702, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to: radio frequency circuitry, a touch display screen, audio circuitry, and a power supply.

[0141] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0142] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the device calibration and sound source tracking methods of the above method embodiments.

[0143] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program, which is loaded and executed by a processor to implement the device calibration and sound source tracking methods of the above method embodiments.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for calibrating equipment, characterized in that, A sound source tracking system is used, the sound source tracking system including an image acquisition device and at least two audio acquisition devices; the image acquisition device includes an image acquisition component, the image acquisition range of the image acquisition component including the installation location of the at least two audio acquisition devices; during sound source tracking, the audio acquisition device is used to determine the angle data of the currently emitting target sound source relative to the audio acquisition device, and sends the device identifier of the audio acquisition device and the angle data to the image acquisition device for sound source tracking; wherein, different audio acquisition devices acquire audio data of target sound sources in different areas; the method includes: Before performing sound source tracking, the image acquisition device is controlled to scan the current sound source tracking environment to obtain an environmental image; Identify each audio acquisition device in the environmental image and obtain the device image area for each audio acquisition device; Using the device image region of each audio acquisition device, the image acquisition posture corresponding to the device image region, and the intrinsic and extrinsic parameters of the image acquisition device, the relative positional relationship of the audio acquisition device relative to the image acquisition device is determined, including: identifying the key point positions of key points on the audio acquisition device in the device image region, and the pixel information of the audio acquisition device in the device image region; determining the posture angle of the audio acquisition device relative to the image acquisition device based on the key point positions; using the pixel information, the image acquisition posture, and the intrinsic and extrinsic parameters of the image acquisition device, determining the orientation and distance of the audio acquisition device relative to the image acquisition device, wherein the relative positional relationship includes the orientation, the distance, and the posture angle; wherein, determining the posture angle of the audio acquisition device relative to the image acquisition device based on the key point positions includes: determining the deflection angle of the plane where each key point is located relative to the acquisition plane based on the difference between the first distance between the positions of each key point when they are not parallel and the second distance between the corresponding key point positions when they are parallel, thereby obtaining the posture angle; The step of determining the orientation and distance of the audio acquisition device relative to the image acquisition device using the pixel information, the image acquisition posture, and the intrinsic and extrinsic parameters of the image acquisition device includes: using the focal length of the image acquisition component and the pixel position in the pixel information based on the principle of triangle similarity, transforming the pixel position to the camera coordinate system of the image acquisition component to obtain a first coordinate position; using the intrinsic and extrinsic parameters of the image acquisition device to transform the first coordinate position to the world coordinate system established based on the image acquisition device to obtain a second coordinate position; and determining the orientation and distance of the audio acquisition device relative to the image acquisition device based on the second coordinate position and the image acquisition posture. A mapping relationship is established between each device identifier and the relative positional relationship, so that after the image acquisition device obtains the angle data carrying the device identifier, it uses the mapping relationship to determine the relative positional relationship of the device identifier, and uses the relative positional relationship and the angle data to track the target sound source.

2. The method according to claim 1, characterized in that, The device outline of the audio acquisition device differs from the target outlines of other targets in the sound source tracking environment; the process of identifying each audio acquisition device in the environmental image and obtaining the device image region of each audio acquisition device includes: Detect the contours of each target in the environmental image; The image region of the target whose contour matches the contour of the device is determined to obtain the device image region of the audio acquisition device.

3. The method according to claim 1, characterized in that, The audio acquisition device is equipped with a calibration mark. The process of identifying each audio acquisition device in the environmental image and obtaining the device image area for each audio acquisition device includes: Detect the calibration markers in the environmental image to obtain the marker image region of the calibration markers; The device image region is determined based on the identified image region.

4. The method according to claim 1, characterized in that, The process of establishing the mapping relationship between each device identifier and the relative positional relationship includes: For each device identifier, a mapping instruction carrying the device identifier is sent to each audio acquisition device. The mapping instruction is used to trigger the audio acquisition device corresponding to the device identifier to perform a preset mapping action. The image acquisition device is controlled to recognize the preset mapping action; Upon recognizing the preset mapping action, a mapping relationship is established between the relative position of the audio acquisition device executing the preset mapping action and the device identifier.

5. The method according to claim 4, characterized in that, The process of establishing a mapping relationship between the relative position of the audio acquisition device performing the preset mapping action and the device identifier includes: Obtain the image acquisition posture corresponding to the recognition of the preset mapping action; Determine the relative positional relationship that matches the image acquisition posture corresponding to the preset mapping action; Establish a mapping relationship between the matching relative positional relationship and the device identifier.

6. The method according to claim 4, characterized in that, The step of controlling the image acquisition device to recognize the preset mapping action includes: controlling the image acquisition device to acquire images with each unmapped relative position relationship and recognizing the preset mapping action; Accordingly, when the preset mapping action is identified, a mapping relationship is established between the relative position of the audio acquisition device performing the preset mapping action and the device identifier, including: when the preset mapping action is identified, establishing a mapping relationship between the currently used relative position and the device identifier.

7. A sound source tracking method, characterized in that, A sound source tracking system, the sound source tracking system comprising an image acquisition device and at least two audio acquisition devices; the image acquisition device comprising an image acquisition component, the image acquisition range of the image acquisition component including the installation location of the at least two audio acquisition devices; the method comprising: The angle data sent by the audio acquisition device, which carries the device identifier of the audio acquisition device, is obtained; the angle data is determined by the audio acquisition device after acquiring the audio data of the target sound source currently emitting sound. A pre-established mapping relationship is obtained between each device identifier and the relative positional relationship between the audio acquisition device and the image acquisition device. The relative positional relationship is obtained by controlling the image acquisition device to scan the current sound source tracking environment to obtain an environmental image, identifying each audio acquisition device in the environmental image, and obtaining the device image region of each audio acquisition device. The device image region of each audio acquisition device, the image acquisition posture corresponding to the device image region, and the intrinsic and extrinsic parameters of the image acquisition device are used to determine the relationship. The relative positional relationship of the device identifier mapping is determined using the mapping relationship. Using the relative positional relationship and the angle data, the image acquisition device is controlled to perform sound source tracking on the target sound source; The mapping relationship is established using the device calibration method as described in claim 1.

8. An electronic device, characterized in that, The device includes a processor and a memory; the memory stores a program, which is loaded and executed by the processor to implement the device calibration method as described in any one of claims 1 to 6; or to implement the sound source tracking method as described in claim 7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement the device calibration method as described in any one of claims 1 to 6; or to implement the sound source tracking method as described in claim 7.

Images