Directing control method, device and equipment

Abstract

The invention discloses a director control method, device and equipment, and relates to the technical field of communication. According to the application, the terminal equipment collects a sound signal through a microphone to carry out sound source positioning, and collects a target image through a camera component; the terminal equipment locates a sound image matching area of the sound source object in the target image according to the sound signal, and then further determines the sound source object from the sound image matching area according to the sound signal; and the terminal equipment executes a program director operation on the sound source object. The sound and image combined mode can more accurately position the sound source object, thereby realizing accurate director control.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a director control method, device and equipment. Background Art

[0002] Directing refers to controlling the camera to shoot the sound source object (people or objects) in the scene based on real-time shooting requirements to output video images during video shooting. For example, in a video conference, the camera can be controlled to shoot the current speaker, and when the speaker changes, the camera can be controlled to shoot the new speaker. During the directing process, in order to obtain a video image containing the sound source object, the camera shooting direction can be adjusted, and a partial capture can be made in the video image.

[0003] With the development of computer technology, automatic directing has developed rapidly and is gradually replacing manual directing. At present, automatic directing can be performed through intelligent directing equipment. The processing process of automatic directing by intelligent directing equipment can be: the intelligent directing equipment measures the sound signal of the sound source object (which can be a speaker) through the microphone array, obtains the direction and distance of the sound source point relative to the microphone, and then directs the sound source object. However, since there will be angle deviations when positioning based on sound signals, if there are dense people around the sound source object, the sound source object may not be accurately located, resulting in poor accuracy of directing. Summary of the invention

[0004] The embodiments of the present application provide a director control method, device and equipment for improving the accuracy of director control.

[0005] In a first aspect, an embodiment of the present application provides a director control method, which can be executed by a terminal device, and the terminal device can include a camera component and a microphone. The terminal device collects a target image in an environment through the camera component, and collects a sound signal emitted by a sound source object in the environment through the microphone; the terminal device locates a sound-image matching area of ​​the sound source object in the target image according to the sound signal; and determines the sound source object from multiple objects included in the sound-image matching area according to the sound signal; the terminal device performs a director operation on the sound source object.

[0006] Through the above method, when the terminal device performs director control, the sound signal is collected by the microphone to locate the sound source, and the target image is collected by the camera component. The sound and image are matched with the target image based on the result of the sound signal positioning, and the sound and image matching area is determined from the target image. The sound and image matching area includes all possible sound source objects identified based on the sound positioning technology, and then the sound source object is further determined from the sound and image matching area. This combination of sound and image can locate the sound source object more accurately.

[0007] In a possible implementation, the terminal device determines a reference position of the sound source object relative to the terminal device based on the sound signal; the terminal device performs sound and image matching on the reference position and the target image, and determines a sound and image matching area from the target image, wherein the sound and image matching area includes the sound source object.

[0008] Through the above method, the terminal device locates the sound source of the sound source object according to the sound signal, determines the reference position of the sound source object through sound source positioning, and then determines the sound-image matching area including the sound source object from the target image according to the reference position. The sound-image matching area includes multiple objects. Taking into account that the sound source localization technology may have positioning deviations, the sound-image matching area where the sound source object may exist is determined from the target image, and the sound source object is further determined from the sound source matching area, so that the sound source object can be accurately determined.

[0009] In a possible implementation, the terminal device extracts voiceprint features from the sound signal; based on the extracted voiceprint features, determines the features of the sound source object from the correspondence between the object features and the voiceprint features; the terminal device determines the sound source object from a plurality of objects included in the sound and image matching area based on the features of the sound source object.

[0010] Through the above method, since the correspondence between the object feature and the voiceprint feature is the binding relationship between the voiceprint feature of the object determined in the historical broadcast process and the object feature, the terminal device determines the object feature corresponding to the voiceprint feature of the sound signal received this time from the correspondence between the object feature and the voiceprint feature, and the object feature is the feature of the sound source object determined in the historical broadcast process. After determining the feature of the sound source object, the terminal device can accurately determine the sound source object from multiple objects included in the sound and image matching area.

[0011] In a possible implementation, the terminal device determines, based on the features of the sound source object and the features of the multiple objects, an object whose feature similarity with the features of the sound source object is greater than a threshold from among the multiple objects as the sound source object.

[0012] Through the above method, the terminal device can accurately determine the sound source object from multiple objects included in the sound and image matching area by calculating the feature similarity.

[0013] In a possible implementation, the terminal device determines a reference object from the multiple objects according to a reference position located based on a sound signal; if the feature similarity between the feature of the reference object and the feature of the sound source object is greater than a threshold, the terminal device determines that the reference object is the sound source object.

[0014] Through the above method, the reference object determined by the terminal device according to the reference position is likely to be the sound source object that emits the sound signal. The terminal device first matches the features of the reference object with the features corresponding to the voiceprint features of the sound signal stored in the corresponding relationship. When the feature similarity between the features of the reference object and the target features is greater than a threshold, the reference object can be directly, quickly, and accurately determined to be the sound source object.

[0015] In a possible implementation, if the feature similarity between the features of the reference object and the features of the sound source object is not greater than a threshold, the terminal device determines that other objects whose feature similarity with the features of the sound source object is greater than the threshold are the sound source objects, and the other objects are objects among the multiple objects other than the reference object.

[0016] Through the above method, if the feature similarity between the feature of the reference object and the feature of the sound source object is not greater than the threshold, it means that there may be a positioning deviation in the sound source localization process, resulting in the determined reference object not being the sound source object. At this time, the terminal device can further determine the sound source object from other objects in the sound and image matching area based on the feature similarity. In this way, even if there is a deviation in sound source localization, the terminal device can still accurately determine the sound source object.

[0017] In a possible implementation, if the feature similarities between the features of the multiple objects and the features of the sound source object are not greater than a threshold, the terminal device determines that the reference object is the sound source object, or determines that the multiple objects are the sound source object.

[0018] Through the above method, when the feature similarities between the features of multiple objects included in the sound and image matching area and the sound source object do not meet the similarity requirements, the terminal device can directly use the reference object determined based on the sound source localization technology as the sound source object, or the multiple objects included in the sound and image matching area, to avoid the inability to locate the sound source object or the incorrect location of the sound source object, thereby improving the reliability of the director's control.

[0019] In a possible implementation, the terminal device updates the object features corresponding to the voiceprint features of the sound signal in the corresponding relationship according to the determined features of the sound source object.

[0020] Through the above method, after locating the sound source object each time, the terminal device updates the correspondence between the object features and the voiceprint features to ensure that the correspondence between the object features and the voiceprint features is in a dynamically updated state.

[0021] In a possible implementation, if the feature similarities between the features of the multiple objects and the features of the sound source object are not greater than a threshold, the sound source object is determined from the multiple objects based on the screening parameters corresponding to each of the multiple objects; the screening parameters represent the degree of participation of the object in the historical directing process.

[0022] Through the above method, if the feature similarities between the features of the multiple objects and the features of the sound source object are not greater than the threshold, the terminal device can accurately determine the sound source object based on the screening parameters.

[0023] In a possible implementation, the terminal device updates the object features corresponding to the voiceprint features of the sound signal in the corresponding relationship according to the features of the multiple objects.

[0024] Through the above method, after locating the sound source object, the terminal device updates the correspondence between the object features and the voiceprint features to ensure that the correspondence between the object features and the voiceprint features is in a dynamically updated state.

[0025] In a possible implementation, the terminal device performs lip movement detection on multiple objects included in the audio-visual matching area, and after determining that no lip movement event is detected, determines the sound source object from the multiple objects included in the audio-visual matching area based on the sound signal.

[0026] Through the above method, since the sound and image matching area is an area determined according to the sound signal of the sound source object detected by the terminal device, the multiple objects included in the sound and image matching area are likely to be the sound source objects; at this time, lip movement detection is performed on the multiple objects in the sound and image matching area. If it is determined that the lip movement detection fails, it means that the multiple objects in the sound and image matching area in the current target image are not speaking. In this case, the terminal device further determines the sound source object from the sound and image matching area based on the feature similarity, thereby providing compatible solutions in various situations and improving the reliability of the director's control.

[0027] In a possible implementation, if the terminal device detects a lip movement event in the sound and image matching area, an object corresponding to the lip movement event among the multiple objects is determined as the sound source object.

[0028] Through the above method, if a lip movement event is detected, it means that there is a speaking object among the multiple objects included in the sound image matching area, and the speaking object is the sound source object, so that the sound source object can be quickly and accurately determined.

[0029] In the second aspect, the embodiment of the present application further provides a director control device, which has the function of implementing the behavior of the terminal device in the method example of the first aspect above. The beneficial effects can be found in the description of the first aspect and will not be repeated here. The functions can be implemented by hardware or by executing corresponding software through hardware. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the structure of the director control device includes a signal acquisition module, a positioning identification module, and a director control module. These modules can execute the corresponding functions in the method example of the first aspect above. Please refer to the detailed description in the method example for details, which will not be repeated here.

[0030] In a third aspect, an embodiment of the present application further provides a terminal device, comprising at least one processor and a memory; wherein the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the at least one processor, the terminal device executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0031] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0032] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program or instructions. When the computer program or the instructions are run on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the architecture of a system provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the structure of a director control device provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0038] Figure 6A schematic diagram of an optional system architecture provided for an embodiment of the present application;

[0039] Figure 7 A flowchart of a broadcast director control method provided in an embodiment of the present application;

[0040] Fig. 8A A schematic diagram of a microphone array provided in an embodiment of the present application;

[0041] Figure 8B A schematic diagram of sound source localization provided in an embodiment of the present application;

[0042] Fig. 9 A schematic diagram of sound source localization in a conference scene provided in an embodiment of the present application;

[0043] Fig.10 A schematic diagram of a target image provided in an embodiment of the present application;

[0044] Fig.11 A flowchart of a broadcast director control method provided in an embodiment of the present application;

[0045] Fig.12 A flowchart of a broadcast director control method provided in an embodiment of the present application;

[0046] Fig.13 A flowchart of a director control method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0048] In the embodiments of the present application, "multiple" refers to two or more than two. In view of this, in the embodiments of the present application, "multiple" can also be understood as "at least two". "At least one" can be understood as one or more, for example, one, two or more. For example, including at least one means including one, two or more, and there is no restriction on which ones are included. For example, including at least one of A, B and C, then A, B, C, A and B, A and C, B and C, or A and B and C may be included. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / ", unless otherwise specified, generally indicates that the previously associated objects are in an "or" relationship.

[0049] Unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects and are not used to limit the order, timing, priority or importance of the multiple objects.

[0050] The system architecture applicable to the director control solution provided in the embodiment of the present application can be as follows Figure 1 As shown, the system architecture may include a director control device 100, a camera assembly 101, a microphone 102, and at least one remote device 200. The director control device 100 is connected to the at least one remote device 200, and the director control device 100 and the at least one remote device 200 may interact directly based on a communication network, or a micro controller unit (MCU) may be included between the director control device 100 and the remote device 200, and the MCU may be used for real-time synchronous interactive communication between the director control device 100 and the remote device 200.

[0051] Among them, the director control device 100 is connected to the camera component 101 and the microphone 102, and the director control device 100 can directly interact with the camera component 101 and the microphone 102.

[0052] As for the camera component 101, the camera component 101 can capture the target image in the environment and transmit the captured target image to the director control device 100. The present application does not limit the specific type of the camera component 101. For example, the camera component 101 can be a camera (wherein the camera component 101 can be a panoramic camera or a director camera), a camera, or can be other monitoring devices with video or image shooting functions. Any component that can perform video or image acquisition can be used as the camera component 101. For another example, the camera component 101 may include multiple cameras. For example, the camera component 101 includes a panoramic camera and a director camera. The panoramic camera can be used to capture the target image in the environment, and the director camera can perform the director operation under the control of the director control device 100.

[0053] The microphone 102 can collect sounds in the environment. For example, after a sound source object in the environment makes a sound, the microphone 102 can collect the sound corresponding to the sound source in the environment and convert it into a sound signal. The microphone 102 transmits the collected sound signal to the director control device 100. The system architecture of the embodiment of the present application may include one or more microphones 102; for example, multiple microphones 102 can form a microphone array. The embodiment of the present application can flexibly set the position of each microphone in the microphone array according to actual application requirements.

[0054] The director control device 100 can receive the target image captured by the camera component 101 and the sound signal captured by the microphone 102. The director control device 100 can locate the sound source object from the target image according to the sound signal and perform the director operation on the identified sound source object. For example, after the director control device 100 locates the sound source object, the director control device 100 can control the shooting angle of the camera component 101 so that the camera component 101 takes a close-up shot of the sound source object; for another example, after the director control device 100 locates the sound source object and obtains the image captured by the camera component 101, it extracts the area where the sound source object is located in the image to obtain a close-up image of the sound source object.

[0055] In addition, the director control device 100 provides an interactive interface. The user can interact with the director control device 100 through the interactive interface. The user can configure the tasks to be performed for the director control device 100 through the interactive interface. For example, the user can configure the director control tasks of the director control device 100 through the interactive interface. The interactive interface mentioned in the embodiment of the present application refers to a function for interacting with the user provided to the user, and the embodiment of the present application does not limit the specific presentation method of the interactive interface. For example, the interactive interface can be understood as client software deployed on the user's device or a dedicated, user-oriented client device. For another example, the interactive interface can be understood as a website that the user can access or a web page built based on hypertext markup language (HTML).

[0056] The remote device 200 can receive the video generated during the director control device 100's director control process, store the received video or play the video, or send the video sent by the director control device 100 to other devices. The remote device 200 can be any terminal device or server.

[0057] Here, for the sake of convenience, each device or component is described as an independent device. In actual scenarios, the specific deployment form of the device or component in the system is not limited. Figure 1 The director control device 100, camera component 101 and microphone 102 included in the system architecture shown can be deployed on different devices respectively, or partially deployed on the same device (for example, the camera component 101 and microphone 102 are deployed on the same device, and the director control device 100 is deployed on another device), or the director control device 100, camera component 101 and microphone 102 can be deployed on the same device.

[0058] Regarding the form of the director control device 100, the director control device 100 can be implemented by software or hardware. As an example, the implementation of the director control device 100 is described below.

[0059] The director control device 100 is taken as an example of a software functional unit, and the director control device 100 may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, and other computing devices. Furthermore, the computing device may be one or more. The camera component 101 may be a camera component in a computing device, or the camera component 101 may be a camera component external to the computing device; the microphone 102 may be a microphone in a computing device, or the microphone 102 may be a microphone external to the computing device.

[0060] For example, the director control device 100 may include codes running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application can be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region can include multiple AZs.

[0061] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway must be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0062] The director control device 100 is taken as an example of a hardware functional unit, and the director control device 100 may include at least one computing device, such as a server, etc. Alternatively, the director control device 100 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0063] The multiple computing devices included in the director control device 100 can be distributed in the same region or in different regions. The multiple computing devices included in the director control device 100 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the director control device 100 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs. The camera component 101 can be a camera component in the computing device, or the camera component 101 can be a camera component external to the computing device; the microphone 102 can be a microphone in the computing device, or the microphone 102 can be a microphone external to the computing device.

[0064] The structure of the director control device 100 is described below. Figure 2 As shown, the present application also provides a director control device 100, which includes a signal acquisition module 110, a positioning identification module 120, and a director control module 130.

[0065] The signal acquisition module 110 is connected to the camera assembly 101 and the microphone 102 . The signal acquisition module 110 acquires the target image in the environment captured by the camera assembly 101 , and acquires the sound signal emitted by the sound source object in the environment captured by the microphone 102 .

[0066] The positioning and identification module 120 can perform sound-image matching based on the target image and sound signal acquired by the signal acquisition module 110, locate the sound source object in the sound-image matching area in the target image, and further determine the sound source object from multiple objects included in the sound-image matching area based on the sound signal.

[0067] When locating the sound source object in the sound image matching area of ​​the target image, the positioning recognition module 120 can determine the reference position of the sound source object according to the sound signal; for example, the embodiment of the present application can collect the sound signal of the sound source object through multiple microphones. Since the positions of the multiple microphones are different, the time when the sound signal of the sound source object reaches each microphone is different. The positioning recognition module 120 can determine the reference position of the sound source object in the environment according to the time difference when the sound signal of the sound source object reaches each microphone. Then the positioning recognition module 120 performs sound image matching on the reference position and the target image to locate the sound image matching area of ​​the sound source object in the target image; for example, the reference position of the sound source object obtained by the positioning recognition module 120 is the position of the sound source object in the physical space, and the reference position can be converted into the plane position of the sound source object in the target image. According to the determined plane position of the sound source object in the target image, the sound image matching area in the target image is obtained.

[0068] The positioning and identification module 120 can also determine the characteristics of the sound source object from the correspondence between the object characteristics and the voiceprint characteristics according to the voiceprint characteristics of the sound signal; and then determine the sound source object from multiple objects included in the sound image matching area according to the characteristics of the sound source object.

[0069] When determining the sound source object from the multiple objects included in the sound image matching area, the positioning and recognition module 120 may respectively determine the feature similarity between the features of the sound source object and the features of the multiple objects, and select the feature similarity from the features of the sound source object and the features of the multiple objects according to the feature similarity. For example, after the positioning and recognition module 120 determines the feature similarity between the feature of the sound source object and each of the multiple objects included in the sound image matching area, the object with the highest feature similarity among the multiple objects is determined as the sound source object.

[0070] In a possible implementation, the positioning recognition module 120 can determine a reference object from multiple objects, and first determine the feature similarity between the features of the reference object and the features of the sound source object. When the feature similarity between the features of the reference object and the features of the sound source object is greater than a threshold, the positioning recognition module 120 can determine that the reference object is a sound source object. When the feature similarity between the features of the reference object and the features of the sound source object is not greater than a threshold, the positioning recognition module 120 can determine other objects whose feature similarity is greater than the threshold from multiple objects as sound source objects. When the feature similarities between the features of multiple objects included in the sound and image matching area and the features of the sound source object are not greater than a threshold, the positioning recognition module 120 can determine that the reference object is a sound source object, or determine that multiple objects are sound source objects; or, the positioning recognition module 120 can also determine the sound source object from multiple objects according to the screening parameters corresponding to each of the multiple objects, wherein the screening parameters represent the degree of participation of the object in the historical directing process.

[0071] After the positioning and recognition module 120 determines the sound source object, the positioning and recognition module 120 may also update the stored correspondence between the object features and the voiceprint features. For example, the positioning and recognition module 120 updates the object features corresponding to the voiceprint features of the sound signal emitted by the sound source object in the correspondence to the features corresponding to the determined sound source.

[0072] After determining the sound-image matching area of ​​the target image, the positioning and recognition module 120 can also perform lip movement detection on multiple objects included in the sound-image matching area. When a lip movement event is detected, the positioning and recognition module 120 determines the object corresponding to the lip movement event among the multiple objects as the sound source object. Correspondingly, when no lip movement event is detected, the positioning and recognition module 120 can determine the sound source object according to the above-mentioned method of determining the sound source object from the sound-image matching area.

[0073] The director control module 130 may perform a director operation on the sound source object determined by the positioning identification module 120. For example, the director control module 130 may control the shooting angle of the camera assembly so that the camera assembly takes a close-up shot of the sound source object; for another example, the director control module 130 extracts the area where the sound source object is located in the target image to obtain a close-up image of the sound source object.

[0074] Among them, the signal acquisition module 110, the positioning identification module 120, and the director control module 130 can all be implemented by software, or can be implemented by hardware. The implementation of the signal acquisition module 110, the positioning identification module 120, and the director control module 130 is similar to the implementation of the director control device 130. For details, please refer to the above content and will not be repeated here.

[0075] In the following introduction, for the convenience of description, the director control device 100 is taken as an example of a terminal device.

[0076] The terminal device in the embodiment of the present application may be a dedicated conference terminal, or the terminal device may also be a portable device, such as a mobile phone, a tablet computer, a wearable device with wireless communication function (such as a watch, a bracelet, etc.), a vehicle-mounted terminal device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), smart home devices (for example, smart TVs, smart speakers, etc.), smart robots, workshop equipment, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, flight equipment (for example, smart robots, drones, airplanes), etc.

[0077] The following introduces the director control scheme of the embodiment of the present application by taking several specific application scenarios as examples.

[0078] Application scenario 1: video conferencing scenario.

[0079] In application scenario 1, the terminal device may be a conference terminal. Figure 3 As shown, in a video conference scenario, the environment may include a conference terminal and multiple conference participants. During the conference, the conference terminal captures the target image of the conference environment through a camera component, and captures the sound signal emitted by the sound source object in the conference environment through a microphone. The target image captured by the conference terminal may include multiple conference participants. According to the sound signal, the sound source object can be located from the multiple conference participants included in the target image, thereby performing a directing operation on the sound source object.

[0080] Application scenario 2: live broadcast scenario.

[0081] In application scenario 2, the terminal device may be a live broadcast terminal, such as a mobile phone, tablet computer, laptop computer, etc. Figure 4As shown, in a live broadcast scene, the environment may include a live broadcast terminal and multiple live broadcast objects. During the live broadcast process, the live broadcast terminal collects the target image of the live broadcast environment through the camera component, and collects the sound signal emitted by the sound source object in the live broadcast environment through the microphone. The target image collected by the live broadcast terminal may include multiple live broadcast objects. According to the sound signal, the sound source object can be located from the multiple live broadcast objects included in the target image, so as to perform a directing operation on the sound source object.

[0082] Application scenario 3: program recording scenario.

[0083] The program recording scene may include multiple photographed objects, and the program video is obtained by photographing the multiple photographed objects through the camera component of the terminal device. During the program recording process, the terminal device can capture the target image of the recording environment through the camera component, and the target image may include multiple photographed objects; the terminal device collects the sound signal emitted by the sound source object in the recording environment through the microphone, and according to the sound signal, the sound source object can be located from the multiple photographed objects included in the target image, so as to perform the directing operation on the sound source object.

[0084] It should be noted that the above-mentioned specific application scenarios are merely examples of applicable application scenarios of the director control solution of the embodiment of the present application. The director control solution of the embodiment of the present application can also be applied to other application scenarios, and the embodiment of the present application is not limited to this.

[0085] Figure 5 The terminal device 500 includes a camera assembly 501 , a microphone 502 , a speaker 503 , an audio module 504 , a processor 505 , a memory 506 , a communication module 507 , and a display screen 508 .

[0086] The camera component 501 is used to capture static images or videos. For example, the camera component 501 of the terminal device of the embodiment of the present application may include one or more cameras. When the camera component 501 includes one camera, the camera may be a panoramic camera for panoramic shooting of the environment, and when director control is required, close-up shooting of the sound source object can be achieved by adjusting the angle of the panoramic camera. When the camera component 501 includes multiple cameras, the multiple cameras may include a panoramic camera and a director camera, the panoramic camera is used to take a panoramic shot of the environment, and the director camera is used to take a close-up shot of the sound source object.

[0087] The terminal device can implement audio functions through microphone 502, speaker 503, audio module 504, etc. For example, audio playback, recording, sound signal collection, etc. In addition, the microphone 502 of the embodiment of the present application can be a microphone array, which includes multiple microphones arranged side by side. The terminal device uses microphones at different positions in the microphone array to measure the sound signal emitted by the sound source object. Since the time for the sound signal to reach each microphone is delayed to varying degrees, combined with the position of each microphone in the microphone array, the angle and distance of the sound source object relative to the terminal device can be obtained, thereby realizing sound source positioning.

[0088] The processor 505 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated in one or more processors. Among them, the controller may be the nerve center and command center of the intelligent device. The controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. A memory may also be set in the processor 505 for storing instructions and data. In some embodiments, the memory in the processor 505 is a high-speed cache memory. The memory may store instructions or data that the processor 505 has just used or circulated. If the processor 505 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor 505 is reduced, and the efficiency of the system is improved.

[0089] The memory 506 can be used to store program codes, such as enabling the processor 505 to execute the program codes in the memory 506 to implement the director control method of the embodiment of the present application.

[0090] The wireless communication function of the terminal device can be implemented through the communication module 507, which may include an antenna, a mobile communication module, a wireless communication module, a modulation and demodulation processor, and a baseband processor. The antenna is used to transmit and receive electromagnetic wave signals. The antenna in the terminal device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antenna. For example, the antenna can be reused as a diversity antenna of a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0091] The mobile communication module can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to terminal devices. The mobile communication module may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module can receive electromagnetic waves by an antenna, and filter, amplify, and process the received electromagnetic waves, and transmit them to a modulation and demodulation processor for demodulation. The mobile communication module can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna. In some embodiments, at least some of the functional modules of the mobile communication module may be arranged in the processor 505. In some embodiments, at least some of the functional modules of the mobile communication module may be arranged in the same device as at least some of the modules of the processor 505.

[0092] The wireless communication module can provide wireless communication solutions including wireless local area networks (WLAN) (such as WiFi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), NFC, infrared technology (IR), etc., which are applied to smart devices. The wireless communication module can be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 505. The wireless communication module can also receive the signal to be sent from the processor 505, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0093] In some embodiments, the antenna of the terminal device can be coupled to the mobile communication module, or the antenna can be coupled to the wireless communication module, so that the terminal device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0094] The display screen 508 is used to display the display interface of the application, etc. The display screen 508 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the terminal device may include 1 or N display screens 508, where N is a positive integer greater than 1.

[0095] Understandably, Figure 5 The components shown do not constitute a specific limitation on the terminal device. The terminal device in the embodiment of the present application may include Figure 5 In addition, Figure 5 The combination / connection relationship between the components can also be adjusted and modified.

[0096] For example, taking the terminal device as a conference terminal as an example, the communication process between the conference terminal and the remote device is introduced as follows: Figure 6 The system architecture shown includes a conference terminal 600 and a remote device 610 ; the conference terminal 600 and the remote device 610 can communicate with each other via an MCU 620 .

[0097] The conference terminal 600 includes a processor 601 , an audio module 602 , a speaker 603 , a microphone 604 , a camera component 605 , a display screen 606 , a video module 607 and a communication module 608 .

[0098] In some embodiments, the audio module 602 and the video module 607 may be integrated into the processor 601 .

[0099] The audio module 602 may include a sound collection unit, a sound source positioning unit, and a voiceprint recognition unit. The microphone 604 collects sound signals in the environment. The sound collection unit in the audio module 602 obtains the sound signal collected by the microphone 604, and uses the sound source positioning unit to locate the sound source object that emits the sound signal, and determines the reference position of the sound source object relative to the conference terminal; the voiceprint recognition unit can extract voiceprint features from the sound signal. Among them, the functions performed by the sound collection unit can also be performed by the above Figure 2The signal acquisition module 110 in the above embodiment is executed, and the functions performed by the sound source localization unit and the voiceprint recognition unit can also be performed by the above Figure 2 The positioning identification module 120 is executed.

[0100] The camera component 605 collects the target image in the environment. After the processor 601 obtains the target image, it processes the target image through the video module 607.

[0101] The video module 607 may include an image acquisition unit, a feature detection unit, an audio-visual matching unit, a lip movement detection unit, an encoding unit, and a decoding unit. Among them, the image acquisition unit is used to obtain the target image in the environment captured by the camera component 605; the feature detection unit is used to detect the target image and extract the object features of the object included in the target image; the audio-visual matching unit is used to perform audio-visual matching with the target image based on the reference position obtained by locating the sound signal, determine the position of the reference position in the target image, and determine the audio-visual matching area in the target image; the lip movement detection unit is used to perform lip movement detection on the object included in the target image, and determine whether the target image or the audio-visual matching area includes a sound source object that is making a sound; the encoding unit is used to encode the image captured by the camera component; and the decoding unit is used to decode the received image. Among them, the functions performed by the image acquisition unit can also be performed through the above Figure 2 The signal acquisition module 110 in the embodiment is executed, and the functions of the feature detection unit, the sound image matching unit, and the lip movement detection unit can also be performed by the above Figure 2 The positioning identification module 120 is executed.

[0102] The processor 601 may locate the sound source object according to the sound image matching area in the target image, and perform a directing operation on the sound source object.

[0103] The display screen 606 may be used to display video images. For example, the displayed video images may be video images captured during the conference directing process.

[0104] During the director control process, the audio module 602 obtains the audio signal generated by the sound signal in the environment, and the video module 607 obtains the video signal obtained by shooting the environment. The communication module 608 sends the audio signal and the video signal to the remote device 610 through the MCU 620. Optionally, when the decoding capability of the remote device is insufficient, the audio and video signals can be decoded by the MCU 620 and sent to the remote device 610.

[0105] The following embodiments describe in detail the director control method provided in the embodiments of the present application in conjunction with the accompanying drawings.

[0106] The following is combined with Figure 7A broadcast control method provided in an embodiment of the present application is described, and the method comprises the following steps:

[0107] Step 701: The terminal device collects a target image in the environment through a camera component, and collects a sound signal emitted by a sound source object in the environment through a microphone.

[0108] The terminal device of the embodiment of the present application can be deployed in an environment where director control is required, such as a video conference environment, a live broadcast environment, a program recording environment, a teaching environment, etc. In this environment, the terminal device is in a working state. In the working state, the terminal device collects a target image in the environment through a camera component, and the target image may include multiple objects in the environment; at the same time, the terminal device collects sound signals emitted by sound source objects in the environment through a microphone.

[0109] In order to ensure the synchronization of sound and image, the target image and sound signal based on which the sound source object is located in the embodiment of the present application are collected at the same time.

[0110] In a possible implementation, the microphone in the embodiment of the present application may be a microphone array. Fig. 8A As shown, the microphone array includes multiple microphones arranged side by side, and each microphone in the microphone array can independently collect sound signals in the environment.

[0111] Step 702: The terminal device locates the sound-image matching area of ​​the sound source object in the target image according to the sound signal.

[0112] The terminal device can collect sound signals in the environment through a microphone array. Based on the sound signals, the terminal device can locate the sound source object in the environment and determine the reference position of the sound source object relative to the terminal device.

[0113] During implementation, the terminal device can locate the sound source object that emits the sound signal based on the time difference between the sound signal reaching each microphone in the microphone array and the relative positions between the microphones in the microphone array.

[0114] like Figure 8B In the sound source localization schematic diagram shown, each microphone in the microphone array receives the sound signal emitted by the sound source object, and the terminal device can determine the relative position of the sound source object and the reference point in the microphone array (for example, the reference point can be the center point of the microphone array) based on the time difference of the sound signal reaching each microphone and the interval between each microphone in the microphone array. The relative position of the sound source object and the reference point in the microphone array includes but is not limited to distance and / or angle. In the embodiment of the present application, the relative position of the sound source object obtained by the terminal device relative to the reference point in the microphone array can be used as the reference position of the sound source object relative to the terminal device.

[0115] After the terminal device determines the reference position of the sound source object relative to the terminal device, since the reference position determined by the terminal device is the reference position of the sound source object relative to the terminal device in physical space, the reference position is a spatial position. The terminal device can match the reference position determined based on the sound signal with the target image; for example, the terminal device converts the reference position and determines the plane position of the reference position in the target image, thereby achieving sound and image matching based on the sound signal and the target image. The terminal device can use the plane position of the determined reference position in the target image as the target position of the sound source object in the target image determined based on the sound signal.

[0116] Since sound source localization often has certain errors, for example, the sound source object determined based on the sound signal may have an angle deviation of about 3° relative to the reference position of the terminal device. Considering the positioning accuracy of sound source localization, directly determining the sound source object in the target image based on the reference position may lead to positioning errors. For example, Fig. 9 In the conference scene shown, the participants are located on both sides of the long table. If object A emits a sound signal, the conference terminal locates the sound signal collected by the microphone. Object A and object B are relatively close to the reference position of the terminal device. When there is a positioning deviation θ, object B may be identified as the sound source object.

[0117] Based on the above problems, after the terminal device of the embodiment of the present application determines the reference position of the sound source object relative to the terminal device according to the sound signal, the terminal device can determine the sound and image matching area in the target image according to the target position of the sound source object in the target image. For example, the terminal device can determine the sound and image matching area in the target image according to the target position of the sound source object in the target image and the preset positioning angle deviation; for example, the terminal device uses the target position of the sound source object in the target image as a dot, determines the radius according to the preset positioning angle deviation, and determines the sound and image matching area in the target image according to the determined dot and radius; wherein the preset positioning angle deviation can be a preset value, and the preset value can be a value obtained by simulation or an empirical value of a person skilled in the art. For another example, the terminal device can use the target position of the sound source object in the target image as a dot, and determine the sound and image matching area in the target image according to the preset radius; wherein the value of the preset radius can be a preset value, and the value of the preset radius can be a value obtained by simulation or an empirical value of a person skilled in the art. Fig.10 As shown in the target image, it is assumed that point O is the target position of the sound source object in the target image determined based on sound source localization. Fig.10 The circular area shown in is the sound and image matching area, wherein the radius of the sound and image matching area can be determined according to a preset positioning angle deviation.

[0118] Step 703: The terminal device determines the sound source object from a plurality of objects included in the sound image matching area according to the sound signal.

[0119] After the terminal device determines the sound and image matching area from the target image, if the sound and image matching area only includes one object, the terminal device may determine that the object in the sound and image matching area is the sound source object.

[0120] When a plurality of objects are included in the sound image matching area, the terminal device further determines a sound source object from among the plurality of objects included in the sound image matching area.

[0121] For example, the terminal device can perform lip movement detection on multiple objects included in the sound and image matching area. If a lip movement event is detected, the object corresponding to the lip movement event among the multiple objects is determined as the sound source object; if no lip movement event is detected, the sound source object is further determined from the multiple objects included in the sound and image matching area based on the sound signal.

[0122] In implementation, when the terminal device performs lip movement detection, it can detect the movement of the lip area of ​​each object in the image based on the target image and the continuous n frames of images collected before the target object, and extract the continuous lip shape change features to determine whether a lip movement event is detected. If the lip area of ​​an object in the sound and image matching area changes its lip shape, it is determined that a lip movement event is detected for the object, and the object can be determined to be the sound source object.

[0123] Since the sound image matching area is an area determined based on the sound signal of the sound source object detected by the terminal device, the multiple objects included in the sound image matching area are likely to be the sound source object; at this time, lip movement detection is performed on the multiple objects in the sound image matching area. If a lip movement event is detected, it means that there is an object that is speaking among the multiple objects included in the sound image matching area, and the object that is speaking is the sound source object. Therefore, after determining that the sound image matching area includes multiple objects, the terminal device can perform lip movement detection on the multiple objects, and after detecting the lip movement event, the sound source object can be quickly located from the sound image matching area.

[0124] For another example, after determining the sound-image matching area of ​​the target image, the terminal device may further determine the sound source object from the multiple objects included in the sound-image matching area directly according to the sound signal.

[0125] The following describes a process in which the terminal device further determines a sound source object from a plurality of objects included in the sound image matching area according to the sound signal.

[0126] Step 1: The terminal device extracts the voiceprint features of the sound signal and determines the features of the sound source object from the correspondence between the object features and the voiceprint features.

[0127] For the convenience of description, the features of the sound source object determined from the correspondence between the object features and the voiceprint features are referred to as target features below.

[0128] In the embodiment of the present application, the correspondence between object features and voiceprint features can be stored in a feature matching library. The correspondence between object features and voiceprint features in the feature matching library can be generated during multiple director control processes based on the voiceprint features of the sound signal collected by the terminal device and the object features of the determined sound source object, and the correspondence between object features and voiceprint features in the feature matching library can be continuously updated with the director control process.

[0129] The characteristics of an object in the embodiment of the present application may refer to the attributes or characteristics of an object, which are the unique identification characteristics of an object. The characteristics of an object may include the state, behavior, and identification of the object, etc. The characteristics of an object may be used to identify or distinguish different objects. Based on the characteristics of an object, an object may be identified from multiple objects and related operations may be performed.

[0130] When the type of the object is different, the features of the object are also different. For example, when the object is a person, the features of the object may be facial features; when the object is an object, the features of the object may be features that characterize the appearance of the object, such as color, size, outline, etc.

[0131] When the object is a person, the terminal device of the embodiment of the present application detects the face area in the image and obtains the face features, which is done after notifying the user involved in the image and with the user's authorization.

[0132] Step 2: The terminal device determines the sound source object from multiple objects included in the sound and image matching area according to the target feature.

[0133] The terminal device may determine, according to the target feature and features of multiple objects included in the sound and image matching area, an object whose feature similarity with the target feature is greater than a threshold value from the sound and image matching area as a sound source object.

[0134] In some embodiments of the present application, the terminal device can determine the feature similarity between features based on the Euclidean distance. For example, for the similarity between the object feature and the target feature, the terminal device can calculate the Euclidean distance between the feature vector corresponding to the object feature and the feature vector corresponding to the target feature, and use the determined Euclidean distance as the feature similarity. In addition, when determining the sound source object, the embodiment of the present application compares the feature similarity with a threshold value, where the threshold value can be a pre-set value, the value can be an empirical value set by a person skilled in the art, or the value can be a value obtained through multiple simulations.

[0135] For example, the terminal device may determine the feature similarity between the feature of each object in the sound image matching area and the target feature, and take the object whose feature similarity is greater than a threshold as the sound source object. For example, for the objects A, B, and C included in the sound image matching area, the feature similarity between the feature of object A and the target feature is determined, the feature similarity between the feature of object B and the target feature is determined, and the feature similarity between the feature of object C and the target feature is determined. If the feature similarity between the feature of object A and the target feature is greater than the threshold, object A is determined to be the sound source object.

[0136] For another example, the terminal device can determine the sound source object in the sound image matching area according to the characteristics of the reference object and the target characteristics in the sound image matching area. In implementation, after determining the reference position of the sound source object relative to the terminal device, the terminal device converts the reference position to obtain the target position in the target image, and a reference object can be determined according to the target position.

[0137] The step of determining, by the terminal device, the sound source object in the sound image matching area according to the characteristics of the reference object and the target characteristics in the sound image matching area comprises:

[0138] Step 2a: The terminal device determines the feature similarity between the feature of the reference object and the target feature.

[0139] For example, the terminal device may calculate the Euclidean distance between a feature vector corresponding to the feature of the reference object and a feature vector corresponding to the target feature, and use the calculated Euclidean distance as the feature similarity between the feature of the reference object and the target feature.

[0140] Step 2b: If the feature similarity between the feature of the reference object and the feature of the sound source object is greater than a threshold, the terminal device determines that the reference object is the sound source object.

[0141] Since the reference object is an object determined by the terminal device from the sound and image matching area according to the reference position, and the reference position is determined by positioning the sound signal, the reference object determined according to the reference position is likely to be the sound source object that emits the sound signal. The embodiment of the present application takes into account the positioning accuracy of sound source positioning. The embodiment of the present application maintains a feature matching library, which stores the correspondence between the voiceprint features of the sound signal collected during the historical broadcast process and the features of the object finally located. Therefore, the terminal device further matches the features of the reference object with the target features corresponding to the voiceprint features of the sound signal stored in the feature matching library. When the feature similarity between the features of the reference object and the target features is greater than a threshold, the reference object is again determined to be the sound source object. Based on this approach, the determined sound source object is more accurate and more reliable.

[0142] In implementation, after determining that the reference object is the sound source object, the terminal device can update the object features corresponding to the voiceprint features of the sound signal in the corresponding relationship according to the features of the reference object. For example, the terminal device updates the object features corresponding to the voiceprint features of the sound signal in the feature matching library to the features of the reference object.

[0143] Step 2c: If the feature similarity between the features of the reference object and the features of the sound source object is not greater than the threshold, the terminal device determines that other objects whose feature similarity with the features of the sound source object is greater than the threshold are the sound source objects, and the other objects are objects other than the reference object among the multiple objects in the sound and image matching area.

[0144] For example, the terminal device determines the feature similarity between other objects except the reference object among the multiple objects in the sound image matching area and the target feature; for example, the sound image matching area includes objects A, B and C, assuming that object A is the determined reference object, when the feature similarity between the feature of object A and the target feature is not greater than a threshold, the terminal device determines the feature similarity between the feature of object B and the target feature, and determines the feature similarity between the feature of object C and the target feature. The terminal device determines other objects whose feature similarity with the feature of the sound source object is greater than the threshold as the sound source object, for example, if the feature similarity between the feature of object B and the target feature is greater than the threshold, object B is determined to be the sound source object.

[0145] After determining that other objects whose feature similarity with the features of the sound source object is greater than a threshold value are the sound source objects, the terminal device can update the object features corresponding to the voiceprint features of the sound signal in the corresponding relationship according to the features of the other objects. For example, the terminal device determines that object B is the sound source object, and updates the object features corresponding to the voiceprint features of the sound signal in the feature matching library to the features of object B.

[0146] When the terminal device of the embodiment of the present application determines that the feature similarity between the feature of the reference object and the target feature is not greater than the threshold, it indicates that there may be a positioning deviation in the sound source localization process, resulting in the determined reference object not being the sound source object. At this time, the terminal device can further determine the sound source object from other objects in the sound image matching area, because if a positioning deviation occurs, then the sound source object is likely to be other objects in the sound image matching area. In this way, the terminal device further determines the sound source object from other objects in the sound image matching area by calculating the features between the features of other objects in the sound image matching area and the target features. Therefore, even if there is a deviation in the sound source localization, the terminal device can still accurately determine the sound source object.

[0147] Step 2d: If it is determined that the feature similarities between the features of the multiple objects in the sound image matching area and the features of the sound source object are not greater than the threshold, the terminal device can determine the sound source object in one or more of the following ways.

[0148] Processing method 1: The terminal device determines the reference object as the sound source object.

[0149] Since the terminal device of the embodiment of the present application determines that the feature similarities between the features of multiple objects in the sound and image matching area and the target feature are not greater than the threshold, one possible situation is that the target feature corresponding to the voiceprint feature stored in the feature matching library may not be the feature of the object obtained by accurate matching; for example, in the last broadcast process, the sound source object corresponding to the voiceprint feature determined by the terminal device may not be accurate enough, resulting in the target feature corresponding to the voiceprint feature stored in the feature matching library not being the feature of the accurate sound source object. Since the target feature is not the feature of the accurate sound source object, matching the features of each object in the sound and image matching area with the target feature will largely fail to determine the object whose feature similarity is greater than the threshold from the multiple objects in the sound and image matching area. In this case, the terminal device can directly use the reference object determined based on the sound source localization technology as the sound source object.

[0150] In addition, in this case, after determining that the reference object is the sound source object, the terminal device can update the object features corresponding to the voiceprint features of the sound signal in the corresponding relationship according to the features of the reference object. For example, the terminal device updates the object features corresponding to the voiceprint features of the sound signal in the feature matching library to the features of the reference object, thereby ensuring that the object features stored in the feature matching library match the voiceprint features.

[0151] Processing method 2: The terminal device uses multiple objects included in the sound and image matching area as sound source objects.

[0152] When the terminal device of the embodiment of the present application determines that the feature similarities between the features of multiple objects in the sound image matching area and the target features are not greater than the threshold, it is considered that the target features corresponding to the voiceprint features stored in the feature matching library may not be the features of the object obtained by accurate matching. In this case, the terminal device can use the multiple objects included in the sound image matching area as sound source objects. This processing method can quickly determine the sound source object and avoid misjudgment resulting in failure to direct the correct sound source object during the directing process.

[0153] For example, the sound and image matching area includes three objects, namely object A, object B, and object C. Assuming that object A is the determined reference object, and assuming that the feature mark corresponding to the voiceprint feature of the sound signal determined from the feature matching library is the target feature, then when it is determined that the feature similarity between the feature of object A and the target feature is not greater than a threshold, the feature similarity between the feature of object B and the target feature is not greater than a threshold, and the feature similarity between the feature of object C and the target feature is not greater than a threshold, the terminal device regards object A, object B, and object C as sound source objects.

[0154] In implementation, after determining that the multiple objects included in the sound image matching area are all sound source objects, the terminal device can update the object features corresponding to the voiceprint features of the sound signal in the corresponding relationship according to the features of the multiple objects included in the sound image matching area. For example, the object features corresponding to the voiceprint features of the sound signal in the feature matching library are updated to the features of the multiple objects included in the sound image matching area.

[0155] Processing method 3: The terminal device determines the sound source object from multiple objects according to the screening parameters corresponding to each object in the multiple objects.

[0156] Among them, the screening parameters represent the participation of the object in the historical broadcast process.

[0157] When the terminal device of the embodiment of the present application determines that the feature similarities between the features of multiple objects in the sound and image matching area and the target feature are not greater than the threshold, it is considered that the target feature corresponding to the voiceprint feature stored in the feature matching library may not be the feature of the object obtained by the precise match. In this case, the terminal device can further determine the sound source object from the multiple objects in the sound and image matching area based on the screening parameters of each object in the sound and image matching area.

[0158] The embodiment of the present application can create a matching number database, which stores the number of times an object matches the voiceprint feature of a sound signal in the historical broadcast control process. For example, in a broadcast process, if the terminal device determines that object A matches the voiceprint feature and determines object A as the sound source object, the number of times the terminal device determines that object A matches the voiceprint feature of the sound signal increases by 1.

[0159] For example, the number of matches database may include the number of precise matches and the number of fuzzy matches. In the process of director control, if it is determined that an object matches the voiceprint feature of the sound signal (i.e., an object is determined as the sound source object), the match type is determined to be a precise match, and the number of precise matches included in the number of matches database is increased by 1; for example, in the above-mentioned process of determining the sound source object, the case where the reference object is determined as the sound source object, or the case where the other objects other than the reference object in the multiple objects included in the sound image matching area are determined as the sound source objects, these two cases of determining the sound source object are precise matches. If it is determined that multiple objects match the voiceprint feature of the sound signal (i.e., multiple objects are determined as the sound source objects), the match type is determined to be a fuzzy match, and the number of fuzzy matches included in the number of matches database is increased by 1; for example, in the above-mentioned process of determining the sound source object, the processing method of processing method 2 is to use multiple objects included in the sound image matching area as sound source objects, and this case of determining the sound source object is a fuzzy match.

[0160] During the broadcasting process, the precise matching times and fuzzy matching times of the objects in the matching times database are continuously updated as the broadcasting control times increase. The matching times database of the embodiment of the present application can be in any data structure form, and the data structure form of the matching times database is not limited here.

[0161] The terminal device can determine the screening parameters of the object according to the exact matching times and fuzzy matching times of the object in the matching times database.

[0162] For example, the terminal device can perform a weighted summation of the number of precise matches and the number of fuzzy matches of the object to determine the screening parameters of the object; the terminal device can determine the object with a larger screening parameter as the sound source object. For example, the sound and image matching area includes three objects, namely object A, object B, and object C. Assuming that object A is the determined reference object, and assuming that the feature mark corresponding to the voiceprint feature of the sound signal determined from the feature matching library is the target feature, then when it is determined that the feature similarity between the feature of object A and the target feature is not greater than the threshold, the feature similarity between the feature of object B and the target feature is not greater than the threshold, and the feature similarity between the feature of object C and the target feature is not greater than the threshold, the terminal device determines the screening parameters of object A, object B, and object C respectively. Assume that the weight coefficient of the exact match number is set to 0.8 and the weight coefficient of the fuzzy match number is set to 0.2; the exact match number of object A is 4 times and the fuzzy match number is 2 times, then the screening parameter of object A is 4*0.8+2*0.2=3.6; the exact match number of object B is 3 times and the fuzzy match number is 3 times, then the screening parameter of object A is 3*0.8+3*0.2=3; the exact match number of object C is 2 times and the fuzzy match number is 2 times, then the screening parameter of object A is 2*0.8+2*0.2=2; the terminal device determines that object A is the sound source object.

[0163] The process of determining the sound source object by the terminal device described above is the process when the correspondence between the object feature and the voiceprint feature includes the object feature corresponding to the voiceprint feature of the sound signal. When the correspondence between the object feature and the voiceprint feature does not include the object feature corresponding to the voiceprint feature of the sound signal, the terminal device can determine the sound source object in any one of the above-mentioned processing methods 1, 2 and 3, which will not be repeated here.

[0164] Step 704: The terminal device performs a directing operation on the sound source object.

[0165] After the terminal device determines the sound source object based on the above process, it determines the position of the sound source object in the environment, and performs a directing operation on the sound source object according to the position of the sound source object in the environment.

[0166] In practice, the terminal device may perform a variety of different operations on the sound source object:

[0167] Operation method 1: The terminal device adjusts the shooting angle of the camera and directs the sound source object.

[0168] The terminal device can adjust the shooting angle of the camera by controlling the pan / tilt according to the position of the sound source object in the environment, so that the camera can take a close-up shot of the sound source object.

[0169] Control method 2: The terminal device takes a screenshot of the image captured by the camera component.

[0170] After the terminal device captures the image through the camera component, it takes a screenshot of the area where the sound source object is located in the captured image, and enlarges the area where the sound source object is located to generate a new image, thereby obtaining a close-up image of the sound source object and achieving the directing of the sound source object.

[0171] The following is an introduction to the complete director control process of the embodiment of the present application with reference to several examples.

[0172] like Fig.11 The director control process shown in the figure takes the above processing method 1 as an example for introduction, and specifically may include the following steps:

[0173] Step 1101: The terminal device captures a target image in an environment through a camera component.

[0174] Step 1102: The terminal device collects the sound signal emitted by the sound source object through a microphone.

[0175] In some embodiments, step 1101 and step 1102 may be performed simultaneously.

[0176] Step 1103: The terminal device performs object detection on the target image to identify the object included in the target image.

[0177] Step 1104: The terminal device extracts voiceprint features from the sound signal to obtain target voiceprint features.

[0178] Step 1105: The terminal device locates the sound source of the sound signal and determines the reference position of the sound source object relative to the terminal device.

[0179] Among them, step 1103, step 1104 and step 1105 are not subject to execution order.

[0180] Step 1106: The terminal device performs audio-visual matching between the reference position and the target image to determine an audio-visual matching area in the target image.

[0181] Step 1107: If the sound and image matching area includes an object, the terminal device determines that the object is a sound source object.

[0182] Step 1108: The terminal device stores the correspondence between the target voiceprint features and the features of the sound source object in a feature matching library.

[0183] Step 1109: If the audio and video matching area includes multiple objects, the terminal device performs lip movement detection on the multiple objects.

[0184] Step 1110: When an object for which lip movement detection is successful is included in multiple objects, the object for which lip movement detection is successful is used as a sound source object.

[0185] Step 1111: The terminal device stores the correspondence between the target voiceprint features and the object features of the object whose lip movement detection is successful in the feature matching library.

[0186] Step 1112: When the object for which lip movement detection is successful is not included in the multiple objects, the terminal device determines the similarity between the feature of the sound source object corresponding to the target voiceprint feature in the feature matching library and the feature of the reference object.

[0187] The reference object is an object determined from the sound and image matching area according to the reference position of the sound source object relative to the terminal device.

[0188] Step 1113: The terminal device determines whether the similarity is greater than a threshold, and if so, executes step 1114, and if not, executes step 1115.

[0189] Step 1114: The terminal device determines that the reference object is the sound source object.

[0190] Step 1115: The terminal device determines the similarity between the features of each other object in the multiple objects and the features of the sound source object.

[0191] Step 1116: The terminal device determines whether the similarity between the features of other objects and the sound source object is greater than a threshold, and if so, executes step 1117, and if not, executes step 1114.

[0192] Step 1117: The terminal device determines that other objects are sound source objects.

[0193] Step 1118: The terminal device stores the correspondence between the target voiceprint features and the features of the sound source object in a feature matching library.

[0194] During implementation, for the above steps 1108, 1111, and 1118, when the feature matching library does not include the object features corresponding to the target voiceprint features, the correspondence between the target voiceprint features and the features of the sound source object is stored in the feature matching library; when the feature matching library includes the object features corresponding to the target voiceprint features, the object features corresponding to the target voiceprint features are updated to the features of the determined sound source object.

[0195] like Fig.12 The director control process shown in the figure takes the above processing method 2 as an example for introduction, and specifically may include the following steps:

[0196] Step 1201: The terminal device captures a target image in the environment through a camera component.

[0197] Step 1202: The terminal device collects sound signals emitted by the sound source object through a microphone array.

[0198] In some embodiments, step 1201 and step 1202 may be performed simultaneously.

[0199] Step 1203: The terminal device performs object detection on the target image to identify the object included in the target image.

[0200] Step 1204: The terminal device extracts voiceprint features from the sound signal to obtain target voiceprint features.

[0201] Step 1205: The terminal device locates the sound source of the sound signal and determines the reference position of the sound source object relative to the terminal device.

[0202] Among them, step 1203, step 1204 and step 1205 are not distinguished in the order of execution.

[0203] Step 1206: The terminal device performs audio-visual matching between the reference position and the target image to determine an audio-visual matching area in the target image.

[0204] Step 1207: If the sound and image matching area includes an object, the terminal device determines that the object is a sound source object.

[0205] Step 1208: The terminal device stores the correspondence between the target voiceprint features and the features of the sound source object in a feature matching library.

[0206] Step 1209: If the audio and video matching area includes multiple objects, the terminal device performs lip movement detection on the multiple objects.

[0207] Step 1210: When an object for which lip movement detection is successful is included in multiple objects, the object for which lip movement detection is successful is used as a sound source object.

[0208] Step 1211: The terminal device stores the correspondence between the target voiceprint features and the object features of the object whose lip movement detection is successful in the feature matching library.

[0209] Step 1212: When the object for which lip movement detection is successful is not included in the multiple objects, the terminal device determines the similarity between the feature of the sound source object corresponding to the target voiceprint feature in the feature matching library and the feature of the reference object.

[0210] The reference object is an object determined from the sound and image matching area according to the reference position of the sound source object relative to the terminal device.

[0211] Step 1213: The terminal device determines whether the similarity is greater than a preset threshold, and if so, executes step 1214, and if not, executes step 1215.

[0212] Step 1214: The terminal device determines that the reference object is the sound source object.

[0213] Step 1215: The terminal device determines the similarity between the features of each other object in the multiple objects and the features of the sound source object.

[0214] Step 1216: The terminal device determines whether the similarity between the features of other objects and the sound source object is greater than a threshold, and if so, executes step 1217, and if not, executes step 1218.

[0215] Step 1217: The terminal device determines that other objects are sound source objects.

[0216] Step 1218: The terminal device determines that multiple target objects are sound source objects.

[0217] Step 1219: The terminal device stores the correspondence between the target voiceprint features and the features of the sound source object in a feature matching library.

[0218] During implementation, for the above steps 1208, 1212, and 1219, when the feature matching library does not include the object features corresponding to the target voiceprint features, the correspondence between the target voiceprint features and the features of the sound source object is stored in the feature matching library; when the feature matching library includes the object features corresponding to the target voiceprint features, the object features corresponding to the target voiceprint features are updated to the features of the determined sound source object.

[0219] like Fig.13 The director control process shown in the figure takes the above processing method 3 as an example for introduction, and specifically may include the following steps:

[0220] Step 1301: The terminal device captures a target image in the environment through a camera component.

[0221] Step 1302: The terminal device collects sound signals emitted by the sound source object through a microphone array.

[0222] In some embodiments, step 1301 and step 1302 may be performed simultaneously.

[0223] Step 1303: The terminal device performs object detection on the target image to identify the object included in the target image.

[0224] Step 1304: The terminal device extracts voiceprint features from the sound signal to obtain target voiceprint features.

[0225] Step 1305: The terminal device locates the sound source of the sound signal and determines the reference position of the sound source object relative to the terminal device.

[0226] Among them, step 1303, step 1304 and step 1305 are not distinguished in the order of execution.

[0227] Step 1306: The terminal device performs audio-visual matching between the reference position and the target image to determine an audio-visual matching area in the target image.

[0228] Step 1307: If the sound and image matching area includes an object, the terminal device determines that the object is a sound source object.

[0229] Step 1308: The terminal device stores the correspondence between the target voiceprint features and the features of the sound source object in a feature matching library.

[0230] Step 1309: The terminal device updates the number of accurate matches in the matching number database.

[0231] During implementation, the terminal device increases the number of exact matches in the matching number database by 1.

[0232] Step 1310: If the audio and video matching area includes multiple objects, the terminal device performs lip movement detection on the multiple objects.

[0233] Step 1311: When an object for which lip movement detection is successful is included in multiple objects, the object for which lip movement detection is successful is used as a sound source object.

[0234] Step 1312: The terminal device stores the correspondence between the target voiceprint feature and the object feature of the object whose lip movement is successfully detected in the feature matching library.

[0235] Step 1313: The terminal device updates the number of accurate matches in the matching number database.

[0236] During implementation, the terminal device increases the number of exact matches in the matching number database by 1.

[0237] Step 1314: when the multiple target objects do not include the object for which lip movement detection is successful, the terminal device determines the similarity between the feature of the sound source object corresponding to the target voiceprint feature in the feature matching library and the feature of the reference object;

[0238] The reference object is an object determined from the sound and image matching area according to the reference position of the sound source object relative to the terminal device.

[0239] Step 1315: The terminal device determines whether the similarity is greater than a threshold, and if so, executes step 1316, and if not, executes step 1317.

[0240] Step 1316: The terminal device determines that the reference object is the sound source object.

[0241] Step 1317: The terminal device determines the similarity between the features of each other object in the multiple objects and the features of the sound source object.

[0242] Step 1318: The terminal device determines whether the similarity between the features of other objects and the features of the sound source object is greater than a threshold. If so, execute step 1319; if not, execute step 1322.

[0243] Step 1319: The terminal device determines that other objects are sound source objects.

[0244] Step 1320: The terminal device stores the correspondence between the target voiceprint features and the features of the sound source object in a feature matching library.

[0245] Step 1321: The terminal device updates the exact matching number in the matching number database.

[0246] During implementation, the terminal device increases the number of exact matches in the matching number database by 1.

[0247] Step 1322: The terminal device determines the screening parameters of each object, and determines the sound source object according to the screening parameters.

[0248] Step 1323: The terminal device stores the correspondence between the target voiceprint feature and the features of multiple objects in a feature matching library.

[0249] Step 1324: The terminal device updates the fuzzy matching times in the matching times database.

[0250] During implementation, the terminal device increases the number of fuzzy matches in the matching number database by 1.

[0251] During implementation, for the above steps 1308, 1312, 1320, and 1323, when the feature matching library does not include the object features corresponding to the target voiceprint features, the correspondence between the target voiceprint features and the features of the sound source object is stored in the feature matching library; when the feature matching library includes the object features corresponding to the target voiceprint features, the object features corresponding to the target voiceprint features are updated to the features of the determined sound source object.

[0252] Based on the above embodiments and the same concept, the embodiments of the present application also provide a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer, the computer executes the method provided in the above embodiments.

[0253] Based on the above embodiments and the same concept, the embodiments of the present application also provide a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed on a computer, the computer executes the method provided in the above embodiments.

[0254] The method provided in the embodiment of the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. Computer-readable storage media can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. Available media can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., digital video discs (DVDs), or semiconductor media (e.g., SSDs), etc.

[0255] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A broadcast control method, characterized in that: Applied to a terminal device, the method comprises: Capturing a target image in an environment through a camera component of the terminal device, and capturing a sound signal emitted by a sound source object in the environment through a microphone of the terminal device; Locating a sound-image matching area of ​​the sound source object in the target image according to the sound signal; According to the sound signal, determining the sound source object from a plurality of objects included in the sound image matching area; Perform a directing operation on the sound source object.

2. The method according to claim 1, characterized in that The step of locating the sound-image matching area of ​​the sound source object in the target image according to the sound signal comprises: Determining a reference position of the sound source object relative to the terminal device according to the sound signal; Performing sound and image matching on the reference position and the target image to locate a sound and image matching area of ​​the sound source object in the target image.

3. The method according to claim 1, characterized in that The step of determining the sound source object from a plurality of objects included in the sound image matching area according to the sound signal comprises: Extracting voiceprint features of the sound signal, and determining features of the sound source object from a correspondence between object features and voiceprint features; The sound source object is determined from a plurality of objects included in the sound image matching area according to the features of the sound source object.

4. The method according to claim 3, characterized in that The step of determining the sound source object from a plurality of objects included in the sound image matching area according to the features of the sound source object comprises: According to the feature of the sound source object and the features of the multiple objects, an object whose feature similarity with the feature of the sound source object is greater than a threshold is determined from the multiple objects as the sound source object.

5. The method according to claim 4, characterized in that The step of determining, from the multiple objects, an object whose feature similarity with the sound source object is greater than a threshold as the sound source object comprises: determining a reference object from the plurality of objects according to the reference position; If the feature similarity between the feature of the reference object and the feature of the sound source object is greater than a threshold, the reference object is determined to be the sound source object.

6. The method according to claim 5, characterized in that The method further comprises: If the feature similarity between the feature of the reference object and the feature of the sound source object is not greater than a threshold, then other objects whose feature similarity with the feature of the sound source object is greater than the threshold are determined to be the sound source objects, and the other objects are objects among the multiple objects other than the reference object.

7. The method according to claim 4, characterized in that The method further comprises: If the feature similarities between the features of the multiple objects and the features of the sound source object are not greater than a threshold, the reference object is determined to be the sound source object, or the multiple objects are determined to be the sound source object.

8. The method according to any one of claims 4 to 7, characterized in that: The method further comprises: According to the determined features of the sound source object, the object features corresponding to the voiceprint features of the sound signal in the corresponding relationship are updated.

9. The method according to claim 4, characterized in that The method further comprises: If the feature similarities between the features of the multiple objects and the features of the sound source object are not greater than a threshold, the sound source object is determined from the multiple objects based on the screening parameters corresponding to each of the multiple objects; the screening parameters represent the degree of participation of the object in the historical directing process.

10. The method according to claim 9, characterized in that The method further comprises: According to the features of the multiple objects, the object features corresponding to the voiceprint features of the sound signal in the corresponding relationship are updated.

11. The method according to any one of claims 1 to 10, characterized in that: Before determining the sound source object from a plurality of objects included in the sound image matching area according to the sound signal, the method further includes: Lip movement detection is performed on a plurality of objects included in the voice and image matching area, and it is determined that no lip movement event is detected.

12. The method according to claim 11, characterized in that The method further comprises: If a lip movement event is detected, the object corresponding to the lip movement event among the multiple objects is determined as the sound source object.

13. A broadcast director control device, characterized in that: The device is applied to a terminal device, and the device includes: A signal acquisition module, used to capture a target image in an environment through a camera assembly, and to capture a sound signal emitted by a sound source object in the environment through a microphone; A positioning and recognition module, configured to locate the sound source object in the sound image matching area in the target image according to the sound signal; and determine the sound source object from a plurality of objects included in the sound image matching area according to the sound signal; The director control module is used to perform director operations on the sound source object.

14. The device according to claim 13, characterized in that The positioning and identification module is used to: According to the sound signal, a reference position of the sound source object relative to the terminal device is determined; and sound and image matching is performed between the reference position and the target image to locate a sound and image matching area of ​​the sound source object in the target image.

15. The device according to claim 14, characterized in that The positioning and identification module is used to: Extracting voiceprint features of the sound signal, and determining features of the sound source object from a correspondence between object features and voiceprint features; The sound source object is determined from a plurality of objects included in the sound image matching area according to the features of the sound source object.

16. The device according to claim 15, characterized in that The positioning and identification module is used to: According to the feature of the sound source object and the features of the multiple objects, an object whose feature similarity with the feature of the sound source object is greater than a threshold is determined from the multiple objects as the sound source object.

17. The device according to claim 16, characterized in that The positioning and identification module is used to: determining a reference object from the plurality of objects according to the reference position; If the feature similarity between the feature of the reference object and the feature of the sound source object is greater than a threshold, the reference object is determined to be the sound source object.

18. The device according to claim 17, characterized in that The positioning and identification module is also used for: If the feature similarity between the feature of the reference object and the feature of the sound source object is not greater than a threshold, then other objects whose feature similarity with the feature of the sound source object is greater than the threshold are determined to be the sound source objects, and the other objects are objects among the multiple objects other than the reference object.

19. The device according to claim 16, characterized in that The positioning and identification module is also used for: If the feature similarities between the features of the multiple objects and the features of the sound source object are not greater than a threshold, the reference object is determined to be the sound source object, or the multiple objects are determined to be the sound source object.

20. The device according to any one of claims 16 to 19, characterized in that: The positioning and identification module is also used for: According to the determined features of the sound source object, the object features corresponding to the voiceprint features of the sound signal in the corresponding relationship are updated.

21. The device according to claim 16, characterized in that The positioning and identification module is also used for: If the feature similarities between the features of the multiple objects and the features of the sound source object are not greater than a threshold, the sound source object is determined from the multiple objects based on the screening parameters corresponding to each of the multiple objects; the screening parameters represent the degree of participation of the object in the historical directing process.

22. The device according to claim 21, characterized in that The positioning and identification module is also used for: According to the features of the multiple objects, the object features corresponding to the voiceprint features of the sound signal in the corresponding relationship are updated.

23. The device according to any one of claims 13 to 22, characterized in that: Before the positioning and identifying module determines the sound source object from a plurality of objects included in the sound and image matching area according to the sound signal, the positioning and identifying module is further configured to: Lip movement detection is performed on a plurality of objects included in the voice and image matching area, and it is determined that no lip movement event is detected.

24. The device according to claim 23, characterized in that The positioning and identification module is also used for: If a lip movement event is detected, the object corresponding to the lip movement event among the multiple objects is determined as the sound source object.

25. A terminal device, characterized in that: The terminal device includes at least one processor and a memory; The memory is used to store computer program codes, and the computer program codes include computer instructions; when the computer instructions are executed by the at least one processor, the terminal device executes the method according to any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 12.

27. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or the instructions are run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 12.

Images