Audio control method and device, medium and equipment

Abstract

The embodiment of the invention provides an audio control method and device, a medium and equipment, and the method comprises the steps: determining the target position information of a sound production part corresponding to a sound production action when detecting that a target object exists in a current playing image and the target object executes the sound production action; according to the target position information, generating a target audio output parameter required by the audio output device to output the audio; and based on the target audio output parameter, controlling audio output equipment to output audio corresponding to the target position information. According to the audio control method provided by the embodiment of the invention, the target audio output parameter required by the audio output equipment is generated according to the target position information of the sound production part corresponding to the sound production action when the target object is detected to exist in the current playing picture and the target object executes the sound production action; the audio output device is controlled to output the audio corresponding to the target position information, the sound source position and the picture action are synchronized, and the audio-visual feeling of a user is improved.

Description

Technical Field

[0001] The present application relates to the field of smart device technology, in particular to the field of audio control technology, and more particularly to an audio control method, device, medium and equipment. Background Art

[0002] As TV screen sizes continue to increase, the problem of mismatch between sound and picture has also arisen. Especially in some dynamic scenes where the characters move over a large range, when the characters in the playing picture move from the left to the right of the TV screen while talking, the sound from the TV is always locked in the center area of ​​the TV screen, resulting in a mismatch between the sound source position and the picture action, affecting the user's audio-visual experience. Summary of the invention

[0003] The present application provides an audio control method, device, medium and equipment. The audio control method provided by the present application solves the problem that the sound source position does not match the picture action in the current audio control technology, which affects the user's audio-visual experience.

[0004] An embodiment of the present application provides an audio control method, which is applied to an audio output device, including:

[0005] When it is detected that there is a target object in the current playing picture and the target object performs a sound-making action, target position information of a sound-making part corresponding to the sound-making action is determined;

[0006] Generating target audio output parameters required for the audio output device to output audio according to the target position information;

[0007] Based on the target audio output parameter, the audio output device is controlled to output audio corresponding to the target position information.

[0008] Further, in the audio control method described in the embodiment of the present application, when it is detected that there is a target object in the current playback picture and the target object performs a sound-making action, determining the target position information of the sound-making part corresponding to the sound-making action includes:

[0009] Obtaining a screenshot corresponding to the current playback screen, inputting the screenshot into a pre-trained target detection model to perform a target detection operation, and outputting a first detection result;

[0010] If the first detection result indicates that there is a target object in the screenshot, further detecting whether the target object performs a sound-making action and outputting a second detection result;

[0011] If the second detection result indicates that the target object performs a vocalization action, the target position information of the vocalization part corresponding to the vocalization action is determined.

[0012] Furthermore, in the audio control method described in the embodiment of the present application, the further detecting whether the target object performs a sound-making action includes:

[0013] Taking the time when the target object is detected as a reference time, obtaining at least two screenshots of the current playing screen that are located after the reference time and contain the target object;

[0014] Respectively extracting the position coordinates of each sound-producing part in the at least two screenshots;

[0015] It is detected whether the relative position coordinates of each sound-generating part in the at least two screenshots have changed, thereby determining whether the target object performs a sound-generating action.

[0016] Furthermore, in the audio control method described in the embodiment of the present application, after outputting the first detection result, the method further includes:

[0017] If the first detection result indicates that the target object does not exist in the screenshot, generating target audio output parameters required for the audio output device to output audio according to the center position coordinates of the current playing screen;

[0018] Based on the target audio output parameter, the audio output device is controlled to output audio at a position corresponding to the center position coordinates.

[0019] Furthermore, in the audio control method described in the embodiment of the present application, the step of determining the target position information of the sound emitting part corresponding to the sound emitting action includes:

[0020] Based on the coordinates of all key points of the target object, construct an array structure corresponding to the target object, and select target position information corresponding to the sound-producing part from the array structure;

[0021] Generating target audio output parameters required for the audio output device to output audio according to the target position information includes:

[0022] The target position information is converted into a numerical value to obtain a target value within a preset range;

[0023] Determining a target channel volume ratio corresponding to the target value based on a mapping relationship between a pre-created value and a volume ratio of each channel in the audio output device;

[0024] According to the target channel volume ratio, the target audio output parameters corresponding to each channel of the audio output device are adjusted.

[0025] Furthermore, in the audio control method described in the embodiment of the present application, when it is detected that there is a target object in the current playback picture and the target object performs a sounding action, before determining the target position information of the sounding part corresponding to the sounding action, the method also includes:

[0026] In response to a start operation for a preset function, detecting whether a target object exists in the current playing picture and the target object performs a sound-making action;

[0027] When it is detected for the first time that there is a target object in the current playing picture and the target object performs a sound-making action, target position information of a sound-making part corresponding to the sound-making action is determined;

[0028] The preset mark is called to be displayed at the position corresponding to the target position information, and the preset mark is hidden after a preset time period.

[0029] Furthermore, in the audio control method described in the embodiment of the present application, after controlling the audio output device to output the audio corresponding to the target position information, the method further includes:

[0030] Detecting the distance between the current user and the audio output device;

[0031] If the distance value is greater than a first preset threshold, increasing the output volume value of the audio output device;

[0032] If the distance value is less than a second preset threshold, the output volume value of the audio output device is weakened, and the first preset threshold is greater than the second preset threshold.

[0033] Accordingly, another aspect of the present application embodiment further provides an audio control device, which is applied to an audio output device, including:

[0034] A position determination module, for determining target position information of a sounding part corresponding to the sounding action when detecting that a target object exists in the current playing picture and the target object performs a sounding action;

[0035] A parameter generation module, used for generating target audio output parameters required for the audio output device to output audio according to the target position information;

[0036] The audio control module is used to control the audio output device to output audio corresponding to the target position information based on the target audio output parameter.

[0037] Correspondingly, another aspect of the embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute the audio control method as described above.

[0038] Correspondingly, another aspect of an embodiment of the present application further provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the audio control method as described above.

[0039] The embodiments of the present application provide an audio control method, apparatus, medium and equipment. The method determines the target position information of the sound part corresponding to the sound-making action when it is detected that there is a target object in the current playback screen and the target object performs a sound-making action; generates the target audio output parameters required for the audio output device to output audio according to the target position information; and controls the audio output device to output audio at a position corresponding to the target position information based on the target audio output parameters. Utilizing the audio control method provided by the embodiments of the present application, when it is detected that there is a target object in the current playback screen and the target object performs a sound-making action, the target audio output parameters required by the audio output device are generated according to the target position information of the sound part corresponding to the sound-making action, so as to control the audio output device to output audio corresponding to the target position information, thereby achieving synchronization between the sound source position and the screen action and improving the user's audio-visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0041] Figure 1 A flowchart of an audio control method provided in an embodiment of the present application.

[0042] Figure 2 This is a schematic diagram of a specific flow of S102 in the audio control method provided in an embodiment of the present application.

[0043] Figure 3 This is a schematic diagram of a specific flow of S202 in the audio control method provided in an embodiment of the present application.

[0044] Figure 4 This is a schematic diagram of the corresponding processing flow when the first detection result indicates that there is no target object in the screenshot in the audio control method provided in an embodiment of the present application.

[0045] Figure 5 This is a schematic diagram of a specific flow of S103 in the audio control method provided in an embodiment of the present application.

[0046] Figure 6This is a schematic diagram of the processing flow corresponding to the first activation of a preset function in the audio control method provided in an embodiment of the present application.

[0047] Figure 7 The present invention provides an overall flow chart of the audio control method according to an embodiment of the present invention.

[0048] Figure 8 A schematic diagram of the process framework of the audio control method provided in an embodiment of the present application.

[0049] Fig. 9 A schematic diagram of the structure of an audio control device provided in an embodiment of the present application.

[0050] Fig.10 Another structural schematic diagram of the audio control device provided in an embodiment of the present application.

[0051] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0053] An embodiment of the present application provides an audio control method, which generates target audio output parameters required by an audio output device according to target position information of a sound part corresponding to the sound action when it is detected that a target object exists in a currently playing picture and the target object performs a sound action, so as to control the audio output device to output audio corresponding to the target position information, thereby achieving synchronization between the sound source position and the picture action and improving the user's audio-visual experience.

[0054] The term "and / or" in this application can be a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0055] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments described here can be implemented in a sequence other than the content illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or television set comprising a series of steps or modules need not be limited to the steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or television sets. The naming or numbering of the steps in the present application does not mean that the steps in the method flow must be performed in the time / logic sequence indicated by the naming or numbering, and the process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present application.

[0056] The audio control method involved in the embodiment of the present application is mainly applied to electronic devices, which may be televisions, smart phones, tablet computers, laptops, desktop computers, servers, etc., which are not limited here. Optionally, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or an IoT cloud (Internet of Things cloud) that provides the ability to store, process and manage data generated by an IoT television, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, which are not limited here.

[0057] For ease of understanding, the specific process in the embodiments of the present application is described below. Figure 1 , Figure 1A flowchart of an embodiment of an audio control method provided in an embodiment of the present application.

[0058] exist Figure 1 In the illustrated embodiment, the method specifically comprises the following steps:

[0059] S101, when it is detected that there is a target object in the current playing picture and the target object performs a sound-making action, the target position information of the sound-making part corresponding to the sound-making action is determined.

[0060] It should be explained that the target object specifically refers to an object that can make a vocal action, such as a person or an animal, or even any object that is artificially endowed with the ability to make a sound (including virtual characters), which can be called a target object, without limitation here. The vocal action specifically refers to the act of speaking or singing, or even any act that indicates making a sound, without limitation here. The vocal part specifically refers to the part that performs the vocal action, such as the mouth, or other artificially defined vocal parts, without limitation here.

[0061] For example, when it is detected that there is a character A in the current playback screen who is speaking through his mouth (specifically, it can be manifested as the opening and closing of the lips), it means that there is a target object in the current playback screen and the target object is performing a vocal action. At this time, the target position information (x, y) of the vocal part corresponding to the vocal action is determined, which is the key parameter for subsequently controlling the audio output device (such as a TV) to output the audio corresponding to the vocal action.

[0062] S102: Generate target audio output parameters required for the audio output device to output audio according to the target position information.

[0063] It should be explained that the target audio output parameters refer to the key parameters required by the audio output device when outputting audio, such as volume value and frequency response (frequency response refers to the frequency range and gain that the audio device can output), etc. In this embodiment, the target audio output parameters are generated according to the target position information, aiming to highlight the sound effect at the location of the target position information, so as to synchronize the sound source position of the target object in the current playing screen with the screen action, and enhance the user's audio-visual experience, which will be described in detail below.

[0064] S103: Based on the target audio output parameter, control the audio output device to output audio corresponding to the target position information.

[0065] In this embodiment, based on the target audio output parameters corresponding to the target position information, the audio output device is controlled to output the audio corresponding to the target position information, so as to achieve synchronization between the sound source position and the screen action, thereby improving the user's audio-visual experience.

[0066] In some embodiments, Figure 2 As shown, when it is detected that there is a target object in the current playing picture and the target object performs a sounding action, the target position information of the sounding part corresponding to the sounding action is determined, including:

[0067] S201, obtaining a screenshot corresponding to the current playing screen, inputting the screenshot into a pre-trained target detection model to perform a target detection operation, and outputting a first detection result;

[0068] In this embodiment, by obtaining a screenshot corresponding to the current playback screen, and using the pre-trained target detection model to perform a target detection operation on the screenshot, it is determined whether a target object exists in the current playback screen according to the output first detection result. To ensure the real-time and accuracy of the target detection operation, a screenshot corresponding to the current playback screen may be obtained every 60ms.

[0069] It should be noted that the target detection model can be obtained based on conventional neural network training, such as R-CNN (Regions with CNN features) or YOLO (You Only Look Once). Since this solution does not make substantial improvements to the existing models, the specific training process will not be described here.

[0070] S202, if the first detection result indicates that there is a target object in the screenshot, further detecting whether the target object performs a sound-making action, and outputting a second detection result;

[0071] In this embodiment, if it is detected that there is a target object in the current playback screen, since the premise for triggering the target position information detection action of this scheme is that there is a target object in the current playback screen and the target object performs a sound action, after determining that there is a target object in the screenshot according to the first detection result, it is necessary to further detect whether the target object performs a sound action and output a second detection result. Based on the output second detection result, it is determined whether the two conditions of the target object and the target object performing a sound action are simultaneously met in the current playback screen.

[0072] In some embodiments, it is also possible to detect whether the target object performs a vocal action within a preset time period. The main purpose of detecting whether the target object performs a vocal action within the preset time period is to determine the vocal action, because the vocal action is a dynamic effect from a visual perception. For example, the opening and closing of the mouth is usually understood as a speaking behavior, so the limiting condition of "preset time period" is added.

[0073] S203: If the second detection result indicates that the target object performs a vocalization action, determine the target position information of the vocalization part corresponding to the vocalization action.

[0074] In this embodiment, since the second detection result is output based on the first detection result, when the second detection result indicates that the target object performs a vocalization action, an instruction to obtain target position information of the vocalization part corresponding to the vocalization action can be triggered.

[0075] In some embodiments, Figure 3 As shown, the further detecting whether the target object performs a sounding action includes:

[0076] S2021, taking the time when the target object is detected as a reference time, acquiring at least two screenshots of the current playback screen that are located after the reference time and contain the target object;

[0077] S2022, respectively extracting the position coordinates of each sound-producing part in the at least two screenshots;

[0078] S2023, detecting whether the relative position coordinates of each sound-generating part in the at least two screenshots have changed, and then determining whether the target object performs a sound-generating action.

[0079] In this embodiment, by judging whether there is a difference in at least two screenshots captured after the target object is detected, and specifically by comparing whether the position coordinates of each sound-making part in at least two screenshots have changed, it is determined whether the target object performs a sound-making action within a preset time period.

[0080] Exemplarily, y_top and y_bottom are used to represent the y coordinates of the top and bottom of the mouth respectively, d_open represents the degree of opening of the mouth within a preset time period t, and d_open-1 represents the degree of opening of the mouth in the previous frame.

[0081] The formula for lip opening is: d_open = |y_bottom-y_top|

[0082] Calculate the difference between frames: Δd = |d_open-d_open-1|

[0083] When Δd exceeds the threshold value T, it is considered that the mouth performs a vocalization action. In some embodiments, a convolutional neural network (CNN) or other deep learning models can be used to automatically identify vocalization actions from the current playback screen. It can also be combined with speech recognition technology to analyze the sound waveform in the current playback screen, synchronize with the video frame, and then detect the action.

[0084] In some embodiments, Figure 4As shown, after S201, the method further includes the following steps:

[0085] S2011, if the first detection result indicates that there is no target object in the screenshot, generating target audio output parameters required for the audio output device to output audio according to the center position coordinates of the current playing screen;

[0086] In this embodiment, if the target object is not detected in the current playing picture, the target audio output parameters can be generated by running the default values. In this embodiment, the center position coordinates of the current playing picture are selected, and the target audio output parameters required for the audio output device to output audio are generated according to the center position coordinates.

[0087] S2012: Based on the target audio output parameters, control the audio output device to output audio at a position corresponding to the center position coordinates.

[0088] In this embodiment, based on the target audio output parameters, the audio output device is controlled to output the audio at the center of the current playing picture, ensuring that the audio is played in the center unless there are special circumstances.

[0089] In some embodiments, the S102 specifically includes:

[0090] Based on the coordinates of all key points of the target object, construct an array structure corresponding to the target object, and select target position information corresponding to the sound-producing part from the array structure;

[0091] In some embodiments, Figure 5 As shown, the S103 specifically includes:

[0092] S1031, performing numerical conversion processing on the target position information to obtain a target numerical value within a preset range;

[0093] S1032, determining a target channel volume ratio corresponding to the target value based on a mapping relationship between a pre-created value and a volume ratio of each channel in the audio output device;

[0094] S1033: Adjust target audio output parameters corresponding to each channel of the audio output device according to the target channel volume ratio.

[0095] In this embodiment, the acquired facial key points (such as the vertices corresponding to the eyes, nose, and mouth) can be constructed into an array structure, which contains the coordinate information of the key points. First, the coordinates of each key point are normalized to adapt to the specified coordinate range, ensuring that all coordinates use the same scale for easy comparison and calculation. Next, the coordinate information of the sound-generating part (such as the mouth) is extracted, and the target position information is numerically converted (such as linear mapping) to obtain a target value within a preset interval (such as 4-22). After obtaining the target value, based on the mapping relationship between the pre-created value and the volume ratio of each channel in the audio output device, the target channel volume ratio corresponding to the target value is determined, and the target audio output parameters corresponding to each channel of the audio output device are adjusted according to the target channel volume ratio. This is achieved by controlling the gain of different channels, so that the final output audio effect is as if it is emitted from the sound-generating part of the target object, so that the sound source position is synchronized with the picture action, and the user's audio-visual experience is enhanced.

[0096] In some embodiments, Figure 6-Figure 7 As shown, before S101, the method further includes:

[0097] S301, in response to a start operation for a preset function, detecting whether there is a target object in the current playing picture and the target object performs a sound-making action;

[0098] In this embodiment, the preset function refers to a function of controlling the sound source position of the target object to be as consistent as possible with the position where the audio output device outputs the audio, realizing synchronization between the sound source position and the screen action, and improving the user's audio-visual experience. This function can be packaged into a control control for the user to choose whether to turn it on. When the turn-on operation of the preset function is detected, a detection instruction is triggered to determine whether there is a target object in the current playback screen and whether the target object performs a sound-generating action.

[0099] S302, when it is detected for the first time that there is a target object in the current playing picture and the target object performs a sound-making action, determining target position information of a sound-making part corresponding to the sound-making action;

[0100] S303, calling a preset mark to display at a position corresponding to the target position information, and hiding the preset mark after a preset time period.

[0101] In this embodiment, when the preset function is turned on for the first time after the audio output device is turned on, in order to prompt the user that the current function has been turned on, when it is first detected that there is a target object in the current playback screen and the target object performs a sound action, the target position information of the sound part corresponding to the sound action can be determined, and a preset mark (such as an edit box or annotation or animation, etc.) is called to be displayed at the position corresponding to the target position information to prompt the user that the current preset function has been turned on, and the preset mark is hidden after a preset time (for example, 3s) to avoid affecting the user's viewing of the playback screen.

[0102] In some embodiments, after controlling the audio output device to output audio corresponding to the target location information, the method further includes:

[0103] Detecting the distance between the current user and the audio output device;

[0104] If the distance value is greater than a first preset threshold, increasing the output volume value of the audio output device;

[0105] If the distance value is less than a second preset threshold, the output volume value of the audio output device is weakened, and the first preset threshold is greater than the second preset threshold.

[0106] In this embodiment, in order to provide the user with a more comfortable hearing effect and avoid the problem of too loud or too small output audio sound effect caused by the user being too close or too far away from the audio output device, the distance value between the current user and the audio output device is detected, and the distance value can be detected every 1s. The distance value between the user and the audio output device is calculated. Assuming that the best viewing effect is 3 to 5m, if it is less than 3m, the output volume value of the audio output device is weakened by reducing the gain in the reverse direction, satisfying the user's experience while ensuring that the user does not feel that the sound is too loud. If it is greater than 5m, the output volume value of the audio output device is enhanced by increasing the gain, satisfying the user's experience while ensuring that the user does not feel that the sound has become smaller.

[0107] In some embodiments, when it is detected that there is a target object in the current playback screen that is facing away, and the current output audio belongs to the target object, the position of the sound-producing part on the target object in the facing away state can be predicted, and according to the position, the target audio output parameters required for the audio output device to output the audio are generated, and based on the target audio output parameters, the audio output device is controlled to output the audio corresponding to the target position information. This allows the user to truly feel that the output audio is emitted from the sound-producing part of the target object even if the target object is facing away.

[0108] In some embodiments, when it is detected that the area where the target object is located exceeds the maximum threshold of the current playback screen (for example, 90%), it means that the target object is enlarged and covers the current playback screen. At this time, the operation of locating the sound-emitting part is of little significance. In order to reduce the data processing pressure of the system, the current operation of obtaining the position information of the sound-emitting part can be skipped, and the default centered sound strategy can be used to directly output the audio.

[0109] In some embodiments, for ease of understanding, please refer to Figure 8 , Figure 8 This is a schematic diagram of the process framework of the audio control method provided in the embodiment of the present application. The following is an explanation of the terms appearing in the figure:

[0110] UI layer (User Interface layer): It is mainly responsible for the interface design for interaction with users in the system architecture. It is the direct interface for users to interact with the system.

[0111] Framework Layer: In the Android system, the Framework Layer is the core part of the system architecture, located between the Application Layer and the operating system (Linux Kernel). It provides a series of services, libraries, and management functions to help developers develop efficient and feature-rich applications.

[0112] Audio HAL (Audio Hardware Abstraction Layer): Audio HAL is a software layer used to handle interaction with audio hardware. It provides a standardized interface that enables operating systems and applications to communicate with audio hardware without having to deal with hardware details directly. This abstraction layer helps simplify the development and maintenance of audio device drivers and promotes cross-platform compatibility.

[0113] SOC (System on Chip): SOC is an integrated circuit (IC) technology that integrates all or most of the functions of a computer or electronic system into a single chip. SOC usually integrates multiple functional modules, such as central processing unit (CPU), graphics processing unit (GPU), memory controller, input / output ports, various interfaces (such as USB, HDMI, Ethernet), audio and video processing units, DSP modules, and power management units. Here it refers to the TV processing chip.

[0114] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0115] In specific implementation, the present application is not limited by the execution order of the various steps described. If no conflict occurs, some steps can be performed in other orders or simultaneously.

[0116] As can be seen from the above, the audio control method provided in the embodiment of the present application determines the target position information of the sound part corresponding to the sound action when it is detected that there is a target object in the current playback screen and the target object performs a sound action; generates the target audio output parameters required for the audio output device to output audio based on the target position information; and controls the audio output device to output audio corresponding to the target position information based on the target audio output parameters. Utilizing the audio control method provided in the embodiment of the present application, when it is detected that there is a target object in the current playback screen and the target object performs a sound action, the target audio output parameters required by the audio output device are generated based on the target position information of the sound part corresponding to the sound action, so as to control the audio output device to output audio corresponding to the target position information, thereby achieving synchronization of the sound source position with the screen action and enhancing the user's audio-visual experience.

[0117] An embodiment of the present application further provides an audio control device, which can be integrated into an electronic device.

[0118] See also Fig. 9 , Fig. 9 The audio control device 30 may include:

[0119] A position determination module 31 is used to determine the target position information of the sound-generating part corresponding to the sound-generating action when it is detected that there is a target object in the current playing picture and the target object performs a sound-generating action;

[0120] A parameter generating module 32, configured to generate target audio output parameters required for the audio output device to output audio according to the target position information;

[0121] The audio control module 33 is used to control the audio output device to output audio corresponding to the target position information based on the target audio output parameter.

[0122] In some embodiments, the position determination module 31 is used to obtain a screenshot corresponding to the current playback screen, input the screenshot into a pre-trained target detection model to perform a target detection operation, and output a first detection result; if the first detection result indicates that there is a target object in the screenshot, further detect whether the target object performs a sound action, and output a second detection result; if the second detection result indicates that the target object performs a sound action, then determine the target position information of the sound part corresponding to the sound action.

[0123] In some embodiments, the position determination module 31 is used to take the moment when the target object is detected as a reference moment, obtain at least two screen screenshots located after the reference moment and containing the target object from the current playback screen; respectively extract the position coordinates of each sound-making part in the at least two screen screenshots; detect whether the relative position coordinates of each sound-making part in the at least two screen screenshots have changed, and then determine whether the target object performs a sound-making action.

[0124] In some embodiments, the device also includes a control module for generating target audio output parameters required for the audio output device to output audio based on the center position coordinates of the currently playing screen if the first detection result indicates that there is no target object in the screenshot; and based on the target audio output parameters, controlling the audio output device to output audio at a position corresponding to the center position coordinates.

[0125] In some embodiments, the position determination module 31 is used to construct an array structure corresponding to the target object based on all key point coordinates of the target object, and select target position information corresponding to the sound emitting part from the array structure.

[0126] In some embodiments, the parameter generation module 32 is used to perform numerical conversion processing on the target position information to obtain a target numerical value within a preset range; based on a mapping relationship between a pre-created numerical value and the volume ratio of each channel in the audio output device, determine the target channel volume ratio corresponding to the target numerical value; and according to the target channel volume ratio, adjust the target audio output parameters corresponding to each channel of the audio output device.

[0127] In some embodiments, the device also includes an effect display module, which is used to respond to the start operation of a preset function, detect whether there is a target object in the current playback screen and the target object performs a sound action; when it is first detected that there is a target object in the current playback screen and the target object performs a sound action, determine the target position information of the sound part corresponding to the sound action; call the preset mark to display at the position corresponding to the target position information, and hide the preset mark after a preset period of time.

[0128] In some embodiments, the device also includes a volume adjustment module, which is used to detect the distance value between the current user and the audio output device; if the distance value is greater than a first preset threshold, the output volume value of the audio output device is enhanced; if the distance value is less than a second preset threshold, the output volume value of the audio output device is weakened, and the first preset threshold is greater than the second preset threshold.

[0129] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same entity or several entities.

[0130] As can be seen from the above, the audio control device 30 provided in the embodiment of the present application, wherein the position determination module 31 is used to determine the target position information of the sound-emitting part corresponding to the sound-emitting action when it is detected that there is a target object in the current playback screen and the target object performs a sound-emitting action; the parameter generation module 32 is used to generate the target audio output parameters required for the audio output device to output audio according to the target position information; the audio control module 33 is used to control the audio output device to output the audio corresponding to the target position information based on the target audio output parameters. By generating the target audio output parameters required by the audio output device according to the target position information of the sound-emitting part corresponding to the sound-emitting action when it is detected that there is a target object in the current playback screen and the target object performs a sound-emitting action, so as to control the audio output device to output the audio corresponding to the target position information, the sound source position is synchronized with the screen action, and the user's audio-visual experience is enhanced.

[0131] An embodiment of the present application further provides an audio control device, which can be integrated into an electronic device.

[0132] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same entity or several entities.

[0133] See also Fig.10 , Fig.10 Another structural diagram of an audio control device provided in an embodiment of the present application, the audio control device 30 includes a memory 120, one or more processors 180, and one or more applications, wherein the one or more applications are stored in the memory 120 and configured to be executed by the processor 180; the processor 180 may include a position determination module 31, a parameter generation module 32, and an audio control 33. For example, the structure and connection relationship of the above components may be as follows:

[0134] The memory 120 can be used to store applications and data. The applications stored in the memory 120 include executable codes. The applications can be composed of various functional modules. The processor 180 executes various functional applications and data processing by running the applications stored in the memory 120. In addition, the memory 120 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 120 may also include a memory controller to provide the processor 180 with access to the memory 120.

[0135] The processor 180 is the control center of the device, and uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the device and processes data by running or executing applications stored in the memory 120, and calling data stored in the memory 120, so as to monitor the device as a whole. Optionally, the processor 180 may include one or more processing cores; preferably, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs.

[0136] Specifically in this embodiment, the processor 180 will load the executable code corresponding to the process of one or more applications into the memory 120 according to the following instructions, and the processor 180 will run the application stored in the memory 120 to implement various functions:

[0137] A position determination instruction, for determining target position information of a sound-generating part corresponding to the sound-generating action when it is detected that a target object exists in the current playback picture and the target object performs a sound-generating action;

[0138] A parameter generation instruction, used to generate target audio output parameters required for the audio output device to output audio according to the target position information;

[0139] The audio control instruction is used to control the audio output device to output audio corresponding to the target position information based on the target audio output parameter.

[0140] In some embodiments, the position determination instruction is used to obtain a screenshot corresponding to the currently playing screen, input the screenshot into a pre-trained target detection model to perform a target detection operation, and output a first detection result; if the first detection result indicates that there is a target object in the screenshot, then further detect whether the target object performs a sound action, and output a second detection result; if the second detection result indicates that the target object performs a sound action, then determine the target position information of the sound part corresponding to the sound action.

[0141] In some embodiments, the position determination instruction is used to take the moment when the target object is detected as a reference moment, and obtain at least two screen screenshots from the current playback screen that are located after the reference moment and contain the target object; respectively extract the position coordinates of each sound-making part in the at least two screen screenshots; detect whether the relative position coordinates of each sound-making part in the at least two screen screenshots have changed, and then determine whether the target object performs a sound-making action.

[0142] In some embodiments, the program also includes control instructions for generating target audio output parameters required for the audio output device to output audio based on the center position coordinates of the currently playing screen if the first detection result indicates that there is no target object in the screenshot; and based on the target audio output parameters, controlling the audio output device to output audio at a position corresponding to the center position coordinates.

[0143] In some embodiments, the position determination instruction is used to construct an array structure corresponding to the target object based on all key point coordinates of the target object, and select target position information corresponding to the sound emitting part from the array structure.

[0144] In some embodiments, the parameter generation instruction is used to perform numerical conversion processing on the target position information to obtain a target numerical value within a preset range; based on a mapping relationship between a pre-created numerical value and the volume ratio of each channel in the audio output device, determine the target channel volume ratio corresponding to the target numerical value; and according to the target channel volume ratio, adjust the target audio output parameters corresponding to each channel of the audio output device.

[0145] In some embodiments, the program also includes effect display instructions, which are used to respond to the start operation of a preset function, detect whether there is a target object in the current playback screen and the target object performs a sound action; when it is first detected that there is a target object in the current playback screen and the target object performs a sound action, determine the target position information of the sound part corresponding to the sound action; call the preset logo to display at the position corresponding to the target position information, and hide the preset logo after a preset period of time.

[0146] In some embodiments, the program also includes a volume adjustment instruction for detecting the distance value between the current user and the audio output device; if the distance value is greater than a first preset threshold, the output volume value of the audio output device is enhanced; if the distance value is less than a second preset threshold, the output volume value of the audio output device is weakened, and the first preset threshold is greater than the second preset threshold.

[0147] The present application also provides an electronic device. Fig.11 , Fig.11 The electronic device 1200 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device can be used to implement the audio control method provided in the above embodiment. The electronic device 1200 can be a television, a smart phone, or a tablet computer.

[0148] like Fig.11As shown, the electronic device 1200 may include an RF (Radio Frequency) circuit 110, a memory 120 including one or more (only one is shown in the figure) computer-readable storage media, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a transmission module 170, a processor 180 including one or more (only one is shown in the figure) processing cores, and a power supply 190. Those skilled in the art will understand that Fig.11 The structure of the electronic device 1200 shown in the figure does not constitute a limitation on the electronic device 1200, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0149] The RF circuit 110 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other televisions. The RF circuit 110 may include various existing circuit elements for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a user identity module (SIM) card, a memory, etc. The RF circuit 110 can communicate with various networks such as the Internet, an intranet, a wireless network, or communicate with other televisions via a wireless network.

[0150] The memory 120 can be used to store software programs and modules, such as the program instructions / modules corresponding to the audio control method in the above embodiment. The processor 180 executes various functional applications and data processing by running the software programs and modules stored in the memory 120. The vibration reminder mode can be automatically selected according to the current scene of the electronic device to perform audio control, which can not only ensure that scenes such as meetings are not disturbed, but also ensure that users can perceive incoming calls, thereby improving the intelligence of the electronic device. The memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 120 may further include a memory remotely arranged relative to the processor 180, and these remote memories may be connected to the electronic device 1200 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0151] The input unit 130 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 130 may include a touch-sensitive surface 131 and other input devices 132. The touch-sensitive surface 131, also known as a touch display screen or touch pad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 131), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 131 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch position, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 180, and can receive and execute commands sent by the processor 180. In addition, the touch-sensitive surface 131 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 131, the input unit 130 may further include other input devices 132. Specifically, the other input devices 132 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0152] The display unit 140 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 1200, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 140 may include a display panel 141. Optionally, the display panel 141 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 131 may cover the display panel 141. When the touch-sensitive surface 131 detects a touch operation on or near it, it is transmitted to the processor 180 to determine the type of touch event. Subsequently, the processor 180 provides corresponding visual output on the display panel 141 according to the type of touch event. Although in Fig.11 In the embodiment, the touch-sensitive surface 131 and the display panel 141 are implemented as two independent components to realize the input and output functions. However, in some embodiments, the touch-sensitive surface 131 and the display panel 141 can be integrated to realize the input and output functions.

[0153] The electronic device 1200 may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 141 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 141 and / or the backlight when the electronic device 1200 is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 1200, they will not be repeated here.

[0154] The audio circuit 160, the speaker 161, and the microphone 162 can provide an audio interface between the user and the electronic device 1200. The audio circuit 160 can transmit the received audio data to the speaker 161 after converting the received audio data into an electrical signal, which is converted into a sound signal for output by the speaker 161; on the other hand, the microphone 162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 160 and converted into audio data, and then the audio data is processed by the output processor 180, and then sent to another terminal through the RF circuit 110, or the audio data is output to the memory 120 for further processing. The audio circuit 160 may also include an earplug jack to provide communication between an external headset and the electronic device 1200.

[0155] The electronic device 1200 can help users send and receive emails, browse web pages, and access streaming media through the transmission module 170 (such as a Wi-Fi module), which provides users with wireless broadband Internet access. Fig.11 The transmission module 170 is shown, but it is understandable that it is not an essential component of the electronic device 1200 and can be omitted as needed without changing the essence of the invention.

[0156] The processor 180 is the control center of the electronic device 1200. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 120, and calling data stored in the memory 120, it executes various functions of the electronic device 1200 and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 180 may include one or more processing cores; in some embodiments, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 180.

[0157] The electronic device 1200 also includes a power supply 190 for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 180 through a power management system, so that the power management system can realize functions such as managing discharge and power consumption. The power supply 190 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0158] Although not shown, the electronic device 1200 may also include a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit 140 of the electronic device 1200 is a touch screen display, and the electronic device 1200 also includes a memory 120, and one or more programs, wherein one or more programs are stored in the memory 120, and are configured to be executed by one or more processors 180. One or more programs include instructions for performing the following operations:

[0159] A position determination instruction, for determining target position information of a sound-generating part corresponding to the sound-generating action when it is detected that a target object exists in the current playback picture and the target object performs a sound-generating action;

[0160] A parameter generation instruction, used to generate target audio output parameters required for the audio output device to output audio according to the target position information;

[0161] The audio control instruction is used to control the audio output device to output audio corresponding to the target position information based on the target audio output parameter.

[0162] In some embodiments, the position determination instruction is used to obtain a screenshot corresponding to the currently playing screen, input the screenshot into a pre-trained target detection model to perform a target detection operation, and output a first detection result; if the first detection result indicates that there is a target object in the screenshot, then further detect whether the target object performs a sound action, and output a second detection result; if the second detection result indicates that the target object performs a sound action, then determine the target position information of the sound part corresponding to the sound action.

[0163] In some embodiments, the position determination instruction is used to take the moment when the target object is detected as a reference moment, and obtain at least two screen screenshots from the current playback screen that are located after the reference moment and contain the target object; respectively extract the position coordinates of each sound-making part in the at least two screen screenshots; detect whether the relative position coordinates of each sound-making part in the at least two screen screenshots have changed, and then determine whether the target object performs a sound-making action.

[0164] In some embodiments, the program also includes control instructions for generating target audio output parameters required for the audio output device to output audio based on the center position coordinates of the currently playing screen if the first detection result indicates that there is no target object in the screenshot; and based on the target audio output parameters, controlling the audio output device to output audio at a position corresponding to the center position coordinates.

[0165] In some embodiments, the position determination instruction is used to construct an array structure corresponding to the target object based on all key point coordinates of the target object, and select target position information corresponding to the sound emitting part from the array structure.

[0166] In some embodiments, the parameter generation instruction is used to perform numerical conversion processing on the target position information to obtain a target numerical value within a preset range; based on a mapping relationship between a pre-created numerical value and the volume ratio of each channel in the audio output device, determine the target channel volume ratio corresponding to the target numerical value; and according to the target channel volume ratio, adjust the target audio output parameters corresponding to each channel of the audio output device.

[0167] In some embodiments, the program also includes effect display instructions, which are used to respond to the start operation of a preset function, detect whether there is a target object in the current playback screen and the target object performs a sound action; when it is first detected that there is a target object in the current playback screen and the target object performs a sound action, determine the target position information of the sound part corresponding to the sound action; call the preset logo to display at the position corresponding to the target position information, and hide the preset logo after a preset period of time.

[0168] In some embodiments, the program also includes a volume adjustment instruction for detecting the distance value between the current user and the audio output device; if the distance value is greater than a first preset threshold, the output volume value of the audio output device is enhanced; if the distance value is less than a second preset threshold, the output volume value of the audio output device is weakened, and the first preset threshold is greater than the second preset threshold.

[0169] The embodiment of the present application also provides an electronic device, which may be a smart phone, a television, a computer, or other device.

[0170] As can be seen from the above, the embodiment of the present application provides an electronic device 1200, and the electronic device 1200 performs the following steps:

[0171] When it is detected that there is a target object in the current playing picture and the target object performs a sound-making action, target position information of a sound-making part corresponding to the sound-making action is determined;

[0172] Generating target audio output parameters required for the audio output device to output audio according to the target position information;

[0173] Based on the target audio output parameter, the audio output device is controlled to output audio corresponding to the target position information.

[0174] An embodiment of the present application further provides a storage medium, wherein a computer program is stored in the storage medium. When the computer program is executed on a computer, the computer executes the audio control method described in any one of the above embodiments.

[0175] It should be noted that, for the audio control method described in the present application, ordinary testers in the field can understand that all or part of the process of implementing the audio control method described in the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as stored in the memory of an electronic device, and executed by at least one processor in the electronic device, and the execution process may include the process of the embodiment of the audio control method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM, ReadOnly Memory), a random access memory (RAM, Random Access Memory), etc.

[0176] For the audio control device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0177] The audio control method, device, medium and equipment provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application; at the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An audio control method, applied to an audio output device, characterized in that: include: When it is detected that there is a target object in the current playing picture and the target object performs a sound-making action, target position information of a sound-making part corresponding to the sound-making action is determined; Generating target audio output parameters required for the audio output device to output audio according to the target position information; Based on the target audio output parameter, the audio output device is controlled to output audio corresponding to the target position information.

2. The audio control method according to claim 1, characterized in that: When it is detected that there is a target object in the current playing picture and the target object performs a sound-making action, determining the target position information of the sound-making part corresponding to the sound-making action includes: Obtaining a screenshot corresponding to the current playback screen, inputting the screenshot into a pre-trained target detection model to perform a target detection operation, and outputting a first detection result; If the first detection result indicates that there is a target object in the screenshot, further detecting whether the target object performs a sound-making action and outputting a second detection result; If the second detection result indicates that the target object performs a sound-making action, the target position information of the sound-making part corresponding to the sound-making action is determined.

3. The audio control method according to claim 2, characterized in that: The further detecting whether the target object performs a sound-making action includes: Taking the time when the target object is detected as a reference time, obtaining at least two screenshots of the current playing screen that are located after the reference time and contain the target object; Respectively extracting the position coordinates of each sound-producing part in the at least two screenshots; It is detected whether the relative position coordinates of each sound-generating part in the at least two screenshots have changed, thereby determining whether the target object performs a sound-generating action.

4. The audio control method according to claim 2, characterized in that: After outputting the first detection result, the method further includes: If the first detection result indicates that the target object does not exist in the screenshot, generating target audio output parameters required for the audio output device to output audio according to the center position coordinates of the current playing screen; Based on the target audio output parameter, the audio output device is controlled to output audio at a position corresponding to the center position coordinates.

5. The audio control method according to claim 1, characterized in that: The step of determining target position information of the vocalization part corresponding to the vocalization action includes: Based on the coordinates of all key points of the target object, construct an array structure corresponding to the target object, and select target position information corresponding to the sound-producing part from the array structure; Generating target audio output parameters required for the audio output device to output audio according to the target position information includes: The target position information is converted into a numerical value to obtain a target value within a preset range; Determining a target channel volume ratio corresponding to the target value based on a mapping relationship between a pre-created value and a volume ratio of each channel in the audio output device; According to the target channel volume ratio, the target audio output parameters corresponding to each channel of the audio output device are adjusted.

6. The audio control method according to claim 1, characterized in that: Before determining the target position information of the sound-generating part corresponding to the sound-generating action when the target object is detected to exist in the current playing picture and the target object performs a sound-generating action, the method further includes: In response to a start operation for a preset function, detecting whether a target object exists in the current playing picture and the target object performs a sound-making action; When it is detected for the first time that there is a target object in the current playing picture and the target object performs a sound-making action, target position information of a sound-making part corresponding to the sound-making action is determined; The preset mark is called to be displayed at the position corresponding to the target position information, and the preset mark is hidden after a preset time period.

7. The audio control method according to claim 1, characterized in that: After controlling the audio output device to output audio corresponding to the target position information, the method further includes: Detecting the distance between the current user and the audio output device; If the distance value is greater than a first preset threshold, increasing the output volume value of the audio output device; If the distance value is less than a second preset threshold, the output volume value of the audio output device is weakened, and the first preset threshold is greater than the second preset threshold.

8. An audio control device, applied to an audio output device, characterized in that: The audio control device comprises: A position determination module, for determining target position information of a sounding part corresponding to the sounding action when detecting that a target object exists in the current playing picture and the target object performs a sounding action; A parameter generation module, used for generating target audio output parameters required for the audio output device to output audio according to the target position information; The audio control module is used to control the audio output device to output audio corresponding to the target position information based on the target audio output parameter.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the audio control method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The system comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the audio control method according to any one of claims 1 to 7.

Images