Object recognition method, device and equipment based on sound and picture tracking and storage medium

Abstract

The invention discloses an object recognition method, device and equipment based on sound and picture tracking and a storage medium, relates to the technical field of display, and discloses an object recognition method based on sound and picture tracking, which comprises the following steps: dynamically obtaining a current display picture; obtaining the sounding attribute of each object contained in the current display picture, and determining a soundable object according to the sounding attribute; and determining a tracking object from the soundable objects, and carrying out sound and picture tracking on the tracking object. According to the invention, the identification accuracy of the tracking object is improved, and the sound and picture tracking effect is improved.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to an object recognition method, device, equipment and storage medium based on audio and video tracking. Background Art

[0002] As high-definition display technology is changing with each passing day, the boundaries of visual experience are constantly being broadened, and the sound technology that complements it has become an indispensable part of improving the overall viewing experience. As an innovative breakthrough in the field of sound effects, audio and video tracking technology aims to accurately match the movement trajectory of sound and picture, so that the audience feels as if they are in every scene of the film and experience the wonderful feeling of sound following the picture. The tracking object selected as the reference of the sound source movement trajectory is the key factor affecting the improvement of audio and video tracking effect.

[0003] Ideally, the tracking object should be able to accurately reflect the sound source and its dynamic changes in the scene. However, in related technologies, the object with the largest display area in the picture is often directly used as the tracking object for sound and picture tracking. However, this can easily misidentify large-area background elements, such as clouds and trees, as tracking objects, resulting in misalignment of sound and picture movement, which greatly reduces the effect of sound and picture tracking.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide an object recognition method, device, equipment and storage medium based on audio and video tracking, aiming to improve the recognition accuracy of the tracked object in the picture and enhance the audio and video tracking effect.

[0006] To achieve the above objectives, the present application provides an object recognition method based on audio and video tracking, the method comprising:

[0007] Dynamically obtain the current display screen;

[0008] Obtaining the sound properties of each object included in the current display screen, and determining the soundable object according to the sound properties;

[0009] A tracking object is determined from the sound-producing objects, and audio and video tracking is performed on the tracking object.

[0010] Optionally, before the step of obtaining the sound properties of each object contained in the current display screen, the method further includes:

[0011] Determining whether there is a human subject in the current display image;

[0012] In the case where a human object exists in the current display image, determining the human object as the tracking object; or,

[0013] In the case that there is no human object in the current display picture, a step of obtaining the vocalization attribute of each object included in the current display picture is performed.

[0014] Optionally, the step of determining the human object as the tracking object includes:

[0015] In the case that the current display screen includes a plurality of human objects, the human object closest to the center position of the current display screen is determined as the tracking object.

[0016] Optionally, the step of obtaining the sound properties of each object contained in the current display screen includes:

[0017] Identifying a moving object in the currently displayed picture;

[0018] Acquire the sound properties of each of the moving objects.

[0019] Optionally, the step of determining a tracking object from the objects that can make sounds includes:

[0020] In the case that the current display screen includes a plurality of the sound-emitting objects, the sound-emitting object having the largest display area is determined as the tracking object.

[0021] Optionally, before the step of determining the object capable of emitting sound with the largest display area as the tracking object, the step further includes:

[0022] Determining an area threshold according to a display area of ​​a first object, wherein the first object is a voicable object with the largest display area, and the display area of ​​the first object is greater than the area threshold;

[0023] In the case where the current display screen includes a plurality of sound-emitting objects whose display areas are larger than the area threshold, the object closest to the center position of the current display screen is selected from the sound-emitting objects whose display areas are larger than the area threshold, and is used as the tracking object; or

[0024] In the case that the current display screen includes a soundable object whose display area is larger than the area threshold, the step of determining the soundable object with the largest display area as the tracking object is performed.

[0025] Optionally, the step of performing audio and video tracking on the tracking object includes:

[0026] Determine target position information of the tracked object in the current display screen;

[0027] Determine, according to the target position information, area proportion information of the tracking object in the playback area corresponding to each channel;

[0028] Generate channel playback information according to the area proportion information and preset channel configuration information;

[0029] Each of the channels is controlled to perform audio playback based on the channel playback information.

[0030] In addition, to achieve the above-mentioned purpose, the embodiment of the present application further proposes an object recognition device based on sound and picture tracking, the device is applied to an electronic device, and the device includes:

[0031] Acquisition module, dynamically acquires the current display screen;

[0032] A determination module, which obtains the sound properties of each object contained in the current display screen, and determines the soundable object according to the sound properties;

[0033] The audio and video tracking module determines a tracking object from the sound-producing objects and performs audio and video tracking on the tracking object.

[0034] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the object recognition method based on audio and video tracking as described above.

[0035] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the object recognition method based on sound and picture tracking as described above are implemented.

[0036] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also proposes a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the object recognition method based on audio and video tracking as described above.

[0037] One or more technical solutions proposed in the embodiments of the present application have at least the following technical effects: by dynamically acquiring the current display screen, the sound properties of each object contained in the current display screen are obtained, and according to the sound properties, the soundable object is determined, and then the tracking object is determined from the soundable object, so as to achieve audio and video tracking of the tracking object. The embodiments of the present application determine the tracking object from the current display screen through the sound properties, so as to more accurately locate the sounding object that the user is concerned about, accurately reflect the sound source and its dynamic changes in the display screen, improve the audio and video tracking effect, and enhance the user's audio-visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart of the first embodiment of the object recognition method based on audio and video tracking provided by the present application;

[0041] Figure 2 A schematic diagram of a scene provided in Embodiment 1 of the object recognition method based on sound and picture tracking of the present application;

[0042] Figure 3 This is a flow chart of a more complete embodiment of the present application;

[0043] Figure 4 Schematic diagram of the framework structure of the object recognition device based on audio and video tracking in an embodiment of the present application;

[0044] Figure 5 Schematic diagram of the device structure of the hardware operating environment of the electronic device involved in the object recognition method based on audio and video tracking in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] In conventional technology, the tracking object selected as a reference for the motion trajectory of the sound source is a key factor affecting the improvement of the sound and picture tracking effect. Ideally, the tracking object should be able to accurately reflect the sound source and its dynamic changes in the scene, such as character dialogue, vehicle driving, etc. However, in conventional technology, the object with the largest display area in the picture is often directly used as the tracking object for sound and picture tracking. However, this can easily misidentify large-area background elements, such as clouds and trees, as tracking objects, resulting in misalignment of sound and picture movement, which greatly reduces the effect of sound and picture tracking.

[0049] The embodiment of the present application provides a solution, which dynamically obtains the current display screen, and then obtains the sound properties of each object contained in the current display screen, and determines the soundable object according to the sound properties, and then determines the tracking object from the soundable object, so as to achieve audio and video tracking of the tracking object. The embodiment of the present application determines the tracking object from the current display screen through the sound properties, and can more accurately locate the sound object that the user is concerned about, and accurately reflect the sound source and its dynamic changes in the display screen, improve the audio and video tracking effect, and enhance the user's audio-visual experience.

[0050] Reference Figure 1 , Figure 1 This is a flow chart of the first embodiment of the object recognition method based on audio and video tracking of the present application.

[0051] It should be noted that the execution subject of the object recognition method based on audio and video tracking can be an electronic device, and the electronic device can be a display device, such as a television, a computer, a projector, etc. This embodiment is not limited to this. For the convenience of description, the execution subject is omitted to explain each embodiment. The object recognition method based on audio and video tracking includes:

[0052] Step S10, dynamically acquiring the current display screen;

[0053] Step S20, obtaining the sound properties of each object included in the current display screen, and determining the soundable object according to the sound properties;

[0054] Step S30, determining a tracking object from the objects that can make sounds, and performing audio and video tracking on the tracking object.

[0055] In a feasible embodiment, the current display screen is dynamically acquired, and the sound attributes of each object contained in the current display screen are determined, wherein the sound attributes may include: audible and non-audible; objects with the sound attribute of being audible are determined as audible objects. Soundable objects refer to objects that can be visually identified and produce sound in actual situations. Since the tracking object should reflect the source of the sound in the display screen and its dynamic changes, the tracking object can be determined from the soundable objects, and audio and video tracking can be performed; thereby avoiding the occurrence of erroneous identification of large areas of background elements as tracking objects, improving the recognition accuracy of tracking objects, and enhancing the user's sense of presence and immersion when watching videos.

[0056] It should be understood that the current display screen may contain one or more screen elements (i.e., objects), and the tracking object is a key screen element that brings a 3D-like surround feeling to the user in terms of hearing. For example, in videos such as animations, movies, or games, characters or people are tracking objects. By identifying the position and movement trajectory of the characters in the picture, the playback position of the corresponding audio data (such as dialogue, footsteps, etc.) can be adjusted to make the sound seem to be emitted from the position of the character, enhancing the sense of immersion. In addition to the characters, objects or props in the picture can also be tracking objects. When there is an object that is making a sound (such as an alarm clock, a car, etc.) in the picture, the real source of the sound can be simulated by adjusting the playback position of the audio data. In some cases, the entire scene or environment itself can also be regarded as a tracking object. For example, in environments such as forests, valleys, or cities, the playback position and intensity of environmental sound effects (such as wind, birdsong, traffic, etc.) can be adjusted to create an immersive feeling.

[0057] Exemplarily, the current display image is captured according to a preset frame rate.

[0058] Exemplarily, the trained neural network model can be used to identify the type of objects in the current display image and determine the sound properties of each object.

[0059] For example, when the tracking object is determined and the tracking object is included in multiple consecutively acquired current display images, the tracking object can be continuously tracked in audio and video according to the position information of the tracking object in the current display image. When the tracking object disappears from the current display image, the tracking object is re-determined.

[0060] In this embodiment, the audio data weight of each target sound unit playing the target audio data is adjusted according to the distance between the target sound unit and the target position, so as to achieve the effect of matching the sound and the tracking object position, allowing the user to clearly feel that the sound output has actual movement with the movement of the picture, and through the spatial movement of the sound position, an intuitive feeling of sound and picture tracking the sound is achieved.

[0061] Based on the above first embodiment, a second embodiment of the object recognition method based on sound and picture tracking of the present application is proposed. In this embodiment, before the step of obtaining the sound properties of each object included in the current display screen in step S20, the method further includes:

[0062] Step S11, determining whether there is a human object in the current display image;

[0063] In a feasible embodiment, the type of each object in the current display image is identified to determine whether there is an object of human type.

[0064] Step S12, when there is a human object in the current display image, the human object is determined as a tracking object; or,

[0065] Step S13, when there is no human object in the current display picture, a step of obtaining the vocalization attributes of each object included in the current display picture is executed.

[0066] In a feasible embodiment, since people are the subject of emotional expression, by using people as tracking objects, the output content of the device can follow people's movements, expressions and sounds, and more accurately capture and convey emotional information, thereby enhancing the emotional resonance of the audience. Therefore, it is determined that the human object belongs to the highest priority object and is determined as the tracking object. If there is no human object in the current display screen, step S20 is executed to obtain the sound properties of each object contained in the current display screen, as well as subsequent steps.

[0067] Exemplarily, when the tracking object is determined and the tracking object is a non-human object, the step of obtaining the sound properties of each object contained in the current display screen is performed, and when it is re-determined that there is a human object in the current display screen, the human object is replaced with a new tracking object. For example, a vehicle object is used as the tracking object, and when multiple current display screens obtained continuously contain vehicle objects, the vehicle object is always used as the tracking object; but when a human object appears in the current display screen, even if the vehicle object has not disappeared from the current display screen, the human object can be replaced with a new tracking object according to the object priority.

[0068] For example, refer to Figure 2, the electronic device is a television, and the current display screen 100 is a surfing scene, which includes objects: people, clouds, waves, surfboards, trees and beaches ( Figure 2 In the conventional technology, the object with the largest display area, that is, the cloud, is usually determined as the tracking object for audio and video tracking, which may not meet the photographer's intention and the audience's psychological expectations. However, the embodiment of the present application identifies the type of each object, determines that there is a human object 101 in the current display image, and then determines the human object 101 as the tracking object, thereby focusing on the process of the surfing showman, which is more in line with the photographer's intention.

[0069] In one feasible implementation, step S12, determining a human subject as a tracking subject, includes:

[0070] Step S121 : when the current display screen includes a plurality of human objects, a human object closest to the center position of the current display screen is determined as a tracking object.

[0071] In a feasible embodiment, when there is only one human object in the current display screen, the human object can be directly determined as the tracking object. When the current display screen includes multiple human objects, considering that the video shooter usually places the key person who wants to be shown at the current moment in the center of the screen, the distance between each human object in the current display screen and the center of the current display screen can be determined first, and the human object closest to the center position can be determined as the tracking object.

[0072] Exemplarily, the coordinates of each human object in the current display screen are determined, the difference between the coordinates of the human object and the coordinates of the center point of the current display screen is calculated, and the human object with the smallest difference is determined as the tracking object.

[0073] In this embodiment, by taking a person as the tracking object, the output content of the device can follow the person's movements, expressions and sounds, and more accurately capture and convey emotional information, thereby enhancing the audience's emotional resonance.

[0074] Based on the above first and / or second embodiments, a third embodiment of the object recognition method based on sound and picture tracking of the present application is proposed. In this embodiment, step S20, the step of obtaining the sound properties of each object included in the current display screen includes:

[0075] Step S21, identifying a moving object in the current display image;

[0076] Step S22, obtaining the sound properties of each moving object.

[0077] In a feasible embodiment, objects in motion (i.e., moving objects) are determined from multiple objects contained in the current display screen, and then the sound properties of each moving object are determined, so that subsequent tracking objects can be screened from the moving objects that can make sound, thereby enhancing the audio and video tracking effect.

[0078] Exemplarily, the difference values ​​of the objects included in the current display picture between multiple consecutive frames may be determined, and the objects whose difference values ​​are greater than a preset difference threshold are determined as moving objects.

[0079] Exemplarily, whether each object is a moving object and the sound properties of each moving object may be determined through a pre-trained recognition model.

[0080] Exemplarily, a training data set is created, wherein the training data set includes multiple training images; labels are added to each training image, for example, the label of a bird is sounding and moving, the label of a train is sounding and moving, and the wheel position is not sounding but moving, wherein the labels of some images can also be set to unknown categories, corresponding to uncertain or difficult to classify images, which helps the model learn more features during the training process and improve generalization ability; and there need to be multiple training images corresponding to each class label to ensure that the training can proceed normally and reduce overfitting or underfitting problems caused by insufficient data, and the more pictures there are, the more features the model can learn, and the higher the accuracy of the model after training. During the model training process, it is necessary to provide positive eigenvalues ​​(correctly classified pictures) and negative eigenvalues ​​(wrongly classified or uncertain pictures) to improve the generalization ability of the model; and then use the training data to train the model, and continuously adjust and optimize the model parameters to improve the recognition accuracy and obtain the recognition model.

[0081] In one feasible implementation, step S30, the step of determining the tracking object from the sound-emitting objects includes:

[0082] Step S31: When the current display screen includes a plurality of sound-emitting objects, the sound-emitting object with the largest display area is determined as the tracking object.

[0083] In a feasible embodiment, when there is only one soundable object in the current display screen, the soundable object can be directly determined as the tracking object. When the current display screen includes multiple soundable objects, further screening can be performed based on the display area of ​​each object in the current display screen, and the soundable object with the largest display area can be determined as the tracking object.

[0084] In this embodiment, when the current display screen includes multiple sound-emitting objects, the embodiment of the present application further considers that in visual perception, a large area occupies more visual space and is often more conspicuous. Therefore, the photographer will give a certain object a larger display area to more intuitively convey that the object is the most important element in the scene and the content that he wants to focus on. Therefore, the embodiment of the present application determines the sound-emitting object with the largest display area as the tracking object, thereby achieving a sound and picture tracking effect that is more in line with the photographer's intention and the audience's psychological expectations.

[0085] In a feasible implementation manner, before the step of determining the object capable of emitting sound with the largest display area as the tracking object in step S31, the method further includes:

[0086] Step S311, determining an area threshold according to a display area of ​​a first object, wherein the first object is a voicable object with the largest display area, and the display area of ​​the first object is greater than the area threshold;

[0087] In a feasible embodiment, when the current display screen includes multiple sound-emitting objects, although the sound-emitting object with the largest display area can be directly determined as the tracking object; however, the embodiment of the present application further takes into account that there may be multiple objects with large display areas and similar areas in the display screen at the same time. Therefore, in order to further improve the recognition accuracy of the tracking object, an area threshold is set according to the display area of ​​the sound-emitting object with the largest display area (i.e., the first object), wherein the display area of ​​the first object is greater than the area threshold, which is used to screen out multiple objects with similar areas to the first object from the current display screen.

[0088] Exemplarily, the area threshold is determined according to a preset ratio and the display area of ​​the first object. For example, the preset ratio is 90%, and then 90% of the display area of ​​the first object is determined as the area threshold.

[0089] Step S312, when the current display screen includes multiple sound-emitting objects whose display areas are larger than the area threshold, select the object closest to the center of the current display screen from the sound-emitting objects whose display areas are larger than the area threshold and use it as the tracking object; or

[0090] Step S313, when the current display screen includes a soundable object whose display area is larger than the area threshold, the step of determining the soundable object with the largest display area as the tracking object is executed.

[0091] In a feasible embodiment, when the current display screen includes multiple sound-emitting objects whose display areas are greater than the area threshold, it indicates that there are multiple objects with an area similar to that of the first object. In order to further improve the recognition accuracy of the tracked object, the distance between the object and the center position of the display screen is introduced as an object screening condition, that is, from the sound-emitting objects whose display areas are greater than the area threshold, the object closest to the center position of the current display screen is screened out as the tracking object. In the case where there is only one sound-emitting object whose display area is greater than the area threshold in the current display screen, it indicates that the display area of ​​the first object in the display screen is significantly different from that of other objects, and the first object can be directly determined as the tracking object.

[0092] In this embodiment, by using the movement of the object in the current display screen, the sound properties, the display area and the screen position as the screening conditions for the tracking object, the sound object that the user is concerned about can be located more accurately, and the sound source and its dynamic changes in the display screen can be accurately reflected, thereby achieving an audio and video tracking effect that is more in line with the photographer's intentions and the audience's psychological expectations.

[0093] In order to assist in understanding the above technical solution, the following is a more complete embodiment of the object recognition method based on audio and video tracking. Figure 3, step A101, obtain the current display screen. Step A102, identify each object in the display screen, and determine whether there is a human object. Step A103, when there is only one human object in the current display screen, directly determine the human object as the tracking object, and perform audio and video tracking. Step A104, when there are multiple human objects in the current display screen, determine the human object closest to the center of the current display screen as the tracking object, and perform audio and video tracking. Step A105, when there is no human object in the current display screen, determine the moving objects in the current display screen, identify the sound properties of each moving object, and then determine the soundable object based on the sound properties; then determine whether there are multiple soundable objects in the current display screen. Step A106, when there is only one soundable object in the current display screen, determine the object as the tracking object, and perform audio and video tracking. Step A107, when there are multiple sound-emitting objects in the current display screen, further set an area threshold by displaying the sound-emitting object with the largest display area, wherein the display area of ​​the sound-emitting object with the largest display area is greater than the area threshold. Step A108, determine whether there are multiple sound-emitting objects with display areas greater than the area threshold in the current display screen. Step A109, when there is only one sound-emitting object with a display area greater than the area threshold in the current display screen, determine the object (i.e., the sound-emitting object with the largest display area) as the tracking object, and perform audio and video tracking. Step A110, when there are multiple sound-emitting objects with a display area greater than the area threshold in the current display screen, determine the object closest to the center of the current display screen from the sound-emitting objects with a display area greater than the area threshold as the tracking object, and perform audio and video tracking. The above scheme can more accurately locate the sound-emitting objects that the user is concerned about, accurately reflect the sound source and its dynamic changes in the display screen, improve the audio and video tracking effect, and enhance the user's audio-visual experience.

[0094] Based on the above first, second and / or third embodiments, a fourth embodiment of the object recognition method based on audio and video tracking of the present application is proposed. In this embodiment, step S30, the step of performing audio and video tracking on the tracking object includes:

[0095] Step A10, determining the target position information of the tracking object in the current display screen;

[0096] In a feasible embodiment, when the tracking object is determined, the position information (ie, target position information) of the tracking object in the current display image may be further determined.

[0097] Exemplarily, preset screen coordinate information is obtained. The screen coordinate information can represent a coordinate system corresponding to a display device, and the screen coordinate information can be a polar coordinate system. Then, the position distribution information of the tracking object in the current display screen can be further determined through the screen coordinate information, that is, the distribution of the tracking object in the coordinate system corresponding to the display device, and used as the target position information.

[0098] Step A20, determining area proportion information of the tracking object in the playback area corresponding to each channel according to the target position information;

[0099] In a feasible embodiment, a display screen of a display device to which the object recognition method based on audio and video tracking is applied may be provided with multiple channels, and each channel has its own corresponding playback area, wherein the playback area may represent the auditory coverage area formed by the sound through a specific channel in space. In order to achieve the audio and video tracking effect, the area proportion information of the tracked object in the playback area corresponding to each channel, that is, the proportion in each playback area, may be determined according to the target position information of the tracked object.

[0100] Exemplarily, the display device is a television, which includes: a left channel, a left sky channel, a right channel, and a right sky channel; the playback area corresponding to each channel can be determined according to the screen coordinate information corresponding to the television display screen, including: a left channel playback area corresponding to the left channel, a left sky channel playback area corresponding to the left sky channel, a right channel playback area corresponding to the right channel, and a right sky channel playback area corresponding to the right sky channel. Then, according to the target position information of the tracking object, the area proportion information of the playback area corresponding to each channel is determined, including: a proportion of 70% in the left channel playback area, a proportion of 30% in the left sky channel playback area, a proportion of 0% in the right channel playback area, and a proportion of 0% in the right sky channel playback area.

[0101] Step A30, generating channel playback information according to the area proportion information and preset channel configuration information;

[0102] Step A40: Control each channel to play audio based on the channel playing information.

[0103] In a feasible embodiment, the channel configuration information can be used to indicate whether each channel is output, the mapping relationship between the channel output and the area ratio information, etc. Therefore, the actual channel playback information can be further generated according to the area ratio information and the channel configuration information, such as the playback intensity, playback delay and other information, and based on the channel playback information, each channel is controlled to play audio to achieve the sound and picture tracking effect.

[0104] Exemplarily, according to the area proportion information and the channel configuration information, the playback sound intensity information and the playback delay information corresponding to each channel are determined, and according to the playback sound intensity information and the playback delay information, the channel playback information is generated.

[0105] Exemplarily, the display device is a television, which includes: a left channel, a left sky channel, a right channel, and a right sky channel; and the playback areas corresponding to each channel include: a left channel playback area corresponding to the left channel, a left sky channel playback area corresponding to the left sky channel, a right channel playback area corresponding to the right channel, and a right sky channel playback area corresponding to the right sky channel. The area proportion information is that the tracking object accounts for 70% of the left channel playback area, 30% of the left sky channel playback area, 0% of the right channel playback area, and 0% of the right sky channel playback area; and then combined with the channel configuration information, the channel playback information generated includes: the left channel audio playback sound intensity is 0.7, the left sky channel audio playback sound intensity is 0.3, the right channel audio playback sound intensity is 0.1, and the right sky channel audio playback sound intensity is 0.1. Therefore, according to the channel playback information, the left channel can be controlled to play the left channel audio at a sound intensity of 0.7, the left sky channel can be controlled to play the left channel audio at a sound intensity of 0.7, the right channel can be controlled to play the left channel audio at a sound intensity of 0.7, and the right sky channel can be controlled to play the left channel audio at a sound intensity of 0.7. This allows the user to output a larger sound intensity through the left channel and the left sky sound, which is more consistent with the feeling of seeing the tracked object on the left side of the screen, thereby achieving the effect of improving the user's audio-visual experience.

[0106] In this embodiment, by adjusting the parameters of the sound channel, the sound and picture tracking effect is achieved, thereby enhancing the user's audio-visual experience.

[0107] The present application embodiment provides an object recognition device based on audio and video tracking, referring to Figure 4 , the device is applied to electronic equipment, the device comprises:

[0108] An acquisition module 10 dynamically acquires the current display image;

[0109] The determination module 20 obtains the sound properties of each object contained in the current display screen, and determines the soundable object according to the sound properties;

[0110] The audio and video tracking module 30 determines a tracking object from the sound-producing objects, and performs audio and video tracking on the tracking object.

[0111] An embodiment of the present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the object recognition method based on audio and video tracking in the above-mentioned embodiment 1.

[0112] Reference below Figure 5 , respectively show the structural schematic diagrams of electronic devices suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0113] like Figure 5 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0114] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0115] The electronic device provided in the embodiment of the present application adopts the object recognition method based on sound and picture tracking in the above embodiment, which can improve the recognition accuracy of the tracked object and enhance the sound and picture tracking effect. Compared with conventional technology, the beneficial effects of the electronic device provided in the embodiment of the present application are the same as the beneficial effects of the object recognition method based on sound and picture tracking provided in the above embodiment, and other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0116] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0117] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0118] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the object recognition method based on audio and video tracking in the above-mentioned embodiment.

[0119] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium that includes or stores a program that can be used by or in combination with an instruction execution system, system or device. The program code included on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0120] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0121] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device is enabled to: dynamically obtain the current display screen; obtain the sound properties of each object contained in the current display screen, and determine the soundable object based on the sound properties; determine the tracking object from the soundable objects, and perform audio and video tracking on the tracking object.

[0122] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0123] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0125] The readable storage medium provided in the embodiment of the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned object recognition method based on sound and picture tracking, which can improve the recognition accuracy of the tracked object and enhance the sound and picture tracking effect. Compared with conventional technologies, the beneficial effects of the computer-readable storage medium provided in the embodiment of the present application are the same as the beneficial effects of the object recognition method based on sound and picture tracking provided in the above-mentioned embodiment, and will not be repeated here.

[0126] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An object recognition method based on sound and picture tracking, characterized in that: The method includes: Dynamically obtain the current display screen; Obtaining the sound properties of each object included in the current display screen, and determining the soundable object according to the sound properties; A tracking object is determined from the sound-producing objects, and audio and video tracking is performed on the tracking object.

2. The method according to claim 1, characterized in that Before the step of obtaining the sound properties of each object contained in the current display screen, the method further includes: Determining whether there is a human subject in the current display image; In the case where a human object exists in the current display image, determining the human object as the tracking object; or, In the case that there is no human object in the current display picture, a step of obtaining the vocalization attribute of each object included in the current display picture is performed.

3. The method according to claim 2, characterized in that The step of determining the human object as the tracking object comprises: In the case that the current display screen includes a plurality of human objects, the human object closest to the center position of the current display screen is determined as the tracking object.

4. The method according to claim 1, characterized in that The step of obtaining the sound properties of each object contained in the current display screen includes: Identifying a moving object in the currently displayed picture; Acquire the sound properties of each of the moving objects.

5. The method according to claim 1, characterized in that The step of determining the tracking object from the sound-emitting objects comprises: In the case that the current display screen includes a plurality of the sound-emitting objects, the sound-emitting object having the largest display area is determined as the tracking object.

6. The method according to claim 5, characterized in that Before the step of determining the object capable of emitting sound with the largest display area as the tracking object, the method further includes: Determining an area threshold according to a display area of ​​a first object, wherein the first object is a voicable object with the largest display area, and the display area of ​​the first object is greater than the area threshold; In the case where the current display screen includes a plurality of sound-emitting objects whose display areas are larger than the area threshold, the object closest to the center position of the current display screen is selected from the sound-emitting objects whose display areas are larger than the area threshold, and is used as the tracking object; or In the case that the current display screen includes a soundable object whose display area is larger than the area threshold, the step of determining the soundable object with the largest display area as the tracking object is performed.

7. The method according to claim 1, characterized in that The step of performing audio and video tracking on the tracking object comprises: Determine target position information of the tracked object in the current display screen; Determine, according to the target position information, area proportion information of the tracking object in the playback area corresponding to each channel; Generate channel playback information according to the area proportion information and preset channel configuration information; Each of the channels is controlled to perform audio playback based on the channel playback information.

8. An object recognition device based on audio and video tracking, characterized in that: The device comprises: Acquisition module, dynamically acquires the current display screen; A determination module, which obtains the sound properties of each object contained in the current display screen, and determines the soundable object according to the sound properties; The audio and video tracking module determines a tracking object from the sound-producing objects and performs audio and video tracking on the tracking object.

9. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the object recognition method based on audio and video tracking as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the object recognition method based on sound and picture tracking as described in any one of claims 1 to 7 are implemented.

Images