Object tracking method and electronic device

Abstract

Provided in the embodiments of the present application are an object tracking method and an electronic device, which are applied to the technical field of electronics. The method comprises: displaying a first shooting interface, a first area of the first shooting interface being used for displaying a video stream shot by a camera; when the first area displays the nth image frame of the video stream, receiving a first operation input by a user to the nth image frame; in response to the first operation, performing feature extraction on the nth image frame to obtain a template feature, the nth image frame containing an object; according to the template feature, using a first network to determine whether the (n+x)th image frame of the video stream contains the object; if it is determined by means of the first network that the (n+x)th image frame does not contain the object, according to the template feature, using a second network to determine whether the (n+x)th image frame contains the object, the first network and the second network meeting at least one of first preset conditions. The method can improve tracking precision.

Description

Target tracking method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on February 27, 2023, with application number 202310212380.4 and application name “Target Tracking Method and Electronic Device,” the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of electronic technology, and in particular to a target tracking method and electronic equipment. Background Art

[0003] Visual object tracking, also known as target tracking, refers to locating a specific object within each frame of a video stream. Currently, object tracking is used in various fields and scenarios, such as autonomous driving, sports broadcasting, security monitoring, drone tracking, and photography and video recording using mobile phones and other electronic devices.

[0004] In target tracking methods in related technologies, if the target is deformed, moves, or partially occluded during tracking, tracking of the target is easily lost.

[0005] Summary of the Invention

[0006] The present application provides a target tracking method and electronic device, which can improve tracking accuracy and reduce the possibility of target tracking loss for target search in difficult scenes.

[0007] In a first aspect, the present application provides a target tracking method, which is performed by an electronic device and includes:

[0008] A first shooting interface is displayed, where the first area of ​​the first shooting interface is used to display a video stream captured by a camera; when the first area displays the nth frame image of the video stream, a first operation input by the user on the nth frame image is received, where n is a positive integer; in response to the first operation, feature extraction is performed on the nth frame image to obtain a template feature, and the nth frame image contains a target; based on the template feature, whether the n+xth frame image of the video stream contains a target is determined through a first network, where x is a positive integer; if it is determined through the first network that the n+xth frame image does not contain a target, then based on the template feature, whether the n+xth frame image contains a target is determined through a second network, wherein the first network and the second network satisfy at least one of the first preset conditions; the first preset conditions include: the network depth of the second network is greater than the network depth of the first network, the network width of the second network is greater than the network width of the first network, and the resolution of the input image of the second network is greater than the resolution of the input image of the first network.

[0009] Optionally, the first shooting interface may be a shooting preview interface, such as a photo preview interface or a video preview interface. The first shooting interface may also be an interface for actually shooting a video, such as a video recording interface.

[0010] Optionally, the first operation may be a pre-set operation for inputting a tracking instruction, or inputting a tracking and focusing (which may be referred to as focus tracking) instruction, etc. The first operation may be, for example, a click operation.

[0011] The first network and the second network are used to process the image in the video stream according to the template features to determine whether the image contains the target. Optionally, the first network and the second network can be search networks, or tracking networks, etc. The first network and the second network meet at least one of the first preset conditions, that is, at least one of the network depth, network width and resolution of the input image of the second network is greater than that of the first network. Wherein, if the network depth and / or network width of the second network is greater than that of the first network, the search depth of the second network for the target is greater than that of the first network. In other words, the first network is a shallow search network compared to the second network, and / or the first network is a low-resolution tracking network.

[0012] The target tracking method provided by the first aspect of the embodiment of the present application is as follows: after the user inputs a first operation on the nth frame image containing the target, the nth frame image is subjected to feature extraction to obtain template features. Subsequently, for the n+xth frame image after the nth frame, the target is searched for using the first network and the second network based on the template features. The second network has at least one of a network depth, a network width, and an input image resolution greater than that of the first network. Thus, during the target tracking process, the first network is first used to determine whether the n+xth frame image contains the target. If the first network determines that the n+xth frame does not contain the target, the second network is used to determine whether the n+xth frame image contains the target. That is, the second network further increases the depth of the target search or increases the resolution of the input image, or simultaneously improves the search accuracy from multiple dimensions of the search depth and the resolution of the input image. In this way, target tracking in difficult scenes can improve tracking accuracy and reduce the possibility of target tracking loss. At the same time, during the target tracking process, the search results of the first network are used to determine whether to deepen the search depth through the second network or increase the input image resolution. Compared with tracking only through the second network, this method can prevent unnecessary deep searches or high-resolution searches in non-difficult scenarios, thereby reducing the complexity of the algorithm operation, improving the real-time performance of target tracking, and reducing device power consumption. In summary, the method provided in the embodiment of the present application can adaptively select the first network and the second network, thereby achieving a balance between tracking accuracy, power consumption, and real-time performance.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes: if it is determined through the second network that the n+xth frame image does not contain the target, then according to the template features, determining whether the n+xth frame image contains the target through the third network, wherein the second network and the third network meet at least one of the second preset conditions; the second preset conditions include: the network depth of the third network is greater than the network depth of the second network, the network width of the third network is greater than the network width of the second network, and the resolution of the input image of the third network is greater than the resolution of the input image of the second network.

[0014] In this implementation, in addition to the first network and the second network, the method further performs a deep search and / or a high-resolution search for the target through a third network. The first network, the second network, and the third network are in a progressive relationship in one or more of the three dimensions of network depth, network width, and resolution of the input image. In this way, during the target tracking process, when it is determined by the second network that the target is not included in the n+x frame image, the third network is further used to determine whether the target is included, so as to further deepen the depth of the target search and / or increase the resolution of the input image, thereby further improving the target search accuracy. Target tracking in difficult scenes can further improve tracking accuracy and reduce the possibility of target tracking loss.

[0015] In one possible implementation, at least one of the network depth and network width of the second network is greater than that of the first network, and at least one of the network depth and network width of the third network is greater than that of the first network, and the resolution of the input image of the third network is greater than that of the first network and the second network, respectively.

[0016] In other words, the first network is a low-resolution shallow search network, the second network is a low-resolution deep search network, and the third network is a high-resolution deep search network. This creates a more balanced progression in search depth and resolution among the first, second, and third networks, further improving the balance between target tracking accuracy, power consumption, and real-time performance.

[0017] In one possible implementation, based on template features, determining whether the n+xth frame image of the video stream contains a target is performed through a first network, including: selecting a first tracking area from the n+xth frame image; searching the first tracking area through the first network based on the template features, and outputting a first position and a first confidence level corresponding to the first position; if the first confidence level is greater than a first confidence level threshold, determining through the first network that the n+xth frame image contains a target; if the first confidence level is less than or equal to the first confidence level threshold, determining through the first network that the n+xth frame image does not contain a target.

[0018] In this implementation, a first tracking region is selected from the n+xth image frame, and the first network searches the n+xth image frame to determine whether it contains the target. In other words, the first network performs a local search on the n+xth image frame to determine whether it contains the target. Target tracking is generally not a difficult scenario. Therefore, this implementation performs a local search during the first search of the n+xth image frame, which reduces algorithm complexity, improves algorithm efficiency, reduces device power consumption, and improves real-time target tracking.

[0019] In one possible implementation, selecting the first tracking area from the n+xth frame image includes: selecting a partial area in the n+xth frame image according to the position of the target in the n+x-1th frame image of the video stream to obtain the first tracking area.

[0020] In other words, the first tracking area is selected from the n+xth image frame based on the target's position in the previous image frame. As the target moves, its position in the image gradually shifts. Therefore, selecting the first tracking area from the n+xth image frame based on the target's position in the previous image frame and then searching for the target can improve the success rate of target search and, in turn, the success rate of target tracking.

[0021] In a possible implementation, the method further includes: if it is determined through the first network that the target is included in the n+x-th frame image, outputting a first position, where the first position represents the position of the target in the n+x-th frame image.

[0022] That is, when the first network searches for the target in the n+xth frame image, if the search is successful, the first position is output, which is the position of the target in the n+xth frame image. After this step, x = x + 1 can be set, and the process returns to the step of "determining whether the n+xth frame image of the video stream contains the target using the first network based on the template features" to achieve target tracking for the next frame image.

[0023] In one possible implementation, based on the template features, a second network is used to determine whether the n+xth frame image contains a target, including: based on the template features, a second network is used to search the first tracking area, and output a second position and a second confidence level corresponding to the second position; if the second confidence level is greater than a second confidence level threshold, the second network is used to determine that the n+xth frame image contains a target; if the second confidence level is less than or equal to the second confidence level threshold, the second network is used to determine that the n+xth frame image does not contain a target.

[0024] In this implementation, when searching for a target through the second network, the first tracking area is searched, that is, the second network performs a local search on the n+xth frame image. In this way, when searching for a target, the second network only deepens the search depth and does not expand the search range, thereby preventing excessive search, reducing the complexity of the algorithm operation, improving the real-time performance of target tracking, and reducing device power consumption.

[0025] In a possible implementation, the method further includes: if it is determined through the second network that the target is included in the n+x-th frame image, outputting a second position, where the second position represents the position of the target in the n+x-th frame image.

[0026] That is, when the second network searches for the target in the n+xth frame image, if the search is successful, the second position is output, which is the position of the target in the n+xth frame image. After this step, x = x + 1 can be set, and the process returns to the step of "determining whether the n+xth frame image of the video stream contains the target using the first network based on the template features" to achieve target tracking for the next frame image.

[0027] In a possible implementation, the search range of the third network for the (n+x)th frame image is larger than the search range of the first network and the second network for the (n+x)th frame image.

[0028] In one possible implementation, determining whether the n+x-th frame image contains a target is performed through a third network, including: dividing the n+x-th frame image into regions to obtain multiple second tracking regions; searching the multiple second tracking regions separately through the third network according to template features, and outputting multiple third positions and third confidence levels corresponding to each third position; if there is a third target confidence level greater than a third confidence level threshold among the multiple third confidence levels, determining that the n+x-th frame image contains a target through the third network; if there is no third target confidence level among the multiple third confidence levels, determining that the n+x-th frame image does not contain a target through the third network.

[0029] In the above two implementations, when searching for a target through the third network, not only the search depth and / or the resolution of the input image is increased, but also the search range is increased, thereby further improving the accuracy of the search, thereby improving the target tracking accuracy and reducing the possibility of target tracking loss.

[0030] In a possible implementation, a resolution of at least one second tracking area among the multiple second tracking areas is greater than a resolution of the first tracking area.

[0031] In this implementation, the resolution of at least one second tracking area is greater than the resolution of the first tracking area, so that the third network has a higher search accuracy for the target, thereby improving the accuracy of target tracking.

[0032] In one possible implementation, the method further includes: if it is determined through the third network that the n+xth frame image contains a target, outputting a third target position, the third target position being a third position corresponding to a third target confidence, and the third target position representing the position of the target in the n+xth frame image; if it is determined through the third network that the n+xth frame image does not contain a target, stopping tracking the target.

[0033] In this implementation, if the third network successfully searches for the target, the resulting location is determined as the target's location in the n+xth frame. After this step, x = x+1, and the process returns to the step of "determining whether the n+xth frame of the video stream contains the target using the first network based on the template features" to track the target in the next frame.

[0034] If the third network fails to search for the target, tracking of the target may be stopped. Of course, in some other embodiments, target tracking may not be stopped, but instead x = x + 1 is set, and the process returns to the step of "determining whether the n+xth frame of the video stream contains the target through the first network based on the template features" to track the target in the next frame.

[0035] In one possible implementation, the first operation is a click operation, and in response to the first operation, feature extraction is performed on the nth frame image to obtain template features, including: in response to the first operation, determining the click position of the first operation; selecting a target area in the nth frame image based on the click position; and extracting features from the target area to generate template features.

[0036] Optionally, in response to the first operation, an area within a preset range may be selected as the target area with the click position of the first operation as the center.

[0037] In this implementation, the target area is determined based on the click position, and then features are extracted based on the target area to generate template features, so that the generated template features are more accurate, thereby making the tracked target more consistent with the user's intention and improving the user experience.

[0038] In a second aspect, the present application provides a device, which is included in an electronic device and has the function of implementing the electronic device behavior described in the first aspect and possible implementations of the first aspect. The function can be implemented through hardware or through hardware executing corresponding software implementations. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a receiving module or unit, a processing module or unit, etc.

[0039] In one possible implementation, the device includes a display module, a receiving module, and a tracking module. The display module is configured to display a first shooting interface, wherein a first area of ​​the first shooting interface is configured to display a video stream captured by a camera. The receiving module is configured to receive a first operation input by a user on the nth frame of the video stream when the first area displays the nth frame of the video stream, where n is a positive integer. The tracking module is configured to, in response to the first operation, extract features from the nth frame of the video stream to obtain a template feature, where the nth frame of the video stream contains a target; based on the template feature, determine whether the n+xth frame of the video stream contains a target using a first network, where x is a positive integer; if the first network determines that the n+xth frame of the video stream does not contain a target, then determine whether the n+xth frame of the video stream contains a target using a second network based on the template feature, wherein the first network and the second network satisfy at least one of first preset conditions; the first preset conditions include: the network depth of the second network is greater than the network depth of the first network, the network width of the second network is greater than the network width of the first network, and the resolution of the input image of the second network is greater than the resolution of the input image of the first network.

[0040] In a third aspect, the present application provides an electronic device, which includes: a processor, a memory, and an interface; the processor, the memory, and the interface cooperate with each other so that the electronic device executes any one of the methods in the technical solution of the first aspect.

[0041] In a fourth aspect, the present application provides a chip including a processor, wherein the processor is configured to read and execute a computer program stored in a memory to perform the method of the first aspect and any possible implementation thereof.

[0042] Optionally, the chip also includes a memory, and the memory is connected to the processor via circuits or wires.

[0043] Further optionally, the chip also includes a communication interface.

[0044] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes any one of the methods in the technical solution of the first aspect.

[0045] In a sixth aspect, the present application provides a computer program product, which includes: a computer program code, which, when the computer program code runs on an electronic device, enables the electronic device to execute any one of the methods in the technical solution of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG1 is a schematic diagram of an interface for target tracking provided in an embodiment of the present application;

[0047] FIG2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0048] FIG3 is a block diagram of the software and hardware structure of an electronic device provided in an embodiment of the present application;

[0049] FIG4 is a schematic diagram of interface changes for target tracking corresponding to a photo preview scene provided by an embodiment of the present application;

[0050] FIG5 is a schematic diagram of interface changes for target tracking corresponding to an example of a video recording scene provided by an embodiment of the present application;

[0051] FIG6 is a flow chart of a target tracking method according to an embodiment of the present application;

[0052] FIG7 is a schematic diagram showing the principle of a target tracking method provided in an embodiment of the present application;

[0053] FIG8 is a schematic diagram of a search network structure and target tracking process according to an embodiment of the present application;

[0054] FIG9 is a flow chart of another target tracking method provided in an embodiment of the present application;

[0055] FIG10 is a schematic diagram of an example of a UAV target tracking result provided in an embodiment of the present application;

[0056] FIG11 is a schematic diagram of an example of automatic data annotation results provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0058] In the following, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the technical features indicated. Therefore, a feature specified as "first," "second," or "third" may explicitly or implicitly include one or more of the features.

[0059] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in different places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0060] To better understand the embodiments of the present application, the terms or concepts that may be involved in the embodiments are explained below.

[0061] 1. Neural Networks

[0062] A neural network (NN), also known as a connection model or simply a network, is a mathematical model that mimics the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This network relies on the complexity of the system to achieve its purpose of processing information by adjusting the interconnections between its numerous internal nodes.

[0063] Neural networks may include feedforward neural networks, back propagation (BP) neural networks, convolutional neural networks (CNN), and the like.

[0064] 2. Network Width

[0065] Network width refers to the number of channels and filters in a neural network. The wider the network, the better it is at extracting target features. This means a single layer of the network can learn richer features, such as textures in different directions and frequencies. In other words, network width determines the amount of information learned at a given layer.

[0066] 3. Network Depth

[0067] Network depth refers to the number of network layers in a neural network. For example, the network depth of a residual network (resnet) 18 is 18, which means that the network has 18 network layers. The greater the network depth, the stronger the learning ability. Taking a convolutional neural network as an example, the network layers of a convolutional neural network can perform layer-by-layer abstract learning on the input image data. For example, the first layer learns the edge features of the image, the second layer learns the simple shape features, and the third layer learns the features of the target shape. Therefore, the network depth determines the expression (abstraction) ability of the network. The greater the network depth (that is, the deeper the network), the stronger the learning ability.

[0068] In summary, to a certain extent, the deeper and wider the network, the better the network's feature extraction capability and the higher the network's accuracy. In the embodiments of the present application, the target search process based on a deeper network depth and / or a wider network width is referred to as a deeper search depth for the target, or as a deep search.

[0069] The following describes the applicable devices for the technical problems and methods of the embodiments of the present application.

[0070] Object tracking is widely used in various technical fields and application scenarios, such as autonomous driving, sports broadcasting, security monitoring, drone tracking and filming, and mobile phone and other electronic devices taking photos and recording videos. For example, when taking photos or previewing videos, or while recording, the target can be tracked to facilitate preset operations such as focusing and adjusting brightness.

[0071] Exemplarily, FIG1 is a schematic diagram of an interface for target tracking provided in an embodiment of the present application. As shown in FIG1(a), the interface includes a puppy, which can be tracked to focus on the puppy. As shown in FIG1(b), the interface includes a pot of green plants, which can be tracked to perform preset processing such as focusing on the green plants and adjusting the brightness. As shown in FIG1(c), the interface includes a vehicle, which can be tracked to perform preset processing such as focusing on the vehicle and the surrounding area and adjusting the brightness.

[0072] In related technologies, during the target tracking process, if the target is deformed, moved, partially occluded, etc. (hereinafter referred to as difficult scenarios), target tracking may be lost.

[0073] The target tracking method provided in the embodiment of the present application searches for the target based on a local low-resolution shallow search network (hereinafter referred to as the local shallow network, also referred to as the first network), a local low-resolution deep search network (hereinafter referred to as the local deep network, also referred to as the second network) and a global high-resolution deep search network (hereinafter referred to as the global deep network, also referred to as the third network model). The three networks are in a progressive relationship in terms of network depth or width, resolution of the input image, and search range. In this way, in the target tracking process, a low-resolution shallow search is first performed on a local part of the image through a local shallow network. If the search fails, a low-resolution deep search is performed on a local part of the image through a local deep network, that is, the depth of the target search is deepened to improve the search accuracy. If the local deep network still fails to search, a high-resolution deep search is performed on the global part of the image through a global deep network, that is, the resolution of the input image is further increased, and the search range is expanded to further improve the search accuracy. In this way, this method can improve the tracking accuracy for target tracking in difficult scenes and reduce the possibility of target tracking loss. At the same time, during the search process, the search depth is gradually deepened or the search range is expanded according to the search results to prevent unnecessary deep searches, high-resolution searches, or global searches in non-difficult scenarios, so as to reduce the complexity of the algorithm operation, improve the real-time performance of target tracking, and reduce device power consumption.

[0074] The target tracking method provided in the embodiments of the present application can be applied to electronic devices that can shoot video, such as mobile phones, tablet computers, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and drones. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0075] For example, FIG2 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195, etc.

[0076] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0077] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0078] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0079] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0080] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0081] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0082] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0083] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0084] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0085] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0086] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. The structures of Antenna 1 and Antenna 2 in Figure 2 are merely examples. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, Antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0087] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0088] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0089] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0090] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with a network and other devices via wireless communication technology. Wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0091] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0092] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0093] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0094] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0095] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0096] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0097] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0098] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0099] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0100] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0101] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0102] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0103] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0104] As shown in Figure 3, a software system using a layered architecture is divided into several layers, each with distinct roles and divisions of labor. Layers communicate with each other via software interfaces. In some embodiments, the software system can be divided into five layers: from top to bottom, application layer 210, application framework layer 220, hardware abstraction layer 230, driver layer 240, and hardware layer 250.

[0105] The application layer 210 may include camera and gallery applications, as well as calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0106] The application framework layer 220 provides an application access interface and a programming framework for the applications of the application layer 210 .

[0107] For example, the application framework layer 220 includes a camera access interface, which is used to provide camera shooting services through camera management and camera devices.

[0108] The camera management in the application framework layer 220 is used to manage the camera. The camera management can obtain the parameters of the camera, such as determining the working status of the camera.

[0109] The camera device in the application framework layer 220 is used to provide a data access interface between different camera devices and camera management.

[0110] The hardware abstraction layer 230 is used to abstract hardware. For example, the hardware abstraction layer 230 may include a camera hardware abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer may include camera device 1, camera device 2, tracking device, etc. It is understood that camera device 1, camera device 2, and tracking device, etc., may be functional modules implemented through software or a combination of software and hardware. When the electronic device is powered on or while the electronic device is running, corresponding processes are created or processes are started to implement the corresponding functions of camera device 1, camera device 2, and tracking device. Specifically, the tracking device is used to implement target tracking.

[0111] The camera hardware abstraction layer can be connected to the camera algorithm library, and the camera hardware abstraction layer can call the algorithms in the camera algorithm library. In the embodiment of the present application, the camera algorithm library includes a tracker (i.e., a tracking algorithm). Optionally, the tracker can be implemented by the following modules: a region selection module, a region division module, a template feature extraction network, a local shallow network, a local deep network, a global deep network, and a logic judgment module.

[0112] It is understood that whether it is a photo preview, a video preview, or a video recording, the electronic device 100 is continuously acquiring and displaying a video stream, which includes multiple frames of images. Therefore, in the embodiment of the present application, each image in the video stream is referred to as a frame image, and the nth image in the video stream is referred to as the nth frame image.

[0113] The region selection module is used to determine the target region in the nth frame image of the video stream, and is also used to select a partial region of the n+xth frame image according to the position of the target in the n+x-1th frame image to obtain a first tracking region. Wherein, n and x are both positive integers. That is, the region selection module determines the target region in the nth frame image, and selects the first tracking region in each frame image after the nth frame. When selecting the first tracking region in a certain frame image, it is selected based on the position of the target in the previous frame (or the previous frame) image of the frame image. Optionally, the resolution of the first tracking region is greater than the resolution of the target region.

[0114] The region division module is used to divide the n+xth frame image to obtain a plurality of second tracking regions. Optionally, the resolution of the second tracking region is greater than the resolution of the first tracking region.

[0115] The template feature extraction network is used to extract features from the target image and obtain template features.

[0116] The local shallow network, local deep network, and global deep network are all used to search for a target in an image to be processed (referred to as the input image) input to the network based on template features, and output the location of the target and the confidence level corresponding to the location. Optionally, the local shallow network, local deep network, and global deep network can be convolutional neural networks.

[0117] Specifically, during target tracking, the image input to the local shallow network and the local deep network is the first tracking region selected by the region selection module. That is, the local shallow network and the local deep network are used to search the local region. The image input to the global deep network is the second tracking region obtained by the region segmentation module. That is, the global deep network is used to search the global region.

[0118] The resolution of the second tracking area can be greater than the resolution of the first tracking area. That is, the resolution of the input image of the global deep network is higher than the resolution of the input image of the local shallow network and the local deep network. Optionally, at least one of the network width and network depth of the local deep network and the global deep network is greater than that of the local shallow network.

[0119] The logic judgment module is used to perform logic judgment on the confidence of the output of the local shallow network, the local deep network, and the global deep network to determine the target tracking result. The target tracking result can include tracking success and tracking failure (i.e. tracking loss).

[0120] Specifically, the logic judgment module is used to judge the first confidence level output by the local shallow network. If the first confidence level is greater than the first confidence level threshold, the tracking is determined to be successful, and the first position output by the local shallow network is output as the target position; if the first confidence level is less than or equal to the first confidence level threshold, the first tracking area is input into the local deep network for processing.

[0121] The logic judgment module is also used to judge the second confidence level output by the local deep network. If the second confidence level is greater than the second confidence level threshold, the tracking is determined to be successful, and the second position output by the local deep network is output as the target position; if the second confidence level is less than or equal to the second confidence level threshold, the logic judgment module instructs the region division module to divide the image into regions, and inputs each second tracking region obtained by the division into the global deep network for processing.

[0122] The logic judgment module is also used to judge the third confidence level output by the global deep network. If the third confidence level is greater than the third confidence level threshold, the tracking is determined to be successful, and the third position output by the global deep network is output as the position of the target; if the third confidence level is less than or equal to the third confidence level threshold, the tracking is determined to be lost, and tracking of the target is stopped.

[0123] Of course, the camera algorithm library may also include other preset algorithms, such as a purple fringing removal algorithm, a saturation adjustment algorithm, etc., and the embodiments of the present application do not impose any limitations on this.

[0124] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera driver, a digital signal processor driver, and a graphics processor driver.

[0125] The hardware layer 250 may include sensors, image signal processors, digital signal processors, graphics processors, and other hardware devices. The sensors may include sensor 1, sensor 2, etc., and may also include depth sensors (time of flight, TOF) and multispectral sensors.

[0126] The following describes the workflow of the software system of the electronic device 100 in conjunction with the target tracking process in the photo preview scenario.

[0127] When a user taps the camera app icon on touch sensor 180K, the app wakes up and enters photo preview mode. The app then uses the camera access interface to access the various camera devices in the camera hardware abstraction layer. For example, the camera hardware abstraction layer determines that the current zoom factor is less than 0.6 and, therefore, issues a command to the camera device driver to access the wide-angle camera. Simultaneously, the camera algorithm library begins loading its algorithms.

[0128] When the sensor of the hardware layer is called, for example, after calling sensor 1 in the wide-angle camera to obtain the original image, the original image is sent to the image signal processing for preliminary processing such as alignment. After processing, it is returned to the camera hardware abstraction layer through the camera device driver, and then processed using other preset algorithms in the loaded camera algorithm library to obtain frame image 1.

[0129] The camera hardware abstraction layer sends the obtained frame image 1 back to the camera APP via the camera access interface for preview display.

[0130] When a user taps position 1 in frame image 1 on touch sensor 180K, the camera app responds by invoking the tracking device of the camera hardware abstraction layer (HAL) through the camera access interface. The tracking device of the HAL invokes the tracker in the camera algorithm library. The tracker acquires the target image from frame image 1 and extracts template features.

[0131] The camera app continues the above process to acquire the next frame, resulting in frame 2. The tracking device then calls the tracker in the camera algorithm library. Based on the template features extracted from frame 1, the tracker tracks the target in frame 2 and determines the target's position in frame 2. The tracking device in the camera hardware abstraction layer generates a tracking marker corresponding to this position. The tracking marker is used to identify the target's position in frame 2. The tracker and other preset algorithms in the camera algorithm library can be driven by the digital signal processor and the graphics processor to drive the graphics processor to execute the algorithm.

[0132] The camera hardware abstraction layer sends the obtained frame image 2 and tracking identifier back to the camera APP via the camera access interface. The camera APP previews and displays the frame image 2 in the interface of the electronic device, and displays the tracking identifier. In this way, the tracking identifier can be presented around the periphery of the target in the interface, prompting the user that the current target is within the area defined by the tracking identifier. In a specific embodiment, the tracking identifier is used to define a square area. Optionally, the tracking identifier defines the square area completely and continuously, that is, the tracking identifier is a tracking frame. Optionally, the tracking identifier can also define the square area only by four corners. The embodiment of the present application does not impose any restrictions on the specific form of the tracking identifier.

[0133] The above-mentioned process of generating frame image 2 is repeated to achieve target tracking for each frame image.

[0134] For ease of understanding, the following embodiments of this application will take an electronic device having the structure shown in Figures 2 and 3 as an example, and combine the accompanying drawings and application scenarios to specifically explain the target tracking method provided in the embodiments of this application.

[0135] First, the interface of the target tracking method is explained by taking the preview and recording application scenarios of a mobile phone as an example.

[0136] For example, FIG4 is a schematic diagram of the interface changes for target tracking corresponding to a photo preview scene provided by an embodiment of the present application. As shown in FIG4 (a), when a user clicks the icon of the camera app on the mobile phone desktop, the mobile phone opens the camera app and enters the photo preview interface, displaying a preview image of the surrounding environment, as shown in FIG4 (b). In the preview interface, if the user clicks on the preview image position 401 on the screen, the camera app responds to the user's click and, using the target tracking method provided by the embodiment of the present application, determines a target area 402 based on the user's clicked position 401. The target area 402 includes a target 403 to be tracked. After the preview image is subsequently acquired, the target 403 in the preview image is tracked and a tracking mark is displayed. For example, as shown in FIG4 (c), when the target 403 moves, or the user moves the mobile phone (for example, the target 403 moves in FIG4), the position of the target 403 in the screen changes. The target 403 is tracked based on the method provided by the embodiment of the present application, and a tracking mark 404 is displayed in the interface.

[0137] Optionally, when the user clicks the photo control 405 in the photo preview interface, the camera app saves the photo to the album. Afterwards, the target tracking method provided in the embodiment of the present application is continued to track the target 403 and display the tracking mark 404, as shown in Figure 4 (d).

[0138] In one embodiment, when target 403 partially or completely moves out of the picture, tracking is lost and the tracking mark is no longer displayed in the interface, as shown in Figure 4 (e). Optionally, after tracking is lost, when target 403 reappears in the picture, target 403 can be continued to be tracked, or target 403 can be no longer tracked. This embodiment of the application does not impose any restrictions on this. In another embodiment, when the user switches the photo mode, for example, from photo to video, portrait, etc., the camera APP restarts the stream and no longer tracks target 402.

[0139] The video preview scene is similar to the photo preview process shown in FIG4 , and will not be described in detail here.

[0140] For example, FIG5 is a schematic diagram of the interface changes of target tracking corresponding to an example of a video recording scene provided by an embodiment of the present application. As shown in FIG5(a), in the photo preview interface, the user clicks the "recording" control 501, and the camera APP switches the photo mode to the video recording mode. The mobile phone displays the video recording preview interface shown in FIG5(b). The interface displays a preview image of the surrounding environment. When the user clicks the start recording control 502 in the video recording preview interface, the camera APP starts recording the video and enters the interface shown in FIG5(c). During the video recording process, if the user clicks on the position 503 on the screen, the camera APP responds to the user's click and determines the target area 504 according to the position 503 clicked by the user through the target tracking method provided by the embodiment of the present application. The target area 504 includes the target 505 to be tracked. After the frame image is subsequently acquired, the target 505 in the frame image is tracked and the tracking mark is displayed. For example, as shown in Figure (d) in Figure 5, when the target 505 moves, or the user moves the mobile phone (taking the movement of the target 505 in Figure 5 as an example), the position of the target 505 in the screen changes. The target 505 is tracked based on the method provided in the embodiment of the present application, and the tracking mark 506 is displayed in the interface.

[0141] Optionally, in a video recording scenario, when the user clicks a stop recording control, or target tracking is lost, target 505 is no longer tracked.

[0142] The following uses the preview and recording scenes of a mobile phone as an example to illustrate the specific process of the target tracking method to achieve the above tracking effect.

[0143] This embodiment mainly describes the image processing process of the camera hardware abstraction layer and the camera algorithm library during target tracking. The image generation process can be referred to the above embodiment and will not be described in detail.

[0144] For example, Figure 6 is a flow chart of a target tracking method provided in an embodiment of the present application, and Figure 7 is a principle diagram of a target tracking method provided in an embodiment of the present application. Please refer to Figures 6 and 7 together. The method includes steps S101 to S133.

[0145] Among them, steps S101 to S105 are the process of extracting features and generating template features, hereinafter referred to as the template feature generation phase. Steps S106 to S114 are the process of searching for targets based on a local shallow network, hereinafter referred to as the local shallow search phase. Steps S115 to S121 are the process of searching for targets based on a local deep network, hereinafter referred to as the local deep search phase. Steps S122 to S133 are the process of searching for targets based on a global deep network, hereinafter referred to as the local deep search phase. The following describes each phase in turn.

[0146] 1. Template feature generation stage

[0147] S101 : The nth frame image of the video stream is displayed on the interface of the mobile phone. In response to a user clicking operation on position 1 in the nth frame image, a tracking device of the camera hardware abstraction layer calls a tracker in the camera algorithm library.

[0148] Optionally, the tracking device may carry the nth frame image and information of position 1 when calling the tracker.

[0149] S102 : The region selection module in the tracker responds to the call of the camera tracking device and determines the target region in the nth frame image according to position 1.

[0150] In this embodiment, the nth frame image is any frame image in the video stream. n is a positive integer. The nth frame image is the frame image that triggers the target tracking process and serves as the basis for subsequent target tracking. Therefore, it is also called the initial frame. The frame image after the initial frame in the video stream that is used for target tracking based on the initial frame is called the tracking frame.

[0151] Optionally, position 1 may be, for example, position 401 in FIG. 4( b ) or position 503 in FIG. 5( b ) .

[0152] In a specific embodiment, the region selection module may select an area within a preset range 1 around position 1 as the target region. For example, a circular area with a preset radius around position 1 may be selected as the target region. Alternatively, a square area with a preset area around position 1 may be selected as the target region. Referring to FIG. 7 , in a specific embodiment, the target region may be as shown in FIG. 7 .

[0153] It should be noted that, in this embodiment, the execution of steps S101 and S102 is triggered by the user's click operation on position 1. In some other embodiments, depending on the application scenario, the execution of these two actions can also be triggered by other means. For example, it can be triggered by target detection. Taking face recognition as an example, when the electronic device detects a face in the nth frame image, the tracking device calls the tracker to track the target. In this case, "determine the target area in the nth frame image according to position 1" in step S102 can also be replaced by: "determine the target area in the nth frame image according to the position of the center point of the detected target". Specifically, the area selection module can calculate the position of the center point of the detected target, and select the area within the surrounding preset range 1 with the center point as the target area. In addition, in this case, step S103 can be omitted and step S104 can be executed directly.

[0154] S103, the area selection module performs target detection on the target area to determine whether there is a target in the target area; if so, execute step S104; if not, end the process.

[0155] Optionally, the method for the region selection module to perform target detection may be as follows: detecting edge features in the target region based on an edge detection algorithm, and then determining whether there is a target in the target region based on the edge features.

[0156] Optionally, the target feature extraction network can also use a salient object detection algorithm to determine whether an object exists in the target region. For example, based on a multi-level feature aggregation network, convolutional features from multiple levels are used as salient cues for salient object detection.

[0157] If the region selection module detects the target and determines that part or all of a person, animal, plant, or object is present in the target region, then the target region is determined to contain a target, and step S104 is executed. For example, if a cup, cat, flower, person, or hand is present in the target region, then the target region is determined to contain a target. If any of these targets are not present in the target region, then the target region is determined to contain no target, and target tracking is abandoned. For example, if the target region only includes the middle area of ​​a carpet with no or repeated patterns, then the target region is determined to contain no target.

[0158] In a specific embodiment, after the target detection in this step, it is determined that the target area shown in FIG. 7 contains a target (dog), and therefore, step S104 is continued to be executed.

[0159] It's understandable that if the target area doesn't contain a target, there's little point in performing target tracking. Even if you do, it'll quickly be lost. Therefore, in this step, target tracking is abandoned for target areas where no target exists, simplifying the algorithm, improving its efficiency, and enhancing the user experience.

[0160] It should be noted that step S103 is an optional step and may not be executed, that is, step S104 is directly executed after step S102 is completed.

[0161] S104: The region selection module inputs the target region into the template feature extraction network.

[0162] S105: The template feature extraction network extracts features from the target area, generates template features, and initializes the tracker.

[0163] Optionally, initializing the tracker may include setting the tracker's state to a tracking state. The tracking state indicates that the target tracking process has been initiated. It is understood that various modules in the tracker may perform related operations based on the tracker's state. Upon receiving a call from the tracking device, if the tracker's state is determined to be a tracking state, the relevant modules in the tracker perform target tracking based on the template features.

[0164] It can be understood that while executing steps S101 to S105 above, the electronic device can obtain the next frame of the video stream, that is, the n+1 frame, according to the process described in the above embodiment. Since the time required to execute steps S101 to S105 is generally short, steps S101 to S105 can be completed before the electronic device obtains the n+1 frame, or at the same time as the electronic device obtains the n+1 frame. Therefore, the template features can be obtained according to step S105, and the target can be tracked starting from the n+1 frame. In this embodiment, the tracking of the target starting from the n+1 frame after step S105 will be explained as an example.

[0165] However, in other embodiments, after step S105, the target may be tracked starting from other frames. For example, the target may be tracked starting from the n+2th frame image, the n+5th frame image, and so on. Optionally, when steps S101 to S105 require a long time, or when the method further includes other steps in addition to steps S101 to S105, resulting in a long time required to generate the template features, after the template features are generated, the target may be tracked starting from the frame image whose generation time is closest to the current moment. This embodiment of the present application does not impose any limitation on this.

[0166] 2. Local shallow search stage

[0167] S106 : After the tracking device in the camera hardware abstraction layer obtains the n+1th frame image (tracking frame), it calls the tracker in the camera algorithm library.

[0168] Optionally, the tracking device can carry the (n+1)th frame image when calling the tracker.

[0169] Please refer to FIG. 7 . In a specific embodiment, the (n+1)th frame image may be as shown in FIG. 7 .

[0170] S107 : In response to the call of the camera tracking device, the region selection module in the tracker selects a partial region of the (n+1)th frame image according to the position of the target in the nth frame image to obtain a first tracking region.

[0171] As described above, after generating the template features, the template feature extraction network sets the tracker state to tracking. Therefore, in this embodiment, upon receiving a call from the camera tracking device, the region selection module determines that the tracker state is tracking and then selects the first tracking region from the (n+1)th frame.

[0172] Optionally, the region selection module may select a first tracking region based on the position of the target's center point in the nth frame. Specifically, the region selection module may select a region within a preset range 2 in the n+1th frame, using the position of the target's center point in the nth frame as the center, to obtain the first tracking region. The area of ​​the first tracking region is smaller than the area of ​​the n+1th frame, that is, the first tracking region is a local region in the n+1th frame.

[0173] Optionally, preset range 2 can be larger than preset range 1, so that the area of ​​the first tracking area is larger than the area of ​​the target area. That is, the resolution of the first tracking area can be higher than that of the target area. The higher the resolution of the first tracking area, the greater the probability of a successful search in subsequent searches based on the first tracking area, thereby increasing the success rate of target tracking.

[0174] Of course, the region selection module may also select the first tracking region based on the position of the edge or vertex of the target in the nth frame image, but the present embodiment does not impose any limitation on this.

[0175] S108. The region selection module inputs the first tracking region into the local shallow network.

[0176] S109. The local shallow network searches for a target in the first tracking area according to the template features, and outputs a first position and a first confidence level corresponding to the first position; the first position is used to represent the position of the target in the (n+1)th frame image.

[0177] S110. The local shallow network sends the first position and the first confidence to the logic judgment module.

[0178] S111. The logic judgment module determines whether the first confidence level is greater than a first confidence level threshold; if so, execute step S112; if not, execute step S115.

[0179] If the first confidence level is greater than the first confidence threshold, it indicates that the local shallow network has successfully searched for the target, i.e., tracking has been successful, and step S112 is executed to output the first position. If the first confidence level is less than or equal to the first confidence threshold, it indicates that the local shallow network has failed to search for the target, and step S115 is executed to enter the local deep search phase.

[0180] S112. The logic judgment module generates tracking success information; the tracking success information is used to indicate that the target tracking is successful.

[0181] S113 : The logic judgment module returns the first position and tracking success information to the tracking device of the camera hardware abstraction layer.

[0182] S114: After receiving the tracking success information, the tracking device generates a first tracking identifier corresponding to the first location according to the first location.

[0183] Afterwards, the (n+1)th frame image and the first tracking mark are displayed on the interface of the electronic device.

[0184] 3. Local deep search stage

[0185] S115 , the logic judgment module inputs the first tracking area into the local deep network.

[0186] S116. The local deep network searches for a target in the first tracking area according to the template features, and outputs a second position and a second confidence level corresponding to the second position; the second position is used to represent the position of the target in the (n+1)th frame image.

[0187] Referring to Figure 7 , in a specific embodiment, the first tracking area may be shown as 701. It can be seen that the input to the local shallow network and the local deep network is a portion of the (n+1)th frame image, i.e., the local shallow network and the local deep network search for a portion of the (n+1)th frame image.

[0188] S117. The local deep network sends the second position and the second confidence to the logic judgment module.

[0189] S118. The logic judgment module determines whether the second confidence level is greater than the second confidence level threshold; if so, execute step S119; if not, execute step S122.

[0190] Optionally, the second confidence threshold may be the same as or different from the first confidence threshold.

[0191] If the second confidence level is greater than the second confidence threshold, it indicates that the local deep network has successfully searched for the target, i.e., tracking has been successful, and step S119 is executed to output the second position. If the second confidence level is less than or equal to the second confidence threshold, it indicates that the local deep network has failed to search for the target, and step S122 is executed to enter the global deep search phase.

[0192] S119. The logic judgment module generates tracking success information; the tracking success information is used to indicate that the target tracking is successful.

[0193] S120 : The logic judgment module returns the second position and tracking success information to the tracking device of the camera hardware abstraction layer.

[0194] S121: After receiving the tracking success information, the tracking device generates a second tracking identifier corresponding to the second location according to the second location.

[0195] Afterwards, the (n+1)th frame image and the second tracking mark are displayed on the interface of the electronic device.

[0196] 4. Global deep search stage

[0197] S122: The logic judgment module instructs the area division module to perform area division.

[0198] S123 : The region division module divides the (n+1)th frame image to obtain a plurality of second tracking regions.

[0199] Optionally, the region division module can divide the n+1th frame image according to a preset division method. For example, the n+1th frame image can be divided into multiple second tracking regions according to a grid division method. This method is simple and convenient and can improve the operating efficiency of the algorithm.

[0200] Optionally, the region segmentation module can also segment the n+1th frame image based on the features of the n+1th frame image. For example, the n+1th frame image can be segmented based on pixel clustering methods, region proposal network (RPN) methods, etc., taking full account of the features in the image to generate multiple second tracking regions. This can improve the integrity of the target in the second tracking region, thereby improving the accuracy of target tracking.

[0201] Optionally, the area of ​​the second tracking area can be larger than the first tracking area, that is, the resolution of the second tracking area is greater than that of the first tracking area. In this way, when a global deep search is performed based on the second tracking area, the probability of successful search is greater, thereby increasing the success rate of target tracking.

[0202] S124. The region division module inputs each second tracking region into the global deep network.

[0203] S125. The global deep network performs target search on each second tracking area according to the template features, and outputs multiple third positions and third confidences corresponding to each third position; the third position is used to represent the position of the target in the (n+1)th frame image.

[0204] Referring to Figure 7 , in a specific embodiment, the multiple second tracking regions obtained after the n+1 frame image is divided can be shown as 702 in Figure 7 . It can be seen that the deep tracking network searches each second tracking region 702 separately, that is, the deep tracking network performs a global search of the n+1 frame image.

[0205] S126. The global deep network sends each third position and each third confidence to the logic judgment module.

[0206] S127. The logic judgment module determines whether there is a third target confidence greater than a third confidence threshold in the third confidence level; if so, execute step S128; if not, execute step S131.

[0207] Optionally, the third confidence threshold may be the same as or different from the first confidence threshold and the second confidence threshold.

[0208] If the third target confidence level exists, it indicates that the global deep network has successfully searched for the target, i.e., tracking is successful, and step S128 is executed to output the third target position corresponding to the third target confidence level. If the third target confidence level does not exist, i.e., all third confidence levels are less than or equal to the third confidence level threshold, it indicates that the global deep network has failed to search for the target, i.e., tracking has failed, and step S122 is executed to stop tracking.

[0209] S128. The logic judgment module generates tracking success information; the tracking success information is used to indicate that the target tracking is successful.

[0210] S129 . The logic judgment module returns the third target position and tracking success information to the tracking device of the camera hardware abstraction layer.

[0211] S130: After receiving the tracking success information, the tracking device generates a third tracking identifier corresponding to the third target location according to the third target location.

[0212] Afterwards, the (n+1)th frame image and the third tracking mark are displayed on the interface of the electronic device.

[0213] Referring to FIG. 7 , in a specific embodiment, the third tracking identifier displayed in the mobile phone interface may be shown as 703 in FIG. 7 .

[0214] S131. The logic judgment module generates tracking failure information; the tracking success information is used to indicate target tracking failure, that is, tracking loss.

[0215] S132: The logic judgment module returns each third position and tracking failure information to the tracking device of the camera hardware abstraction layer.

[0216] Optionally, in this step, the logic judgment module may also only send the tracking failure information to the tracking device of the camera hardware abstraction layer without sending each third position, so as to simplify the algorithm and improve the algorithm operation efficiency.

[0217] S133: After receiving the tracking failure information, the tracking device stops tracking.

[0218] After receiving a tracking failure message, the tracking device does not generate a tracking indicator, nor does it display the tracking indicator in the interface. Furthermore, the tracking device stops tracking the current target and, after the next frame is generated, no longer calls the tracker in the camera's algorithm library.

[0219] It is understood that during the tracking process, if a preset exit tracking condition is met, the tracking device will also stop tracking. The preset exit tracking condition can be, for example, switching to a photo taking mode as described in the above embodiment.

[0220] Furthermore, if target tracking is successful and the preset tracking exit condition is not met, target tracking is performed for any subsequent frame in the video stream, namely the n+xth frame, using the same method as above steps S106 to S133. That is, after above steps S114, S121, and S130, n is set to n+1, and the process returns to step S106.

[0221] The target tracking method provided in this embodiment searches for targets based on a local shallow network, a local deep network, and a global deep network. The three networks are in a progressive relationship in terms of network depth or network width, the resolution of the input image, and the search range. In this way, during the target tracking process, a low-resolution shallow search is first performed on a local part of the image through a local shallow network. If the search fails, a low-resolution deep search is performed on the local part of the image through a local deep network, that is, the network depth and / or width of the search is deepened to improve the search accuracy. If the local deep network still fails to search, a high-resolution deep search is performed on the global part of the image through a global deep network, that is, the resolution of the input image is further increased, and the search range is expanded to further improve the search accuracy. In this way, target search in difficult scenes can improve tracking accuracy and reduce the possibility of target tracking loss. At the same time, during the search process, the search depth is gradually deepened according to the search results, or the input image resolution is increased, or the search range is expanded. Compared with tracking only through the global deep network, this method can prevent unnecessary deep search, high-resolution search, or global search in non-difficult scenarios, so as to reduce the complexity of the algorithm operation, improve the real-time performance of target tracking, and reduce device power consumption.

[0222] In summary, the method provided in the embodiment of the present application can adaptively select a local shallow network, a local deep network, and a global deep network, thereby achieving a balance between tracking accuracy, power consumption, and real-time performance.

[0223] It should be noted that in some embodiments, target tracking can also be performed only through a local shallow network and a global deep network. Specifically, in the embodiment shown in Figure 6 above, if the judgment result of step "S111, the logic judgment module determines whether the first confidence level is greater than the first confidence level threshold" is no, step S122 is executed, and steps S115 to S121 are no longer executed. In this way, the algorithm can be simplified, the algorithm operation efficiency can be improved, and the power consumption can be reduced, thereby improving the real-time performance of target tracking.

[0224] Of course, in some other embodiments, the target can also be tracked only through a local shallow network and a local deep network. That is to say, in the target tracking method provided in the embodiment of the present application, the search network may include all of the three networks of the local shallow network, the local deep network and the global deep network, and may also include two of the three networks. The search network may also include more networks than these three networks, and the embodiment of the present application does not impose any restrictions on this. In summary, the number of networks in the search network is multiple (greater than or equal to 2), and the multiple networks present an increasing effect in at least one of the search range, the resolution of the input image, the depth of the network, and the width of the network. In this way, when tracking the target through multiple networks, the search intensity is gradually increased according to the search results to improve the search accuracy. In this way, the tracking accuracy can be improved for target searches in difficult scenes and the possibility of target tracking loss is reduced. At the same time, during the search process, the search intensity is gradually deepened and increased according to the search results to prevent excessive search in non-difficult scenes, so as to reduce the complexity of the algorithm operation, improve the real-time performance of target tracking, and reduce the power consumption of the device.

[0225] It can be understood that the structures of the local shallow network, the local deep network and the global deep network can be the same or different. For example, Figure 8 is a schematic diagram of the structure of a search network and the target tracking process provided in an embodiment of the present application. As shown in Figure 8, in this embodiment, the local shallow network, the local deep network and the global deep network can all include a search feature extraction network, a feature adjustment layer, a correlation matching layer, a positioning branch and a classification branch. Among them, the search feature area network and the template feature extraction network in each network share weights. That is to say, the network structure, the structure and filter weights of the search feature area network and the template feature extraction network in each network are the same, but the resolution of the input image can be different.

[0226] The target tracking provided by the embodiment of the present application is further explained below in conjunction with the structures of each network.

[0227] For example, FIG9 is a flowchart of another target tracking method provided in an embodiment of the present application. Please refer to FIG8 and FIG9 together. The method includes:

[0228] 1. Template feature generation stage

[0229] S201: The region selection module selects a target region z in the initial frame.

[0230] Specifically, referring to the above step S102, the region selection module can select the target region z with the position 1 clicked by the user as the center, or with the position of the center point of the detected target as the center. The position of the position 1 or the center point of the target is represented as c0 = (x, y), and the target region z can be represented as: z∈R h×w×3 That is, the dimension of the target area z is h×w×3, where h represents the height value h, w represents the width value w, and 3 represents the number of channels 3 (R, G, B three channels).

[0231] S202, template feature extraction network Extract features from the target area to obtain template features

[0232] 2. Local shallow search stage

[0233] For any frame tracking frame x, the local shallow search phase is first entered, that is, the following steps S203 to S208 are performed:

[0234] S203 : The region selection module selects a partial region in the tracking frame x according to the position of the target in the previous image frame to obtain a first tracking region x.

[0235] Specifically, when tracking frame x is the first tracking frame (i.e., the first image frame after the initial frame), the region selection module selects a first tracking region x in tracking frame x based on the target's position in the initial frame. In this case, the target's position in the initial frame can be determined based on the target detection result, and the first tracking region x can be further selected. Optionally, the resolution of the first tracking region x is greater than that of the target region, i.e., H>h and / or W>w.

[0236] When tracking frame x is the next tracking frame after the first tracking frame, the region selection module selects the first tracking region x in tracking frame x based on the target's position in the previous tracking frame (tracking frame x-1). In this case, the target's position in the previous tracking frame can be determined based on the tracking results of the previous tracking frame, and the first tracking region x is then selected.

[0237] Optionally, the region selection module can determine the position of the target's center point in the previous frame image based on the target's position in the previous frame image, and then select the first tracking region x with the position of the target's center point in the previous frame image as the center. The position of the target's center point is represented by c z =(x,y), the first tracking area x can be expressed as: x∈R H×W×3 That is, the dimension of the first tracking area x is H×W×3, where H represents the height value, W represents the width value, and 3 represents the number of channels (R, G, and B channels).

[0238] In a specific embodiment, the target area z of the previous frame or the first tracking area x-1 of the previous frame can be enlarged by a preset multiple based on the position of the target center point in the previous frame image to obtain the first tracking area x of the tracking frame x. For example, the position of the target in the tracking frame x-1 is (100, 100, 200, 200), and the preset enlargement multiple is 2. Then, based on the position of the target center point in the tracking frame x-1, the position of the first tracking area x selected in the tracking frame x is (50, 50, 250, 250), that is, the first tracking area x is x∈R 200×200×3 .

[0239] S204. Search feature extraction network in local shallow network Extract features from the first tracking area x and obtain search features

[0240] S205, feature adjustment layer μ in the local shallow network s Search features Make adjustments to obtain the adjusted search features

[0241] S206. At the same time, the feature adjustment layer μ in the local shallow network s Template Features Make adjustments to obtain the adjusted template features

[0242] S207, Correlation Matching Layer ω in Local Shallow Network s The adjusted template features and adjusted search features Perform relevant matching to obtain matching features

[0243] Optionally, the specific implementation method of the correlation matching layer can be cross-correlation matching based on depthwise separable convolution, or cross-correlation matching based on an encoding-decoding structure (transformer), etc., and this embodiment of the application does not impose any restrictions on this. Similarly, this embodiment of the application does not impose any restrictions on the specific structure and implementation method of other sub-networks in the network. They can be designed according to actual needs as long as they can achieve their functions.

[0244] S208, matching features After positioning branch h s and the confidence branch ρ s, obtain multiple position coordinates b1 = (l, t, b, r) and the confidence level ρ1 corresponding to each position coordinate b1, and output the first position b according to each position coordinate b1 and the corresponding confidence level ρ1 t1 and the first confidence ρ t1 .

[0245] The position coordinate b1 is used to define an area. l represents the x-coordinate value of the upper left vertex of the area, t represents the y-coordinate value of the upper left vertex of the area, b represents the x-coordinate value of the lower right vertex of the area, and r represents the y-coordinate value of the lower right vertex of the area.

[0246] The maximum value ρ among multiple confidence levels ρ1 t1 As the output of the local shallow network, the first confidence ρ is obtained t1 ; The first confidence ρ t1 The corresponding position coordinate b t1 As the output of the local shallow network, the first position b is obtained t1 .

[0247] S209: The logic judgment module judges the first confidence level ρ t1 Is it greater than the first confidence threshold σ1?

[0248] If the first confidence level ρ t1 If the confidence level is greater than the first confidence threshold σ1, it means that there is a target in the first tracking area, and the local shallow network successfully searches for the target, that is, the tracking is successful. Then the logic judgment module sends the first position b to the tracking device. t1 and tracking success information, the electronic device displays the first position b t1 The corresponding tracking mark is shown in Figure 8. Afterwards, the target tracking of the next frame x+1 is started, ie, x=x+1 is set, and the process returns to step S203.

[0249] If the first confidence level ρ t1 If the confidence value is less than or equal to the first confidence threshold σ1, it means that there is no target in the first tracking area, and the local shallow network fails to search for the target, then the local deep search phase is entered, that is, the following steps S210 to S215 are executed.

[0250] 3. Local deep search stage

[0251] S210, Search Feature Extraction Network in Local Deep Network Extract features from the first tracking area x and obtain search features

[0252] S211, Feature adjustment layer μ in local deep network dSearch features Make adjustments to obtain the adjusted search features

[0253] S212, at the same time, the feature adjustment layer μ in the local deep network d Template Features Make adjustments to obtain the adjusted template features

[0254] S213, Correlation Matching Layer ω in Local Deep Network d The adjusted template features and adjusted search features Perform relevant matching to obtain matching features

[0255] S214, matching features After positioning branch h d and the confidence branch ρ d , obtain multiple position coordinates b2 = (l, t, b, r) and the confidence level ρ2 corresponding to each position coordinate b2, and output the second position b according to each position coordinate b2 and the corresponding confidence level ρ2 t2 and the second confidence ρ t2 .

[0256] The maximum value ρ among multiple confidence levels ρ2 t2 As the output of the local deep network, the second confidence ρ is obtained t2 ; The second confidence ρ t2 The corresponding position coordinate b t2 As the output of the local deep network, the second position b is obtained t2 .

[0257] S215: The logic judgment module judges the second confidence level ρ t2 Is it greater than the second confidence threshold σ2?

[0258] If the second confidence level ρ t2 If the confidence level is greater than the second confidence threshold σ2, it means that there is a target in the first tracking area, and the local deep network successfully searches for the target, that is, the tracking is successful. Then the logic judgment module sends the second position b to the tracking device. t2 and tracking success information, the electronic device displays the second location b t2 The corresponding tracking mark is shown in Figure 8. Afterwards, the target tracking of the next frame x+1 is started, ie, x=x+1 is set, and the process returns to step S203.

[0259] If the second confidence level ρ t2 If the confidence value is less than or equal to the second confidence threshold σ2, it means that there is no target in the first tracking area, and the local deep network fails to search for the target, then the global deep search phase is entered, that is, the following steps S217 to S224 are executed.

[0260] 4. Global deep search stage

[0261] S217 : The region division module divides the tracking frame x to obtain a plurality of second tracking regions.

[0262] The region selection module inputs each second tracking region into the global deep network. For any second tracking region i, the global deep network performs the following steps S218 to S224:

[0263] S218, Search Feature Extraction Network in Global Deep Network Extract features from the second tracking area i and obtain the search features

[0264] S219, Feature Adjustment Layer μ in Global Deep Network n Search features Make adjustments to obtain the adjusted search features

[0265] S220, at the same time, the feature adjustment layer μ in the global deep network n Template Features Make adjustments to obtain the adjusted template features

[0266] S221, Correlation matching layer ω in global deep network n The adjusted template features and adjusted search features Perform relevant matching to obtain matching features

[0267] S222, matching features After positioning branch h n and the confidence branch ρ n , obtain multiple position coordinates b3 = (l, t, b, r) and the confidence level ρ3 corresponding to each position coordinate b3, and output the third position b according to each position coordinate b3 and the corresponding confidence level ρ3 t3 and the third confidence ρ t3 .

[0268] The maximum value ρ among multiple confidence levels ρ3 t3 As the output of the global deep network, the third confidence ρ is obtained t3 ; The third confidence ρ t3 The corresponding position coordinate b t3 As the output of the global deep network, the third position b is obtained t3 .

[0269] S223, the logic judgment module determines the third confidence level ρ t3 Is it greater than the third confidence threshold σ3?

[0270] If the third confidence level ρ t3 If the value is greater than the third confidence threshold σ3, it means that there is a target in the second tracking area i, then the tracking is successful, and the electronic device displays the third position b t3 The corresponding tracking mark (as shown in FIG8 ) ends the tracking of the current tracking frame x and performs target tracking of the next tracking frame x+1, that is, setting x=x+1, and returns to step S203 .

[0271] If the third confidence level ρ t3 If the confidence value is less than or equal to the third confidence threshold σ3, it indicates that there is no target in the second tracking area i, and step S224 is executed.

[0272] S224. Determine whether all second tracking areas have been traversed. If not, set i = i + 1 and return to step S218, i.e., perform a global deep search on the next second tracking area i + 1. If so, determine that tracking has failed, i.e., tracking is lost, the electronic device does not display the tracking mark, and tracking stops.

[0273] Optionally, in this step, if no target exists in any second tracking area i and it is determined that all second tracking areas have not been traversed, the process can directly return to step S218 to search the next second tracking area. That is, steps S218 to S224 are executed repeatedly until tracking is successful or lost (i.e., a tracking result is obtained).

[0274] As a possible implementation, if there is no target in the second tracking area i and it is determined that all second tracking areas have not been traversed, target tracking can be started for the next tracking frame x+1. While tracking the target in the next tracking frame x+1, steps S218 to S224 are performed once to search the next second tracking area i+1. Similarly, while tracking the target in each tracking frame, a second tracking area is searched until a tracking result is obtained.

[0275] Specifically, in tracking frame x, steps S218 to S224 are executed to search for the second tracking region i among the multiple second tracking regions. If the search is successful, a tracking indicator is displayed, tracking for tracking frame x is terminated, and target tracking for tracking frame x+1 is then initiated. If no target exists in second tracking region i, and it is determined that all second tracking regions have not been traversed, target tracking for tracking frame x+1 is first initiated. While tracking the target in tracking frame x+1, steps S218 to S224 can be executed to search for the second tracking region i+1. If the search is successful, a tracking indicator is displayed, tracking for tracking frame x+1 is terminated, and target tracking for tracking frame x+2 is then initiated. If no target exists in second tracking region i+1, the search for second tracking region i+2 continues as described above while tracking the target in tracking frame x+2, and so on, until a tracking result is obtained.

[0276] That is to say, in this embodiment, when performing a global deep search through the global deep network, the global deep search process can be split and delayed and performed simultaneously with the tracking of subsequent tracking frames. This can prevent video freezes caused by the long global deep search time and improve user experience.

[0277] The specific structures of the local shallow network, local deep network and global deep network in this application are described below.

[0278] As described above, at least one of the network width and network depth of the local deep network and the global deep network is greater than that of the local shallow network. In other words, the local deep network and the global deep network are deeper and / or wider than the local shallow network. The resolution of the input image to the global deep network can be higher than that of the local shallow network and the local deep network. Moreover, the global deep network searches the entire image, while the local shallow network and the local deep network search the local part of the image.

[0279] In a specific embodiment, the resolution, number of channels and network depth of the three networks can be shown in Table 1 below:

[0280] Table 1

[0281] It should be noted that the data provided in Table 1 is only an example and does not constitute any limitation to these three networks.

[0282] Optionally, the network depth and width of the search feature extraction network, feature adjustment layer, correlation matching layer, positioning branch, and confidence branch of the three networks in the embodiment of the present application can be designed and adjusted according to actual needs. Table 2 gives an implementation example:

[0283] Table 2

[0284] In this case, conv represents a convolutional layer, maxpool represents a maximum pooling layer, stage represents a stage, center crop represents a center crop, and corrconv represents a correlated convolutional layer.

[0285] The target tracking method provided in the embodiments of the present application searches for the target layer by layer through a local shallow network, a local deep network, and a global deep network, thereby achieving a balance between target tracking accuracy, power consumption, and real-time performance. Moreover, the local shallow network, local deep network, and global deep network can adjust the network structure, depth, width, etc. according to actual usage scenarios and needs, facilitating network expansion and adaptation, meeting the usage requirements of different platforms, and improving the applicability of the method.

[0286] The tracking effect of the target tracking method provided in the embodiment of the present application is described below with reference to the scenarios and drawings.

[0287] Scenario 1: Drone target tracking

[0288] For example, Figure 10 is a schematic diagram of a drone target tracking result provided in an embodiment of the present application. In this embodiment, the drone uses the method provided in an embodiment of the present application to track a target person in an image. In four consecutive image frames, the tracking results are shown in Figures 10 (a), 10 (b), 10 (c), and 10 (d), respectively. As can be seen from the figures, this method can accurately track the target.

[0289] Scenario 2: Automatic Data Labeling

[0290] Exemplarily, Figure 11 is a schematic diagram of an example of automatic data annotation results provided by an embodiment of the present application. In this embodiment, the target tracking method provided by an embodiment of the present application is applied to a driving recorder installed in a vehicle. When the vehicle is moving, the driving recorder obtains video data of the road in real time, and uses the target tracking method to track and annotate the target vehicle in each frame image. The tracking and annotation results are shown in Figure 11 (a), Figure 11 (b), Figure 11 (c), and Figure 11 (d). As can be seen from the figure, this method can accurately track and annotate the target.

[0291] The above describes in detail an example of a target tracking method provided by an embodiment of the present application. It is understandable that, in order to implement the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0292] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each function can be divided into various functional modules, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0293] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0294] The electronic device provided in this embodiment is used to execute the above-mentioned target tracking method, and thus can achieve the same effect as the above-mentioned implementation method.

[0295] When integrated, the electronic device may also include a processing module, a storage module, and a communication module. The processing module may be used to control and manage the operation of the electronic device. The storage module may be used to support the execution of program code and data stored in the electronic device. The communication module may be used to support communication between the electronic device and other devices.

[0296] The processing module may be a processor or a controller. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module may be a memory. The communication module may specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip, or a Wi-Fi chip.

[0297] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment may be a device having the structure shown in FIG. 2 .

[0298] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the target tracking method of any of the above embodiments.

[0299] An embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the target tracking method in the above-mentioned embodiment.

[0300] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the target tracking method in the above-mentioned method embodiments.

[0301] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0302] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0303] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0304] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0305] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0306] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0307] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A target tracking method, the method being executed by an electronic device, characterized in that: The method comprises: Display a first shooting interface, where a first area of ​​the first shooting interface is used to display a video stream captured by a camera; When the first area displays the nth frame image of the video stream, receiving a first operation input by a user on the nth frame image, where n is a positive integer; In response to the first operation, extracting features from the n-th frame image to obtain template features, wherein the n-th frame image contains a target; Determine, by a first network, whether the n+xth frame image of the video stream contains the target according to the template feature, where x is a positive integer; If it is determined through the first network that the n+xth frame image does not contain the target, then according to the template feature, determining through the second network whether the n+xth frame image contains the target, wherein the first network and the second network satisfy at least one of the first preset conditions; The first preset condition includes: the network depth of the second network is greater than the network depth of the first network, the network width of the second network is greater than the network width of the first network, and the resolution of the input image of the second network is greater than the resolution of the input image of the first network.

2. The method according to claim 1, characterized in that: The method further comprises: If it is determined through the second network that the n+xth frame image does not contain the target, then according to the template feature, determining through a third network whether the n+xth frame image contains the target, wherein the second network and the third network satisfy at least one of the second preset conditions; The second preset condition includes: the network depth of the third network is greater than the network depth of the second network, the network width of the third network is greater than the network width of the second network, and the resolution of the input image of the third network is greater than the resolution of the input image of the second network.

3. The method according to claim 2, characterized in that At least one of the network depth and the network width of the second network is greater than that of the first network, and at least one of the network depth and the network width of the third network is greater than that of the first network, and the resolution of the input image of the third network is greater than that of the first network and the second network, respectively.

4. The method according to claim 2 or 3, characterized in that: The determining, by a first network, whether the n+xth frame image of the video stream contains the target according to the template feature includes: Selecting a first tracking area from the n+xth frame image; Searching the first tracking area through the first network according to the template feature, and outputting a first position and a first confidence level corresponding to the first position; If the first confidence level is greater than a first confidence level threshold, determining through the first network that the n+xth frame image contains the target; If the first confidence is less than or equal to the first confidence threshold, it is determined through the first network that the n+xth frame image does not contain the target.

5. The method according to claim 4, characterized in that The selecting a first tracking area from the n+xth frame image includes: According to the position of the target in the (n+x-1)th frame image of the video stream, a partial area in the (n+x)th frame image is selected to obtain the first tracking area.

6. The method according to claim 4 or 5, characterized in that: The method further comprises: If it is determined through the first network that the target is included in the n+x-th frame image, the first position is output, where the first position represents the position of the target in the n+x-th frame image.

7. The method according to any one of claims 4 to 6, characterized in that The determining, by a second network, whether the n+xth frame image contains the target according to the template feature includes: searching the first tracking area through the second network according to the template feature, and outputting a second position and a second confidence level corresponding to the second position; If the second confidence level is greater than a second confidence level threshold, determining through the second network that the n+xth frame image contains the target; If the second confidence is less than or equal to the second confidence threshold, determining through the second network that the n+xth frame image does not contain the target.

8. The method according to claim 7, characterized in that The method further comprises: If it is determined through the second network that the target is included in the n+x-th frame image, the second position is output, and the second position represents the position of the target in the n+x-th frame image.

9. The method according to any one of claims 4 to 8, characterized in that The search range of the third network for the n+xth frame image is larger than the search range of the first network and the second network for the n+xth frame image.

10. The method according to claim 9, characterized in that The determining, by a third network, whether the n+xth frame image contains the target comprises: Dividing the n+xth frame image into regions to obtain a plurality of second tracking regions; According to the template feature, searching the plurality of second tracking areas respectively through the third network, and outputting a plurality of third positions and a third confidence level corresponding to each of the third positions; If there is a third target confidence greater than a third confidence threshold among the multiple third confidences, determining through the third network that the n+xth frame image contains the target; If the third target confidence does not exist in the multiple third confidences, it is determined through the third network that the n+xth frame image does not contain the target.

11. The method according to claim 10, characterized in that A resolution of at least one second tracking area among the plurality of second tracking areas is greater than a resolution of the first tracking area.

12. The method according to claim 10 or 11, characterized in that: The method further comprises: If it is determined by the third network that the target is included in the n+x-th frame image, a third target position is output, where the third target position is a third position corresponding to the third target confidence level, and the third target position represents the position of the target in the n+x-th frame image; If it is determined through the third network that the n+xth frame image does not contain the target, tracking of the target is stopped.

13. The method according to any one of claims 1 to 12, characterized in that The first operation is a click operation, and in response to the first operation, extracting features from the n-th frame image to obtain template features includes: In response to the first operation, determining a click position of the first operation; According to the click position, selecting a target area in the nth frame image; Feature extraction is performed on the target area to generate the template feature.

14. An electronic device, characterized in that: include: Processors, memory, and interfaces; The processor, the memory and the interface cooperate with each other so that the electronic device executes the method as claimed in any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 13.

16. A target tracking method, the method being executed by an electronic device, characterized in that: The method comprises: Display a first shooting interface, where a first area of ​​the first shooting interface is used to display a video stream captured by a camera; When the first area displays the nth frame image of the video stream, receiving a first operation input by a user on the nth frame image, where n is a positive integer; In response to the first operation, extracting features from the n-th frame image to obtain template features, wherein the n-th frame image contains a target; Determine, by a first network, whether the n+xth frame image of the video stream contains the target according to the template feature, where x is a positive integer; If it is determined through the first network that the n+xth frame image does not contain the target, then according to the template feature, determining through the second network whether the n+xth frame image contains the target, wherein the first network and the second network are both convolutional neural networks, and the first network and the second network satisfy at least one of the first preset conditions; The first preset condition includes: the network depth of the second network is greater than the network depth of the first network, the network width of the second network is greater than the network width of the first network, and the resolution of the input image of the second network is greater than the resolution of the input image of the first network.

17. The method according to claim 16, characterized in that The method further comprises: If it is determined through the second network that the n+xth frame image does not contain the target, then according to the template feature, determining through a third network whether the n+xth frame image contains the target, wherein the second network and the third network satisfy at least one of the second preset conditions; The second preset condition includes: the network depth of the third network is greater than the network depth of the second network, the network width of the third network is greater than the network width of the second network, and the resolution of the input image of the third network is greater than the resolution of the input image of the second network.

18. The method according to claim 17, characterized in that At least one of the network depth and the network width of the second network is greater than that of the first network, and at least one of the network depth and the network width of the third network is greater than that of the first network, and the resolution of the input image of the third network is greater than that of the first network and the second network, respectively.

19. The method according to claim 17 or 18, characterized in that The determining, by a first network, whether the n+xth frame image of the video stream contains the target according to the template feature includes: Selecting a first tracking area from the n+xth frame image; Searching the first tracking area through the first network according to the template feature, and outputting a first position and a first confidence level corresponding to the first position; If the first confidence level is greater than a first confidence level threshold, the first network is used to determine the n+xth frame. The image contains the target; If the first confidence is less than or equal to the first confidence threshold, it is determined through the first network that the n+xth frame image does not contain the target.

20. The method according to claim 19, characterized in that The selecting a first tracking area from the n+xth frame image includes: According to the position of the target in the (n+x-1)th frame image of the video stream, a partial area in the (n+x)th frame image is selected to obtain the first tracking area.

21. The method according to claim 19 or 20, characterized in that The method further comprises: If it is determined through the first network that the target is included in the n+x-th frame image, the first position is output, where the first position represents the position of the target in the n+x-th frame image.

22. The method according to any one of claims 19 to 21, characterized in that The determining, by a second network, whether the n+xth frame image contains the target according to the template feature includes: searching the first tracking area through the second network according to the template feature, and outputting a second position and a second confidence level corresponding to the second position; If the second confidence level is greater than a second confidence level threshold, determining through the second network that the n+xth frame image contains the target; If the second confidence is less than or equal to the second confidence threshold, determining through the second network that the n+xth frame image does not contain the target.

23. The method according to claim 22, characterized in that The method further comprises: If it is determined through the second network that the target is included in the n+x-th frame image, the second position is output, and the second position represents the position of the target in the n+x-th frame image.

24. The method according to any one of claims 19 to 23, characterized in that The search range of the third network for the n+xth frame image is larger than the search range of the first network and the second network for the n+xth frame image.

25. The method according to claim 24, characterized in that The determining, by a third network, whether the n+xth frame image contains the target comprises: Dividing the n+xth frame image into regions to obtain a plurality of second tracking regions; According to the template feature, searching the plurality of second tracking areas respectively through the third network, and outputting a plurality of third positions and a third confidence level corresponding to each of the third positions; If there is a third target confidence greater than a third confidence threshold among the multiple third confidences, determining through the third network that the n+xth frame image contains the target; If the third target confidence does not exist in the multiple third confidences, it is determined through the third network that the n+xth frame image does not contain the target.

26. The method according to claim 25, characterized in that A resolution of at least one second tracking area among the plurality of second tracking areas is greater than a resolution of the first tracking area.

27. The method according to claim 25 or 26, characterized in that The method further comprises: If it is determined by the third network that the target is included in the n+x-th frame image, a third target position is output, where the third target position is a third position corresponding to the third target confidence level, and the third target position represents the position of the target in the n+x-th frame image; If it is determined through the third network that the n+xth frame image does not contain the target, tracking of the target is stopped.

28. The method according to any one of claims 16 to 27, characterized in that The first operation is a click operation, and in response to the first operation, extracting features from the n-th frame image to obtain template features includes: In response to the first operation, determining a click position of the first operation; According to the click position, selecting a target area in the nth frame image; Feature extraction is performed on the target area to generate the template feature.

Images