A target tracking method and an electronic device

Abstract

This application provides a target tracking method. The method includes: The electronic device classifies the tracking targets into specified targets and unspecified targets. For the tracking of specified targets, the electronic device can automatically identify the features of the specified targets in the image frames collected by the camera and continuously track the specified targets in subsequent image frames. For the tracking of unspecified targets, the electronic device requires the user to specify the area of the unspecified target. The electronic device identifies the features of the unspecified target based on the area specified by the user. After that, the electronic device can automatically identify the unspecified target in subsequent image frames and continuously track the unspecified target. This method realizes different tracking methods for specified targets and unspecified targets, solves the problem that current electronic devices cannot track unspecified targets, and realizes all-object tracking.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to an object tracking method and an electronic device. Background Art

[0002] With the continuous development of image processing technology, object tracking is widely used in fields such as intelligent video surveillance, autonomous driving, and unmanned supermarkets. Generally, an electronic device can obtain the features of a specified object in one or more pre-acquired images and save the features of the specified object. Then, the electronic device can perform feature matching between the features of candidate objects in the subsequently captured image frames and the features of the specified object. If the features of the candidate object match the features of the saved tracking object, the electronic device can determine that the candidate object in the current image frame is the tracking object and mark the position of the tracking object.

[0003] However, the existing object tracking methods are for tracking some specified objects (such as people) in the captured images, and for some non-specified objects (such as animals), they cannot be recognized and tracked. Summary of the Invention

[0004] This application provides an object tracking method and an electronic device. This method realizes different tracking methods for specified objects and non-specified objects, solves the problem that current electronic devices cannot track non-specified objects, and realizes object tracking for all things.

[0005] In a first aspect, this application provides an object tracking method, including: an electronic device displays a first shooting interface, and the first shooting interface displays an image captured in real time by a camera; the electronic device determines whether the image captured in real time by the camera includes a specified object that matches a preset object feature template; when the image captured in real time by the camera does not include a specified object that matches the preset object feature template, the electronic device receives and responds to a first input operation on the image captured in real time by the camera, and determines a first area selected by the user on the image captured in real time by the camera; the electronic device extracts first image features in the first area on the image captured in real time by the camera; after extracting the first image features, the electronic device, based on the first image features, determines a second area in the image captured in real time by the camera whose similarity to the first image features is greater than a first value; the electronic device, based on the second area, crops a first cropped image from the image captured in real time by the camera, and the first cropped image includes the second area; the electronic device displays a second shooting interface, and the second shooting interface displays the first cropped image.

[0006] This method realizes different tracking methods for specified objects and non-specified objects, solves the problem that current electronic devices cannot track non-specified objects, and realizes object tracking for all things.

[0007] In combination with the first aspect, in a possible implementation, after the electronic device displays the second shooting interface and the first cropped image is displayed on the second shooting interface, the method further includes: the electronic device extracts second image features in a second area of the image captured by the camera in real time; after the second image features are extracted, the electronic device determines, based on the second image features, a third area in the image captured by the camera in real time whose similarity to the second image features is greater than a first value; the electronic device crops a second cropped image from the image captured by the camera in real time based on the third area, where the second cropped image includes the third area; the electronic device displays a third shooting interface, and the third shooting interface displays the second cropped image. In this way, the electronic device can update the features of the tracking target every preset number of frames. Since the tracking target deforms due to movement, the features of the tracking target will also change.

[0008] In combination with the first aspect, in a possible implementation, after the electronic device displays the second shooting interface and the first cropped image is displayed on the second shooting interface, the method further includes: the electronic device extracts second image features in a second area of the image captured by the camera in real time; the electronic device performs a weighted operation on the second image features and the first image features to obtain third image features; after the third image features are obtained, the electronic device determines, based on the third image features, a fourth area in the image captured by the camera in real time whose similarity to the third image features is greater than a first value; the electronic device crops a third cropped image from the image captured by the camera in real time based on the fourth area, where the third cropped image includes the fourth area; the electronic device displays a fourth shooting interface, and the fourth shooting interface displays the third cropped image. In this way, the electronic device performs a weighted operation on the features of the tracking target in every two frames of images to obtain new features as the features of the tracking target. Since the tracking target deforms due to movement, the features of the tracking target will also change.

[0009] In combination with the first aspect, in a possible implementation, after the electronic device displays the second shooting interface and the first cropped image is displayed on the second shooting interface, the method further includes: the electronic device receives a second input operation from the user; in response to the second input operation, the electronic device displays a fifth shooting interface, and the fifth shooting interface displays the image captured by the camera in real time. In this way, the electronic device can end target tracking according to the user's needs. The second input operation can be clicking an end tracking function control in the second shooting interface, or the second input operation can also be a voice control operation, etc. After ending target tracking, what the electronic device displays is the image captured in real time, and the image captured in real time is no longer cropped.

[0010] In combination with the first aspect, in a possible implementation, the method further includes: based on the position of the first region in the image captured by the camera in real time, the electronic device determines a fifth region in the image captured by the camera in real time, and the fifth region includes the first region; the electronic device determines, from the image captured by the camera in real time, a second region whose similarity to the first image feature is greater than a first value based on the first image feature, specifically including: the electronic device determines, from the fifth region in the image captured by the camera in real time, a second region whose similarity to the first image feature is greater than the first value based on the first image feature. The electronic device determines the search region in the next frame of image based on the first region, that is, the fifth region is the search region. Specifically, the electronic device uses the center point of the first region as the center and M times the size of the first region as the search region, that is, the fifth region. The electronic device performs an operation on each pixel point in the fifth region with the feature of the tracking target to obtain the response value of each pixel point in the fifth region. If the maximum response value of the pixel points in the fifth region is greater than a preset value, it indicates that there is a tracking target in this frame of image, and the pixel point corresponding to the maximum response point in this frame of image is the center point of the tracking target in this frame of image.

[0011] In combination with the first aspect, in a possible implementation, before the electronic device receives and responds to a first input operation on the image captured in real time by the camera, the method further includes: the electronic device receives and responds to a click operation within a sixth area on the image captured in real time by the camera, and displays a first target box on the sixth area of the image captured in real time by the camera; after the electronic device displays the first target box within the sixth area of the image captured in real time by the camera, the electronic device acquires the image captured in real time by the camera and displays a second target box on the image captured in real time by the camera; the image content within the second target box includes the image content within the first target box; the electronic device receives and responds to a first input operation on the image captured in real time by the camera, and determines a first area selected by the user on the image captured in real time by the camera, specifically including: the electronic device receives and responds to a click operation on the second target box on the image captured in real time by the camera, and the area of the second target box on the image captured in real time by the camera is the first area; the first area includes the sixth area. In this way, for non-specified targets, since the electronic device cannot automatically identify non-specified targets in the image, the electronic device can receive the user's manual confirmation of the position of the non-specified target in the image frame, that is, the electronic device receives and responds to a click operation within the sixth area on the image captured in real time by the camera, and displays a first target box on the sixth area of the image captured in real time by the camera. The first target box is used to enclose the position coordinates of the user's click. Then, the first target box gradually enlarges until it completely encloses the image of the non-specified target, that is, the electronic device acquires the image captured in real time by the camera and displays a second target box on the image captured in real time by the camera; the image content within the second target box includes the image content within the first target box. After the second target box completely encloses the entire image of the non-specified target, the electronic device can receive the user's click operation so that the size of the second target box no longer changes, that is, the electronic device receives and responds to a click operation on the second target box on the image captured in real time by the camera, and the area of the second target box on the image captured in real time by the camera is the first area; the first area includes the sixth area.

[0012] In a possible implementation, after the electronic device confirms that the second target box completely encloses the entire image of the non-specified target, the electronic device does not need to receive a click operation on the second target box on the image captured in real time by the camera to make the size of the second target box no longer change, and the electronic device can automatically control the size of the second target box to no longer change.

[0013] In another possible implementation, after the electronic device receives and responds to a click operation in the sixth area on the image captured in real time by the camera and displays a first target box on the sixth area of the image captured in real time by the camera, the first target box can receive a dragging operation by the user to obtain a second target box, and the image included in the second target box is a complete image of a non-specified target.

[0014] In combination with the first aspect, in a possible implementation, the electronic device receives and responds to a first input operation on the image captured in real time by the camera, and determines a first area selected by the user on the image captured in real time by the camera, specifically including: the electronic device receives and responds to a sliding operation on the image captured in real time by the camera, and determines the area of the sliding trajectory of the sliding operation as the first area. That is, the electronic device can receive the sliding operation of the user on the image frame to determine the position of the non-specified target in the image frame.

[0015] In combination with the first aspect, in a possible implementation, the method further includes: the electronic device displays a first tracking box in the eighth area on the first cropped image, and the image content in the eighth area is the same as the image content in the second area, and the image size in the eighth area is different from the image size in the second area. The electronic device displays a tracking box to identify the position of the tracking target in the image. Before and after the image is cropped, the image size of the tracking target displayed by the electronic device is different. The electronic device displays the image after cropping the image captured in real time, and the eighth area is the area where the tracking box is displayed for the cropped image, and the second area is the image before cropping, including the complete image of the tracking target.

[0016] In combination with the first aspect, in a possible implementation, the method further includes: when the image captured in real time by the camera includes a specified target that matches a preset target feature template, the electronic device displays marking information around the specified target on the image captured in real time by the camera; the electronic device receives and responds to a second input operation on the image captured in real time by the camera, and the electronic device obtains the image captured in real time by the camera; the electronic device determines, based on the preset target feature template, a ninth area in the image captured in real time by the camera whose similarity to the target feature template is greater than a second value; the electronic device crops a fourth cropped image from the image captured in real time by the camera based on the ninth area, and the fourth cropped image includes the ninth area; the electronic device displays a sixth shooting interface, and the sixth shooting interface displays the first cropped image. The image frame displayed by the electronic device may include a specified target. When the electronic device automatically identifies the specified target in the image frame, the electronic device can display marking information (such as a target box or category) around the specified target, and the electronic device receives a click operation by the user to determine one of the multiple specified targets as the tracking target.

[0017] In combination with the first aspect, in a possible implementation, the method further includes: after extracting the first image feature, the electronic device does not determine an area with a similarity greater than the first value to the first image feature from the continuous T frames of images captured by the camera in real time; the electronic device displays a seventh shooting interface, and the seventh shooting interface displays the images captured by the camera in real time. In this way, when the electronic device does not track a non-specified target in the continuous T frames of images captured by the camera in real time, that is, it is determined that the electronic device fails to track the non-specified target, the electronic device will display the images captured by the electronic device in real time.

[0018] In a second aspect, the present application provides another electronic device, including one or more processors, one or more memories, and a camera; the one or more memories and the camera are coupled to the one or more processors, the one or more memories are used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute a target tracking method in any possible implementation of the first aspect above.

[0019] In a third aspect, the present application provides a computer-readable storage medium, including instructions, when the instructions run on an electronic device, causing the electronic device to execute a target tracking method in any possible implementation of the first aspect above.

[0020] This method realizes different tracking methods for specified targets and non-specified targets, solves the problem that current electronic devices cannot track non-specified targets, and realizes all-object tracking. Description of the Drawings

[0021] Figure 1 It is a schematic structural diagram of an electronic device 100 provided by an embodiment of the present application;

[0022] Figure 2 It is a software structure block diagram of an electronic device 100 provided by an embodiment of the present application;

[0023] Figures 3 - 5 Exemplarily shown is a schematic diagram of a set of interfaces for opening a camera application provided by an embodiment of the present application;

[0024] Figures 6 - 7 Exemplarily shown is a schematic diagram of a set of interfaces for opening a target tracking function provided by an embodiment of the present application;

[0025] Figures 8 - 10 Exemplarily shown is a schematic diagram of a set of interfaces for opening a target tracking function provided by an embodiment of the present application;

[0026] Figures 11 - 13 Exemplarily shown is a schematic diagram of a set of interfaces for determining a tracking target provided by an embodiment of the present application;

[0027] Figures 14 - 16 Exemplarily shown is a schematic diagram of the interface of a set of electronic devices 100 provided in an embodiment of the present application when target tracking is successful;

[0028] Figures 17 - 18 Exemplarily shown is a schematic diagram of the interface of a set of electronic devices 100 provided in an embodiment of the present application when target tracking fails;

[0029] Figure 19 Exemplarily shown is a schematic diagram of the interface of a method for ending target tracking provided in an embodiment of the present application;

[0030] Figures 20 - 21 Exemplarily shown is another schematic diagram of the interface of a set of electronic devices 100 provided in an embodiment of the present application when target tracking is successful;

[0031] Figure 22 Exemplarily shown is a schematic flowchart of a target tracking method provided in an embodiment of the present application;

[0032] Figures 23 - 31 Exemplarily shown is a schematic diagram of the interface of a process in which a set of electronic devices 100 provided in an embodiment of the present application tracks a certain non - designated target;

[0033] Figures 32 - 33 Exemplarily shown is another schematic diagram of the interface of a process in which a set of electronic devices 100 provided in an embodiment of the present application tracks a certain non - designated target;

[0034] Figure 34 Exemplarily shown is another schematic flowchart of a target tracking method provided in an embodiment of the present application. Detailed implementation manners

[0035] Hereinafter, the technical solutions in the embodiments of the present application will be clearly and elaborately described with reference to the accompanying drawings. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; the "and / or" in the text is merely a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

[0036] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" is two or more than two.

[0037] The term "user interface (UI)" in the specification, claims and drawings of this application is a media interface for interaction and information exchange between an application or an operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in specific computer languages such as Java and Extensible Markup Language (XML). The interface source code is parsed and rendered on a terminal device and finally presented as content recognizable by the user, such as controls like images, texts, and buttons. A control, also known as a widget, is a basic element of the user interface. Typical controls include a toolbar, a menu bar, an input box, a button, a scrollbar, an image, and a text. The attributes and content of the controls in the interface are defined by tags or nodes. For example, XML passes through <textview> 、 <imgview> 、 <videoview>Nodes etc. are used to specify the controls included in the interface. One node corresponds to one control or property in the interface, and after being parsed and rendered, the node presents as visible content to the user. In addition, in the interfaces of many applications, such as hybrid applications, there are usually web pages included. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hyper text markup language (HTML), cascading stylesheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as recognizable content to the user by a browser or a web page display component similar to the browser function. The specific content included in the web page is also defined by tags or nodes in the web page source code. For example, HTML uses 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0038] A commonly used form of the user interface is the graphical user interface (GUI), which refers to the user interface related to computer operations displayed in a graphical manner. It can be interface elements such as a window or a control displayed on the display screen of an electronic device.

[0039] Figure 1 Shows a schematic structural diagram of the electronic device 100.

[0040] The device type of the electronic device 100 can include mobile phones, televisions, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, etc. The embodiments of this application do not impose special restrictions on the device type of the electronic device 100.

[0041] Hereinafter, the embodiments of this application will be described by taking the electronic device 100 as a mobile phone as an example. This application does not make any limitations here.

[0042] It should be understood that Figure 1 The illustrated electronic device 100 is merely an example, and the electronic device 100 may have more or fewer components than those Figure 1 shown, may combine two or more components, or may have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0043] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0044] It can be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0045] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0046] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.

[0047] A memory can also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can hold the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0048] The charging management module 140 is configured to receive a charging input from a charger. Herein, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input from the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0049] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as the battery capacity, the number of battery cycles, and the battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be provided in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be provided in the same device.

[0050] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0051] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0052] The mobile communication module 150 may provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 may receive electromagnetic waves through the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 may also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation.

[0053] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor.

[0054] The wireless communication module 160 may provide solutions for wireless communications such as wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the electronic device 100. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves through the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive the signal to be transmitted from the processor 110, perform frequency modulation and amplification on it, and convert it into electromagnetic waves through the antenna 2 for radiation.

[0055] The electronic device 100 realizes the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change the display information.

[0056] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0057] The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, etc.

[0058] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera sensor. The optical signal is converted into an electrical signal, and the camera sensor transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0059] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the sensor. The sensor can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The sensor converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0060] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0061] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0062] The NPU is a neural-network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission pattern between human brain neurons, it can quickly process input information and can also continuously learn on its own. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0063] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to achieve the data storage function.

[0064] The internal memory 121 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.). The data storage area can store the data created during the use of the electronic device 100 (such as audio data, phone book, etc.). In addition, the internal memory 121 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0065] The electronic device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and the application processor, etc. Such as music playback, recording, etc.

[0066] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0067] The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or hands-free calls through the speaker 170A.

[0068] The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be listened to by placing the receiver 170B close to the human ear.

[0069] The microphone 170C, also known as the "microphone" or "transmitter", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak by placing the mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0070] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0071] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor may include at least two parallel plates having conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, the instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, the instruction to create a new short message is executed.

[0072] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the angle of jitter of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and enables the lens to offset the jitter of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenarios.

[0073] The barometric pressure sensor 180C is used to measure barometric pressure. In some embodiments, the electronic device 100 calculates the altitude according to the barometric pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0074] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case.

[0075] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0076] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance through infrared or laser.

[0077] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode may be an infrared light-emitting diode. The electronic device 100 emits infrared light outward through the light-emitting diode. The electronic device 100 uses the photodiode to detect the infrared reflected light from nearby objects.

[0078] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touch.

[0079] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering of incoming calls, etc.

[0080] The temperature sensor 180J is used to detect temperature.

[0081] The touch sensor 180K, also known as the "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180K is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a different position from the display screen 194.

[0082] The bone conduction sensor 180M can acquire vibration signals.

[0083] The keys 190 include a power-on key, volume keys, etc. The keys 190 can be mechanical keys. They can also be touch keys. The electronic device 100 can receive key inputs and generate key signal inputs related to the user settings and function controls of the electronic device 100.

[0084] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and can also be used for touch vibration feedback. For example, touch operations on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. Touch operations on different areas of the display screen 194 can also correspond to different vibration feedback effects for the motor 191. Different application scenarios (such as time reminders, receiving messages, alarms, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0085] The indicator 192 can be an indicator light, which can be used to indicate the charging status, power change, or can also be used to indicate messages, missed calls, notifications, etc.

[0086] The SIM card interface 195 is used to connect to the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the electronic device 100. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0087] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of the present invention, the Android system with a layered architecture is taken as an example to exemplarily illustrate the software structure of the electronic device 100.

[0088] Figure 2 It is the software structure block diagram of the electronic device 100 in the embodiments of the present invention.

[0089] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom are the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0090] The application layer can include a series of application packages.

[0091] As Figure 2 shown, the application packages can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0092] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0093] As Figure 2 shown, the application framework layer may include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc.

[0094] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0095] The content provider is used to store and obtain data, and make this data accessible to application programs. The data may include videos, images, audio, dialed and answered calls, browsing history and bookmarks, phone books, etc.

[0096] The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

[0097] The telephone manager is used to provide the communication function of the electronic device 100. For example, the management of call status (including connection, disconnection, etc.).

[0098] The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc.

[0099] The notification manager enables application programs to display notification information in the status bar. It can be used to convey message of the notification type, and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify the completion of a download, message reminder, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or a scroll bar text, such as the notification of a background running application program, and can also be a notification that appears on the screen in the form of a dialogue window. For example, prompting text information in the status bar, emitting a prompt sound, vibrating the electronic device, flashing the indicator light, etc.

[0100] Android Runtime includes a core library and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system.

[0101] The core library contains two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core library of Android.

[0102] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0103] The system library may include multiple functional modules. For example: surface manager, Media Libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), tracking templates, etc.

[0104] The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

[0105] The media library supports the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media library can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0106] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0107] The 2D graphics engine is a drawing engine for 2D drawing.

[0108] The tracking template is used to represent one or more features corresponding to the tracking target.

[0109] The kernel layer is the layer between hardware and software. The kernel layer at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0110] Currently, since only recognition algorithms or models for some specified targets are generally pre-installed in the electronic device, the electronic device can only recognize the specified target or object (such as a person) and track the specified target or object in subsequent image frames. Therefore, the present application provides a target tracking method, which classifies the tracking target into a specified target and an unspecified target. For the tracking of the specified target, the electronic device 100 can automatically identify the features of the specified target in the image frame collected by the camera and continuously track the specified target in subsequent image frames. For the tracking of the unspecified target, the electronic device 100 needs the user to specify the area of the unspecified target. The electronic device 100 recognizes the features of the unspecified target according to the area specified by the user. After that, the electronic device 100 can automatically recognize the unspecified target in subsequent image frames and continuously track the unspecified target. This method realizes different tracking methods for the specified target and the unspecified target, solves the problem that the current electronic device 100 cannot track the unspecified target, and realizes the tracking of all things.

[0111] The designated target means that the electronic device 100 is pre - installed with recognition algorithms or models for some designated targets. The electronic device 100 can automatically recognize the targets in the image frame, then mark the positions of the targets in the image frame, and focus on the targets during shooting. The designated targets are pre - installed in the electronic device 100. Exemplarily, the designated targets can include but are not limited to people, buses, cars, bags, the moon, flowers, bowls, water cups, vases, houses, etc.

[0112] The non - designated target means that the electronic device 100 does not have pre - installed recognition algorithms or models for some designated targets. Therefore, the electronic device 100 cannot automatically recognize non - designated targets in the image frame. In this application, other targets other than the designated targets pre - installed in the electronic device 100 are called non - designated targets, such as animals.

[0113] The tracking target is that the electronic device 100 receives a user's selection operation on a target in the Nth frame image. The target selected by the user is the tracking target. After that, the electronic device 100 continuously tracks the target selected by the user in subsequent consecutive image frames, marks the position of the target selected by the user in the subsequent consecutive image frames, and focuses on the target during shooting to make the target selected by the user clearer. The tracking target can be any one of the designated targets or non - designated targets.

[0114] Embodiment 1

[0115] The electronic device 100 can automatically recognize the designated target in the image frame and continuously track the designated target in subsequent consecutive image frames. Specifically, the user interface of the electronic device 100 displays an image frame, which includes one or more designated targets. The electronic device 100 determines one of the one or more designated targets as the tracking target according to the user's instruction and displays a tracking frame. The electronic device 100 extracts the features of the tracking target in the tracking frame. After that, in subsequent consecutive image frames, the electronic device 100 uses the center of the tracking frame in the previous frame as the search center and M times the size of its tracking frame as the search area to track the tracking target. If the electronic device 100 matches the features in the search area that match the features of the tracking target, the electronic device 100 marks the position of the tracking target in this frame image and focuses on the tracking target during shooting to make the tracking target clearer. The electronic device 100 adopts this method to track the tracking target. If the electronic device 100 does not match the features in the search area that match the features of the tracking target in subsequent consecutive P frame images, the target tracking fails, and the electronic device 100 stops tracking the tracking target.

[0116] In some embodiments, the image frame displayed on the user interface of the electronic device 100 may be the one displayed when the electronic device 100 is recording a video or taking a photo. In one possible implementation, the electronic device 100 may have the camera application opened to record a video or take a photo. In another possible implementation, the electronic device 100 may also have other applications with a photo-taking function (such as WeChat, Weibo, etc.) opened to record a video or take a photo. The present application does not make any limitation here.

[0117] In some other embodiments, the image frame displayed on the user interface of the electronic device 100 may also be sent to the electronic device 100 by other devices (such as drones, surveillance cameras, etc.).

[0118] It can be understood that if the image frame displayed on the user interface of the electronic device 100 is obtained by other devices (such as drones, surveillance cameras, etc.) and sent to the electronic device 100. Before the electronic device 100 displays the image frame obtained by other devices (such as drones, surveillance cameras, etc.), the electronic device 100 establishes a communication connection with other devices (such as drones, surveillance cameras, etc.). In one possible implementation, the electronic device 100 may establish a communication connection with a video surveillance device through an application. In another possible implementation, the electronic device 100 may establish a communication connection with other devices (such as drones, surveillance cameras, etc.) through Bluetooth, Wireless Local Area Network (WLAN), etc.

[0119] In the following embodiments of the present application, an example is given where the electronic device 100 is tracking a specified target while recording a video.

[0120] Please refer to Figures 3 - 21 , Figures 3 - 21 which exemplarily shows the process of the electronic device 100 tracking a specified target.

[0121] Figure 3 Exemplarily shown is the user interface 30 of the electronic device 100. The user interface 30 may include icons of some applications. For example, the icon 309 of the clock, the icon 311 of the calendar, the icon 313 of the gallery, the icon 317 of the file manager, the icon 319 of the email, the icon 321 of the music, the icon 105 of the file manager, the icon 325 of Huawei Video, the icon 327 of Sports Health, the icon 329 of the weather, the icon 330 of the camera, the icon 331 of the contacts, the icon 332 of the phone, and the icon 333 of the messages. In some embodiments, the user interface 30 may include more or fewer icons of applications. In some embodiments, the user interface 30 may include some related to Figure 3 Icons of different applications shown, such as icons of instant messaging applications, etc. There is no limitation here.

[0122] Exemplarily, such as Figure 4 As shown, the icon 330 of the camera can receive a user click operation. In response to the user click operation, the electronic device 100 displays as Figure 5 shown in the user interface 40. The user interface 40 exemplarily shows some function icons. For example, aperture icon 410, night scene icon 402, portrait icon 403, photo taking icon 404, video recording icon 405, professional icon 406, etc. multi-icon 407, flashlight icon 408, filter icon 409 and settings icon 410, current captured photo thumbnail control 412, shooting control 413, front and rear camera switching control 414. The user interface 40 also includes an image frame 411 captured by the camera application.

[0123] When the electronic device 100 enables the video recording function, the user wants to continuously track a specified target in the image frame displayed on the electronic device 100, so that the picture of a specified target captured is clearer. Any one of the following two methods can be used to enable the target tracking function of the electronic device 100.

[0124] Method 1:

[0125] As Figure 6 shown, the settings icon 410 can receive a user click operation. In response to the user click operation, the electronic device 100 displays as Figure 7 shown in the user interface 50. The user interface 50 includes a shooting mute control 415, the shooting mute function is enabled; a timed shooting control 416, the timed shooting function is disabled; a voice control photo taking control 417, the voice control photo taking function is disabled; a target tracking control 418, the target tracking function is disabled; a volume key function control 419, the function of the volume key is the shutter; a quick capture control 420 when the screen is off, that is, when the electronic device 100 receives a double click on the volume down key by the user in the locked screen state, the electronic device 100 starts the camera and takes a photo; the user interface 50 also includes an image adjustment control 421. Among them, when the target tracking function is disabled, the target tracking control 418 can receive a user click operation to enable the target tracking function.

[0126] Method 2:

[0127] As Figure 8 shown, the electronic device 100 can receive a leftward sliding operation of the user on the right boundary of the user interface 40. In response to this user operation, the electronic device 100 can display as Figure 9 The user interface 60 shown. The user interface 60 includes a target tracking control 601. The target tracking control 601 can receive a user click operation to enable the target tracking function. In response to the user clicking on the target tracking control 601, the electronic device enables the target tracking function.

[0128] In some embodiments, the electronic device 100 can receive a user's operation of swiping right on the left boundary of the user interface 40 to display the user interface including the target tracking control 601, can also receive a user's operation of swiping up on the lower boundary of the user interface 40 to display the user interface including the target tracking control 601, and can also receive a user's operation of swiping down on the upper boundary of the user interface 40 to display the user interface including the target tracking control 601. This application does not limit the position of the user's swiping operation.

[0129] In some embodiments, the display position of the user interface 60 can be at the top of the user interface 40, can also be at the bottom of the user interface 40, can also be in the middle of the user interface 40, etc. This application does not limit the display position of the user interface 60.

[0130] Figures 10 - 12 Exemplarily shows the process of an electronic device 100 determining a tracking target.

[0131] Figure 10 Exemplarily shows a schematic diagram of the Nth frame image 111 obtained by the electronic device 100 after enabling the target tracking function. The user interface 40 exemplarily shows the specified target automatically recognized by the electronic device 100 and the position coordinates of the specified target. The user interface 40 exemplarily shows the specified target 1 (house) and the specified target 2 (person) automatically recognized by the electronic device 100. The Nth frame image 111 can also include more other specified targets, and this application does not limit this again. The Nth frame image 111 can also include target frames for each specified target. The target frames are used to enclose the target objects and represent the position coordinates of the target objects. Exemplarily, the target frame 1101 is used to enclose the specified target 1 (house), and the target frame 1102 is used to enclose the specified target 2 (person).

[0132] In some embodiments, the Nth frame image 111 may not display the category of the specified target automatically recognized by the electronic device 100. For example, the Nth frame image 111 does not display the specified target 1 (house) and the specified target 2 (person), and this application does not limit this here.

[0133] Figure 11 Exemplarily shows the position coordinates of the target frame 1101 and the target frame 1102. In the embodiments of this application, the position coordinates of the target object enclosed by the target frame are represented by the four vertex coordinates of the target frame. As Figure 12 As shown in the figure, with the left boundary of the Nth frame image 111 as the X-axis, the lower boundary of the Nth frame image 111 as the Y-axis, and the intersection of the left boundary and the lower boundary of the Nth frame image 111 as the origin, a coordinate system is established. The position coordinates of the target box 1101 are determined by the coordinates of point A, point B, point C, and point D, that is, the position coordinates of the specified target 1 (house) are determined. The position coordinates of the target box 1102 are determined by the coordinates of point E, point F, point G, and point H, that is, the position coordinates of the specified target (person) are determined.

[0134] In some embodiments, the electronic device 100 may not display the target box. That is, in Figure 10 and Figure 11 , the electronic device 100 does not display the target box 1101 and the target box 1102. This is not limited herein.

[0135] As Figure 12 shown, in some embodiments, after the electronic device 100 automatically identifies the specified target and the position coordinates of the specified target, the user interface 40 will also display a prompt box 1103. The content of the prompt box 1103 includes "Please select the target to track", and the prompt box 1103 is used to prompt the user to select the target that needs to be tracked.

[0136] As Figure 12 shown, the specified target 2 (person) can receive the click operation of the user. In response to the click operation of the user, the electronic device 100 determines that the specified target 2 (person) is the target to be tracked. After determining that the specified target 2 (person) is the target to be tracked, the electronic device 100 will track the specified target 2 (person) in subsequent consecutive image frames. Of course, the specified target 1 (house) can also receive the click operation of the user. In response to the click operation of the user, the electronic device 100 determines that the specified target 1 (house) is the target to be tracked. After determining that the specified target 1 (house) is the target to be tracked, the electronic device 100 will track the specified target 1 (house) in subsequent consecutive image frames.

[0137] After the electronic device 100 determines that the specified target 2 (person) is the target to be tracked, the electronic device 100 will mark the position of the specified target 2 (person) in this image frame in the user interface 40.

[0138] In some embodiments, after the electronic device 100 determines that the specified target 2 (person) is the target to be tracked, the user interface 40 will display a tracking box for the specified target 2 (person). The tracking box will mark the position of the specified target 2 (person) in this image frame. The tracking box 1110 is centered on the center of the target box 1102 and encloses the target to be tracked. The image enclosed within the tracking box 1110 includes the complete image of the target to be tracked.

[0139] In some embodiments, after the electronic device 100 marks the position of the tracking target in the image frame in the user interface 40, the type corresponding to the tracking target may be displayed at or around the position of the designated target.

[0140] It should be noted that only when the coordinates of the position clicked by the user are within the position coordinates of the target box of each designated target, will the electronic device 100 display the tracking box and track the tracking target selected by the user in subsequent consecutive image frames. When the coordinates of the position clicked by the user are not within the position coordinates of the target box of each designated target, the electronic device does not display the tracking box, indicating that the user has not selected the target to be tracked. The user needs to click on the target to be tracked again. When the coordinates of the position clicked by the user are within the position coordinates of the target box of the designated target, the electronic device 100 confirms that the designated target clicked by the user is the tracking target, and the electronic device 100 will track the tracking target selected by the user in subsequent consecutive image frames.

[0141] In other embodiments, when the electronic device 100 does not display the target box 1101 and the target box 1102, the electronic device 100 may receive a user's frame-drawing operation to circle the designated target 2 (person) to select the tracking target. The electronic device 100 may also use other methods to confirm the tracking target, which is not limited herein.

[0142] In response to the tracking target selected by the user, the electronic device 100 will extract the features of the tracking target and save the features of the tracking target.

[0143] Figure 13 Exemplarily shows the tracking target selected by the user and the features corresponding to the tracking target. As Figure 13 As shown, in response to the user's operation of selecting a tracking target, the electronic device 100 extracts the features of the specified target 2 (person) and saves the features of the specified target 2 (person). The electronic device 100 can extract the features of the tracking target image through a feature extraction algorithm, obtain and save the features of the tracking target image (such as texture features, contour features, color features, etc.). Specifically, the electronic device 100 can use the feature vector obtained by extracting the features of the tracking target image to represent the features of the tracking target corresponding to the tracking target image. The features of the specified target 2 (person) in the Nth frame image can be represented by the feature vector F1(N). The feature vector can represent other features such as the color feature, texture feature, and contour feature of the tracking target. For example, the feature vector F1(N) can represent one or more of the texture feature, contour feature, color feature, etc. of the specified target 2 (person). Here, the specific form and size of the feature vector F1(N) of the specified target 2 (person) are not limited. For example, F1(N) can be a feature vector [0.5, 0.6, 0.8, …, 0.9, 0.7, 0.3] containing n values. Among them, n is an integer, which can be 128, 256, 512, etc., and the size of n is not limited. After the electronic device 100 extracts the features of the specified target 2 (person), it saves the features of the specified target 2 (person) to the tracking template.

[0144] The tracking template can be used to represent one or more features of the tracking target. After the electronic device extracts the features of the tracking target, the electronic device saves the features of the tracking target to the tracking template. In subsequent consecutive video frames, when the electronic device matches the features of the specified target in the image frame with the features of the tracking target in the tracking template, if the match is successful, the electronic device determines the specified target as the tracking target in the image frame.

[0145] After the user specifies the tracking target, the electronic device 100 will track the tracking target in the image frames after the Nth frame image.

[0146] Next, how the electronic device 100 tracks the tracking target will be introduced.

[0147] The electronic device 100 can track the tracking target in any one of the following two ways.

[0148] Method 1:

[0149] In consecutive image frames after the Nth frame image, the electronic device 100 uses the center of the tracking box in the previous frame image as the search center, and uses M times the size of its tracking box as the search area to track the tracking target. The electronic device 100 obtains the response value of each pixel point in the search area based on the characteristics of the tracking target and each pixel point in the search area. If the maximum response value corresponding to the pixel point in the search area is greater than the preset response value, there is a tracking target in the search area. Then the electronic device 100 marks the position of the tracking target in this frame of image. After that, the electronic device 100 automatically focuses on the tracking target to make the tracking target clearer in the captured image.

[0150] Figures 14 - 16 Exemplarily shows a schematic diagram of successful target tracking by the electronic device 100.

[0151] Specifically, Figure 14 Exemplarily shows the (N + 1)th frame image 171 obtained by the electronic device 100. The (N + 1)th frame image 171 is the original image obtained by the electronic device 100.

[0152] The electronic device 100 uses the center of the tracking box 1110 in the Nth frame image as the search center, and uses M times the size of the tracking box 1110 as the search area 1105 to track the tracking target in the (N + 1)th frame image 171, where M is a constant greater than or equal to 1.

[0153] Exemplarily, if the coordinates of the center point of the tracking box 1110 in the Nth frame image are (x, y), and the size of the tracking box 1110 is P * Q, then the coordinates of the center point of the search area 1105 in the (N + 1)th frame image are (x, y), and the size of the search area 1105 is (P * M) * (Q * M).

[0154] The electronic device 100 obtains the response value of each pixel point in the search area based on the characteristics of the tracking target and each pixel point in the search area.

[0155] Specifically, the electronic device 100 performs a convolution operation on the characteristics of the tracking target and each pixel in the search area 1105 in the (N + 1)th frame image to obtain the response value of each pixel. The response value represents the probability that each pixel point in the search area 1105 in the (N + 1)th frame image is the center point of the final tracking target.

[0156] Not limited to the convolution operation, the electronic device 100 can also use other methods to obtain the response value of each pixel point in the search area, which is not limited in this application.

[0157] When the maximum response value corresponding to the pixel points within the search area 1105 in the (N + 1)-th frame image is greater than the preset response value, it indicates that there is a tracking target within the search area 1105, and the target tracking is successful. The electronic device 100 uses the pixel point corresponding to the maximum response value as the center point of the tracking target in the (N + 1)-th frame image to determine the position of the tracking target. After that, the electronic device shall give the tracking frame of the tracking target in the (N + 1)-th frame image according to the center point of the tracking target in the (N + 1)-th frame image.

[0158] It can be understood that the size of the tracking frame of the tracking target in the (N + 1)-th frame image can be different from or the same as that of the tracking frame of the tracking target in the N-th frame image, and this application does not make any limitations here.

[0159] After the electronic device 100 tracks the tracking target in the (N + 1)-th frame image, the electronic device 100 will mark the position of the tracking target in the (N + 1)-th frame image on the user interface 40.

[0160] As Figure 15 shown, after the electronic device 100 determines the center point of the tracking target in the (N + 1)-th frame image, the electronic device 100 will use the center point of the tracking target as the center and crop the obtained (N + 1)-th frame image 171 with a cropping frame of a preset size to obtain an image 1701. Generally, the preset size of the cropping frame is larger than the size of the tracking frame 1110. Exemplarily, the preset size of the cropping frame can be half or three-quarters of the size of the original image obtained by the electronic device 100, and this application does not make any limitations on the preset size of the cropping frame.

[0161] As Figure 16 shown, Figure 16 is a UI diagram of the electronic device 100 displaying the image 1701. The electronic device 100 displays the image 1701 and also displays the tracking frame 1702 of the tracking target. The tracking frame 1702 uses the pixel point corresponding to the maximum response value within the search area 1105 as the center point.

[0162] It can be understood that the size of the tracking frame 1702 can be different from the size of the tracking frame 1110. Because in each frame image, the size of the tracking target may be different. For example, the tracking target may be deformed due to movement, resulting in the image of the tracking target being different in each frame image, and thus the size of the tracking target in each frame image is different.

[0163] In some embodiments, after the electronic device 100 marks the position of the tracking target in the (N + 1)-th frame image on the user interface 40, the type corresponding to the specified target can be displayed at or around the position of the tracking target.

[0164] Exemplarily, as Figure 16 As shown, the electronic device 100 will display a specified target 2 (person) icon on the user interface 40, and the specified target 2 (person) icon is used to mark the type of the tracking target.

[0165] In a possible implementation, when the maximum response value corresponding to the pixel points in the search area 1105 in the (N + 1)-th frame image is greater than the preset response value, it indicates that there is a tracking target in the search area 1105, which means that the target tracking is successful. The electronic device 100 can display a prompt, and the prompt is used to prompt the user that the target tracking is successful.

[0166] As Figure 16 shown, if the target tracking is successful, the user interface 40 displays a prompt box 1703, and the prompt box 1703 is used to prompt the user that the target tracking is successful. The prompt content of the prompt box 1703 includes "Target tracking successful".

[0167] It can be understood that in order to enable the user to more clearly observe the motion state of the tracking target, the user interface 40 may also not display the prompt, so that the image displayed by the electronic device 100 is more concise.

[0168] Figures 17 - 18 An exemplary schematic diagram of the target tracking failure of the electronic device 100 is shown.

[0169] If the maximum response value corresponding to the pixel points in the search area of the electronic device 100 is less than the preset value in the consecutive P frame images according to the above method, the target tracking fails, where p is a positive integer greater than or equal to 1.

[0170] Specifically, Figure 17 Exemplarily, the (N + 1)-th frame image 181 obtained by the electronic device 100 is shown. The (N + 1)-th frame image 181 is the original image obtained by the electronic device 100.

[0171] The electronic device 100 uses the center of the tracking frame 1110 in the N-th frame image as the search center, and uses M times the size of the tracking frame 1110 as the search area 1105 to track the tracking target in the (N + 1)-th frame image 181, where M is a constant greater than or equal to 1.

[0172] Exemplarily, if the coordinates of the center point of the tracking frame 1110 in the N-th frame image are (x, y), and the size of the tracking frame 1110 is P * Q, then the coordinates of the center point of the search area 1801 in the (N + 1)-th frame image are (x, y), and the size of the search area 1801 is (P * M) * (Q * M).

[0173] The electronic device 100 performs a convolution operation on the features of the tracking target and each pixel in the search area 1801 of the (N + 1)-th frame image 181 to obtain the response value of each pixel. The response value represents the probability that each pixel point in the search area 1801 of the (N + 1)-th frame image 181 is the center point of the final tracking target.

[0174] When the maximum response value corresponding to the pixel points in the search area 1801 of the (N + 1)-th frame image 181 is less than the preset response value, it indicates that there is no tracking target in the search area.

[0175] When the electronic device 100 performs tracking according to the above method in P consecutive frame images after the N-th frame image (for example, the (N + P)-th frame image), and the maximum response value corresponding to the pixel points in the search area in the P consecutive frame images is less than the preset response value, then the tracking of the tracking target fails.

[0176] After the tracking of the tracking target fails, the electronic device 100 will stop tracking the tracking target, and the electronic device 100 will display the (N + P)-th original image obtained by the electronic device 100 on the user interface 40, that is, the electronic device 100 stops tracking the tracking target when displaying the (N + P)-th original image obtained by the electronic device 100.

[0177] As Figure 18 shown, Figure 18 is a UI diagram of the electronic device 100 displaying the obtained (N + P)-th original image 182.

[0178] In a possible implementation manner, when the target tracking fails, the electronic device 100 will display a prompt, and the prompt is used to prompt the user that the target tracking fails.

[0179] As Figure 18 shown, if the electronic device 100 fails to track the tracking target according to the above method in P consecutive frame images, the user interface 40 displays a prompt box 1802, and the prompt box 1802 is used to prompt the user that the target tracking fails. The prompt content of the prompt box 2101 includes "Target tracking failed".

[0180] After the electronic device stops tracking the tracking target, when the user wants to track a specified target in an image frame again, the specified target in the image frame can receive the user's click operation. In response to the user's click operation, when the position coordinates of the user's click are within the position coordinates of the specified target, the electronic device 100 will determine the specified target clicked by the user as the tracking target, and extract the features of the specified target clicked by the user as the features of the tracking target. The electronic device 100 will track the tracking target in the consecutive image frames after the (N + P)-th frame image. Specifically, please refer to Figures 12 - 18 the embodiments shown, and the present application will not be elaborated again.

[0181] In a possible implementation, after the electronic device 100 enables the tracking mode, the electronic device 100 can receive an active trigger operation from the user to stop tracking the tracking target.

[0182] As Figure 19 shown, when the electronic device 100 enables the tracking mode and tracks the tracking target in the (N + 1)-th frame image, the user interface 40 can display an exit tracking mode control 1707. The exit tracking mode control 1707 can receive a click operation from the user to cause the electronic device 100 to stop tracking the tracking target. In this way, the user can choose to exit the tracking mode according to their own needs, improving the user experience.

[0183] It can be understood that the electronic device 100 can also end the tracking of the tracking target in other ways, not limited to the user clicking the exit tracking mode control to end the tracking of the tracking target, and the present application does not make a limitation here.

[0184] Method 2:

[0185] In consecutive image frames after the N-th frame image of the electronic device 100, for example, in the (N + 1)-th frame image, the electronic device 100 can automatically identify the positions of one or more specified targets in the (N + 1)-th frame image and display target frames corresponding to the specified targets at the positions of the one or more specified targets. The electronic device 100 extracts the features of the specified targets in the one or more target frames and matches the features of the one or more specified targets with the features of the tracking target in the tracking template. If the features of one specified target match the features of the tracking target, the electronic device 100 successfully tracks the target and marks the position of the tracking target in this frame image. Optionally, the electronic device 100 can also identify the types of one or more specified targets in the (N + 1)-th image frame and display the types corresponding to the respective specified targets at or around the positions of the specified targets. After that, the electronic device 100 focuses on the tracking target during shooting to make the captured tracking target clearer.

[0186] Figures 20 - 21 An exemplary schematic diagram showing the successful target tracking of the electronic device 100 is shown.

[0187] Specifically, Figure 20 Exemplarily shown is the (N + 1)-th frame image 191 obtained by the electronic device 100. The (N + 1)-th frame image 191 is the original image obtained by the electronic device 100.

[0188] The electronic device 100 can automatically identify the specified target in the (N + 1)-th frame image 191 and display the target box of the specified target. The user interface 40 exemplarily shows the target box 1601 of the specified target 1 (house); the target box 1602 of the specified target 2 (person); the target box 1603 of the specified target 3 (person).

[0189] In some embodiments, the (N + 1)-th frame image may also display the type icons of each specified target. This application does not make any limitations here.

[0190] In some embodiments, the electronic device 100 may also not display the target box of the specified target. This application does not make any limitations here.

[0191] First, the electronic device 1000 will extract features of the identified specified target in the (N + 1)-th frame image 191.

[0192] Figure 21 Exemplarily shows the images of multiple specified targets in the (N + 1)-th frame image 191 and the features respectively corresponding to the multiple specified targets. The features of the specified target 1 (house) can be represented by the feature vector F2(N), F2(N) = [x1, x2, x3, …, xn]. The feature vector F2(N) can represent the texture features, contour features, etc. of the specified target 1 (house). The features of the specified target 2 (person) can be represented by the feature vector F3(N), F3(N) = [x1, x2, x3, …, xn]. The feature vector F3(N) can represent the texture features, contour features, etc. of the specified target 2 (person). The features of the specified target 3 (person) can be represented by the feature vector F4(N), F4(N) = [x1, x2, x3, …, xn]. The feature vector F4(N) can represent the texture features, contour features, etc. of the specified target 3 (person).

[0193] Secondly, the electronic device 100 will match the features of the multiple specified targets extracted from the (N + 1)-th frame image 191 with the features of the saved tracking target.

[0194] Specifically, the electronic device 100 will match the feature vector F2(N) of the specified target 1 (house), the feature vector F3(N) of the specified target 2 (person), and the feature vector F4(N) of the specified target 3 (person) with the feature vector F1(N) of the saved tracking target. If the feature vector F3(N) of the specified target 2 (person) matches successfully with the feature vector F1(N) of the tracking target, the electronic device 100 will display that the target tracking is successful.

[0195] The successful matching of the feature vector F3(N) of the specified target 2 (person) with the feature vector F1(N) of the tracking target means that the electronic device 100 can calculate the Euclidean distance D between the feature vector F3(N) of the specified target 2 (person) and the feature vector F1(N) of the tracking target. If the Euclidean distance D1 between the feature vector F3(N) and the feature vector F1(N) is less than the preset Euclidean distance D, the electronic device 100 determines that the feature vector F3(N) of the specified target 2 (person) and the feature vector F1(N) of the tracking target are successfully matched.

[0196] When the feature of a specified target in the (N + 1)-th frame image 191 matches the feature of the tracking target, the electronic device 100 will crop the (N + 1)-th frame image 191 according to a cropping frame of a preset size and display the cropped image on the user interface 40. Specifically, please refer to Figures 15 - 16 the embodiments shown, and details are not described herein again in this application.

[0197] In some embodiments, when there is no feature matching the feature of the tracking target in each of the consecutive P frame images (such as the (N + P)-th frame image) after the N-th frame image according to the above method, the target tracking fails.

[0198] If the electronic device 100 only recognizes the specified target 1 (house) and the specified target 3 (person) in the (N + 1)-th frame image.

[0199] First, the electronic device 100 will extract features of the recognized specified target 1 (house) and the specified target 3 (person) in the (N + 1)-th frame image.

[0200] Specifically, reference can be made to Figure 21 , where the feature of the specified target 1 (house) can be represented by the feature vector F2(N), F2(N) = [x1, x2, x3,..., xn]. The feature vector F2(N) can represent the texture feature, contour feature, etc. of the specified target 1 (house). The feature of the specified target 3 (person) can be represented by the feature vector F4(N), F4(N) = [x1, x2, x3,..., xn]. The feature vector F4(N) can represent the texture feature, contour feature, etc. of the specified target 3 (person).

[0201] Second, the electronic device 100 will match the specified target 1 (house) and the specified target 3 (person) extracted from the (N + 1)-th frame image with the features of the saved tracking target.

[0202] Specifically, the electronic device 100 matches the feature vector F2(N) of the specified target 1 (house) and the feature vector F4(N) of the specified target 3 (person) with the feature vector F1(N) of the saved tracking target. If the feature vector F2(N) of the specified target 1 (house) and the feature vector F4(N) of the specified target 3 (person) both fail to match the feature vector F1(N) of the tracking target, the electronic device 100 displays that the target tracking fails.

[0203] Specifically, the failure of the feature vector F2(N) of the specified target 1 (house) to match the feature vector F1(N) of the tracking target means that the electronic device 100 can calculate the Euclidean distance D2 between the feature vector F2(N) of the specified target 1 (house) and the feature vector F1(N) of the tracking target. If the Euclidean distance D2 between the feature vector F2(N) and the feature vector F1(N) is greater than the preset Euclidean distance D, the electronic device 100 determines that the feature vector F2(N) of the specified target 1 (house) fails to match the feature vector F1(N) of the tracking target. The failure of the feature vector F4(N) of the specified target 3 (person) to match the feature vector F1(N) of the tracking target means that the electronic device 100 can calculate the Euclidean distance D3 between the feature vector F4(N) of the specified target 3 (person) and the feature vector F1(N) of the tracking target. If the Euclidean distance D3 between the feature vector F4(N) and the feature vector F1(N) is greater than the preset Euclidean distance D, the electronic device 100 determines that the feature vector F4(N) of the specified target 3 (person) fails to match the feature vector F1(N) of the tracking target.

[0204] By analogy, if there are no features matching the tracking target in the consecutive P frames of images (such as the (N + P)-th frame of image) after the N-th frame of image by the electronic device 100 according to the above method, the target tracking fails.

[0205] After the tracking of the tracking target fails, the electronic device 100 stops tracking the tracking target, and the electronic device 100 will display the (N + P)-th frame of original image obtained by the electronic device 100 on the user interface 40. Specifically, reference can be made to Figure 18 the embodiments shown, which will not be elaborated here.

[0206] Based on the above embodiments, a target tracking method provided by an embodiment of the present application is introduced. As Figure 22 shown, the method includes:

[0207] S2201. The electronic device 100 displays a first user interface, and the first user interface displays the N-th frame of image, and the N-th frame of image includes one or more specified targets.

[0208] The Nth frame image in the first user interface displayed by the electronic device 100 can be displayed in real time when the electronic device 100 is recording a video or taking a photo. In one possible implementation, the electronic device 100 can be the camera application being opened to record a video or take a photo. In another possible implementation, the electronic device 100 can also be other applications with a photo-taking function (such as WeChat, Weibo, etc.) being opened to record a video or take a photo. This application does not make any limitations here.

[0209] In some other embodiments, the image frames displayed on the user interface of the electronic device 100 can also be sent to the electronic device 100 by other devices (such as drones, surveillance cameras, etc.).

[0210] It can be understood that after the electronic device displays the first user interface, the electronic device can enable the target tracking function. Specifically, please refer to Figures 6 - 9 the embodiments shown, and this application will not repeat them here.

[0211] S2202. The electronic device 100 determines a tracking target among one or more specified targets.

[0212] When the image captured in real time by the camera includes a specified target that matches the preset target feature template, the electronic device displays marking information around the specified target in the image captured in real time by the camera. The marking information can be a target box or a category.

[0213] There can be multiple ways for the electronic device 100 to determine a tracking target among one or more specified targets.

[0214] For example, in one possible implementation, the electronic device can detect multiple specified targets (such as a house, a person, etc.) in the Nth frame image. The electronic device 100 can display a target box to enclose the targets automatically recognized by the electronic device 100. The user can click on the target box (the second input operation) to select the target object to be tracked. In response to the user operation, the electronic device 100 can determine the specified target within the target box as the tracking target and display a tracking box.

[0215] Specifically, reference can be made to Figures 10 - 12 the process of the electronic device 100 determining the tracking target described therein, and this application will not repeat it here.

[0216] In one possible implementation, when the electronic device 100 does not display the target boxes of multiple specified targets, the electronic device 100 can receive a user's circle-drawing operation to enclose the target to be tracked. In response to the user operation, the electronic device 100 determines that the specified target within the indication box is the tracking target.

[0217] The above-mentioned target box and tracking box can be rectangular boxes, square boxes, diamond boxes, etc. The shapes of the target box and the tracking box are not limited herein.

[0218] S2203. The electronic device 100 extracts features of the tracking target and saves the features of the tracking target.

[0219] Specifically, the electronic device 100 can extract features of the tracking target image through a feature extraction algorithm, obtain and save the features of the tracking target image (such as texture features, contour features, etc.). Specifically, the electronic device 100 can use the feature vector obtained by extracting features from the tracking target image as the feature of the tracking target corresponding to the tracking target image. The electronic device 100 can save the features of the tracking target into a tracking template, and the tracking template can be used to represent the features of the tracking target.

[0220] It can be understood that there can be multiple tracking targets in the embodiments of the present application. When there are multiple tracking targets, the electronic device 100 can extract and save the tracking target features for each tracking target image according to the steps of S2502 - S2503 respectively. The electronic device 100 can save the tracking target images corresponding to multiple tracking targets and the tracking target features corresponding to the tracking target images.

[0221] Specifically, the electronic device 100 can determine two tracking targets, namely tracking target 1 and tracking target 2, from one or more specified targets in the Nth frame image. The electronic device extracts the features of tracking target 1 and tracking target 2 and saves the features of tracking target 1 and tracking target 2.

[0222] The electronic device 100 can determine tracking target 1 and tracking target 2 in any of the following ways.

[0223] In a possible implementation manner, the electronic device 100 can detect multiple specified targets (such as houses, people, etc.) in the Nth frame image. The electronic device 100 can display target boxes to enclose the targets automatically recognized by the electronic device 100. The user can click on target box 1 to select the target object to be tracked (such as tracking target 1) and click on target box 2 to select the target object to be tracked (such as tracking target 2). In response to the user operation, the electronic device 100 can determine the specified target within the target box as the tracking target, that is, determine tracking target 1 and tracking target 2.

[0224] In another possible implementation, when the electronic device 100 does not display the target boxes of multiple specified targets, the electronic device 100 can receive the user's circle-drawing operation to circle the target to be tracked. In response to the user operation, the electronic device 100 determines that the specified target in the indication box is the tracking target. Exemplarily, the electronic device 100 can receive the user's circle-drawing operation to circle the tracking target 1 and the tracking target 2. The electronic device 100 determines that the tracking target 1 and the tracking target 2 are the tracking targets.

[0225] In a possible implementation, the electronic device can track the tracking target according to a preset target feature template without extracting the features of the tracking target in consecutive image frames.

[0226] Optionally, the electronic device 100 can update the features of the tracking target corresponding to the tracking target saved by the electronic device 100 every preset number of frames.

[0227] For example, the preset number of frames can be 10, and the electronic device 100 updates the features corresponding to the tracking target saved by the electronic device 100 every 10 frames. The preset number of frames can be configured by the system of the electronic device 100 or can be custom-set. The preset number of frames can be 2, the preset number of frames can be 5, and the preset number of frames can also be 10. This application does not make any limitation here.

[0228] The electronic device 100 can use any one of the following two methods to update the features of the tracking target corresponding to the tracking target saved by the electronic device 100.

[0229] Method 1: The feature vector of the tracking target selected by the electronic device 100 in the Nth frame image is F1. When the electronic device 100 determines that the specified target 1 in the (N + 1)th frame image is the tracking target selected by the user in the Nth frame image, the electronic device 100 extracts the feature of the specified target 1 as the feature vector F2. The electronic device 100 clears the feature vector F1 of the tracking target and saves the feature vector F2 of the specified target 1, and uses the feature vector F2 of the specified target 1 as the feature of the tracking target.

[0230] Method 2: The feature vector of the tracking target selected by the electronic device 100 in the Nth frame image is F1. When the electronic device 100 determines that the specified target 1 in the (N + 1)th frame image is the tracking target selected by the user in the Nth frame image, the electronic device 100 extracts the feature of the specified target 1 as the feature vector F2. The electronic device 100 performs a weighted operation on the feature vector F1 of the tracking target and the feature vector F2 of the specified target 1 to obtain the feature vector F3, and the electronic device 100 uses the feature vector F3 as the feature of the tracking target.

[0231] It should be noted that this application can also use other methods to update the features of the tracking target, and this application does not make any limitation here.

[0232] S2204. The second user interface displayed by the electronic device 100, where the (N + 1)-th frame image is displayed in the second user interface, and there is one or more specified targets in the (N + 1)-th frame image.

[0233] S2205. With the center point of the tracking box of the tracking target in the N-th frame image as the center and M times the size of the tracking box of the tracking target in the N-th frame image as the search area, if a feature matching the feature of the tracking target is matched within the search area, the electronic device 100 marks the position of the tracking target in the (N + 1)-th frame image.

[0234] The electronic device determines a ninth area (search area) with a similarity greater than a second value to the target feature template from the images captured by the camera in real time based on a preset target feature template; the electronic device crops a fourth cropped image from the images captured by the camera in real time based on the ninth area, and the fourth cropped image includes the ninth area. The electronic device displays a sixth shooting interface, and the sixth shooting interface displays the first cropped image.

[0235] Optionally, the electronic device 100 can also identify the type of the tracking target in the (N + 1)-th image frame and display the type corresponding to the tracking target at or around the position of the tracking target in the (N + 1)-th image frame. Of course, the electronic device 100 may also not display the type corresponding to the tracking target at or around the position of the tracking target, and this application does not make a limitation here.

[0236] The electronic device determines a ninth area (search area) with a similarity greater than a second value to the target feature template from the images captured by the camera in real time based on a preset target feature template, specifically including:

[0237] The electronic device 100 takes the center point of the target box of the tracking target in the N-th frame image as the center and M times the size of the target box of the tracking target in the N-th frame image as the search area (ninth area), performs a convolution operation on each pixel point within the search area in the (N + 1)-th frame image, and obtains the response value of each pixel point within the search area in the (N + 1)-th frame image.

[0238] If the maximum response value corresponding to the pixel point is greater than a preset response value, there is a tracking target in the (N + 1)-th frame image, and the electronic device 100 takes the pixel point corresponding to the maximum response value as the center point of the tracking target in the (N + 1)-th frame image and displays a tracking box.

[0239] Exemplarily, if the coordinates of the center point of the tracking box in the N-th frame image are (x, y) and the size of the tracking box is P * Q, then the coordinates of the center point of the search area in the (N + 1)-th frame image are (x, y), and the size of the search area is (P * M) * (Q * M).

[0240] The electronic device 100 performs a convolution operation on the feature of the tracking target and each pixel in the search area of the (N + 1)-th frame image to obtain the response value of each pixel. The response value represents the probability that each pixel point in the search area of the (N + 1)-th frame image is the center point of the tracking target.

[0241] When the maximum response value corresponding to the pixel point in the search area of the (N + 1)-th frame image is greater than the preset response value, it indicates that there is a tracking target in the search area and the target tracking is successful. The pixel point corresponding to the maximum response value is used as the center point of the tracking target in the (N + 1)-th frame image to determine the position of the tracking target. After that, the electronic device displays the tracking frame of the tracking target in the (N + 1)-th frame image according to the center point of the tracking target in the (N + 1)-th frame image.

[0242] It can be understood that the size of the tracking frame of the tracking target in the (N + 1)-th frame image can be different from or the same as that of the tracking frame of the tracking target in the N-th frame image, and this application does not make any limitation here.

[0243] When the electronic device 100 tracks the tracking target in the (N + 1)-th frame image and determines the center point of the tracking target in the (N + 1)-th frame image, the electronic device 100 crops the obtained (N + 1)-th frame image with a cropping frame of a preset size centered on the center point of the tracking target to obtain a display image. Generally, the preset size of the cropping frame is larger than the size of the tracking frame. Exemplarily, the preset size of the cropping frame can be half or three-quarters of the size of the original image obtained by the electronic device 100, and this application does not make any limitation on the preset size of the cropping frame.

[0244] After that, the electronic device 100 displays the display image on the user interface. Specifically, please refer to Figures 15 - 16 the embodiments shown, and this application will not elaborate here.

[0245] If the response values corresponding to the pixel points in the search area of the electronic device 100 in consecutive P frame images are all less than the preset value according to the above method, the target tracking fails, where P is a positive integer greater than or equal to 1.

[0246] Exemplarily, if the coordinates of the center point of the tracking frame in the N-th frame image are (x, y) and the size of the tracking frame is P * Q, then the coordinates of the center point of the search area of the (N + 1)-th frame image are (x, y), and the size of the search area is (P * M) * (Q * M).

[0247] The electronic device 100 performs a convolution operation on the feature of the tracking target and each pixel in the search area of the (N + 1)-th frame image to obtain the response value of each pixel. The response value represents the probability that each pixel point in the search area of the (N + 1)-th frame image is the center point of the final tracking target.

[0248] If the maximum response value corresponding to the pixel points within the search area in the (N + 1)-th frame image is less than the preset response value, it indicates that there is no tracked target within the search area.

[0249] When the electronic device 100 performs tracking in the consecutive P frame images after the N-th frame image according to the above method, and the maximum response value corresponding to the pixel points within the search area in the consecutive P frame images is less than the preset response value, then the tracking of the tracked target fails, where P is a positive integer greater than or equal to 1.

[0250] After the tracking of the tracked target fails, the electronic device 100 will stop tracking the tracked target, and the electronic device 100 will display the (N + P)-th original image obtained by the electronic device 100 on the user interface. Specifically, please refer to Figures 17 - 18 the embodiments shown, and details are not described herein again in this application.

[0251] After the electronic device stops tracking the tracked target, when the user wants to track a specified target in an image frame again, the specified target in the image frame can receive the user's click operation. In response to the user's click operation, when the position coordinates of the user's click are within the position coordinates of the specified target, the electronic device 100 will determine the specified target clicked by the user as the tracked target, and extract the features of the specified target clicked by the user as the features of the tracked target. The electronic device 100 will track the tracked target in the consecutive image frames after the (N + P)-th frame image. Specifically, please refer to Figures 12 - 16 the embodiments shown, and details are not described herein again in this application.

[0252] In a possible implementation manner, after the electronic device 100 turns on the tracking mode, the electronic device 100 can receive the user's active trigger operation to stop tracking the tracked target.

[0253] When the electronic device 100 turns on the tracking mode and tracks the tracked target in the (N + 1)-th frame image, the user interface can display an exit tracking mode control. The exit tracking mode control can receive the user's click operation to make the electronic device 100 stop tracking the tracked target. In this way, the user can choose to exit the tracking mode according to their own needs, improving the user experience.

[0254] It can be understood that the electronic device 100 can also adopt other ways to end the tracking of the tracked target, not limited to the user clicking the exit tracking mode control to end the tracking of the tracked target, and this application does not make any limitations herein.

[0255] The electronic device 100 can also adopt other ways to determine whether there is a tracked target in the (N + 1)-th frame image.

[0256] Specifically, in consecutive image frames after the Nth frame image, such as the (N + 1)th frame image, the electronic device 100 can automatically identify one or more specified targets in the image frame and display target boxes for one or more specified targets. The electronic device 100 extracts the features of the specified targets in one or more target boxes and matches the features of the one or more specified targets with the features of the tracking target in the tracking template. If the features of one specified target match the features of the tracking target, the electronic device 100 succeeds in target tracking and gives the tracking box for this specified target.

[0257] If in consecutive P frame images (such as the (N + P)th frame image) after the Nth frame image of the electronic device 100, according to the above method, there are no features matching the features of the tracking target in each frame image, then the target tracking fails.

[0258] Specifically, please refer to Figures 20 - 21 the embodiments shown, and the present application will not repeat them here.

[0259] Embodiment 2

[0260] Since there is no pre-set recognition algorithm or model for non-specified targets in the electronic device 100, the electronic device 100 cannot automatically identify non-specified targets in the image frame. When the user needs to track a non-specified target in the image frame displayed by the electronic device 100, the user needs to manually circle the area of the non-specified target image. After that, the electronic device 100 extracts the features of the non-specified target image and saves them. Then, in subsequent consecutive image frames, the electronic device 100 uses the center of the previous frame image as the search center and an M-fold of the size of its tracking box as the search area to track the tracking target. If the electronic device 100 matches a feature in the search area that matches the feature of the tracking target, the electronic device 100 marks the position of the tracking target in this frame image and focuses on the tracking target during shooting to make the captured tracking target clearer. The electronic device 100 uses this method to track the tracking target. If the electronic device 100 does not match a feature in the search area that matches the feature of the tracking target in consecutive P frame images, then the target tracking fails, and the electronic device 100 stops tracking the tracking target.

[0261] In some embodiments, the image frame displayed on the user interface of the electronic device 100 can be the one displayed when the electronic device 100 is recording a video or taking a photo. In one possible implementation, the electronic device 100 can be recording a video or taking a photo with the camera application program enabled. In another possible implementation, the electronic device 100 can also be recording a video or taking a photo with other applications having a photo-taking function (such as WeChat, Weibo) enabled. The present application does not make any limitations here.

[0262] In some other embodiments, the image frames displayed on the user interface of the electronic device 100 can also be sent to the electronic device 100 by other devices (such as drones, surveillance cameras, etc.).

[0263] It can be understood that if the image frames displayed on the user interface of the electronic device 100 are acquired by other devices (such as drones, surveillance cameras, etc.) and sent to the electronic device 100. Before the electronic device 100 displays the image frames acquired by other devices, the electronic device 100 establishes a communication connection with other devices. In one possible implementation, the electronic device 100 can establish a communication connection with other devices through an application. In another possible implementation, the electronic device 100 can establish a communication connection with other devices through Bluetooth, Wireless Local Area Network (WLAN), etc.

[0264] In the following embodiments of the present application, the case where the electronic device 100 is tracking a certain non-specified target while recording a video is described.

[0265] Please refer to Figures 23 - 33 , Figures 23 - 33 which exemplarily shows a schematic diagram of the interface of the process of the electronic device 100 tracking a certain non-specified target.

[0266] Specifically, for how the electronic device 100 enables the target tracking function, please refer to the embodiments shown in Figures 6 - 10 and details are not elaborated herein in the present application.

[0267] Figure 23 Exemplarily shown is a schematic diagram of the Nth frame of image 111 acquired by the electronic device 100 after enabling the target tracking function. The Nth frame of image 111 is the original image acquired by the electronic device 100.

[0268] The Nth frame of image 111 exemplarily shows the category of the specified target automatically recognized by the electronic device 100 and the position coordinates of the specified target. The Nth frame of image 111 exemplarily shows the specified target 1 (house), the specified target 2 (person), and the non-specified target (animal). The Nth frame of image 111 may also include more other specified targets and / or non-specified targets, which are not limited herein in the present application. The Nth frame of image 111 may also include target frames for each specified target, and the target frames are used to enclose the target object and represent the position coordinates of the target object. Exemplarily, the target frame 1101 is used to enclose the specified target 1 (house), and the target frame 1102 is used to enclose the specified target 2 (person).

[0269] In some embodiments, the electronic device 100 may not display the target frame. That is, in Figure 23 In this case, the electronic device 100 does not display the target frame 1101 and the target frame 1102. No specific limitation is made here.

[0270] As Figure 23 shown, when the user wants to track a non-specified target (animal), the electronic device 100 can receive a click operation by the user on the image of the non-specified target (animal). In response to the user's click operation, the electronic device 100 will display a first target frame in the user's click area, and the first target frame is used to enclose the user's click position.

[0271] As Figure 24 shown, Figure 24 An exemplary schematic diagram of the first target frame 2501 is shown. Exemplarily, when the user clicks on the head of the non-specified target (animal), the electronic device 100 will display the first target frame 2501 on the head of the non-specified target (animal).

[0272] Of course, the user can also click on other parts of the non-specified target (animal), and no limitation is made here in this application.

[0273] In response to the user's click operation, the electronic device 100 will extract the features of the image of the non-specified target (animal) within the first target frame 2501 and save the features of the image.

[0274] Figure 25 An exemplary illustration shows the tracking target selected by the user and the features corresponding to the tracking target. As Figure 25 shown, the electronic device 100 extracts the features of the image of the non-specified target (animal) within the first target frame 2501 in response to the user's click operation, and saves the features of the image of the non-specified target (animal) within the first target frame 2501. Feature extraction can be represented by using a feature vector F5(N) for the features of the image of the non-specified target (animal) within the first target frame 2501 in the Nth frame image. For example, F5(N) = [x1, x2, x3, …, xn]. The feature vector F5(N) can represent the texture features, contour features, etc. of the image of the non-specified target (animal) within the first target frame 2501. Here, the specific form and size of the feature vector F5(N) of the image of the non-specified target (animal) within the first target frame 2501 are not limited. For example, F5(N) can be a feature vector [0.5, 0.6, 0.8, …, 0.9, 0.7, 0.3] containing n values. Among them, n is an integer, which can be 128, 256, 512, etc., and the size of n is not limited. After the electronic device 100 extracts the features of the image of the non-specified target (animal) within the first target frame 2501, it saves the features of the image of the non-specified target (animal) within the first target frame 2501 to the tracking template of the electronic device 100. The tracking template can be used to represent multiple features of the tracking target.

[0275] After that, the size of the first target box 2501 will gradually increase from the preset minimum value until it increases to enclose the complete image of the non-specified target (animal), and then the size of the first target box 2501 will no longer change.

[0276] As Figure 26 shown, for example, when the electronic device 100 displays the (N + 1)-th frame image 171, the size of the first target box 2501 expands to the second target box 2701, and the electronic device 100 will extract the features of the image of the non-specified target (animal) within the second target box 2701 and save the features of the image.

[0277] Figure 27 Exemplarily shown are the tracking target selected by the user and the features corresponding to the tracking target. As Figure 27 shown, the electronic device 100 extracts the features of the image of the non-specified target (animal) within the second target box 2701 and saves the features of the image of the non-specified target (animal) within the second target box 2701. It can be understood that the second target box 2701 is larger than the first target box 2501, the image of the non-specified target (animal) within the second target box 2701 is more complete than the image of the non-specified target (animal) within the first target box 2501, and the image of the non-specified target (animal) within the second target box 2701 includes the image of the non-specified target (animal) within the first target box 2501. Feature extraction may be represented by the feature vector F6(N) for the features of the image of the non-specified target (animal) within the second target box 2701 in the (N + 1)-th frame image. For example, F6(N) = [x1, x2, x3, …, xn]. The feature vector F6(N) may represent the texture features, contour features, etc. of the image of the non-specified target (animal) within the second target box 2701. Here, the specific form and size of the feature vector F6(N) for the image of the non-specified target (animal) within the second target box 2701 are not limited. For example, F6(N) may be a feature vector [0.6, 0.8, 0.9, …, 0.9, 0.7, 0.3] containing n values. Among them, n is an integer, which may be 128, 256, 512, etc., and the size of n is not limited.

[0278] After the electronic device 100 extracts the features of the image of the non-specified target (animal) within the second target box 2701, in one possible implementation, the electronic device 100 clears the features of the image of the non-specified target (animal) within the first target box 2501 and saves the features of the image of the non-specified target (animal) within the second target box 2701; in another possible implementation, the electronic device 100 performs a weighted operation on the features of the image of the non-specified target (animal) within the first target box 2501 and the features of the image of the non-specified target (animal) within the second target box 2701, and uses the result as the features of the image of the non-specified target (animal) within the second target box 2701.

[0279] When the second target box 2701 has not completely enclosed the image of the non-specified target (animal), the size of the second target box 2702 continues to increase until it completely encloses the image of the non-specified target (animal).

[0280] As Figure 28 shown, for example, when the electronic device 100 displays the (N + 2)-th frame image 2901, the size of the second target box 2702 expands to the third target box 2901, and the electronic device 100 extracts the features of the image of the non-specified target (animal) within the third target box 2901 and saves the features of the image.

[0281] Figure 29 Exemplarily shows the tracking target selected by the user and the features corresponding to the tracking target. As Figure 29 As shown, the electronic device 100 extracts features of the image of the non-specified target (animal) within the third target box 2901 and saves the features of the image of the non-specified target (animal) within the third target box 2901. It can be understood that the third target box 2901 is larger than the second target box 2701, the image of the non-specified target (animal) within the third target box 2901 is more complete than the image of the non-specified target (animal) within the second target box 2701, the image of the non-specified target (animal) within the third target box 2901 includes the image of the non-specified target (animal) within the second target box 2701, and the image of the non-specified target (animal) within the third target box 2901 can be a complete image including the non-specified target (animal). Feature extraction can represent the features of the image of the non-specified target (animal) within the third target box 2901 in the (N + 2)-th frame image with the feature vector F7(N). For example, F7(N) = [x1, x2, x3, …, xn]. The feature vector F7(N) can represent the texture features, contour features, etc. of the image of the non-specified target (animal) within the third target box 2901. Here, the specific form and size of the feature vector F7(N) of the image of the non-specified target (animal) within the third target box 2901 are not limited. For example, F7(N) can be a feature vector [0.7, 0.9, 0.8, …, 0.9, 0.7, 0.3] containing n values. Among them, n is an integer, which can be 128, 256, 512, etc., and the size of n is not limited.

[0282] After the electronic device 100 extracts the features of the image of the non-specified target (animal) within the third target box 2901, in one possible implementation, the electronic device 100 clears the features of the image of the non-specified target (animal) within the second target box 2701 and saves the features of the image of the non-specified target (animal) within the third target box 2901; in another possible implementation, the electronic device 100 performs a weighted operation on the features of the image of the non-specified target (animal) within the second target box 2701 and the features of the image of the non-specified target (animal) within the third target box 2901, and uses the result as the features of the image of the non-specified target (animal) within the third target box 2901.

[0283] In some embodiments, the electronic device 100 may not display the second target box 2701 and only display the first target box 2501 and the third target box 2901. This application does not make any limitations here.

[0284] It should be noted that the size of the first target box 2501 can be directly enlarged to the third target box 2901 to completely enclose the image of the non-specified target (animal). The first target box 2501 can also be enlarged to the third target box 2901 after passing through multiple target boxes to completely enclose the image of the non-specified target (animal). The present application does not limit the number of multiple target boxes, and the number of multiple target boxes can be one (such as the second target box 2701), or two, three, etc.

[0285] As Figure 30 shown, after the electronic device 100 displays the third target box 2901, the electronic device 100 detects that the third target box 2901 has completely enclosed the image of the non-specified target (animal). The electronic device 100 can receive the operation of the user clicking on the third target box 2901, and in response to the user's click operation, the third target box 2901 will no longer change.

[0286] As Figure 31 shown, in response to the user's click operation, the target box of the non-specified target (animal) will no longer change, and the electronic device 10 will display the tracking box 3301 of the non-specified target (animal) in the (N + 2)-th frame image.

[0287] In a possible implementation manner, after the electronic device 100 displays the third target box 2901, the electronic device 100 may also not receive the operation of the user clicking on the third target box 2901. After the electronic device 100 uses an algorithm or model (such as an edge algorithm) to detect that the third target box 2901 has completely enclosed the image of the non-specified target (animal), the third target box 2901 will no longer change, and the tracking box 3301 of the non-specified target (animal) will be displayed.

[0288] After that, the electronic device 100 will track the non-specified target (animal) in consecutive video frames subsequent to the (N + 2)-th frame image. The method by which the electronic device 100 tracks the non-specified target is the same as the method by which the electronic device 100 tracks the specified target in Embodiment 1. Specifically, please refer to Embodiment 1, and the present application will not elaborate herein.

[0289] Figures 23 - 31 Exemplarily shows how the electronic device 100 identifies the non-specified target (animal) and extracts the features of the non-specified target (animal). The following introduces several other methods by which the electronic device 100 identifies the non-specified target (animal) and extracts the features of the non-specified target (animal).

[0290] Method 1;

[0291] As Figure 32 shown, Figure 32 Exemplarily shown is a schematic diagram of the Nth frame image 111 obtained by the electronic device 100 after enabling the target tracking function. The Nth frame image 111 is the original image obtained by the electronic device 100.

[0292] As Figure 32 shown, when the user wants to track a non-specified target (animal), the electronic device 100 can receive the user's manual operation of drawing a circle to completely enclose the image of the non-specified target (animal). After the user finishes drawing the circle, the electronic device 100 will display a target box 3301 as Figure 31 shown.

[0293] The electronic device 100 will extract the features of the image of the non-specified target (animal) within the target box 3301 and save the features of the non-specified target (animal) to the tracking template.

[0294] After that, the electronic device 100 will track the non-specified target (animal) in the consecutive video frames subsequent to the Nth frame image. The method by which the electronic device 100 tracks the non-specified target is the same as the method by which the electronic device 100 tracks the specified target in the first embodiment. Specifically, please refer to Figures 17 - 18 the embodiment shown, and details are not described herein again in this application.

[0295] Method 2;

[0296] When the user wants to track a non-specified target (animal), the electronic device 100 can receive a click operation of the user on the image of the non-specified target (animal). In response to the user's click operation, the electronic device 100 will display a first target box in the user's click area, and the first target box is used to enclose the user's click position.

[0297] Please refer to Figure 24 , Figure 24 which exemplarily shows a schematic diagram of the first target box 2501. Exemplarily, when the user clicks on the head of the non-specified target (animal), the electronic device 100 will display the first target box 2501 on the head of the non-specified target (animal). In response to the user's click operation, the electronic device 100 will extract the features of the image of the non-specified target (animal) within the first target box 2501 and save the features of the image.

[0298] As Figure 33 shown, the electronic device 100 can receive a right boundary rightward dragging operation of the user on the first target box 2501. In response to the user's dragging operation, the first target box 2501 gradually enlarges as the user operates on the right boundary of the first target box 2501 to drag it rightward. When the first target box 2501 enlarges to completely enclose the non-specified target (animal), the user releases the finger, and the first target box 2501 will no longer change.

[0299] Please refer to Figure 30 , after the first target box 2501 becomes large enough to completely enclose the non-designated target (animal), the electronic device 100 displays a tracking box 3101 for the non-designated target (animal). Moreover, the electronic device 100 extracts features from the image of the non-designated target (animal) within the tracking box 3101. After the electronic device 100 extracts the features of the image of the non-designated target (animal) within the tracking box 3101, in one possible implementation, the electronic device 100 clears the features of the image of the non-designated target (animal) within the first target box 2501 and saves the features of the image of the non-designated target (animal) within the tracking box 3101; in another possible implementation, the electronic device 100 performs a weighted operation on the features of the image of the non-designated target (animal) within the first target box 2501 and the features of the image of the non-designated target (animal) within the tracking box 3101, and uses the result as the features of the image of the non-designated target (animal) within the tracking box 3101.

[0300] Based on the above embodiments, another target tracking method provided by the second embodiment of the present application is introduced. As Figure 34 shown, this method includes:

[0301] S3401. The electronic device 100 displays a first user interface, and a Nth frame image is displayed in the first user interface, and one or more non-designated targets are included in the Nth frame image.

[0302] The Nth frame image in the first user interface displayed by the electronic device 100 may be the image displayed when the electronic device 100 is recording a video or taking a photo. In one possible implementation, the electronic device 100 may have the camera application program opened to record a video or take a photo. In another possible implementation, the electronic device 100 may also have other application programs with a photo-taking function (such as WeChat, Weibo, etc.) opened to record a video or take a photo, and the present application does not make a limitation here.

[0303] In some other embodiments, the image frame displayed on the user interface of the electronic device 100 may also be sent to the electronic device 100 by other devices (such as a drone, a surveillance camera, etc.).

[0304] It can be understood that after the electronic device displays the first user interface, the electronic device may turn on the target tracking function. Specifically, please refer to the embodiments shown in Figures 6 - 10 , and the present application will not elaborate again.

[0305] S3402. The electronic device 100 determines a tracking target from one or more non-designated targets. The electronic device 100 includes a first target box, and the image enclosed by the first target box is the first partial image of the non-designated target.

[0306] The electronic device receives and responds to a click operation within a sixth region on an image captured in real time by the camera, and displays a first target frame on the sixth region of the image captured in real time by the camera; after the electronic device displays the first target frame on the sixth region of the image captured in real time by the camera, the electronic device acquires the image captured in real time by the camera and displays a second target frame on the image captured in real time by the camera; the image content within the second target frame includes the image content within the first target frame; the electronic device receives and responds to a click operation on the second target frame on the image captured in real time by the camera, and the region of the second target frame on the image captured in real time by the camera is a first region; the first region includes the sixth region. In this way, for non-designated targets, since the electronic device cannot automatically identify non-designated targets in the image, the electronic device can receive the position of the non-designated target manually confirmed by the user in the image frame, that is, the electronic device receives and responds to a click operation within the sixth region on the image captured in real time by the camera, and displays a first target frame on the sixth region of the image captured in real time by the camera. The first target frame is used to enclose the position coordinates of the user's click. Subsequently, the first target frame gradually enlarges until it completely encloses the image of the non-designated target, that is, the electronic device acquires the image captured in real time by the camera and displays a second target frame on the image captured in real time by the camera; the image content within the second target frame includes the image content within the first target frame. After the second target frame completely encloses the entire image of the non-designated target, the electronic device can receive the click operation of the user, so that the size of the second target frame no longer changes, that is, the electronic device receives and responds to a click operation on the second target frame on the image captured in real time by the camera, and the region of the second target frame on the image captured in real time by the camera is a first region; the first region includes the sixth region.

[0307] The electronic device 100 displays the Nth frame image, and there is one or more designated targets and one or more non-designated targets in the Nth frame image.

[0308] When the user tracks a non-designated target, the electronic device 100 can receive the operation of the user clicking on the non-designated target. In response to the operation of the user clicking on the non-designated target, the electronic device 100 displays a first target frame, and the first target frame includes a first part of the image of the non-designated target. It can be understood that the first part of the image of the non-designated target is a part of the non-designated target. Specifically, please refer to Figures 23 - 24 the embodiments shown, and the present application will not be elaborated herein.

[0309] S3403. The electronic device 100 extracts and saves the features of the first part of the image of the non-designated target.

[0310] Specifically, the electronic device 100 can extract features from the first partial image of the non-specified target through a feature extraction algorithm, obtain and save the features of the first partial image of the non-specified target (such as texture features, contour features, etc.). Specifically, the electronic device 100 can use a feature vector obtained by extracting features from the first partial image of the non-specified target as the feature corresponding to the first partial image of the non-specified target. The electronic device 100 can save the features of the first partial image of the non-specified target to a tracking template, and the tracking template can be used to represent one or more features of the tracking target.

[0311] Specifically, please refer to Figure 25 the embodiments shown, and details are not described herein again for this application.

[0312] S3404. The second user interface displayed by the electronic device 100, where the second user interface displays the (N + 1)-th frame image, and the (N + 1)-th frame image includes a second target box, and the image enclosed by the second target box is the second partial image of the non-specified target.

[0313] The electronic device 100 displays the (N + 1)-th frame image, and there is one or more specified targets and one or more non-specified targets in the (N + 1)-th frame image.

[0314] Since the first partial image of the non-specified target included within the first target box is a part of the complete image of the non-specified target, the first target box will gradually expand until it encloses the entire complete image of the non-specified target.

[0315] The size of the second target box is larger than the size of the first target box, and the image of the non-specified target enclosed by the second target box is more than the image of the non-specified target enclosed by the first target box.

[0316] The second partial image of the non-specified target includes the first partial image of the non-specified target, and the second partial image of the non-specified target is a part of the complete image of the non-specified target.

[0317] Optionally, the first target box can automatically extend to the second target box, or the electronic device can also receive a sliding operation on the first target box by the user to expand the first target box to the second target box, and this application does not make a limitation here.

[0318] S3405. The electronic device 100 extracts and saves features from the second partial image of the non-specified target.

[0319] Specifically, the electronic device 100 can extract features from the second partial image of the non-specified target through a feature extraction algorithm, obtain and save the features of the second partial image of the non-specified target (such as texture features, contour features, etc.). Specifically, the electronic device 100 can use a feature vector obtained by extracting features from the second partial image of the non-specified target as the feature corresponding to the second partial image of the non-specified target. The electronic device 100 can save the features of the second partial image of the non-specified target into a tracking template, and the tracking template can be used to represent one or more features of the tracking target.

[0320] Specifically, please refer to Figures 26 - 27 the embodiments shown, and details are not described herein again in this application.

[0321] Optionally, the electronic device 100 can update the features of the non-specified target in any of the following ways:

[0322] Method 1: The electronic device 100 clears the features of the first partial image of the non-specified target and saves the features of the second partial image of the non-specified target as the features of the second partial image of the non-specified target.

[0323] Method 2: The electronic device 100 performs a weighted operation on the features of the first partial image of the non-specified target and the features of the second partial image of the non-specified target as the features of the second partial image of the non-specified target.

[0324] S3406. The electronic device 100 displays a third user interface, and the third user interface displays the (N + 2)-th frame image, and the (N + 2)-th frame image includes a third target box, and the image enclosed by the third target box is the complete image of the non-specified target.

[0325] The electronic device displays a first shooting interface, and the first shooting interface displays the image captured by the camera in real time;

[0326] The electronic device determines whether the image captured by the camera in real time includes a specified target that matches a preset target feature template; when the image captured by the camera in real time does not include a specified target that matches a preset target feature template, the electronic device receives and responds to a first input operation on the image captured by the camera in real time, and determines the first area selected by the user on the image captured by the camera in real time.

[0327] The electronic device 100 displays the (N + 2)-th frame image, and the (N + 2)-th frame image has one or more specified targets and one or more non-specified targets.

[0328] Since the second partial image of the non-specified target included in the second target box is a part of the complete image of the non-specified target, the second target box will gradually become larger until it completely encloses the complete image of the non-specified target.

[0329] The size of the third target box is larger than that of the second target box, and the image of the non-designated target enclosed by the third target box is the complete image (the first area) of the non-designated target.

[0330] Optionally, the second target box can automatically extend to the third target box, or the electronic device can also receive a dragging operation (the first input operation) applied by the user on the second target box to expand the second target box to the third target box. This application does not make a limitation here.

[0331] After the electronic device 100 displays the third target box 2901, the electronic device 100 detects that the third target box 2901 has completely enclosed the image of the non-designated target (animal). The electronic device 100 can receive an operation in which the user clicks on the third target box 2901. In response to the user's click operation (the first input operation), the third target box 2901 no longer changes.

[0332] Alternatively, after the electronic device 100 displays the third target box 2901, the electronic device 100 may not receive an operation in which the user clicks on the third target box 2901. After the electronic device 100 uses an algorithm or model (such as an edge algorithm) to detect that the third target box 2901 has completely enclosed the image of the non-designated target (animal), the third target box 2901 no longer changes, and a tracking box 3301 of the non-designated target (animal) is displayed.

[0333] S3407. The electronic device 100 extracts and saves features of the third part of the image of the non-designated target.

[0334] The electronic device extracts the first image features in the first area on the image collected in real time by the camera.

[0335] Specifically, the electronic device 100 can extract features of the third part of the image of the non-designated target through a feature extraction algorithm, obtain and save the features (the first image features) of the third part of the image of the non-designated target (such as texture features, contour features, etc.). Specifically, the electronic device 100 can use a feature vector obtained by extracting features from the third part of the image of the non-designated target as the feature corresponding to the third part of the image of the non-designated target. The electronic device 100 can save the features of the third part of the image of the non-designated target to a tracking template, and the tracking template can be used to represent one or more features of the tracking target.

[0336] Specifically, please refer to Figures 28 - 29 the embodiments shown, and this application will not elaborate here.

[0337] Optionally, the electronic device 100 can update the features of the non-designated target in any of the following ways:

[0338] Method 1: Specifically, the electronic device extracts the second image feature in the second area of the image captured by the camera in real time; after extracting the second image feature, the electronic device determines, based on the second image feature, a third area in the image captured by the camera in real time, where the similarity to the second image feature is greater than the first value; based on the third area, the electronic device crops a second cropped image from the image captured by the camera in real time, and the second cropped image includes the third area; the electronic device displays a third shooting interface, and the third shooting interface displays the second cropped image.

[0339] That is, the electronic device 100 clears the features of the second part of the image of the non-specified target and saves the features of the third part of the image of the non-specified target as the features of the third part of the image of the non-specified target, that is, the image of the tracking target.

[0340] Method 2: The electronic device extracts the second image feature in the second area of the image captured by the camera in real time; the electronic device performs a weighted operation on the second image feature and the first image feature to obtain a third image feature; after obtaining the third image feature, the electronic device determines, based on the third image feature, a fourth area in the image captured by the camera in real time, where the similarity to the third image feature is greater than the first value; based on the fourth area, the electronic device crops a third cropped image from the image captured by the camera in real time, and the third cropped image includes the fourth area; the electronic device displays a fourth shooting interface, and the fourth shooting interface displays the third cropped image.

[0341] That is, the electronic device 100 performs a weighted operation on the features of the second part of the image of the non-specified target and the features of the third part of the image of the non-specified target as the features of the third part of the image of the non-specified target, that is, the image of the tracking target.

[0342] After the electronic device 100 displays the third target box, the third target box completely frames the non-specified target that the user needs to track. Therefore, the features of the third part of the image within the third target box can be used as the features of the tracking target. After that, the electronic device 100 will track the non-specified target.

[0343] S3403 - S3407 describe the method of automatically enlarging the size of the target box until the entire image of the non-specified target is completely framed to obtain the features of the tracking target (i.e., the non-specified target). The following introduces several other methods for the electronic device 100 to identify the non-specified target (animal) and extract the features of the non-specified target (animal).

[0344] Method 1: The electronic device receives and responds to a sliding operation on the image captured by the camera in real time, and determines the area of the sliding trajectory of the sliding operation as the first area.

[0345] When the user wants to track a non-specified target, the electronic device 100 can receive the user's operation of manually drawing a circle to completely enclose the image of the non-specified target (animal). After the user finishes drawing the circle, the electronic device 100 will display a third target frame. The image enclosed by the third target frame is the complete image of the non-specified target. The electronic device 100 will extract and save the features of the image enclosed by the third target frame as the features of the tracking target.

[0346] Method 2: When the user wants to track a non-specified target, the electronic device 100 can receive the user's operation of clicking on the non-specified target. In response to the user's operation of clicking on the non-specified target, the electronic device 100 displays a first target frame, and the first target frame includes a first partial image of the non-specified target. It can be understood that the first partial image of the non-specified target is a part of the non-specified target. The electronic device 100 extracts and saves the features of the first partial image of the non-specified target.

[0347] After that, the electronic device 100 receives the user's dragging operation on the first target frame, so that the size of the first target frame is enlarged to a third target frame. The third partial image of the non-specified target enclosed by the third target frame is the complete image of the non-specified target. The electronic device 100 will extract the features of the third partial image enclosed by the third target frame. After the electronic device 100 extracts the features of the third partial image enclosed by the third target frame, the method of updating the features of the tracking target can be any one of the following;

[0348] Method 1: The electronic device 100 clears the features of the first partial image of the non-specified target and saves the features of the third partial image of the non-specified target as the features of the third partial image of the non-specified target, that is, the image of the tracking target.

[0349] Method 2: The electronic device 100 performs a weighted operation on the features of the first partial image of the non-specified target and the features of the third partial image of the non-specified target as the features of the third partial image of the non-specified target, that is, the image of the tracking target.

[0350] After the electronic device 100 displays the third target frame, the third target frame completely encloses the non-specified target that the user needs to track. Therefore, the features of the third partial image within the third target frame can be used as the features of the tracking target. After that, the electronic device 100 will track the non-specified target.

[0351] S3408. The electronic device 100 displays a fourth user interface, and a (N + 3)-th frame image is displayed in the fourth user interface, and there is one or more non-specified targets in the (N + 3)-th frame image.

[0352] S3409. The electronic device 100 takes the center point of the third target box in the (N + 2)-th frame image as the center and M times the third target box in the (N + 2)-th frame image as the search area. If a feature matching the feature of the tracking target is found within the search area, the electronic device 100 marks the position of the tracking target.

[0353] After extracting the first image feature, the electronic device determines, based on the first image feature, a second area in the images captured by the camera in real time, where the similarity between the second area and the first image feature is greater than a first value. Based on the second area, the electronic device crops a first cropped image from the images captured by the camera in real time, and the first cropped image includes the second area. The electronic device displays a second shooting interface, and the second shooting interface displays the first cropped image.

[0354] Optionally, the electronic device 100 can also identify the types of one or more specified targets in the (N + 1)-th image frame and display the types corresponding to the respective specified targets at or around the positions of the specified targets. Of course, the electronic device 100 may also not display the types corresponding to the respective specified targets at or around the positions of the specified targets, and this application does not make any limitation here.

[0355] In a possible implementation, the electronic device 100 takes the center point of the third target box in the (N + 2)-th frame image as the center and M times the third target box in the (N + 2)-th frame image as the search area (the second area), and performs a convolution operation on each pixel point within the search area in the (N + 3)-th frame image to obtain the response value of each pixel point within the search area in the (N + 3)-th frame image. If the maximum response value corresponding to a pixel point is greater than a preset response value, there is a tracking target in the (N + 3)-th frame image. The electronic device 100 takes the pixel point corresponding to the maximum response value as the center point of the tracking target in the (N + 3)-th frame image and displays a tracking box.

[0356] Exemplarily, if the coordinates of the center point of the third target box in the (N + 2)-th frame image are (x, y) and the size of the third target box is P * Q, then the coordinates of the center point of the search area in the (N + 3)-th frame image are (x, y), and the size of the search area is (P * M) * (Q * M).

[0357] The electronic device 100 performs a convolution operation on the feature of the tracking target and each pixel within the search area in the (N + 3)-th frame image to obtain the response value of each pixel, and the response value represents the probability that each pixel point within the search area in the (N + 3)-th frame image is the center point of the tracking target.

[0358] When the maximum response value corresponding to the pixel points within the search area in the (N + 3)-th frame image is greater than the preset response value, it indicates that there is a tracking target within the search area and the target tracking is successful. Taking the pixel point corresponding to the maximum response value as the center point of the tracking target in the (N + 3)-th frame image to determine the position of the tracking target. After that, the electronic device 100 displays a tracking frame of the tracking target in the (N + 3)-th frame image based on the center point of the tracking target in the (N + 3)-th frame image.

[0359] In some embodiments, if in consecutive P frame images of the electronic device 100, the response values corresponding to the pixel points within the search area are all less than the preset value according to the above method, then the target tracking fails, where P is a positive integer greater than or equal to 1.

[0360] Exemplarily, if the coordinates of the center point of the third target box in the (N + 2)-th frame image are (x, y) and the size of the third target box is P * Q, then the coordinates of the center point of the search area in the (N + 1)-th frame image are (x, y), and the size of the search area is (P * M) * (Q * M).

[0361] The electronic device 100 performs a convolution operation on the feature of the tracking target and each pixel within the search area in the (N + 3)-th frame image to obtain the response value of each pixel. The response value represents the probability that each pixel point within the search area in the (N + 3)-th frame image serves as the center point of the final tracking target.

[0362] When the maximum response value corresponding to the pixel points within the search area in the (N + 3)-th frame image is less than the preset response value, it indicates that there is no tracking target within the search area.

[0363] When the electronic device 100 performs tracking according to the above method in consecutive P frame images after the (N + 2)-th frame image, and the maximum response value corresponding to the pixel points within the search area in the consecutive P frame images is less than the preset response value, then the tracking of the tracking target fails, where P is a positive integer greater than or equal to 1.

[0364] That is, after the electronic device extracts the first image feature, based on the first image feature, no area with a similarity greater than the first value to the first image feature is determined from the consecutive T frame images captured by the camera in real time; the electronic device displays a seventh shooting interface, and the seventh shooting interface displays the images captured by the camera in real time.

[0365] When the electronic device 100 stops tracking the tracking target, and the user wants to track a certain non-specified target in the image frame again, the electronic device 100 can receive the area where the user manually circles the non-specified target image. In response to the user's operation, the electronic device 100 will determine the non-specified target as the tracking target, and the electronic device 100 extracts and saves the feature of the non-specified target image. The electronic device 100 will track the tracking target in the consecutive image frames after the (N + P + 2)-th frame image. Specifically, please refer to Figures 14 - 18 For the embodiments shown, the present application will not be elaborated again.

[0366] In a possible implementation manner, after the electronic device 100 turns on the tracking mode, the electronic device 100 can receive an active trigger operation from the user to stop tracking the tracking target.

[0367] The electronic device receives a second input operation from the user; in response to the second input operation, the electronic device displays a fifth shooting interface, and the fifth shooting interface displays the image captured by the camera in real time. The second input operation can also be a voice control operation, etc.

[0368] When the electronic device 100 turns on the tracking mode and tracks the tracking target in the (N + 3)-th frame image, the user interface can display an exit tracking mode control. The exit tracking mode control can receive a click operation from the user (the second input operation) to make the electronic device 100 stop tracking the tracking target. In this way, the user can choose to exit the tracking mode according to their own needs, improving the user experience.

[0369] It can be understood that the electronic device 100 can also end the tracking of the tracking target in other ways, not limited to the user clicking the exit tracking mode control to end the tracking of the tracking target. The present application does not make a limitation here.

[0370] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the various embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A target tracking method, characterized in that, the method includes: The electronic device displays a first shooting interface, and the image captured by the camera in real time is displayed on the first shooting interface; The electronic device determines whether the image captured by the camera in real time includes a specified target that matches a preset target feature template; When the image captured by the camera in real time does not include the specified target that matches the preset target feature template, the electronic device receives and responds to a first input operation on the image captured by the camera in real time, and determines a first area selected by the user on the image captured by the camera in real time; The electronic device extracts a first image feature in the first area on the image captured by the camera in real time; After extracting the first image feature, the electronic device determines, based on the first image feature, a second area in the image captured by the camera in real time whose similarity to the first image feature is greater than a first value; The electronic device crops a first cropped image from the image captured by the camera in real time based on the second area, and the first cropped image includes the second area; The electronic device displays a second shooting interface, and the second shooting interface displays the first cropped image.

2. The method according to claim 1, characterized in that, after the electronic device displays the second shooting interface and the second shooting interface displays the first cropped image, the method further includes: The electronic device extracts a second image feature in the second area on the image captured by the camera in real time; After extracting the second image feature, the electronic device determines, based on the second image feature, a third area in the image captured by the camera in real time whose similarity to the second image feature is greater than the first value; The electronic device crops a second cropped image from the image captured by the camera in real time based on the third area, and the second cropped image includes the third area; The electronic device displays a third shooting interface, and the third shooting interface displays the second cropped image.

3. The method according to claim 1, characterized in that, after the electronic device displays the second shooting interface and the second shooting interface displays the first cropped image, the method further includes: The electronic device extracts a second image feature in the second area on the image captured by the camera in real time; The electronic device performs a weighted operation on the second image feature and the first image feature to obtain a third image feature; After obtaining the third image feature, the electronic device determines, based on the third image feature, a fourth area in the image captured by the camera in real time whose similarity to the third image feature is greater than the first value; The electronic device crops a third cropped image from the image captured by the camera in real time based on the fourth area, and the third cropped image includes the fourth area; The electronic device displays a fourth shooting interface, and the fourth shooting interface displays the third cropped image.

4. The method according to claim 1, characterized in that, After the electronic device displays the second shooting interface and the second shooting interface displays the first cropped image, the method further includes: The electronic device receives a second input operation from the user; In response to the second input operation, the electronic device displays a fifth shooting interface, and the fifth shooting interface displays an image captured by the camera in real time.

5. The method according to claim 1, wherein, the method further includes: Based on the position of the first area in the image captured by the camera in real time, the electronic device determines a fifth area in the image captured by the camera in real time, and the fifth area includes the first area; The electronic device determines a second area from the image captured by the camera in real time, and the similarity between the second area and the first image feature is greater than a first value based on the first image feature. Specifically, it includes: The electronic device determines the second area from the fifth area in the image captured by the camera in real time, and the similarity between the second area and the first image feature is greater than the first value based on the first image feature.

6. The method according to claim 1, wherein, Before the electronic device receives and responds to a first input operation on the image captured by the camera in real time, the method further includes: The electronic device receives and responds to a click operation within a sixth area on the image captured by the camera in real time, and displays a first target frame on the sixth area of the image captured by the camera in real time; After the electronic device displays the first target frame within the sixth area of the image captured by the camera in real time, the electronic device acquires the image captured by the camera in real time and displays a second target frame on the image captured by the camera in real time; the image content within the second target frame includes the image content within the first target frame; The electronic device receives and responds to a first input operation on the image captured by the camera in real time, and determines a first area selected by the user on the image captured by the camera in real time. Specifically, it includes: The electronic device receives and responds to a click operation on the second target frame on the image captured by the camera in real time, and the area of the second target frame on the image captured by the camera in real time is the first area; the first area includes the sixth area.

7. The method according to claim 1, wherein, The electronic device receives and responds to a first input operation on the image captured by the camera in real time, and determines a first area selected by the user on the image captured by the camera in real time. Specifically, it includes: The electronic device receives and responds to a sliding operation on the image captured by the camera in real time, and determines the area of the sliding trajectory of the sliding operation as the first area.

8. The method according to claim 1, wherein, the method further includes: The electronic device displays a first tracking frame in an eighth area on the first cropped image, the image content in the eighth area is the same as the image content in the second area, and the image size in the eighth area is different from the image size in the second area.

9. The method according to claim 1, wherein, the method further includes: when the specified target matching the preset target feature template is included in the image captured by the camera in real time, the electronic device displays marking information around the specified target in the image captured by the camera in real time; the electronic device receives and responds to a second input operation on the image captured by the camera in real time, and the electronic device acquires the image captured by the camera in real time; the electronic device determines, based on the preset target feature template, a ninth region in the image captured by the camera in real time, where the similarity between the ninth region and the target feature template is greater than a second value; the electronic device crops a fourth cropped image from the image captured by the camera in real time based on the ninth region, and the fourth cropped image includes the ninth region; the electronic device displays a sixth shooting interface, and the sixth shooting interface displays the first cropped image.

10. The method according to any one of claims 1-8, wherein, the method further includes: after the electronic device extracts the first image feature, based on the first image feature, no region with a similarity greater than the first value to the first image feature is determined in T consecutive frames of images captured by the camera in real time; the electronic device displays a seventh shooting interface, and the seventh shooting interface displays the image captured by the camera in real time.

11. An electronic device, wherein, it includes one or more processors, one or more memories, and a camera; the one or more memories and the camera are coupled to the one or more processors, the one or more memories are used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, including instructions, wherein, when the instructions run on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 10.

Images