A person tracking display method and electronic device

Abstract

A method for tracking and displaying a person. In this method, an electronic device intelligently detects a person in a video stream, and cuts the video stream according to the number and position of the detected targets and outputs the video stream for display. By implementing the technical solution provided by the present application, the electronic device can automatically complete the tracking of a person in a video scene, and the output video screen can display more details of the person.

Description

Technical Field

[0001] The present application relates to the field of terminal and image processing technology, and in particular to a person tracking display method and electronic equipment. Background Art

[0002] With the development of network technology, more and more users have become accustomed to remote video interaction. For example, employees can conduct online meetings through real-time video recording; in special periods, students can take remote classes through real-time video recording; even if they are thousands of miles apart, family members can easily communicate online through real-time video recording and share their latest news.

[0003] However, currently in the process of shooting a person video, because the camera is not facing the person, or because the person is in a moving state, the person may be at the edge of the shooting screen. The person shot is small, and it is difficult to clearly show the details of the person. Summary of the invention

[0004] The present application provides a person tracking display method and electronic device, which automatically completes person tracking in a video scene, and the output video screen displays more person details.

[0005] In a first aspect, an embodiment of the present application provides a method for tracking and displaying a person, comprising: an electronic device performs target detection on the latest frame in a video stream to obtain the number of targets and the positions of the targets in the latest frame, where the targets are the persons in the latest frame; the electronic device determines a cropping area based on the number of targets and the positions of the targets in the latest frame, where the cropping area covers the positions of the targets in the latest frame and is smaller than the range of the latest frame; the electronic device outputs a cropped frame based on the cropping area.

[0006] In the embodiment of the present application, the electronic device detects the target of the person in the video stream, obtains the number of targets and the position in the latest frame of the video stream based on the detection, intelligently determines the cropping area that is smaller than the size of the latest frame and covers the target, and outputs the cropped frame according to the cropping area. Thus, the automatic tracking of the person is achieved. Since the cropping area is smaller than the size of the latest frame in the video stream, the person in the output video is larger than that in the original video, and it can display more details of the person.

[0007] In combination with the first aspect, in some embodiments, the electronic device determines the cropping area based on the number of targets and the positions of the targets in the latest frame, specifically including: when the number of targets is 1, the electronic device determines the cropping area based on the position of the 1 target in the latest frame; the cropping area is centered on the position of the 1 target in the latest frame, covers the position of the 1 target in the latest frame, and is smaller than the screen range of the latest frame; when the number of targets is multiple, the electronic device determines the cropping area based on the positions of multiple targets in the latest frame; the cropping area covers the positions of the multiple targets in the latest frame and is smaller than the screen range of the latest frame.

[0008] In the above embodiment, the electronic device determines the cropping area in different ways according to the number of targets obtained after target detection by the electronic device. Thus, the appropriate cropping area can be intelligently determined according to the number of photographed objects, so that the final cropped frame image better meets user expectations.

[0009] In combination with some embodiments of the first aspect, in some embodiments, the electronic device determines the cropping area based on the positions of multiple targets in the latest frame, specifically including: the electronic device determines the protagonist among the multiple targets; the electronic device determines the cropping area based on the positions of the multiple targets in the latest frame and the position of the protagonist in the latest frame; when there is no protagonist among the multiple targets, the cropping area covers the positions of the multiple targets in the latest frame and is smaller than the screen range of the latest frame; when there is a protagonist among the multiple targets, the cropping area is centered on the position of the protagonist in the latest frame, covers the positions of the multiple targets in the latest frame, and is smaller than the screen range of the latest frame.

[0010] In the above embodiment, when multiple targets are detected, the electronic device can determine the protagonist among the multiple targets, and then intelligently determine the appropriate cropping area based on whether there is a protagonist and the position of the protagonist in the latest frame, so that the final cropped frame is more in line with the actual scene requirements.

[0011] In combination with some embodiments of the first aspect, in some embodiments, the electronic device determines the protagonist among the multiple targets, specifically including: if the protagonist among the multiple targets has not been determined in the historical frame picture, the electronic device performs posture analysis on the multiple targets to determine the protagonist among the multiple targets, and the historical frame picture is the frame picture before the latest frame picture in the video stream; if the protagonist among the multiple targets has been determined in the historical frame picture, the electronic device tracks and determines the protagonist among the multiple targets based on the position and feature information of the protagonist in the historical frame picture.

[0012] In the above embodiment, the electronic device can determine the protagonist in the latest frame in different ways according to whether the protagonist has been determined in the historical frame. In the case where the protagonist has been determined in the historical frame, it is no longer necessary to determine the protagonist through posture analysis, but only to track the position and feature information of the protagonist in the historical frame to determine the protagonist among multiple targets, so that there is no need to continuously perform posture analysis on multiple targets in each frame, which reduces the power consumption of the electronic device and improves the efficiency of determining the protagonist.

[0013] In combination with some embodiments of the first aspect, in some embodiments, the electronic device performs posture analysis on the multiple targets to determine the protagonist among the multiple targets, specifically including: the electronic device determines that the target among the multiple targets that maintains a preset protagonist action for a preset protagonist duration is the protagonist among the multiple targets.

[0014] In the above embodiment, the protagonist is determined by maintaining the preset protagonist action for a preset protagonist duration. On the one hand, misjudgment of the target action is avoided, and the accuracy of determining the protagonist is improved. On the other hand, the user can independently choose whether to maintain the preset protagonist action, which improves the interaction performance between the electronic device and the user.

[0015] In combination with some embodiments of the first aspect, in some embodiments, the electronic device tracks and determines the protagonist among the multiple targets based on the position and feature information of the protagonist in the historical frame, specifically including: the electronic device determines a candidate target in the latest frame, the candidate target being a target among the multiple targets whose position is within a preset distance threshold from the protagonist in the previous frame; when there is one candidate target, the electronic device determines the candidate target as the protagonist among the multiple targets; when there are multiple candidate targets, the electronic device determines the candidate target among the multiple candidate targets whose feature information is closest to the feature information of the protagonist in the historical frame as the protagonist among the multiple targets.

[0016] In the above embodiment, the candidate target is determined by the distance from the position of the protagonist in the previous frame, and the protagonist among multiple candidate targets is determined by the proximity to the characteristic information of the protagonist in the historical frame. While ensuring the accuracy of determining the protagonist, the power consumption of the electronic device is reduced.

[0017] In combination with some embodiments of the first aspect, in some embodiments, the method further includes: the electronic device records feature information of a protagonist among the multiple targets.

[0018] In the above embodiment, after determining the protagonist in the latest frame, the characteristic information of the protagonist is recorded, which can facilitate subsequent tracking and determination of the protagonist in the newly generated frame, without the need to perform posture analysis on each frame, thereby reducing the power consumption of the electronic device.

[0019] In combination with some embodiments of the first aspect, in some embodiments, the electronic device performs target detection on the latest frame in the video stream to obtain the number of targets and the positions of the targets in the latest frame, specifically including: the electronic device downsamples the original latest frame in the video stream to obtain the latest frame, and the resolution of the latest frame is less than the resolution of the original latest frame; the electronic device performs target detection on the latest frame to obtain the number of targets and the positions of the targets in the latest frame; the electronic device outputs a cropped frame according to the cropped area, specifically including: the electronic device crops and upsamples the original latest frame according to the cropped area in the latest frame to obtain a cropped frame, and the resolution of the cropped frame is equal to the resolution of the original latest frame; the electronic device outputs the cropped frame.

[0020] In the above embodiment, before target detection, the frame is downsampled to reduce the computational load of the electronic device, and upsampled after cropping to improve the resolution of the cropped frame, making the details of the person clearer.

[0021] In combination with some embodiments of the first aspect, in some embodiments, before the step of the electronic device outputting the cropped frame image, the method further includes: the electronic device performing distortion correction on the cropped frame image.

[0022] In the above embodiment, the cropped frame is subjected to distortion correction, and the deformation of the image captured by the wide-angle camera is corrected, so that the video image can more realistically reflect the captured object.

[0023] In combination with some embodiments of the first aspect, in some embodiments, before the step of the electronic device performing target detection on the latest frame in the video stream to obtain the number of targets and the positions of the targets in the latest frame, the method also includes: the electronic device merging the video streams captured by multiple cameras to obtain the video stream.

[0024] In the above embodiment, the video streams captured by multiple cameras are merged and then processed, thereby improving the viewing angle range of the video picture.

[0025] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a camera, one or more processors and a memory; the camera is used to shoot and obtain a video stream; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute: perform target detection on the latest frame in the video stream to obtain the number of targets and the position of the targets in the latest frame, the target being a person in the latest frame; determine a cropping area based on the number of targets and the position of the targets in the latest frame, the cropping area covers the position of the target in the latest frame and is smaller than the screen range of the latest frame; output a cropped frame based on the cropping area.

[0026] In combination with some embodiments of the second aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to cause the electronic device to execute: when the number of the target is 1, the cropping area is determined according to the position of the target in the latest frame; the cropping area is centered on the position of the target in the latest frame, covers the position of the target in the latest frame, and is smaller than the screen range of the latest frame; when the number of the target is multiple, the cropping area is determined according to the positions of multiple targets in the latest frame; the cropping area covers the positions of the multiple targets in the latest frame and is smaller than the screen range of the latest frame.

[0027] In combination with some embodiments of the second aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to cause the electronic device to execute: determining a protagonist among the multiple targets; determining a cropping area based on the positions of the multiple targets in the latest frame and the position of the protagonist in the latest frame; when there is no protagonist among the multiple targets, the cropping area covers the positions of the multiple targets in the latest frame and is smaller than the screen range of the latest frame; when there is a protagonist among the multiple targets, the cropping area is centered on the position of the protagonist in the latest frame, covers the positions of the multiple targets in the latest frame, and is smaller than the screen range of the latest frame.

[0028] In combination with some embodiments of the second aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to enable the electronic device to execute: when the protagonist among the multiple targets has not been determined in the historical frame, performing posture analysis on the multiple targets to determine the protagonist among the multiple targets, and the historical frame is the frame before the latest frame in the video stream; when the protagonist among the multiple targets has been determined in the historical frame, tracking and determining the protagonist among the multiple targets based on the position and feature information of the protagonist in the historical frame.

[0029] In combination with some embodiments of the second aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to cause the electronic device to execute: determining that the target among the multiple targets that maintains a preset protagonist action for a preset protagonist duration is the protagonist among the multiple targets.

[0030] In combination with some embodiments of the second aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to enable the electronic device to execute: determine a candidate target in the latest frame, the candidate target being a target among the multiple targets whose position is within a preset distance threshold from the protagonist in the previous frame; when there is one candidate target, determine the candidate target as the protagonist among the multiple targets; when there are multiple candidate targets, determine the candidate target among the multiple candidate targets whose feature information is closest to the feature information of the protagonist in the historical frame as the protagonist among the multiple targets.

[0031] In combination with some embodiments of the second aspect, in some embodiments, the one or more processors are specifically used to call the computer instruction to cause the electronic device to execute: recording feature information of the protagonist among the multiple targets.

[0032] In combination with some embodiments of the second aspect, in some embodiments, the one or more processors are specifically used to call the computer instructions to enable the electronic device to execute: downsampling the original latest frame in the video stream to obtain the latest frame, the resolution of the latest frame is smaller than the resolution of the original latest frame; performing target detection on the latest frame to obtain the number of targets and the positions of the targets in the latest frame; cropping and upsampling the original latest frame according to the cropped area in the latest frame to obtain a cropped frame, the resolution of the cropped frame is equal to the resolution of the original latest frame; and outputting the cropped frame.

[0033] In combination with some embodiments of the second aspect, in some embodiments, the one or more processors are further used to call the computer instruction to cause the electronic device to execute: performing distortion correction on the cropped frame image.

[0034] In combination with some embodiments of the second aspect, in some embodiments, the one or more processors are further used to call the computer instruction to enable the electronic device to execute: merging the video streams captured by multiple cameras to obtain the video stream.

[0035] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions so that the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0036] It is understood that the chip system may include a Figure 5 The processor 110 in the electronic device 100 shown may also include multiple Figure 5 The processor 110 in the electronic device 100 shown in FIG. 1 may further include one or more other chips in the chip system, for example, Figure 5 The image signal processing chip in the camera 193 in the electronic device 100 shown may also include an image display chip in the display screen 194 , etc., which is not limited here.

[0037] In a fourth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0038] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions. When the instructions are executed on an electronic device, the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0039] It is understandable that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer program product provided in the fourth aspect, and the computer storage medium provided in the fifth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the relationship between the video stream and the frame picture in the embodiment of the present application;

[0041] Figure 2 This is a schematic diagram of the effect of using the method for tracking and displaying a person in an embodiment of the present application;

[0042] Figure 3 This is another schematic diagram of the effect of using the method for tracking and displaying a person in an embodiment of the present application;

[0043] Figure 4 This is another schematic diagram of the effect of using the person tracking display method in the embodiment of the present application;

[0044] Figure 5 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0045] Figure 6 is a software structure block diagram of the electronic device provided in the embodiment of the present application;

[0046] Figure 7 is an exemplary schematic diagram of a video shooting interface in an embodiment of the present application;

[0047] Figure 8 is an exemplary schematic diagram of a shooting setting interface in an embodiment of the present application;

[0048] Fig. 9 is an exemplary schematic diagram of target detection in the person tracking display method of an embodiment of the present application;

[0049] Fig.10 is an exemplary schematic diagram of determining a cropping area in the person tracking display method of an embodiment of the present application;

[0050] Fig.11 is an exemplary schematic diagram cut out and outputted in the person tracking display method of an embodiment of the present application;

[0051] Fig.12 is another exemplary schematic diagram of performing target detection in the person tracking display method of an embodiment of the present application;

[0052] Fig.13 is another exemplary schematic diagram of determining a cropping area in the person tracking display method in an embodiment of the present application;

[0053] Fig.14 is another exemplary schematic diagram cut out and output in the person tracking display method of an embodiment of the present application;

[0054] Fig.15 is an exemplary schematic diagram of determining a protagonist in a character tracking and display method in an embodiment of the present application;

[0055] Fig.16 is another exemplary schematic diagram for determining a cropping area in the person tracking display method of an embodiment of the present application;

[0056] Fig.17 is another exemplary schematic diagram cut out and output in the person tracking display method of an embodiment of the present application;

[0057] Fig.18is an exemplary schematic diagram of determining a candidate target in the person tracking and display method of an embodiment of the present application;

[0058] Fig.19 It is another exemplary schematic diagram of determining candidate targets in the person tracking display method in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0060] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0061] Since the embodiments of the present application involve the application of image processing technology, in order to facilitate understanding, the relevant terms and concepts involved in the embodiments of the present application are first introduced below.

[0062] (1)SSD (single shot multibox detector) algorithm:

[0063] The SSD algorithm is an artificial neural network algorithm for target detection. While ensuring accuracy, it also improves the detection speed compared to other target detection algorithms and can meet the requirements of real-time detection.

[0064] To use the SSD algorithm for target detection, you need to first use labeled samples as a training set to train it and obtain an SSD algorithm that meets your needs.

[0065] For example, in an embodiment of the present application, a series of images with labeled characters can be used to train the SSD algorithm, so that the SSD algorithm can detect the characters in the input image and output the position information of the characters in the input image. Generally, in the SSD algorithm, the position information of the characters in the input image can be represented by a prediction box that frames the characters in the input image. The specific size of the prediction box can be pre-set, or it can be automatically adjusted according to preset rules based on the size of the detected target, which is not limited here.

[0066] According to different target detection usage scenarios, the characters in the embodiments of the present application may include only people, people and family pets (such as cats, dogs, etc.), or more objects related to people, which are not limited here. For different target requirements, it is only necessary to use samples with labels on the targets to be identified as training sets to train the corresponding SSD algorithm.

[0067] (2) Image downsampling (subsampled) and upsampling (upsampling):

[0068] Image downsampling is the process of reducing the size of an image. The main purpose of downsampling is to reduce the resolution of the target detection image and reduce the computing load of artificial intelligence (AI).

[0069] For example, if the resolution of an image is M*N, it is downsampled by a factor of s to obtain an image with a resolution of (M / s)*(N / s). Of course, s should be a common divisor of M and N.

[0070] Image upsampling is the process of enlarging an image so that it can be displayed on a higher resolution display device. Image enlargement is almost always done using the interpolation method, that is, inserting new elements between pixels based on the original image using a suitable interpolation algorithm.

[0071] In the embodiment of the present application, the purpose of downsampling the image is to reduce the image resolution, thereby reducing the computational load of image processing. For example, when using the SSD algorithm to detect an object on an image, or when performing posture detection on an image, the computational load of a small-resolution image is smaller and the operation is faster when processing.

[0072] In the embodiment of the present application, the image is upsampled because the resolution of the frame in the obtained video stream is reduced after the original video stream is cropped according to the determined cropping area. Therefore, the frame in the cropped video stream is upsampled so that the resolution of the frame in the obtained video stream is restored to the resolution of the frame in the original video stream.

[0073] (3) Video stream and frame:

[0074] A video stream is composed of frames. For example, a video stream with a frame rate of 24 frames per second (FPS) means that the video stream is composed of 24 frames per second.

[0075] Figure 1 It is a schematic diagram of the relationship between the video stream and the frame picture. In the embodiment of the present application, the video stream is generated by real-time shooting of the camera, so the video stream obtained by the shooting will continuously generate new frame pictures over time. Assuming that the shooting starts at time T0, a frame picture will be generated from time T1 to the latest time T13, where the frame pictures generated from time T1 to T12 can be called historical frame pictures, and the latest frame picture generated at time T13 can be called the latest frame picture.

[0076] The character tracking display method in the embodiment of the present application will process each latest frame, and the processed frame can form a new video stream.

[0077] In the prior art, when shooting a video, the person may be at the edge of the shooting screen because the camera is not facing the person or the person is in a moving state. The person shot is small and the details of the person cannot be clearly shown.

[0078] By using the character tracking display method in the embodiment of the present application, the user does not need to manually adjust the camera or video screen. In the video scene, the electronic device can automatically complete the character tracking, and the output video screen can display more character details, optimizing the user experience in the video scene.

[0079] The character tracking and display method in the embodiment of the present application has good application effects in different application scenarios.

[0080] For example, in a scene where the subject is a single person, Figure 2 The figure is a schematic diagram of the effect of using the character tracking display method in the embodiment of the present application. Figure 2 (a) is the original video. The single person is located at the edge of the video and is small, making it difficult to observe the details. After being processed by the person tracking display method in the embodiment of the present application, the output is as follows: Figure 2 In the video shown in (b), the output video shows the details of the person, retains the background near the person, and ensures that the person is in the center of the video.

[0081] For example, in a scene where the target is multiple people, Figure 3 , which is another schematic diagram of the use effect of the person tracking display method in the embodiment of the present application. Figure 3(a) is the original video. The multiple characters are located at the edge of the video, and the details of each character cannot be observed. After being processed by the character tracking display method in the embodiment of the present application, the output is as follows: Figure 3 In the video shown in (b), the output video contains all the characters in the original video and retains the character details as much as possible, which improves the user's video experience.

[0082] For example, when there are multiple cameras shooting at the same time, Figure 4 , which is another schematic diagram of the use effect of the person tracking display method in the embodiment of the present application. Figure 4 (a) is the original video shot by two cameras, in which a person is located at the junction of the two cameras, and the person's movements cannot be fully observed. After being processed by the person tracking display method in the embodiment of the present application, the output is as follows Figure 4 In the video shown in (b), the output video contains all the characters in the videos shot by the original two cameras, and the movements of each character can be clearly and completely observed, which improves the range of the video viewing angle.

[0083] The following first introduces an exemplary electronic device 100 provided in an embodiment of the present application.

[0084] Figure 5 It is a schematic diagram of the structure of the electronic device 100 provided in an embodiment of the present application.

[0085] The embodiment is described in detail below by taking the electronic device 100 as an example. It should be understood that the electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0086] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194 and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0087] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0088] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0089] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0090] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0091] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0092] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0093] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0094] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0095] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0096] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0097] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0098] The SIM interface can be used to communicate with the SIM card interface 195 to implement the function of transmitting data to the SIM card or reading data in the SIM card.

[0099] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0100] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0101] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0102] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0103] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0104] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0105] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0106] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0107] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0108] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, applications required for at least one function (such as face recognition function, fingerprint recognition function, mobile payment function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as face information template data, fingerprint information template, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0109] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0110] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0111] In the embodiment of the present application, the user operation collected by the pressure sensor 180A and / or the touch sensor 180K can trigger the camera 193 in the electronic device 100 to record a video. The recorded video can be displayed on the display screen 194 after the processor 110 calls the operation instruction stored in the internal memory 121 for processing.

[0112] Figure 6 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0113] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0114] The application layer can include a series of application packages.

[0115] like Fig.10 As shown, the application package may include a person tracking display module, a camera, a gallery, a calendar, calls, maps, navigation, WLAN, Bluetooth, music, video, short messages and other applications (also referred to as applications).

[0116] In the embodiment of the present application, after the camera application is started, the character tracking and display module in the application package can be called to execute the character tracking and display method in the embodiment of the present application to track and display the character in the video scene.

[0117] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0118] like Figure 4 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0119] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0120] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0121] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0122] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0123] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0124] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications from applications running in the background, or a notification that appears on the screen in the form of a dialogue interface. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0125] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0126] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0127] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0128] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0129] The surface manager is used to manage the display subsystem and provides the fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.

[0130] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0131] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0132] A 2D graphics engine is a drawing engine for 2D drawings.

[0133] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, sensor driver, and virtual card driver.

[0134] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with capturing a photo scene.

[0135] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input event (including touch coordinates, timestamp of the touch operation and other information). The original input event is stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch single-click operation, and the control corresponding to the single-click operation is the control of the camera application icon as an example, the camera application calls the interface of the application framework layer, starts the camera application, and then starts the camera driver by calling the kernel layer, and captures a static image or video through the camera 193. In the embodiment of the present application, the camera application can call the character tracking display module in the application package to process the video captured by the camera 193, and the camera application can output the processed video to the display screen 194.

[0136] The following describes the person tracking display method in the embodiment of the present application in combination with the software and hardware architecture of the electronic device 100. Depending on the number of shooting targets and the mode selected by the user, the person tracking display method has the following situations, which are described below respectively:

[0137] 1. The shooting target is a single target.

[0138] like Figure 7 The figure is an exemplary schematic diagram of the video shooting interface of the electronic device in the embodiment of the present application.

[0139] The video shooting interface 700 may be a video shooting interface displayed after the user opens the camera application and clicks the video recording control. It may also be a video shooting interface displayed after the video chat function is started by other applications in the electronic device 100, such as a video conferencing application, a chat application, etc., which is not limited here.

[0140] The video shooting interface 700 may include a shooting screen display area 701 , a setting control 702 and a recording control 703 .

[0141] The captured image display area 701 is used to display the image captured by the camera 193 in the electronic device 100 .

[0142] The setting control 702 is used to trigger the display of the shooting setting interface;

[0143] The recording control 703 is used to control the start, pause and stop of video recording.

[0144] When the electronic device 100 receives a user click Figure 7 After the operation of the setting control 702 in the electronic device 100, the electronic device 100 can display the shooting setting interface. Figure 8 The figure is an exemplary schematic diagram of the shooting setting interface in the embodiment of the present application.

[0145] The shooting setting interface 800 may include a person tracking control 801, a deformity correction control 802, a protagonist mode control 803, and a video stitching control 804. It is understandable that in actual applications, the shooting setting interface 800 may also include many other functional controls, such as a portrait beautification control, a time-lapse shooting control, etc., which are not limited here.

[0146] The person tracking control 801 is used to trigger the person tracking function. After the person tracking function is activated, the electronic device 100 may execute the person tracking display method in the embodiment of the present application on the video stream captured by the camera 193.

[0147] The deformity correction control 802 is used to trigger the start-up picture deformity correction function. After the picture deformity correction function is started, the picture in the video stream will be corrected before being sent to the display screen 194 for display, so that the output picture is more stable and clearer.

[0148] The protagonist mode control 803 is used to trigger the protagonist mode. The protagonist mode is to determine one of the characters as the protagonist when the shooting target is multiple characters and display it in the center area of ​​the picture.

[0149] The video splicing control 804 is used to trigger the video stream splicing function. After the video stream splicing function is activated, the electronic device 100 can combine the video streams captured by multiple cameras 193 into one video stream for processing and display.

[0150] If the current shooting target is a single target and the user is Figure 8In the shooting setting interface 800 shown, the person tracking function is activated by clicking the person tracking control 801. Then, when shooting a video, the electronic device 100 will perform target detection on the frame pictures in the currently shot video stream to determine the number of targets and the positions of the targets in the frame pictures.

[0151] like Fig. 9 FIG. 1 is an exemplary schematic diagram of target detection in the human tracking display method of an embodiment of the present application. The electronic device 100 can detect the target in the frame image and determine the coordinates (Xm, Ym) of the center point of the target in the frame image.

[0152] It is understandable that starting the person tracking function by clicking on the person tracking control 801 is only an example. In actual applications, there may be many ways to start the person tracking function. For example, the electronic device may turn on the person tracking function by default, or may control the start of the person tracking function through preset gesture operation instructions, etc., which is not limited here.

[0153] Understandably, Fig. 9 This is just an example of determining the target position through target detection. In actual applications, there are many different ways to represent the position of the target in the frame. For example, if the frame is established with an XY axis coordinate system, the position of the target in the frame can be represented by feeding back the maximum X value, minimum X value, maximum Y value and minimum Y value of the target area. A frame can also be used to frame the target area, and the position of the target in the frame can be represented by feeding back the coordinates of the frame. This is not limited here.

[0154] There are many options for artificial intelligence neural networks used for target detection, such as Faster R-CNN algorithm, R-FCN algorithm and SSD algorithm, etc. Preferably, in the embodiment of the present application, a trained SSD algorithm for detecting people can be used for target detection.

[0155] It is understandable that target detection on the frame image is mainly performed by the processor 110 in the electronic device 100 to perform data processing.

[0156] Optionally, in the embodiment of the present application, before performing target detection on the frame picture in the currently captured video stream, the electronic device 100 may first downsample the frame picture to reduce the resolution of the frame picture. Then, target detection is performed on the downsampled frame picture, thereby reducing the computational load of target detection on the frame picture.

[0157] When the electronic device 100 detects a target and determines that there is only one target in the frame, the electronic device can determine a cropping area based on the position of the target in the frame. The cropping area is centered on the target and is smaller than the image range of the original frame. Fig.10 FIG. 1 is an exemplary schematic diagram of determining a cropping area in a method for displaying a person tracking in an embodiment of the present application. The electronic device 100 may determine a screen range centered on the target and covering the target with an X / Y axis length of 1 / 2 (or other values) of the original frame as the cropping area. For example, Fig.10 In the example, the X-axis length of the frame image is X1, and the Y-axis length is Y1. The electronic device can determine that the area 1001 centered on the target and covering the target with an X-axis length of X1 / 2 and a Y-axis length of Y1 / 2 is the cropping area.

[0158] It is understandable that the electronic device 100 can automatically determine the cropping area according to the position of the target in the frame according to the preset composition rule. The preset composition rule can be a factory preset rule, a rule added by the user, or a trained artificial intelligence model, which is not limited here.

[0159] The electronic device 100 can crop the frame image according to the cropping area and output it. Fig.11 , which is an exemplary schematic diagram of cropping and outputting in the method for displaying character tracking in an embodiment of the present application. Compared with the frame in the original video stream, in the frame displayed on the display screen 194 after cropping and outputting, the character is located at the center of the display screen 194 and is larger than in the original frame, so it can display more character details.

[0160] It should be noted that when cropping a frame, the frame in the captured video stream is cropped. If the frame is downsampled before target detection in order to reduce the computational load, then the coordinates of the cropped area in the frame in the captured video stream need to be determined based on the coordinates of the cropped area in the downsampled frame. Then, cropping is performed on the frame in the captured video stream according to the cropped area.

[0161] Since the resolution of the cropped frame is smaller than the resolution of the original frame, upsampling is required to restore the resolution of the cropped frame to the resolution of the original frame.

[0162] 2. The shooting target is multiple targets, and the user turns on the protagonist mode.

[0163] If the current shooting target is multiple targets, the user Figure 8 In the shooting setting interface 800 shown in FIG. 8 , the person tracking function is activated by clicking the person tracking control 801, and the user Figure 8In the shooting setting interface 800 shown, the protagonist mode is activated by clicking the protagonist mode control 803. Then, when shooting a video, the electronic device 100 will perform target detection on the frame pictures in the currently shot video stream to determine the number of targets and the positions of the targets in the frame pictures.

[0164] It is understandable that starting the protagonist mode by clicking on the protagonist mode control 803 is only an example. In actual applications, there may be many ways to start the protagonist. For example, the electronic device may turn on the protagonist mode by default, or may control the start of the protagonist mode through preset gesture operation instructions, etc., which is not limited here.

[0165] like Fig.12 FIG. 1 is another exemplary schematic diagram of target detection in the method for tracking and displaying a person in an embodiment of the present application. Through target detection, the electronic device 100 can determine that the number of targets is 3, and can obtain the coordinates of each target in the frame.

[0166] The protagonist mode of the electronic device 100 is turned on. In the protagonist mode, when there are multiple objects to be photographed, the electronic device 100 will display the protagonist of the multiple objects in the central area of ​​the frame.

[0167] When the shooting target is multiple targets and the user turns on the protagonist mode, the electronic device 100 processes the frame image in three cases, which are described below respectively:

[0168] Case 1: The electronic device has not yet determined the protagonist from multiple targets.

[0169] To determine the protagonist from multiple targets, it is necessary to determine whether there is a target among the multiple targets that meets the preset protagonist condition through posture analysis. Specifically, the electronic device can perform posture analysis on multiple targets in the frame. When it is determined that a certain target maintains the preset protagonist action for a preset protagonist duration Tz, the target is determined to be the protagonist among the multiple targets.

[0170] The preset protagonist action may be factory preset or set by the user, which is not limited here. For example, the preset protagonist action may be an OK gesture action, or may be identifying that the elbow key point is higher than the shoulder key point, the angle between the wrist-elbow-shoulder key points is greater than a certain angle, etc. There may also be many other actions or gestures as preset protagonist actions, which are not limited here.

[0171] It is understandable that since posture analysis to identify the protagonist from multiple targets requires at least a preset protagonist duration Tz, and it is possible that the target will perform the preset protagonist action after a period of time in the shooting, therefore, in the early stage of video shooting, there is a period of time when the protagonist cannot be identified from multiple targets. At this time, the electronic device 100 can determine the cropping area based on the number of targets determined by target detection and the position of the targets in the frame. The cropping area is smaller than the screen range of the original frame and covers all detected targets. Fig.13 FIG. 1 is another exemplary schematic diagram of determining a cropping area in the method for tracking and displaying a person in an embodiment of the present application. The electronic device 100 determines an area 1301 covering the multiple targets as the cropping area.

[0172] The electronic device 100 can crop the frame image according to the cropping area and output it. Fig.14 , which is another exemplary schematic diagram of cropping and outputting in the method for displaying human tracking in an embodiment of the present application. Compared with the frame images in the original video stream, in the frame images that are cropped and output and displayed on the display screen 194, each person is larger than in the original frame images, and all targets detected in the original frame images are fully displayed, so it can display more details of the person.

[0173] Case 2: The electronic device determines the protagonist from multiple targets in the latest frame.

[0174] If a target among the multiple targets in the latest frame has been maintaining the preset protagonist action in the historical frames, and the duration of maintaining the preset protagonist action just reaches the preset protagonist duration Tz in the latest frame, the electronic device 100 can determine in the latest frame that the target that has been maintaining the preset protagonist action for the preset protagonist duration Tz is the protagonist among the multiple targets. Fig.15 As shown, it is an exemplary schematic diagram for determining the protagonist in the character tracking display method of an embodiment of the present application. Assume that the preset protagonist action is a one-handed-on-hips action. The captured video stream continuously generates new frames over time T. In the frames generated at times T1 and T2, the electronic device 100 does not detect that any target maintains the preset protagonist action. At time T3, the electronic device 100 determines that the middle target among the three targets maintains the preset protagonist action through posture analysis. The electronic device 100 continues to perform posture detection on each frame generated thereafter, and determines that in the latest frame generated until time T13, the middle target maintains the preset protagonist action for a preset protagonist duration. Then, the electronic device 100 determines that the middle target is the protagonist among the multiple targets. After determining that a target in the frame is the protagonist among multiple targets, the electronic device 100 can record the characteristic information of the target.

[0175] After determining the main character among the multiple targets in the frame, the electronic device 100 can determine a cropping area according to the number of the multiple targets, the positions of the multiple targets in the frame, and the position of the main character in the frame. The cropping area is smaller than the image range of the original frame, covers all detected targets, and is centered on the main character. Fig.16 FIG. 1 is another exemplary schematic diagram of determining a cropping area in the method for tracking and displaying a person in an embodiment of the present application. The electronic device 100 determines an area 1601 covering the multiple targets and centered on the main character as the cropping area.

[0176] The electronic device 100 can crop the frame image according to the cropping area and output it. Fig.17 As shown, another exemplary schematic diagram of cropping and outputting in the character tracking display method of an embodiment of the present application is shown. Compared with the frame picture in the original video stream, the frame picture displayed on the display screen 194 after cropping and outputting is displayed with the protagonist in the middle as the center, and each character is larger than in the original frame picture. Not only all the targets detected in the original frame picture are fully displayed, more character details are displayed, but also the main characters can be identified at a glance, which improves the user experience in the video scene.

[0177] Case 3: The electronic device has identified the protagonist from the historical frames and tracks and identifies the protagonist in the latest frame.

[0178] If the electronic device 100 has already determined the protagonist in the historical frame, then in the newly generated frame, the electronic device 100 no longer needs to determine the protagonist in the latest frame based on posture analysis. It only needs to track and determine the protagonist in the latest frame based on the feature information of the protagonist in the historical frame and the position information of the protagonist in the previous frame.

[0179] Specifically, the electronic device 100 may first determine that the target in the latest frame is within a preset distance threshold s from the position of the protagonist in the previous frame as a candidate target. If the number of candidate targets is 1, the electronic device 100 may determine that the candidate target is the protagonist in the latest frame. If the number of candidate targets is multiple, the electronic device 100 may determine that the candidate target whose feature information is closest to the feature information of the protagonist in the historical frame is the protagonist in the latest frame.

[0180] like Fig.18As shown, it is an exemplary schematic diagram of determining candidate targets in the character tracking display method of an embodiment of the present application. In the historical frame images generated from time T1 to T13, the electronic device 100 has determined the protagonist among multiple targets. In the previous frame image of the latest frame image generated at time T14, that is, the frame image generated at time T13, the coordinates of the center point of the protagonist are (Xz, Yz). It can be understood that due to factors such as camera deflection or target movement, the position of the target in the frame image generated at time T14 may not be exactly the same as that in the frame image generated at time T13. As shown in FIG. Fig.18 As shown, in the frame generated at time T14, there is only one candidate target within the preset distance threshold s from the position of the protagonist in the previous frame. When the electronic device 100 determines that there is only one candidate target in the latest frame, the electronic device 100 can determine that the candidate target is the protagonist in the latest frame. The electronic device 100 can record the characteristic information of the protagonist.

[0181] like Fig.19 As shown, it is another exemplary schematic diagram of determining candidate targets in the character tracking display method in the embodiment of the present application. In the frame generated at time T13, the coordinates of the center point of the protagonist are (Xz, Yz). In the frame generated at time T14, there are 2 candidate targets within the preset distance threshold s from the position of the protagonist in the previous frame. At this time, the electronic device 100 will compare the characteristic information of the protagonist recorded when the protagonist is determined in the historical frame with the characteristic information of the two candidate targets, and determine the protagonist in the frame generated at time T14 by the candidate target whose characteristic information is closest to the characteristic information of the protagonist in the historical frame.

[0182] Specifically, in the historical frame, whenever a protagonist is identified, the electronic device can record the characteristic information of the protagonist. After multiple candidate targets are identified in the latest frame, the electronic device can use the recorded characteristic information of the protagonist to perform a cosine distance comparison with the characteristic information of the candidate targets in the latest frame, and lock the one with the closest distance as the protagonist.

[0183] After the protagonist is determined in the latest frame, the electronic device 100 can also record the position information and feature information of the protagonist, so as to facilitate tracking and determining the protagonist in the subsequently generated frame.

[0184] After determining the main character among the multiple objects in the frame, the electronic device 100 can determine the cropping area according to the number of the multiple objects, the positions of the multiple objects in the frame, and the position of the main character in the frame, and crop and output it. Fig.16 and Fig.17 , which is similar to the above case 2 in determining the cropping area, cropping and outputting, and will not be repeated here.

[0185] 3. The shooting target is multiple targets, and the user has not turned on the protagonist mode.

[0186] If the current shooting target is multiple targets, the user Figure 8 In the shooting setting interface 800 shown in FIG. 1 , the person tracking function is activated by clicking the person tracking control 801, but the user does not Figure 8 In the shooting setting interface 800 shown, the protagonist mode control 803 is clicked, and the protagonist mode is not started. Then, when shooting a video, the electronic device 100 will perform target detection on the frame pictures in the currently shot video stream to determine the number of targets and the positions of the targets in the frame pictures.

[0187] Then, according to the number of targets determined by target detection and the position of the targets in the frame, the cropping area is determined, cropped and output. For specific methods, please refer to the above Fig.13 and Fig.14 , which is similar to the above case 1 in determining the cropping area, cropping and outputting, and will not be repeated here.

[0188] It is understandable that, when the target is multiple targets, the electronic device may also perform downsampling before target detection to reduce the frame resolution to reduce the processing load, as in the case where the target is a single target. After cropping, upsampling may also be performed to restore the resolution of the cropped frame to the resolution of the original frame, which will not be elaborated here.

[0189] In the embodiment of the present application, in the video shooting scene, when the electronic device 100 cuts and outputs the processed frame according to the cut area, it can be output to the display screen 194 of the electronic device 100 for display. In the video conference or video chat scene, the electronic device 100 can also output the processed frame to the electronic device of the communication peer through the mobile communication module 150 and / or the wireless communication module 160, so that it is displayed on the display screen of the electronic device of the communication peer. The specific output object can be determined according to the actual use scenario and is not limited here.

[0190] In some embodiments of the present application, the above-mentioned character tracking display method can also be used to process the recorded video, and according to the generation time of the frame in the video, the frame being processed is used as the above-mentioned latest frame, and the frame that has been processed is used as the above-mentioned historical frame. The specific processing method is the same as the above-mentioned character tracking display method, and will not be repeated here.

[0191] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0192] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0193] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0194] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A method for tracking and displaying a person, characterized in that: include: The electronic device displays a video shooting interface in response to the first user operation, wherein the video shooting interface includes a picture shot by a camera; After turning on the person tracking control and the protagonist mode, the electronic device performs target detection on the latest frame in the video stream to obtain the number of targets and the positions of the targets in the latest frame, wherein the targets are the people in the latest frame; The electronic device determines a cropping area according to the number of the targets and the positions of the targets in the latest frame, where the cropping area covers the positions of the targets in the latest frame and is smaller than the image range of the latest frame; The electronic device displays the cropped frame image according to the cropped area; The electronic device determines the cropping area according to the number of the targets and the positions of the targets in the latest frame, specifically including: when the number of the targets is multiple, if the protagonist among the multiple targets is not determined in the historical frame, the electronic device determines that the target among the multiple targets that maintains a preset protagonist action for a preset protagonist duration is the protagonist among the multiple targets, and the historical frame is a frame in the video stream before the latest frame; the electronic device determines the cropping area according to the positions of the multiple targets in the latest frame and the position of the protagonist in the latest frame.

2. The method according to claim 1, characterized in that The electronic device determines the cropping area according to the number of the objects and the positions of the objects in the latest frame, and specifically includes: When the number of the targets is one, the electronic device determines the cropping area according to the position of the target in the latest frame; the cropping area is centered on the position of the target in the latest frame, covers the position of the target in the latest frame, and is smaller than the screen range of the latest frame.

3. The method according to claim 1, characterized in that: When there is no protagonist among the multiple targets, the cropped area covers the positions of the multiple targets in the latest frame and is smaller than the screen range of the latest frame; when there is a protagonist among the multiple targets, the cropped area is centered on the position of the protagonist in the latest frame, covers the positions of the multiple targets in the latest frame, and is smaller than the screen range of the latest frame.

4. The method according to claim 3, characterized in that: The method further comprises: If the protagonist among the multiple targets has been determined in the historical frame, the electronic device tracks and determines the protagonist among the multiple targets according to the position and feature information of the protagonist in the historical frame.

5. The method according to claim 4, characterized in that The electronic device tracks and determines the protagonist among the multiple targets according to the position and feature information of the protagonist in the historical frame, specifically including: The electronic device determines a candidate target in the latest frame, wherein the candidate target is a target among the multiple targets whose position is within a preset distance threshold from the main character in the previous frame; When there is only one candidate target, the electronic device determines that the candidate target is a protagonist among the multiple targets; When there are multiple candidate targets, the electronic device determines a candidate target whose feature information is closest to feature information of the protagonist in the historical frame as the protagonist among the multiple targets.

6. The method according to any one of claims 3 to 5, characterized in that The method further comprises: The electronic device records feature information of a protagonist among the multiple targets.

7. The method according to any one of claims 1 to 5, characterized in that The electronic device performs target detection on the latest frame in the video stream to obtain the number of targets and the positions of the targets in the latest frame, specifically including: The electronic device downsamples the original latest frame in the video stream to obtain the latest frame, wherein the resolution of the latest frame is smaller than the resolution of the original latest frame; The electronic device performs target detection on the latest frame to obtain the number of targets and the positions of the targets in the latest frame; The electronic device displays the cropped frame image according to the cropped area, specifically comprising: The electronic device crops and upsamples the original latest frame according to the cropped area in the latest frame to obtain a cropped frame, wherein the resolution of the cropped frame is equal to the resolution of the original latest frame; The electronic device displays the cropped frame image.

8. The method according to claim 7, characterized in that Before the step of outputting the cropped frame image by the electronic device, the method further includes: The electronic device performs distortion correction on the cropped frame image.

9. The method according to any one of claims 1 to 5 and 8, characterized in that: Before the step of the electronic device detecting targets in the latest frame of the video stream to obtain the number of targets and the positions of the targets in the latest frame, the method further comprises: The electronic device combines the video streams captured by multiple cameras to obtain the video stream.

10. An electronic device, characterized in that: The electronic device comprises: a camera, one or more processors and a memory; The camera is used for shooting to obtain a video stream; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute: In response to the first user operation, displaying a video shooting interface through the display screen, wherein the video shooting interface includes a picture shot by the camera; After turning on the person tracking control and the protagonist mode, performing target detection on the latest frame in the video stream to obtain the number of targets and the positions of the targets in the latest frame, wherein the target is the person in the latest frame; Determine a cropping area according to the number of the targets and the positions of the targets in the latest frame, where the cropping area covers the positions of the targets in the latest frame and is smaller than the image range of the latest frame; According to the cropping area, displaying the cropped frame image through the display screen; The method of determining the cropping area according to the number of targets and the positions of the targets in the latest frame specifically includes: when there are multiple targets, if the protagonist among the multiple targets is not determined in the historical frame, then determining the target among the multiple targets that maintains a preset protagonist action for a preset protagonist duration as the protagonist among the multiple targets, and the historical frame is a frame in the video stream before the latest frame; determining the cropping area according to the positions of the multiple targets in the latest frame and the position of the protagonist in the latest frame.

11. The electronic device according to claim 10, characterized in that: The one or more processors are specifically configured to call the computer instructions so that the electronic device executes: When the number of the targets is 1, the cropping area is determined according to the position of the target in the latest frame; the cropping area is centered on the position of the target in the latest frame, covers the position of the target in the latest frame, and is smaller than the screen range of the latest frame.

12. The electronic device according to claim 10, characterized in that: When there is no protagonist among the multiple targets, the cropped area covers the positions of the multiple targets in the latest frame and is smaller than the screen range of the latest frame; when there is a protagonist among the multiple targets, the cropped area is centered on the position of the protagonist in the latest frame, covers the positions of the multiple targets in the latest frame, and is smaller than the screen range of the latest frame.

13. The electronic device according to claim 12, characterized in that: The one or more processors are further configured to call the computer instructions so that the electronic device executes: When the protagonist among the multiple targets has been determined in the historical frame, the protagonist among the multiple targets is tracked and determined according to the position and feature information of the protagonist in the historical frame.

14. The electronic device according to claim 13, characterized in that: The one or more processors are specifically configured to call the computer instructions so that the electronic device executes: Determine a candidate target in the latest frame, wherein the candidate target is a target among the multiple targets whose position is within a preset distance threshold from the main character in the previous frame; When there is only one candidate target, determining the candidate target as a protagonist among the multiple targets; When there are multiple candidate targets, the candidate target whose feature information is closest to the feature information of the protagonist in the historical frame is determined as the protagonist among the multiple targets.

15. The electronic device according to claim 12 or 14, characterized in that: The one or more processors are specifically configured to call the computer instructions so that the electronic device executes: The characteristic information of the protagonist among the multiple targets is recorded.

16. The electronic device according to any one of claims 10 or 14, characterized in that: The one or more processors are specifically configured to call the computer instructions so that the electronic device executes: Downsampling the original latest frame in the video stream to obtain the latest frame, wherein the resolution of the latest frame is smaller than the resolution of the original latest frame; Performing target detection on the latest frame to obtain the number of targets and the positions of the targets in the latest frame; According to the cropping area in the latest frame, the original latest frame is cropped and up-sampled to obtain a cropped frame, wherein the resolution of the cropped frame is equal to the resolution of the original latest frame; The cropped frame image is displayed through the camera.

17. The electronic device according to claim 16, characterized in that: The one or more processors are further configured to call the computer instructions so that the electronic device executes: Distortion correction is performed on the cropped frame image.

18. The electronic device according to any one of claims 10 to 14 and 17, characterized in that: The one or more processors are further configured to call the computer instructions so that the electronic device executes: The video streams captured by multiple cameras are merged to obtain the video stream.

19. A chip system, applied to an electronic device, the chip system comprising one or more processors, the processors being used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-9.

20. A computer program product comprising instructions, characterized in that When the computer program product is run on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 9.

21. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 9.

Images