Object tracking method and electronic device

Abstract

Provided in the embodiments of the present application are an object tracking method and an electronic device. The method comprises: an electronic device displaying a first user interface, wherein the first user interface comprises a first image; in response to a click operation input for an object in the first image, the electronic device displaying on the first image a bounding box of the object; the electronic device cropping an area corresponding to the bounding box on the first image to obtain a template image of the object; and the electronic device tracking the object in a second image on the basis of the template image. In the method, a bounding box of an object is determined, and an area corresponding to the bounding box on a first image is used as a template image to track the object, so that the obtained template image is more accurate, thereby improving the accuracy of object tracking, and reducing the probability of false positives and false negatives.

Description

Target pursuit method and electronic device

[0001] The present application claims priority to the Chinese patent application No. 202410729177.9, filed on June 5, 2024, and entitled "A target pursuit method and electronic device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, in particular to the field of computer vision, and more particularly to a target pursuit method and electronic device. BACKGROUND

[0003] Target pursuit technology can realize continuous positioning of an object defined in a current frame in subsequent frames in a video or a photograph, and is widely applied in intelligent security, smart photographing of a mobile phone, and the like. Click-to-prompt target pursuit is an interactive target pursuit technology, and a user can realize automatic pursuit of a target by clicking the target in an image. For example, when the user uses a mobile phone to take a photograph, after the user clicks a car in a photograph preview interface of the mobile phone for the first time, the user does not need to click the car again to realize subsequent pursuit of the car. The accuracy of target pursuit directly affects the interactive experience of the user, and in some scenarios, even a safety accident can be caused.

[0004] Currently, a click-to-prompt target pursuit method is to obtain a template image of a target to be pursued by performing fixed-size cropping on an image centered on the coordinates of the click. However, in a complex background and a small object scenario, fixed-size cropping causes other objects to appear in the template image. For example, as shown in a photographing interface of FIG. 1A, after the user clicks a dog in a preview image, the electronic device performs fixed-size cropping on the preview image in response to the user operation, as shown in a photographing interface of FIG. 1B. The dog to be pursued is included in a cropping frame 101a, and part of a cat is also included. When the image cropped by the cropping frame 101a is used as the template image, since the template image includes other objects such as the cat, the target clicked by the user cannot be accurately understood, and problems of target mis-pursuit and target missed-pursuit occur. SUMMARY

[0005] Embodiments of the present application provide a target pursuit method and an electronic device. The method uses an outer frame of a target as a cropping frame, so that the template image is more accurate, the accuracy of target pursuit is improved, and the probability of target mis-pursuit and target missed-pursuit is reduced.

[0006] In a first aspect, an embodiment of the present application provides a target pursuit method applied to an electronic device, the method comprising: displaying, by the electronic device, a first user interface, the first user interface comprising a first image; displaying, by the electronic device, a bounding box of a target in the first image in response to a click operation input for the target in the first image; cropping, by the electronic device, a region corresponding to the bounding box in the first image to obtain a template image of the target; and pursuing, by the electronic device, the target in a second image based on the template image, the second image being an image of the electronic device subsequent to the first image.

[0007] In the embodiment of the present application, the electronic device can determine the bounding box of the target clicked by the user, the size of the bounding box is closer to the size of the target to be pursued, and the template image obtained by the electronic device based on the bounding box is more accurate. The subsequent target pursuit based on the accurate template image can improve the accuracy of target pursuit and reduce the probability of target mis-pursuit and target missed-pursuit.

[0008] With reference to the first aspect, in a possible implementation, the electronic device displays the bounding box of the target in the first image in response to the click operation input for the target in the first image, comprising: the electronic device obtains the coordinates of the click position and the first image in response to the click operation input for the target in the first image; the electronic device determines a segmentation mask of the target based on the coordinates of the click position and the first image, the segmentation mask being used to indicate a region where a pixel point of the target in the first image is located; the electronic device determines the bounding box based on the segmentation mask; and the electronic device displays the bounding box on the first image.

[0009] In the embodiment of the present application, the electronic device combines the coordinates of the click position and the first image, and based on the coordinates of the click position, the segmentation mask of the target in the first image can be accurately obtained, so that the bounding box determined by the electronic device is more accurate.

[0010] With reference to the first aspect, in a possible implementation, the bounding box is a bounding rectangular box of the target.

[0011] With reference to the first aspect, in a possible implementation, the bounding box is determined based on the segmentation mask, comprising: the electronic device determines a bounding rectangular box based on a maximum position point and a minimum position point of the segmentation mask in a first direction and a maximum position point and a minimum position point of the segmentation mask in a second direction, the first direction being perpendicular to the second direction; and the bounding rectangular box is a rectangle surrounded by a straight line along the first direction and passing through the maximum position point in the first direction, a straight line along the first direction and passing through the minimum position point in the first direction, a straight line along the second direction and passing through the maximum position point in the second direction, and a straight line along the second direction and passing through the minimum position point in the second direction.

[0012] In the embodiments of the present application, the circumscribed rectangle is used as the circumscribed frame of the target, which can reduce the calculation amount of the electronic device, so that the electronic device can respond faster.

[0013] Optionally, the circumscribed rectangle can be a minimum circumscribed rectangle. Since the size of the minimum circumscribed rectangle is closer to the size of the target, the template image of the target obtained is more accurate.

[0014] In combination with the first aspect, in a possible implementation, the electronic device determines the segmentation mask of the target based on the coordinates of the click position and the first image, including: the electronic device encodes the coordinates of the click position to obtain encoded coordinates; the electronic device up-samples the encoded coordinates to obtain a coordinate feature map, which has the same size as the first image; and the electronic device inputs the coordinate feature map spliced with the first image into an interactive segmentation model to obtain the segmentation mask.

[0015] In the embodiments of the present application, the coordinate feature map obtained by encoding and up-sampling the coordinates of the click position has the same size as the first image, which can realize the splicing of the coordinates of the click position and the first image. The electronic device splices the coordinate feature map and the first image before inputting them into the interactive segmentation model, which realizes the early fusion of the two. Then, the spliced feature map is input into the interactive segmentation model, and the coordinate feature map and the first image after splicing are processed by the interactive segmentation model to obtain the segmentation mask of the target clicked by the user. This can improve the accuracy of the obtained segmentation mask, and further make the cropping frame obtained based on the segmentation mask more accurate.

[0016] In combination with the first aspect, in a possible implementation, the interactive segmentation model is trained with the spliced sample image and the coordinate feature map corresponding to the sample image as input, and a preset segmentation mask as a label, the preset segmentation mask being a region where the pixel points of the first target in the sample image are located; and the coordinate feature map corresponding to the sample image is obtained based on the coordinates of the position of the first target in the sample image.

[0017] In the embodiments of the present application, the interactive segmentation model can be trained with the spliced sample image and the coordinate feature map corresponding to the sample image as input, and a preset segmentation mask as a label, which can improve the accuracy of the interactive segmentation model, and make the interactive segmentation model more accurately identify the segmentation mask of the target clicked by the user.

[0018] In combination with the first aspect, in a possible implementation, before the electronic device displays the first user interface, the method further includes: the electronic device displays a second user interface, the second user interface including a follow-all switch control; and the electronic device, in response to a user operation input for the follow-all switch control, opens the function of prompting the target to follow the click, the user operation being used to instruct the electronic device to open the function of prompting the target to follow the click.

[0019] In the embodiments of the present application, the second user interface of the electronic device can include an all-things pursuit switch control, so that the user can select whether to start the function of clicking the target for pursuit, prevent false touch, and improve user experience.

[0020] In combination with the first aspect, in a possible implementation, the electronic device pursues the target in the second image based on the template image, including: the electronic device displays the second image; and the electronic device displays a detection frame on the second image, the detection frame being used to indicate a position or a region of the target on the second image.

[0021] In the embodiments of the present application, the electronic device can display the detection frame on the second image to indicate the target clicked by the user, thereby improving user experience.

[0022] In combination with the first aspect, in a possible implementation, the electronic device pursues the target in the second image based on the template image, including: the electronic device performs feature extraction on the template image to obtain features of the template image; the electronic device performs feature extraction on the second image to obtain features of the second image; the electronic device finds a target region in the second image, the target region being a region in the features of the second image that has the highest similarity with the features of the template image; and the electronic device determines that a position corresponding to the target region in the second image is the target when a similarity between features in the target region and the features of the template image is greater than or equal to a preset value.

[0023] In the embodiments of the present application, the electronic device can determine whether the position corresponding to the target region in the second image is the target based on the preset value, so that the result of pursuit by the electronic device is more accurate.

[0024] In combination with the first aspect, in a possible implementation, the detection frame is displayed on the position corresponding to the target region on the second image, and the detection frame is a figure determined by the electronic device based on an edge of the target region.

[0025] In combination with the first aspect, in a possible implementation, the method further includes: when the similarity between the features in the target region and the features of the template image is less than the preset value, the electronic device stops pursuing the target.

[0026] In the second aspect, the embodiments of the present application provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the electronic device performs the method described in the first aspect and any possible implementation manner of the first aspect.

[0027] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions, when the instructions are executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes instructions, when the computer program product is executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0029] FIGS. 1A-1B are schematic diagrams of a shooting interface provided by an embodiment of the present application;

[0030] FIGS. 2A-2F are schematic diagrams of a user interface provided by an embodiment of the present application;

[0031] FIG. 3 is a schematic flow diagram of a target pursuit method provided by an embodiment of the present application;

[0032] FIG. 4 is a schematic diagram of a segmentation mask provided by an embodiment of the present application;

[0033] FIG. 5 is a schematic flow diagram of coordinate coding provided by an embodiment of the present application;

[0034] FIG. 6 is a schematic flow diagram of training an interactive segmentation model provided by an embodiment of the present application;

[0035] FIG. 7 is a schematic flow diagram of obtaining a segmentation mask provided by an embodiment of the present application;

[0036] FIG. 8 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application;

[0037] FIG. 9 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] In the embodiments of the present application, the word “exemplary” or “for example” is used to mean serving as an example, instance, or illustration. Any embodiment or design presented as “exemplary” or “for example” in the embodiments of the present application is not necessarily to be construed as preferred or advantageous over other embodiments or design solutions. In fact, a word “exemplary” or “for example” is used to present concepts in a concrete manner.

[0039] In the embodiments of the present application, unless otherwise specified, “multiple” means two or more. For example, multiple nodes mean two or more nodes. “At least one” means any number, for example, one, two, or more.

[0040] "A and / or B" can be only A, only B, or include A and B. "At least one of A, B, and C" can be only A, only B, only C, or include A and B, include B and C, include A and C, or include A, B, and C. The terms "first", "second", "third", "fourth" and the like in this application are only used to distinguish different objects, and are not used to indicate priority or importance of the objects.

[0041] The embodiments of the present application are described below with reference to the accompanying drawings of the embodiments of the present application.

[0042] In order to improve the accuracy of the target pursuit prompted by the click, the embodiments of the present application provide a target pursuit method. In this method, when determining the cropping frame, the electronic device can input the coordinate feature map obtained by encoding the coordinates of the user's click position on the current image (also referred to as the first image) and the feature map obtained by splicing the first image into an interactive segmentation model, and obtain the segmentation mask of the target clicked by the user through the interactive segmentation model. Further, the bounding box of the target is determined based on the segmentation mask, and the cropping frame is set as the bounding box, so that the size of the cropping frame is closer to the size of the target to be pursued, and the template image obtained by cropping is more accurate. With the accurate template image, the target pursuit in the subsequent frame can improve the accuracy of target pursuit and reduce the probability of target mis-pursuit and target missed-pursuit.

[0043] Further, the embodiments of the present application provide that after obtaining the template image, the features of the template image are compared with the features of the image (also referred to as the second image) subsequent to the first image, to find the region (also referred to as the target region) in the features of the first image that has the highest similarity with the features of the template image, and identify the position corresponding to the target region in the second image as the target. This method can achieve the pursuit of the target based on the above feature comparison method regardless of the type of target clicked by the user, and thus can achieve the pursuit of all things.

[0044] The application scenario of the target pursuit method provided by the embodiments of the present application is exemplarily described below in combination with the user interfaces shown in FIGS. 2A-2F.

[0045] (1) The user opens the camera application

[0046] When the electronic device displays the desktop, the user can open the camera application. The electronic device detects the user operation of opening the camera application, and in response to the user operation, displays the preview interface of the camera application.

[0047] The user interface 21 shown in FIG. 2A is a desktop displayed by an electronic device according to an embodiment of the present application. The user interface 21 can include at least one application icon (for example, a browser icon, a video icon, a phone icon, a settings icon, and a camera icon 210a).

[0048] The user can click the icon 210a of the camera, and the electronic device, upon detecting a user operation on the icon 210a of the camera, in response to the user operation, can open the camera application, display a preview interface of the camera application, which can be the user interface 22 as shown in FIG. 2B.

[0049] It should be noted that the interface diagram of the electronic device shown in FIG. 2A is an exemplary display of the embodiments of the present application, and the interface diagram of the electronic device can also be in other styles, and the positions of the icons of the applications and the names of the corresponding applications can be adjusted according to the preferences of the user, which is not limited in the present application.

[0050] (2) The user opens the click prompt target pursuit function

[0051] When the electronic device displays the preview interface of the camera application, the user can open the click prompt target pursuit function.

[0052] Exemplarily, as shown in FIG. 2B, FIG. 2B is a user interface 22 provided by an embodiment of the present application. In the user interface 22, the electronic device can display the specific controls as shown in FIG. 2A, which will be described in detail as follows:

[0053] The preview area 220 is used to display the image captured by the camera.

[0054] The all-pursuit switch control 220a is used to implement the function of opening the click prompt target pursuit.

[0055] The shooting mode menu 221 can include options of aperture, night scene, person, photograph 221a, video recording, movie, and professional, etc. Different camera modes can implement different shooting functions. The camera mode pointed to by the "triangle" in the shooting mode menu 221 is used to indicate the initial or user-selected camera mode. As shown in FIG. 2B, the "triangle" points to "photograph", indicating that the current camera is in the photograph mode.

[0056] The gallery control 222a is used for the user to view the photographed pictures and videos. Specifically, the electronic device, in response to a user operation on the gallery control 222a, displays the photographed pictures and videos.

[0057] The shooting control 222b is used to implement the shooting of images or the recording of videos. Specifically, the electronic device, in response to a user operation on the shooting control 222b, can shoot images, start recording videos, or end recording videos.

[0058] The conversion camera control 222c is used to implement the switching between the front camera and the rear camera.

[0059] FIG. 2B illustrates an example of the all things pursuit switch control 220a being in an off state. In turn, as shown in FIG. 2B, the user can click the all things pursuit switch control 220a, and in response to the user operation, the electronic device starts the click prompt target pursuit function, and displays the user interface 23 shown in FIG. 2C. FIG. 2C is a schematic diagram of a user interface photographed when the click prompt target pursuit function is started.

[0060] It should be noted that the user interface of the electronic device shown in FIG. 2B is an example of the present application, and the user interface of the electronic device can also be in other styles, which are not limited in the present application.

[0061] It should be noted that the display of the pattern and position of the all things pursuit switch control described above is an example of the present application, which is not limited in the present application; after the user opens the camera and enters the shooting interface, the all things pursuit switch can be in an on state or an off state.

[0062] (3) The user selects a target to be pursued

[0063] As shown in FIG. 2C, the all things pursuit switch control 230a in the user interface 23 is in an on state, at this time, the user can click a target such as a "puppy" in the image currently displayed in the preview area 230, and the electronic device can respond to the user operation, and can input the coordinates of the user's click position on the current image (also referred to as a first image) and the feature map after splicing the first image into an interactive segmentation model, to obtain a segmentation mask of the target clicked by the user through the interactive segmentation model, and then determine a cropping frame based on the segmentation mask, and then display the cropping frame on the current image, as shown in the user interface 24 in FIG. 2D. Compared with the user interface 23, the user interface 24 shows a cropping frame 240a at the click position, and the cropping frame 240a is used to obtain a template image of the target clicked by the user.

[0064] (4) The electronic device pursues the target in the subsequent image

[0065] After the electronic device displays the first image described above, the electronic device continuously captures images through the camera, the image displayed in the preview area is updated in real time, and the target "puppy" is pursued in real time, and a detection frame for indicating the position of the target is displayed in real time in the subsequent image, to prompt the user of the position or area of the target in the image.

[0066] Exemplarily, the electronic device displays a user interface 25 as shown in FIG. 2E, in which the position of the target "puppy" in the image displayed by the user interface 25 changes. Specifically, a subsequently acquired image of the first image can be displayed, taking the second image acquired subsequently as an example. When the electronic device is tracking the puppy in the preview area, the puppy is located at the upper left side of the zebra crossing. When the puppy moves in the subsequent frames captured by the electronic device, the detection frame 250a continues to track the puppy as the puppy moves, so that the detection frame 250a is always attached to the puppy.

[0067] Exemplarily, the detection frame 250a changes in size as the target "puppy" changes in size, as shown in FIG. 2E, when the puppy moves from the upper left side of the zebra crossing to the lower left side, the puppy is close to the camera, the image of the puppy becomes larger, and the detection frame 250a becomes larger as the target "puppy" becomes larger.

[0068] FIG. 2F exemplarily shows a user interface 26, as shown in FIG. 2F, when the puppy leaves the road, the puppy in the preview area of the electronic device disappears, that is, the target object in the preview area disappears, and the electronic device can stop tracking the target "puppy". The electronic device can also display prompt information in the currently displayed interface to prompt that the target "puppy" disappears.

[0069] It should be noted that the target tracking method provided in the embodiments of the present application can also be applied in the recording interface or the playing interface of the electronic device. Exemplarily, the user can click a target in the image captured by the camera and displayed in the recording interface or the image played in the playing interface, and the electronic device responds to the user operation to track the target clicked by the user.

[0070] The following describes a target tracking method provided in the embodiments of the present application.

[0071] FIG. 3 exemplarily shows a target tracking method provided in the embodiments of the present application, which can include the following steps or all the steps:

[0072] S101: The electronic device displays a user interface UI1, and the user interface UI1 includes a universal tracking switch control.

[0073] Exemplarily, the user interface UI1 can be the user interface 22 as shown in FIG. 2B, and the user interface UI1 can display the image currently captured by the camera.

[0074] S102: The electronic device receives a user operation input to the universal tracking switch control, and in response to the user operation, the function of clicking to prompt target tracking is enabled.

[0075] The above user operation is used to instruct the electronic device to enable the function of clicking to prompt target tracking.

[0076] Exemplarily, as shown in FIG. 2B, the user can click the everything pursuit switch control 220a, and accordingly, the electronic device, in response to the user operation, starts the function of clicking to prompt target pursuit, and displays the user interface 23 as shown in FIG. 2C.

[0077] It should be noted that after the electronic device starts the function of clicking to prompt target pursuit, the electronic device can perform part or all of the following S103-S114.

[0078] It should be understood that in the embodiments of the present application, the user operation can be a touch operation of the user, or a non-contact operation (for example, a gesture in the air), or a voice instruction of the user, and the embodiments of the present application do not make specific limitation on this.

[0079] S103: The electronic device displays a user interface UI2, which includes a current preview image, also referred to as a first image.

[0080] Exemplarily, the user interface UI2 can be the user interface 23 as shown in FIG. 2C, and the image displayed in the preview area of the user interface 23 is the first image.

[0081] S104: The electronic device acquires the coordinates of the clicking position and the first image in response to the clicking operation input on the target on the first image.

[0082] The coordinates of the clicking position are the coordinates of the position clicked by the user in the first image, including the horizontal coordinate x and the vertical coordinate y of the clicking position.

[0083] It should be noted that the image is collected by the corresponding sensor of the electronic device, and the format of the image can be an RGB (red green blue, RGB) format image, or a YUV format image, and the embodiments of the present application do not make limitation on this.

[0084] S105: The electronic device determines the segmentation mask of the target based on the coordinates of the clicking position and the first image.

[0085] The segmentation mask of the target is used to indicate the region where the pixel point of the target in the first image is located. Exemplarily, as shown in FIG. 4, the first image and the segmentation mask of the target “puppy” in the first image, wherein the black region in the segmentation mask of the target “puppy” is used to indicate the region where the target “puppy” is located.

[0086] In an implementation, the electronic device can encode the coordinates of the click position, up-sample the encoded coordinates to obtain a coordinate feature map of the coordinates, and input the coordinate feature map and the first image into the interactive segmentation model after splicing to obtain the segmentation mask map of the target. The specific implementation of the interactive segmentation model can refer to the training method of the interactive segmentation model described below, and the specific implementation of step S105 can refer to the method described in FIG. 7 below.

[0087] S106: The electronic device determines a crop frame of the target in the segmentation mask map based on the segmentation mask map.

[0088] The crop frame refers to the circumscribed frame of the target calculated based on the segmentation mask map, which is used to crop the first image to obtain a template image containing the target.

[0089] Based on different geometric shapes, the shape of the circumscribed frame can be a rectangle, an ellipse, a polygon, etc. Among them, when the circumscribed frame is displayed on the first image, the circumscribed frame surrounds the target in the first image, and the edges of the circumscribed frame have at least three intersection points with the edges of the target, and the three intersection points are not on the same straight line. Since the circumscribed frame has a small difference from the size of the target, the template image obtained by taking the circumscribed frame as the crop frame is more accurate.

[0090] Exemplarily, the shape of the circumscribed frame is a rectangle, also known as a circumscribed rectangular frame. Optionally, at this time, the crop frame can be the maximum circumscribed rectangular frame or the minimum circumscribed rectangular frame of the target.

[0091] The electronic device can determine the size and position of the target in the first image through the above crop frame. In some embodiments, the electronic device can determine the maximum position point (also referred to as point A) and the minimum position point (also referred to as point B) of the segmentation mask map in a first direction, and determine the maximum position point (also referred to as point C) and the minimum position point (also referred to as point D) of the segmentation mask map in a second direction, wherein the first direction is perpendicular to the second direction, and then determine the circumscribed rectangular frame of the target as the rectangle surrounded by the straight line along the first direction and passing through the point A, the straight line along the first direction and passing through the point B, the straight line along the second direction and passing through the point C, and the straight line along the second direction and passing through the point D.

[0092] As described above, the first direction can be any direction on the surface of the segmentation mask map. It can also be that the first direction can be parallel to the x-axis of the first image, that is, the direction corresponding to the x-axis parallel to the x-coordinate x of the first image.

[0093] In some embodiments, the circumscribed rectangular frame can be represented by the coordinates of the points A, B, C, and D corresponding to the first image, and can also be represented by the coordinates of the center point of the circumscribed rectangular frame and the width and height of the circumscribed rectangular frame.

[0094] In another implementation, the circumscribed rectangle can be derived according to a rotating bounding box algorithm or the like, which is not limited in the embodiments of the present application.

[0095] It should be noted that, in the embodiments of the present application, the circumscribed rectangle obtained by the electronic device is taken as an example for illustration, and the circumscribed rectangle obtained by the electronic device can also be other circumscribed rectangles, which are not limited in the embodiments of the present application.

[0096] S107: The electronic device displays the circumscribed rectangle in the region corresponding to the circumscribed rectangle on the first image.

[0097] The region corresponding to the circumscribed rectangle is a region in the first image corresponding to the circumscribed rectangle on the segmentation mask image.

[0098] For example, as shown in FIG. 2D, the electronic device can display the circumscribed rectangle 240a at the position of the puppy in response to the user operation of clicking the puppy.

[0099] S108: The electronic device crops the region corresponding to the circumscribed rectangle on the first image to obtain a template image.

[0100] The template image is an image in the region corresponding to the circumscribed rectangle on the first image.

[0101] In some embodiments, the electronic device can crop the region corresponding to the circumscribed rectangle on the first image, and take the image of the region as a template image for the electronic device to pursue the target at the next moment.

[0102] S109: The electronic device extracts features of the template image to obtain features of the template image.

[0103] The electronic device can put the template image into a model such as a convolutional neural network to extract features of the template image by stacking multiple convolutional layers and pooling layers, which are not limited in the embodiments of the present application.

[0104] S110: The electronic device extracts features of the second image to obtain features of the second image.

[0105] The electronic device can put the second image into a model such as a convolutional neural network to extract features of the template image by stacking multiple convolutional layers and pooling layers.

[0106] In some embodiments, the model for extracting features of the template image can share parameters with the model for extracting features of the second image to increase the running speed.

[0107] The second image is an image obtained by the electronic device after displaying the first image. In some embodiments, the second image can be an image obtained by the electronic device at the first frame after displaying the first image.

[0108] S111: The electronic device finds the target region in the features of the second image.

[0109] The target region is a region in the features of the second image that has the highest similarity to the features of the template image.

[0110] The similarity of a region in the features of the second image to the features of the template image is used to measure the degree of similarity between the features in the region and the features of the template image. The higher the similarity, the more similar the features of the region are to the features of the template image. In some embodiments, the electronic device can find the target region by comparing the features of the template image to the features in multiple regions in the features of the second image respectively, calculating the similarity of the features in the multiple regions to the features of the template image respectively, and finding the region in the multiple regions that has the highest similarity to the features of the template image.

[0111] For example, the multiple regions can be all regions, and the electronic device can find the region in the second image that has the highest similarity to the features of the template image by using a sliding window method. The electronic device can slide the window from left to right and from top to bottom in the features of the second image by using windows of different sizes, obtain the features in each window, calculate the similarity of the features in each window to the features of the template image, and find the region in the features of the second image that has the highest similarity to the features of the template image, i.e., the target region.

[0112] It should be noted that the similarity of the features of the template image to the features of the multiple regions in the features of the second image can be calculated by a similarity metric (such as Euclidean distance, cosine similarity, etc.), and the method of measuring the similarity is not limited in the embodiments of the present application.

[0113] S112: The electronic device determines whether the similarity of the features in the target region to the features of the template image is greater than or equal to a preset value. If yes, S113 is performed, and if no, S114 is performed.

[0114] The electronic device can set the preset value to determine whether the second image contains the target. If the similarity is greater than the preset value, the second image contains the target, and the electronic device displays a detection frame in the region corresponding to the target region in the features of the second image. The detection frame is used to indicate the position or region of the target in the second image.

[0115] If the similarity is less than the preset value, the electronic device does not detect the target in the second image, and the electronic device stops searching for the target, i.e., S114 is performed.

[0116] In some embodiments, the electronic device can obtain the bounding box based on the edge of the target region. Specifically, the bounding box can be a graphic such as a rectangle determined by the electronic device based on the edge of the target region. The bounding box is used to indicate the position or region of the target on the second image.

[0117] It should be noted that the preset value can be greater than or equal to 0.5 and less than 1, such as 0.7, 0.8, 0.95, etc. Here, the value range of the similarity is taken as an example to illustrate the value range of 0-1.

[0118] S113: The electronic device displays the bounding box on the region corresponding to the target region in the second image, and performs S110.

[0119] For example, as shown in the user interface 25 in FIG. 2E, when the puppy moves in the subsequent frames photographed by the electronic device, if the electronic device finds the region with the highest similarity to the features of the template image in the features of the second image and displays the bounding box 250a at the puppy.

[0120] S114: The electronic device stops pursuing the target.

[0121] For example, as shown in the user interface 26 in FIG. 2F, when the puppy disappears in the current image, the electronic device stops pursuing the target, and the electronic device stops displaying the bounding box.

[0122] In other embodiments, after the electronic device stops pursuing the target, the electronic device can also continue to display the bounding box in the region corresponding to the last frame image before the target disappears. The embodiments of the present application do not limit this.

[0123] Before introducing the specific implementation of S105, the interactive segmentation model involved and its training method are first described.

[0124] The training device can obtain a plurality of sample images and a coordinate feature map corresponding to each sample image. The coordinate feature map is obtained by encoding the coordinates of the position of the sample target in the sample image and then upsampling.

[0125] The training device can vectorize encode the coordinates of the position of the sample target in the sample image, and perform upsampling to obtain a coordinate feature map. For example, FIG. 5 is a flowchart of coordinate encoding provided by an embodiment of the present application. As shown in FIG. 5, the training device can first vectorize encode the coordinates (x, y) of the position of the sample target in the sample image to obtain a first vector and a second vector of d dimensions, where the first vector is a vector obtained by encoding the coordinate x, and the second vector is a vector obtained by encoding the coordinate y. Then, the training device groups the first vector and the second vector into a matrix of 2 rows and d columns. Finally, the matrix is upsampled to the same size as the sample image to obtain a sample coordinate matrix map. Wherein, d is a positive integer greater than 1.

[0126] In some embodiments, the training device can vectorize encode the coordinates of the position of the sample image by word embedding vectorization encoding, and the training device can also vectorize encode by one-hot encoding and the like, which is not limited by the embodiments of the present application.

[0127] In some embodiments, the interaction segmentation model can be a convolutional neural network, a deep neural network, a residual neural network, etc. For example, the interaction segmentation model can include a feature down-sampling module and a feature up-sampling module, as shown in FIG. 6, the interaction segmentation model can be used to process the concatenated feature map through the feature down-sampling module and the feature up-sampling module to obtain a segmentation mask map of the sample target. Wherein, the feature down-sampling module is used to extract the features of the concatenated feature map, which is used for perceptual understanding of the sample target, and the feature up-sampling module is used to restore the small feature map of the down-sampling module to the image size to accurately locate the size and position of the sample target.

[0128] FIG. 6 is a flowchart of training an interaction segmentation model provided by an embodiment of the present application. As shown in FIG. 6, the training device can concatenate the sample image and the coordinate feature map corresponding to the sample image and input it into the interaction segmentation model to be trained, process it through the interaction segmentation model, and obtain a segmentation mask map of the sample target.

[0129] In some embodiments, the interaction segmentation model is trained by taking the concatenated feature map of the sample image and the coordinate feature map corresponding to the sample image as sample data, and taking a preset segmentation mask map as a label. A defined loss function is used to calculate the loss between the segmentation mask map output by the trained interaction segmentation model and the preset segmentation mask map, and the preset segmentation mask map is the region where the pixel points of the sample target in the sample image are located. Based on the loss function value, the parameters of the current trained interaction segmentation model are adjusted through a back propagation algorithm until the loss function value converges to a preset value, and the training of the interaction segmentation model is ended. In this way, the interaction segmentation model is trained.

[0130] As shown in FIG. 7 is a flowchart of a specific implementation of S105 provided by the embodiment of the present application, i.e., the electronic device obtaining the segmentation mask of the target based on the coordinates of the click position and the first image. S105 can include the following steps or all of them.

[0131] S201: The electronic device encodes the coordinates of the click position and up-samples the encoded coordinates to obtain a coordinate feature map.

[0132] The electronic device can vectorize the horizontal coordinate x and the vertical coordinate y of the click position and map the vectorization to obtain the encoded coordinates as shown in FIG. 7, i.e., the coordinate encoding matrix. After obtaining the coordinate encoding matrix, the electronic device can up-sample the coordinate encoding matrix to the same size as the first image to obtain the coordinate feature map. The specific implementation is the same as the process of encoding and up-sampling the position of the sample target in the sample image, which will not be described here.

[0133] S202: The electronic device concatenates the coordinate feature map with the first image to obtain a concatenated feature map (also referred to as a target feature map).

[0134] For example, as shown in FIG. 7, when the first image is an RGB format image, the first image has 3 channels. The electronic device can concatenate the 3 channels of the RGB format image with the coordinate feature map to combine the information of the user click position with the image information, so that the interactive segmentation model can consider both kinds of information at the same time, thereby improving the accuracy of the interactive segmentation model. In this way, the interactive segmentation model can better understand the relationship between the user click position and the image content.

[0135] It should be noted that the image format is not limited to the RGB format, and the image format can also be YUV, etc., which is not limited by the embodiment of the present application.

[0136] S203: The electronic device inputs the concatenated feature map into the interactive segmentation model to output the segmentation mask of the target.

[0137] The interactive segmentation model can be a pre-trained machine learning algorithm model. The electronic device can extract the segmentation mask of the target from the first image based on the interactive segmentation model, obtain a template image used as the target to pursue subsequent frames based on the segmentation mask, and improve the accuracy of target pursuit.

[0138] The device related to the embodiment of the present application will be introduced below.

[0139] FIG. 8 is a hardware structure diagram of the electronic device provided by the embodiment of the present application. The electronic device can also be the training device described above.

[0140] As shown in FIG. 8, the electronic device 100 performs the method shown in FIGS. 2A-7, which can include a processor 110, an external memory interface 120, an internal memory 126, a camera 130, a display screen 140, an audio module 150, a speaker 150A, a receiver 150B, a microphone 150C, a headset jack 150D, a sensor module 160, and the like. The sensor module 160 can include a pressure sensor 160A, a distance sensor 160F, a proximity light sensor 160G, a touch sensor 160K, an ambient light sensor 160L, and the like.

[0141] The processor 110 can include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU), and the like. Different processing units can be independent devices or integrated into one or more processors.

[0142] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of fetching and executing instructions.

[0143] The processor 110 can also be provided with a memory for storing instructions and data.

[0144] In the embodiments of the present application, the processor 110 can be used to execute the various embodiments described above to achieve the target pursuit method provided by the present application.

[0145] The internal memory 126 can include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs).

[0146] The electronic device achieves display functions through a GPU, a display screen 140, and an application processor, and the like. The GPU is a microprocessor for image processing, connected to the display screen 140 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0147] The display screen 140 is configured to display images, videos, and the like. The display screen 140 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a miniled, a microLed, a micro-oled, a quantum dot light emitting diodes (QLED), and the like.

[0148] The electronic device can implement the photographing function through the ISP, the camera 130, the video codec, the GPU, the display screen 140, and the application processor, and the like.

[0149] The ISP is configured to process data fed back by the camera 130. The camera 130 is configured to capture still images or videos. In the embodiments of the present application, the camera 130 can be a telephoto camera or a main camera, a wide-angle camera, and the like.

[0150] The digital signal processor is configured to process digital signals. In addition to processing digital image signals, the digital signal processor can also process other digital signals. For example, when the electronic device selects a frequency point, the digital signal processor is configured to perform Fourier transform on the frequency point energy, and the like.

[0151] The video codec is configured to compress or decompress digital videos. The electronic device can support one or more video codecs. In this way, the electronic device can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and the like.

[0152] The NPU is a neural-network (NN) computing processor. By drawing on the structure of a biological neural network, for example, by drawing on the transmission mode between human brain neurons, the NPU can quickly process input information and can also constantly self-learn. Through the NPU, the electronic device can implement intelligent cognition applications, such as image recognition, face recognition, voice recognition, text understanding, and the like.

[0153] The electronic device can implement an audio function through the audio module 150, the speaker 150A, the receiver 150B, the microphone 150C, the earphone interface 150D, and the application processor, etc. For example, music playing, recording, etc.

[0154] The structure of the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device can include more or fewer components than those shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware. For example, the electronic device can further include a key, a motor, an indicator, and a subscriber identification module (SIM) card interface, etc. For another example, the sensor module can further include a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a fingerprint sensor, a temperature sensor, a bone conduction sensor, etc.

[0155] FIG. 9 is a schematic diagram of a software and hardware structure of the electronic device according to an embodiment of the present application.

[0156] As shown in FIG. 9, the software and hardware structure framework of the electronic device according to the present application can include an application layer, an application framework layer (FWK), a system library, a hardware abstraction layer (HAL), a kernel layer, and a hardware layer.

[0157] The application layer can include a series of application packages. As shown in FIG. 9, the application layer can include camera, gallery, WLAN, and setting applications (also referred to as applications). The application layer can further include calendar, call, map, navigation, Bluetooth, music, video, short message, etc. applications (not shown in the figure).

[0158] The gallery application is used to display images or videos, etc. in the electronic device.

[0159] The application framework layer provides an application programming interface (API) and a programming framework for the applications of the application layer. The application framework layer includes some pre-defined functions. The application framework layer can include a window manager, a content provider, a view system, and a resource manager, etc.

[0160] The runtime (Runtime) is responsible for the scheduling and management of the system. The runtime includes a core library and a virtual machine. The core library includes two parts: one part is a function function that a programming language (for example, java language) needs to call, and the other part is a core library of the system.

[0161] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the programming files (e.g., java files) of the application layer and the application framework layer into binary files. The virtual machine is used to perform functions such as management of object life cycle, stack management, thread management, management of security and exceptions, and garbage collection.

[0162] The system library can include a plurality of functional modules. For example, a surface manager, media libraries, a three-dimensional image processing library (e.g., OpenGL ES), a two-dimensional graphics engine (e.g., SGL), etc.

[0163] The surface manager is used to manage the display subsystem and provides fusion of two-dimensional (2-Dimensional, 2D) and three-dimensional (3-Dimensional, 3D) layers for a plurality of applications.

[0164] The media library supports playback and recording of a plurality of commonly used audio, video formats, and static image files, etc. The media library can support a plurality of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0165] The three-dimensional image processing library is used to implement 3D graphics drawing, image rendering, composition, and layer processing, etc.

[0166] The 2D graphics engine is a drawing engine for 2D drawing.

[0167] The hardware abstraction layer (HAL) is an interface layer between the operating system kernel and the upper layer software, and its purpose is to abstract the hardware. The hardware abstraction layer is an abstract interface of the device kernel driver, and is used to implement an application programming interface for providing access to the underlying device to a higher level Java API framework. The HAL includes a plurality of library modules, such as the camera HAL shown in FIG. 9 and the audio HAL, display screen, Bluetooth, etc. not shown in FIG. 9. Each of the library modules implements an interface for a specific type of hardware component. It can be understood that the camera HAL can provide an interface for a camera application to access hardware components such as a camera, and the camera HAL can include a video acquisition module. The video acquisition module is used to acquire images from the camera.

[0168] The kernel layer is the foundation of the Android operating system, and the ultimate functions of the Android operating system are completed through the kernel layer. The kernel layer can include display drivers, camera drivers, and audio drivers, etc.

[0169] The hardware layer can include a display, a microphone, a camera, a memory, an audio encoder, and a video encoder, and the like. The microphone collects raw audio data, the camera collects images, and the memory can store videos, images, and the like. The audio encoder is hardware for performing audio encoding, and the video encoder is hardware for performing video encoding.

[0170] The embodiments of the present application further provide a computer readable storage medium. All or part of the processes in the above method embodiments can be directed by computer instructions related hardware to complete, and the computer instructions can be stored in the above computer readable storage medium. When the computer instructions are executed, part or all of the methods performed by the electronic device or the training device in the above method embodiments can be implemented. The computer readable storage medium can be the memory in the electronic device or the training device in any of the above embodiments. The computer readable storage medium can also be an external storage device on the above device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Further, the computer readable storage medium can include both the above internal storage unit and the external storage device. The computer readable storage medium is used to store the above computer program and other programs and data required. The computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0171] The embodiments of the present application further provide a computer program product, which contains a computer program. When the computer program product runs on a computer, it makes the computer execute part or all of the methods performed by the electronic device or the training device in the above embodiments.

[0172] In the above embodiments, all or part of the processes can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the processes can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state disk), etc.

[0173] Those of ordinary skill in the art can understand that all or part of the processes in the above-described embodiments can be implemented by a computer program instructing relevant hardware, which can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-described embodiments.

[0174] It should be noted that the flowcharts described in the embodiments of the present application are only one embodiment. The steps in each flowchart can be modified or changed in various ways without departing from the spirit of the present application, such as executing the steps in the flowchart in different order, or deleting, adding, or modifying some steps.

[0175] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A target pursuit method characterized by comprising: The method is applied to an electronic device, and the method comprises: The electronic device displays a first user interface, and the first user interface comprises a first image; The electronic device displays a bounding box of a target in the first image in response to a click operation input for the target in the first image; The electronic device crops a region corresponding to the bounding box on the first image to obtain a template image of the target; The electronic device searches for the target in a second image based on the template image, and the second image is an image obtained by the electronic device after the first image.

2. The method of claim 1, wherein, The electronic device displays a bounding box of a target in the first image in response to a click operation input for the target in the first image, comprising: The electronic device obtains the coordinates of the click position and the first image in response to the click operation input for the target in the first image; The electronic device determines a segmentation mask of the target based on the coordinates of the click position and the first image, and the segmentation mask is used to indicate a region where a pixel point of the target in the first image is located; The electronic device determines the bounding box based on the segmentation mask; The electronic device displays the bounding box on the first image.

3. The method of claim 2, wherein, The bounding box is a bounding rectangular box of the target.

4. The method of claim 3, wherein, The determination of the bounding box based on the segmentation mask comprises: The electronic device determines the bounding rectangular box based on maximum and minimum position points of the segmentation mask in a first direction and maximum and minimum position points of the segmentation mask in a second direction, the first direction being perpendicular to the second direction; the bounding rectangular box is a rectangle surrounded by a straight line along the first direction and passing through the maximum position point in the first direction, a straight line along the first direction and passing through the minimum position point in the first direction, a straight line along the second direction and passing through the maximum position point in the second direction, and a straight line along the second direction and passing through the minimum position point in the second direction.

5. The method according to any one of claims 2 to 4, wherein, The determination of the segmentation mask of the target based on the coordinates of the click position and the first image comprises: The electronic device encodes the coordinates of the click position to obtain encoded coordinates; The electronic device up-samples the encoded coordinates to obtain a coordinate feature map, and the coordinate feature map has the same size as the first image; The electronic device inputs the coordinate feature map spliced with the first image into an interactive segmentation model to obtain the segmentation mask.

6. The method of claim 5, wherein, The interactive segmentation model is trained by taking the spliced sample image and the coordinate feature map corresponding to the sample image as input and taking a preset segmentation mask as a label, and the preset segmentation mask is a region where a pixel point of a first target in the sample image is located. The coordinate feature map corresponding to the sample image is obtained based on the coordinates of the position of the first target in the sample image.

7. The method according to any one of claims 1 to 6, wherein Before the electronic device displays the first user interface, the method further comprises: The electronic device displays a second user interface, the second user interface including an everything pursuit switch control; The electronic device, in response to a user operation input for the everything pursuit switch control, opens a function of pursuing a click prompt target, the user operation being used to instruct the electronic device to open the function of pursuing the click prompt target.

8. The method according to any one of claims 1 to 7, wherein, The electronic device pursues the target in a second image based on the template image, including: The electronic device displays the second image; The electronic device displays a detection frame on the second image, the detection frame being used to indicate a position or an area of the target on the second image.

9. The method of claim 8, wherein, The electronic device pursues the target in a second image based on the template image, further including: The electronic device extracts features of the template image to obtain features of the template image; The electronic device extracts features of the second image to obtain features of the second image; The electronic device finds a target area in the second image, the target area being an area with the highest similarity to the features of the template image among the features of the second image; The electronic device determines that a position corresponding to the target area in the second image is the target when a similarity between features in the target area and the features of the template image is greater than or equal to a preset value.

10. The method of claim 9, wherein, The detection frame is displayed on the position corresponding to the target area in the second image, the detection frame being a graph determined by the electronic device based on edges of the target area.

11. The method of claim 9 or 10, wherein, The method further includes: The electronic device stops pursuing the target when the similarity between the features in the target area and the features of the template image is less than the preset value.

12. An electronic device, comprising: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to cause the electronic device to perform the method of any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The instructions, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1-11.

14. A computer program product, characterised in that, The computer program product, when executed on an electronic device, causes the electronic device to perform the method of any one of claims 1-11.

Images