Target detection method and electronic device

Abstract

The present application provides a method and electronic device for target detection, and relates to the field of image processing. The present application provides a method for target detection, including: obtaining a first image; inputting the first image into a first model, obtaining a detection frame of each object in the first image output by the first model, the first model is generated by performing knowledge distillation on the first student network using a first teacher network and a first sample set; according to the detection frame of each object in the first image, obtaining an object image of each object in the first image; according to each object image and a second model, determining the category information of each object in the first image, the second model is generated by performing knowledge distillation on the second student network using a second teacher network and a second sample set. By adopting the method in the present application, the electronic device can accurately identify objects in the image, and when a new object category is added, there is no need to retrain the model, which improves the flexibility of the electronic device in identifying new categories.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and electronic device for target detection. Background Art

[0002] The purpose of object detection is to find all the objects of interest in an image and determine the category and location of the object. With the continuous development of artificial intelligence technology, terminal devices have also widely used object detection technology. Usually, a mobile terminal (such as a mobile phone, vehicle-mounted equipment, etc.) is equipped with an object detection model, which is used to perform object detection on the input image.

[0003] However, due to the large number of parameters in the target detection model, the model is large in size, which increases the difficulty of deploying it on mobile terminals. At the same time, when new categories need to be added, a large number of images labeled with new categories are required to retrain the target detection model, which increases time and labor costs. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a target detection method and electronic device, so that the electronic device can accurately identify objects in an image. When new object categories are added, there is no need to retrain the model, which reduces the time and labor costs of the electronic device for target detection and improves the flexibility of the electronic device in identifying new categories.

[0005] In a first aspect, the present application provides a method for target detection, which is applied to a first electronic device, the method comprising: obtaining a first image; inputting the first image into a first model, obtaining a first result output by the first model, the first result comprising a detection box for each object in the first image, the detection box for each object being used to indicate a position and range of the object, the first model being generated after knowledge distillation is performed on a first student network using a first teacher network and a first sample set, the first teacher network being a Everything Segmentable Model (SAM), the first sample training set comprising a plurality of first sample images and a first verification image corresponding to the first sample image, the first verification image comprising a detection box for each object; obtaining an object image for each object in the first image according to the detection box for each object in the first image; determining category information for each object in the first image according to each object image and the second model, the second model being generated after knowledge distillation is performed on a second student network using a second teacher network and a second sample set, the second sample set comprising a second sample image and text information of an object category corresponding to the second sample image, the second teacher network comprising an image text model, the image text model being used to determine a correspondence between image features and text features.

[0006] In this way, the first electronic device makes the first model deployed in the first electronic device a lightweight model by means of knowledge distillation, and at the same time, the first model also retains the accuracy of any object in the segmented image of SAM, thereby improving the accuracy of the detection frame of each object in the image determined by the first electronic device. In addition, the first electronic device obtains the second model by means of knowledge distillation of the second training model using the second teacher network, so that the deployed second model is a lightweight model, while retaining the accuracy of the second teacher network in identifying the category of each object image. Since the second teacher network calculates the correspondence between image features and text, that is, the second model is also used to determine the correspondence between image features and text, therefore, when there is text information of a newly added object category, there is no need to retrain the second model, and only the text information of the newly added object category needs to be re-entered in the first electronic device, thereby improving the flexibility of target detection and reducing the time cost and manpower cost of updating the identifiable category of target detection. In addition, the first model and the second model are deployed in the first electronic device to perform target detection on the image. The first model can be used to segment any object in the image, thereby determining the detection frame of each object, thereby increasing the types of objects that can be detected by the first electronic device. The second model determines the category of each object. In this example, the segmentation and type determination of the object are handled by two models, reducing the difficulty of training the target detection model.

[0007] According to the first aspect, before determining the category information of each object in the first image based on each object image and the second model, the method also includes: storing the newly added text features sent by the second electronic device, the newly added text features being the features obtained after the second electronic device performs feature extraction on the text information of the newly added category; determining the category information of each object in the first image based on each object image and the second model, including: the second model obtains the image features of each object image based on each input object image; the second model obtains the similarity between each object image and each text feature based on the stored text features and the image features of each object image; and determining the category information of the object in each object image based on the text features corresponding to the maximum similarity in each object image.

[0008] In this way, the first electronic device does not need to extract features from the text information of the newly added category, and can directly store the newly added text features sent by the second electronic device, thereby increasing the speed at which the first electronic device determines the category information of each object image.

[0009] According to the first aspect, before determining the category information of each object in the first image according to each object image and the second model, the method further includes: storing new object category text information sent by the second electronic device or new object category text information input by the user; determining the category information of each object in the first image according to each object image and the second model, including: the second model obtains the image feature of each object image and the text feature of each object category text information according to each object image input and the stored object category text information; the second model obtains the similarity between each object image and each text feature according to each text feature and the image feature of each object image; and determines the category information of the object in each object image according to the text feature corresponding to the maximum similarity in each object image. In this way, the first electronic device can receive the text information of the new object category input by the user or the new object category text information sent by the second electronic device, and the second model extracts the features of the new object category text information, and the second model determines the category of the object in each object image by obtaining the similarity between each object image and each text feature. This method does not require retraining the second model, thereby improving the flexibility of target detection.

[0010] According to the first aspect, the method further includes: displaying a preview interface of the first image; in response to a first selection operation of the user on the preview interface of the first image, determining a first target object selected by the user in the first image according to the detection frames of each object in the first image; acquiring and storing category information of the first target object for the first electronic device to determine the object to be focused in the next frame of the image; displaying the detection frame of the first target object in the preview interface of the first image, and performing a focusing operation with the first target object as the focus. In this way, the first electronic device can determine the object to be focused in the first image by detecting the detection frames of each object in the first image and the first selection operation input by the user, perform a focusing operation on the first target object and display the detection frame of the first target object in the first image, so as to prompt the user of the object to be focused, and store the category information of the first target object at the same time, so as to improve the accuracy of determining the target object in the next frame of the image.

[0011] According to the first aspect, the method further includes: when the first electronic device captures the second image, acquiring the second image, the first image being the previous frame of the second image; inputting the second image into the first model, acquiring the second result output by the first model, the second result including the detection frame of each object in the second image; acquiring the image of each object in the second image according to the detection frame of each object in the second image; determining the category information of each object in the second image according to the object image of each object in the second image and the second model; determining the second target object in the second image according to the detection frame and category information of the first target object in the first image, and the detection frame of each object in the second image and the category information of each object, the second target object is the same object as the first target object; displaying the detection frame of the second target object in the preview interface of the second image, and performing a focusing operation with the second target object as the focus. In this way, the second image is the next frame of the first image, and the first electronic device determines the target object in the second image by the category and detection frame of the first target object in the first image. Since the target object in the second image is determined by using dual conditions, the accuracy of determining the object to be focused can be improved.

[0012] According to the first aspect, the method further includes: displaying a detection frame of a target object on a preview interface of the current frame image, the target object being a first target object; in response to a zoom-in operation on the preview interface of the current frame image, determining a target focal length for shooting the next frame image according to the position and range of the detection frame of the target object in the current frame image, so that the first electronic device can fully display the detection frame of the target object in the image shot according to the target focal length, and the target focal length is greater than the focal length for shooting the current frame image. In this way, the first electronic device can accurately determine the target focal length of the first frame image according to the detection frame of the first target object, avoiding the situation where the target object is not fully displayed after zooming in on the focal length.

[0013] According to the first aspect, the method further includes: displaying a preview interface of the first image; obtaining a detection frame of each object in the first image and the category information of each object; determining the scene category of shooting the first image according to the category information and the detection frame of each object; determining a shooting strategy for the first image according to the scene category, the shooting strategy is used to indicate image processing of the first image, and the image processing includes: adjusting white balance and adjusting brightness. In this way, the first electronic device determines the scene category through the dual conditions of the detection frame and the category information, so that the determined scene category of the first image is more accurate, which facilitates the electronic device to perform different image processing for different scene categories, so that the processed image better meets the needs of the user.

[0014] According to the first aspect, the method further includes: displaying a preview interface of the first image; in response to a blur operation input by the user, determining a first target object in the first image according to each object detection frame and category information in the first image; blurring the object image outside the detection frame of the first target object according to the detection frame and category information of the first target object. In this way, the first electronic device can quickly blur the image outside the first target object.

[0015] According to the first aspect, the second target object in the second image is determined based on the detection frame and category information of the first target object in the first image, and the detection frame of each object in the second image and the category information of each object, including: searching for an object of the same category as the first target object in the second image according to the category information of the first target object in the first image and the category information of each object in the second image; obtaining the detection frame with the greatest similarity to the detection frame of the first target object according to the detection frame of the first target object in the first image and the detection frame of the found object as the detection frame of the second target object. In this way, the first electronic device can quickly screen out objects of different categories from the first target object through the category information of the first target object, reduce the scope of the target object in the second image to be determined subsequently, and then compare the detection frame of the found object with the detection frame of the first target object to determine the second target object, which can improve the accuracy of the second target object to be determined.

[0016] In a second aspect, the present application provides an electronic device comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the target detection method corresponding to the first aspect and any one of the implementation methods of the first aspect.

[0017] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0018] In a third aspect, the present application provides a computer-readable medium for storing a computer program. When the computer program is run on an electronic device, the electronic device executes the target detection method corresponding to the above-mentioned first aspect and any one of the implementation methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 is a schematic diagram exemplarily illustrating an application scenario of target detection;

[0021] Figure 2 is a schematic diagram of the structure of an electronic device shown exemplarily;

[0022] Figure 3 is a software structure block diagram of an electronic device exemplarily shown;

[0023] Figure 4 is a flowchart of an exemplary method for target detection;

[0024] Figure 5 is a schematic diagram of exemplarily showing the training of the first model;

[0025] Figure 6 is a schematic diagram of exemplarily showing training of a second model;

[0026] Figure 7 is a flowchart of an exemplary application of target detection;

[0027] Figure 8 is a schematic diagram showing exemplary processing of an image by a first model and a second model;

[0028] Fig. 9 is a schematic diagram showing an exemplary focus on a teddy bear;

[0029] Fig.10 is a schematic diagram showing an exemplary method of tracking focus on a dog;

[0030] Fig.11 is a schematic diagram showing an exemplary zoom focus operation;

[0031] Fig.12 is a schematic diagram showing an exemplary blurred subject image;

[0032] Fig.13 is a schematic diagram exemplarily illustrating image processing according to scene categories. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0035] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0036] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0037] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0038] Before describing the technical solution of the embodiment of the present application, the application scenario of target detection in the embodiment of the present application is first described in conjunction with the accompanying drawings. Figure 1 , user A starts the camera application in the mobile phone, and the camera application of mobile phone A is in preview state. When the camera application is in preview state, the real-time acquired image is transmitted to the image processing module, and the image processing module performs target detection processing on each frame of the image. Optionally, the camera application can perform target detection processing on each frame of the image by default in the preview state.

[0039] Specifically, the image processing module can detect each recognizable object in the current frame (such as the current frame is the first frame) image, determine the category and detection frame of each recognizable object (that is, the range and position of the recognizable object), and the detection frame of the recognizable object can cover the recognizable object, and the detection frame can be a preset shape, such as a rectangle.

[0040] Optionally, when the camera application detects a user's focusing operation (such as clicking an object on the display interface), the camera application can obtain the detection frame and category of each identifiable object in the current frame (such as the first frame) from the image processing module. The camera application can determine the selected target object in the current frame based on the detection frame of each identifiable object and the position of the click operation. The camera application can display the detection frame of the target object on the display interface, and the camera application can also perform a focusing operation on the target object. Figure 1 As shown in Figure 1a, if the image processing module does not recognize the detection frame of the teddy bear, even if the user clicks on the teddy bear on the display interface, the display interface cannot display the detection frame of the teddy bear.

[0041] Optionally, the image processing module may return the category and detection frame of each recognizable object in the first frame image to the camera application, and the camera application processes the first frame image according to the category and detection frame of each recognizable object. For example, in this example, the camera application displays the detection frame of each recognizable object in the preview interface of the first frame image according to the position information of the detection frame of each recognizable object in the first frame image. Figure 1 As shown in 1a in FIG. 1 , the preview interface of the camera displays the first frame image with the detection frame marked, and the image includes three objects, namely a dog, a cat, and a teddy bear. The image processing module recognizes the detection frames of the cat and the dog in the image, and returns the position information of the detection frames of the cat and the dog to the camera application, and the camera application displays the dog detection frame 102 and the cat detection frame 103 in the preview interface of the first frame image.

[0042] like Figure 1 As shown in Figure 1a, the image processing module fails to recognize the teddy bear, the camera application fails to receive the position information of the detection frame of the teddy bear, and cannot display the detection frame of the teddy bear 104 in the preview interface 101. Since the camera application cannot recognize the teddy bear, when the user selects to focus on the teddy bear, the camera application performs an inaccurate focusing operation on the teddy bear, such as the focusing position is 10 pixels away from the teddy bear.

[0043] In order to increase the types of objects that can be recognized by the image processing module, the electronic device usually needs to add a large number of images of objects in different scenes and of different types to retrain the object detection model. For example, the electronic device collects a large number of different scenes containing teddy bears and images containing different types of teddy bears as training data, such as Figure 1 As shown in 1b, the teddy bear in the collected image is labeled, and the target detection model is retrained so that the retrained model can recognize the teddy bear in different images, that is, determine the detection frame of the teddy bear in different images.

[0044] However, if there are objects that cannot be recognized by the target detection model in the electronic device, the number of recognizable objects needs to be increased. For each additional type of recognizable object, the electronic device needs to re-collect a large number of images containing new objects, annotate the new objects in the collected images, and update the target detection model based on the re-collected images and annotated images, which consumes a lot of manpower and time costs and is not conducive to the application and update of the target detection model.

[0045] At the same time, the target detection models deployed in mobile terminals usually have inaccurate detection accuracy and find it difficult to accurately identify the category, range and location of each object.

[0046] The embodiment of the present application provides a method for target detection, which is performed by an electronic device, which may be a mobile phone, a tablet computer, a camera, and the like. The electronic device may acquire a first image, transfer the first image to a first model, determine a detection frame of each object in the first image, and acquire an object image of each object in the first image according to the detection frame of each object in the first image; acquire the category information of each object in the first image according to each object image and a second model; the first model is used to detect the position and range of each object in the image, and the second model is used to determine the category information of each object according to each object image. Optionally, the first model is generated by performing knowledge distillation on the first student network using the first teacher network and the first sample set, the first teacher network is the SAM, the first sample training set includes multiple first sample images and a first verification image corresponding to the first sample image, and the first verification image includes a detection frame of each object. The second model is generated by performing knowledge distillation on the second student network using the second teacher network and the second sample set, the second sample set includes a second sample image and text information of the object category corresponding to the second sample image, the second teacher network is an image text model, and the image text model is used to determine the correspondence between image features and text features.

[0047] In this example, the first student network is distilled by the first teacher network, so that the first model obtained is small in size and easy to deploy in the mobile terminal. Similarly, the second model is generated by distilling the second student network based on the second teacher network. The distillation method makes the second model small in size and easy to deploy in the mobile terminal. At the same time, since the second teacher network is an image text model, the second model can also be paired according to the input text information and image. When there are new recognition types, the number of categories that can be recognized by the mobile terminal can be increased by adding text information of new categories, which is convenient for updating the target detection function.

[0048] Figure 2 This is a schematic diagram of the structure of an electronic device 100 shown in an embodiment of the present application. It should be understood that: Figure 2 The electronic device 100 shown is only one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the drawings, may combine two or more components, or may have a different configuration of components. Figure 2 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0049] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0050] Figure 3 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0051] The layered architecture of the electronic device 100 divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into three layers, from top to bottom, namely, the application layer, the application framework layer, and the kernel layer. It is understandable that Figure 3 The layers in the software structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which is not limited in the present application.

[0052] like Figure 3As shown, the application layer can include a series of application packages. The application package can include applications such as camera, chat application, gallery, calendar, short message, etc. The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some pre-defined functions.

[0053] like Figure 3 As shown, the application framework layer may include a window manager, a resource manager, a content provider, a view system, a phone manager, a notification manager, and the like.

[0054] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0055] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0056] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0057] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0058] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0059] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc.

[0060] The kernel layer is the layer between the hardware and the above software layers. The kernel layer contains at least display drivers, Wi-Fi drivers, Bluetooth drivers, audio drivers, sensor drivers, etc.

[0061] Figure 4 FIG. 1 is a flowchart of an exemplary target detection method. Figure 4 As shown, the target detection process includes:

[0062] Step 401: The electronic device acquires a first image.

[0063] Specifically, in this example, the electronic device is explained by taking a mobile phone as an example. The mobile phone can obtain images captured by the camera. For example, user A starts the camera application in the mobile phone, and when the camera application of mobile phone A is in the preview state, the captured image is transmitted to the image processing module in real time. In this example, the image processing module can be used to perform target detection processing on the acquired image, that is, to determine the category and detection frame of each object in the image. The image processing module may include a first model and a second model. The first model can detect the position and range (i.e., the detection frame) of objects of any category in the image; the second model is used to determine the category of each object based on the object image of each object.

[0064] In some examples, when the camera of the mobile phone is in the shooting state, the target detection process can be performed on each frame of the collected image by default, for example, the mobile phone is in the state of video chatting, recording a video, previewing a taken photo, etc.

[0065] Optionally, the electronic device may also obtain images sent by other devices (such as other mobile phones).

[0066] Step 402: The electronic device transmits the first image to the first model to determine a detection frame for each object in the first image.

[0067] Combine the following Figure 5 The training process of the first model is described in detail.

[0068] Specifically, the first teacher network can adopt the Segment Anything Model (SAM). The first teacher network can be trained in other electronic devices in advance, such as computers with stronger computing power. The training process of the first teacher network will not be repeated here. The trained first teacher network can be used to determine the position and range of any object in the image, to achieve the segmentation of any object, that is, to determine the detection frame of any object.

[0069] The network structure of the first student network can be the same as or different from the network structure of the first teacher network. Figure 5 As shown, the electronic device uses the first teacher network and the first training set to perform knowledge distillation on the first student network to obtain a first model. The knowledge distillation method can be offline distillation or semi-supervised distillation, and the specific distillation process is not repeated. In this example, the first training set includes multiple first sample images and first verification images corresponding to the first sample images, and each object in the first verification image is marked with a detection frame.

[0070] In the process of the first student network learning the knowledge of the first teacher network, the electronic device inputs the first sample image into the first student network and the first teacher network, and the first student network can iteratively train the first student network according to the difference between the output results (including the detection box of each object) and the results output by the first teacher network to obtain the first model.

[0071] Optionally, in the process of the first student network learning the knowledge of the first teacher network, the electronic device inputs the first sample image into the first student network and the first teacher network, and the first student network can iteratively train the first student network according to the difference between the output result (including the detection box of each object) and the result output by the first teacher network, and the difference between the image features extracted by the first student network and the image features extracted by the first teacher network, to obtain the first model. The following example illustrates the process of the electronic device obtaining the difference between the image features extracted by the first student network and the image features extracted by the first teacher network. For example, there are 16 convolutional layers in the first teacher network and 4 convolutional layers in the first student network. The electronic device can obtain an image feature from every 4 convolutional layers in the first teacher network, such as the image features obtained from the 1st to 4th layers are recorded as T11, the image features obtained from the 5th to 8th layers are recorded as T12, the image features obtained from the 9th to 12th layers are recorded as T13, and the image features obtained from the 13th to 16th layers are recorded as T14. The electronic device obtains an image feature from each convolutional layer of the first student network, such as the image feature obtained from the first layer is recorded as T21, the image feature obtained from the second layer is recorded as T22, the image feature obtained from the third layer is recorded as T23, and the image feature obtained from the fourth layer is recorded as T24; the electronic device obtains the difference between T11 and T21, the difference between T12 and T22, the difference between T13 and T23, and the difference between T14 and T24. The four differences are the differences between the image features extracted by the first student network and the image features extracted by the first teacher network.

[0072] The electronic device inputs the acquired first image into the first model, and the first model outputs a detection frame without a category. Figure 5 As shown, the first image includes three objects, and the image is input into the first model to obtain the detection frames of the three objects. The electronic device can determine the object image of each object according to the detection frame of each object, for example, by cropping the first image with the detection frame area to obtain the corresponding object image. Figure 5 As shown, the electronic device can obtain an image of a cat, an image of a dog, and an image of a teddy bear.

[0073] Step 403: The electronic device obtains an object image of each object in the first image according to the detection frame of each object in the first image.

[0074] Exemplarily, the electronic device may use the detection frame of each object in the first image as the cropping area to crop and obtain the object image of each object in the first image. Figure 5 As shown, the first image includes three objects, namely, a cat, a dog, and a teddy bear, and the first model determines the detection frame of each object; the electronic device uses the detection frame of the dog as the first cropping area in the first image, and crops to obtain the object image of the dog. The electronic device uses the detection frame of the cat as the second cropping area in the first image, and crops to obtain the object image of the cat. The electronic device uses the detection frame of the teddy bear as the third cropping area in the first image, and crops to obtain the object image of the teddy bear.

[0075] Step 404: The electronic device determines category information of each object in the first image according to each object image and the second model.

[0076] Exemplarily, the second model may include: an image editor, a text editor, and a classifier. The image editor is used to extract image features from an image, and the text editor is used to extract text features from a text. Optionally, the electronic device may use the image-text contrast learning method as a classifier of the second model. The classifier obtains the similarity between each image feature and text feature based on the image features output by the image editor and the text features output by the text editor.

[0077] In one example, the second teacher network can be pre-trained by other devices. The second teacher network may include an image text editor, for example, an ImageBind model. The ImageBind model can determine the text category corresponding to the input image, and the ImageBind model may include an image editor, a text editor, and a classifier for image text. The network structure of the second student network in the electronic device may be the same as or different from the network structure of the second teacher network. The electronic device uses the second teacher network and the second training set to perform knowledge distillation on the second student network to generate an image editor and a text editor deployed on the mobile terminal, i.e., the image editor and the text editor in the second model. The knowledge distillation method can adopt an offline distillation or a semi-supervised distillation method, and the specific distillation process will not be repeated.

[0078] In this example, the electronic device uses a pre-trained second teacher network to perform knowledge distillation on the second student network to generate a second model, so that the second model is a lightweight model. At the same time, it can retain the characteristics of accurate feature extraction of the image editor and text editor in the second teacher network, and also allows the second model to be flexibly deployed on mobile terminals with limited computing power.

[0079] During the process of knowledge distillation of the second teacher network by the electronic device, the second sample set includes different object images and different categories of text information (such as Figure 6). Figure 6 As shown in , the text information of known categories includes: preset text categories and "other" categories. The "other" category is used to refer to categories other than the currently preset text categories. Figure 6 As shown, the preset text categories include: dog, cat, ... flamingo. In this example, adding the "other" category to the known categories can ensure that each image feature has corresponding category information during the training process, thereby improving the accuracy of the second model being trained.

[0080] The electronic device inputs a second sample image and category text information of a known object into the second student network and the second teacher network, wherein the second sample image includes: object images of different objects, such as Figure 6 The dog image, cat image, and flamingo image shown in . The text information of known categories is as follows Figure 6 As shown, including: dog, cat... flamingo and others. The second student network can iteratively train the second student network according to the difference between the output results (including the image features of each object image, the text features of the known text information, and the similarity between the image features of each object image and the known text features) and the results output by the second teacher network (including the image features of each object image, the text features of the known text information, and the similarity between the image features of each object image and the known text features) to obtain a second model.

[0081] like Figure 6 As shown, the similarity table obtained by the classifier calculation of the second model has a horizontal axis representing the text features of each text and a vertical axis representing the image features of each object image. For example, the similarity between the text features of a dog and the image features of a dog is 1; the similarity between the text features of a dog and the image features of a cat is 0.

[0082] The classifier in the second model can determine the correspondence between the image features and the text features. That is, when the second model recognizes that the object in the object image belongs to another category, the text information of the new object category can be input into the second model without retraining the second model, so that the second model can recognize the category text information corresponding to the object in the object image.

[0083] For example, if Figure 6As shown, the electronic device can add text information of new object categories (such as airplanes, apples... teddy bears), input the text information of the newly added object categories into the second model, and the second model extracts the text features of the text information of the new object categories and stores them. The electronic device inputs different object images into the second model, and the second model extracts the image features of each object image; the second model can calculate the similarity between each image feature and each stored text feature, and the second model determines the category information corresponding to each object image based on the similarity between each image feature and each text feature. For example, the second model can output that the similarity between the image feature of a cat and the text feature of a cat is 1.0, and the similarity between the text feature of a dog is 0, select the text feature with the greatest similarity to the image feature of a cat (i.e., the text feature of a cat), and determine that the category of the cat image is "cat".

[0084] In this example, after the electronic device deploys the second model, the electronic device inputs the object image of each object in the first image into the second model, and the second model obtains the image features of each object image. The electronic device can input the stored text information of the object category into the second model, and the second model determines the category of each object image based on the text features of each stored object category and the image features of each object image. For example, the second model determines that the category corresponding to the image of a cat is a cat.

[0085] In another example, the electronic device may also pre-store text features of text information of pre-stored object categories. When the electronic device inputs the object image of each object in the first image into the second model, the second model determines the category of each object image based on the text features of each stored object category and the image features of each object image. In this manner, the second model does not need to extract text features, which reduces the steps of the second model determining the object category in each object image and improves the speed of determining the object category in each object image.

[0086] In this example, since the first model is obtained based on the knowledge distillation of the first teacher network, the first model is lightweight, and any object in the acquired image can be segmented to obtain the object image of each object in the image. The electronic device can accurately determine the category of each object based on the object image and the second model of each object. Since the second model is obtained based on the knowledge distillation of the second teacher network, the second model is a lightweight model, which is easy to deploy in the mobile terminal. At the same time, the electronic device can add text information of new object categories at any time without retraining the second model, which improves the speed and flexibility of updating target recognition in the mobile terminal.

[0087] Figure 7 The following is a flowchart of an exemplary target detection application, wherein the application process includes:

[0088] Step 701: The electronic device displays a preview interface.

[0089] Specifically, in this example, the electronic device is a mobile phone, and the application process of target detection is described by taking the mobile phone in the preview state of shooting as an example. Fig. 9 As shown in FIG. 9a, the mobile phone starts the camera and is in a shooting preview state, displaying a preview interface 901 of the first image shot. The camera application transmits the first image captured by the camera in real time to the image processing module.

[0090] Step 702: The electronic device obtains a click operation input by the user.

[0091] Exemplarily, in response to a click operation input by the user, the position clicked by the user in the first image (such as recorded as the first position) is obtained. Fig. 9 As shown in 9a, the user clicks on the teddy bear 902 in the first image 901, and the mobile phone can obtain the first position of the user's click operation in the first image and transmit the first position information to the image processing module.

[0092] Meanwhile, in response to the user's click operation, the mobile phone may execute step 703 to detect whether there is an instance object in the first image.

[0093] Step 703: The electronic device detects whether there is an object in the image, if so, executes step 704, otherwise, ends the operation.

[0094] Exemplarily, the first model is used to segment objects present in the image. When there are instance objects in the input image, the first model outputs a detection frame for each instance object. When there are no instance objects in the input image, the result output by the first model is empty. For example, when the entire first image is a solid color image, there is no object in the image. The image is input into the first model, and the result output by the first model is empty. Based on this, if the image processing module in the mobile phone detects that the result output by the first model includes a detection frame, it is determined that there are instance objects in the first image. The image processing module can crop the first image according to the detection frame of each object output by the first model, obtain the object image corresponding to each object, and execute step 704. The training process of the first model is the same as the relevant description in step 402, and will not be repeated here. If the image processing module detects that there is no detection frame in the result output by the first model, it is determined that there are no instance objects in the first image, and no subsequent operations are required.

[0095] like Figure 8As shown, the first image is input into the detector (ie, the first model), and the detector outputs the detection frames of the three objects. The image processor crops the first image according to each detection frame to obtain the object image corresponding to each detection frame.

[0096] Step 704: The electronic device detects category text information that matches the detection frame of the object.

[0097] Exemplarily, the image processing module inputs the object image of each object and any preset text information into the second model, and the second model extracts the image features of each object image and the text features of the pre-stored text information. The second model calculates the similarity between the image features of each object image and each text feature, and outputs the similarity between the image features of each object image and each text feature.

[0098] For each object image, the electronic device may select a text feature corresponding to the maximum similarity in the object image, and obtain text information corresponding to the text feature as the category text information of the object image.

[0099] For example, if Figure 8 As shown, the image processor inputs three object images into the second model, and the image encoder in the second model extracts the image features in each object image. In this example, the image processing module can input pre-stored known text information into the second model, and the text editor in the second model extracts the text features of each text information. Optionally, when there is newly added text information (i.e., text information of unknown category), the known text information and the unknown text information can be input into the second model.

[0100] The second model extracts the text features of each text information. The second model uses the image-text comparison method to calculate the similarity between the image features of each object image and each text feature, and obtains Figure 8 In the similarity table, the similarity between the dog image and the dog text information corresponding to the second row and the first column is 0.89, which is the maximum similarity between the dog image feature and each text feature. The image processing module determines that the category of the dog image is "dog".

[0101] In one example, if Figure 8 As shown, the known categories stored in the mobile phone do not include teddy bears. The mobile phone can obtain text information of the newly added object category from the server. Optionally, the server can pre-extract text features of each newly added text information, send each text feature to the mobile phone, and the mobile phone receives and stores each text feature sent by the server. The second model of the image processing module can directly obtain the text features of each stored text information without the need for the second model to re-extract features for each text information each time. This method can speed up the determination of the object category.

[0102] Step 705: If the electronic device detects that the detection frame of the object matches the category text information, it outputs the detection frame and category text information of the object.

[0103] Exemplarily, the image processing module can output the detection frame and category text information of each object. The image processing module can determine the object selected by the user based on the first position clicked by the user and the detection frame of each object. After determining the object selected by the user, the image processing module can determine the category of the object selected by the user based on the category text information corresponding to the object. The image processing module can return the detection frame of the object selected by the user to the camera application, and the camera application will mark the detection frame of the selected object in the first image and focus on the object selected by the user. Fig. 9 As shown in FIG9 b , in the preview interface 903 , a detection frame 904 of a teddy bear is displayed.

[0104] In another example, the mobile phone can realize accurate focus tracking based on the target detection processing results of each image (i.e., the detection frame and category information of each object). Fig.10 To illustrate the process:

[0105] Specifically, Fig.10 As shown in FIG10a, the camera is in the preview state of shooting or recording, and the mobile phone displays an image 1001, which includes a walking cat and a running dog. If the user wants to focus on the dog during the shooting process (i.e., track the dog), the user can click on the dog 1002 in the image 1001, as shown in FIG10a. Fig.1010a in the figure. In response to the user's click operation, the camera obtains the user's click position in the image 1001 and transmits it to the image processing module. When the camera is in the preview process, the camera transmits the captured image 1001 to the image processing module, and the image processing module transmits the image 1001 to the first model to obtain the detection frame of each object in the image 1001, such as the detection frame of the cat and the detection frame of the dog. The image processing module detects that the first model outputs 2 detection frames and determines that the image 1001 includes objects. The image processing module can crop the image 1001 according to the size of the detection frame of each object to obtain the image of the cat (denoted as cat_1) and the image of the dog (denoted as dog_1). The image processing module inputs the image of the cat and the image of the dog into the second model, and the second model can extract the image features of the cat (cat_feature1) and the image features of the dog (dog_feature1). The second model calculates the similarity between the image feature of the cat and each stored text feature, and calculates the similarity between the image feature of the dog and each stored text feature, obtains the text feature with the greatest similarity to the image feature of the cat, and determines the category information corresponding to the image of the cat as "cat" based on the text feature. Similarly, the image processing module can determine the category information of the image of the dog as "dog".

[0106] The image processing module obtains the detection frame of each object and the location information of the user's click, determines the object selected by the user as the target object, and can add the detection frame of the target object to the image A1 and return to the camera application to display the focused object selected by the user. Fig.10As shown in 10b, the display interface of the mobile phone displays image 1001, and the dog detection frame 1004 is displayed in the image 1001. In this example, the camera can perform a focusing operation based on the dog detection frame. The camera sends the captured image 1005 (the next frame image of image 1001) to the image processing module, and the image processing module transmits the image 1005 to the first model to obtain the detection frame of each object in the image 1005, that is, the cat detection frame and the dog detection frame. The image processing module detects that the first model outputs 2 detection frames and determines that the image 1005 includes an object. The image processing module can crop the image A2 according to the size of the detection frame of each object, and obtain the image of the cat (referred to as cat_2) and the image of the dog (referred to as dog_2). The image processing module inputs the cat image (cat_2) and the dog image (dog_2) into the second model, and the second model can extract the image features of the cat (cat_feature2) and the image features of the dog (dog_feature2). The second model calculates the similarity between the image feature of the cat (cat_feature2) and each stored text feature, and calculates the similarity between the image feature of the dog (dog_feature2) and each stored text feature, obtains the text feature with the greatest similarity to the image feature of the cat (cat_feature2), and determines the category information corresponding to the image of the cat (cat_2) based on the text feature. Similarly, the image processing module can determine the category information of the dog image. The image processing module can search for objects in image 1005 that have the same category as the target object in image 1001 (referred to as candidate objects) based on the category information of the target object in the previous frame image 1001. The image processing module can calculate the similarity between the detection frame of the candidate object and the detection frame of the target object in image A1 (such as the similarity of position and the similarity of range), and obtain the candidate object corresponding to the detection frame with the greatest similarity as the target object of image 1005. The image processing module can add a detection frame of the target object in the image 1005 and return it to the camera application. The camera application performs a focusing operation with the target object in the image 1005 as the focus point and instructs the display screen to display the detection frame 1006 of the target object in the image 1005 to prompt the user to focus on the object. Fig.10 As shown in FIG. 10 c , the display interface of the mobile phone displays an image 1005 , in which a detection frame 1006 focusing on the dog is displayed.

[0107] Similarly, each frame of the image captured by the camera in real time can determine the target object to be focused in the manner shown in image 1005, perform a focusing operation on the determined target object and display a detection frame of the focused object. Fig.10As shown in 10c, the similarity between the detection frame of each object and each detection frame of the previous frame is usually calculated, and the object in the detection frame with the greatest similarity is selected as the target object. When there is an overlap between the two object images, the target object determined by this method may change from a dog to a cat, resulting in a focus tracking error. In this example, the image processing module can determine the target object in the image of the current frame based on the detection frame and category information of each object and the detection frame and category information of the target object in the previous frame image. This method can effectively avoid the problem of incorrectly determining the target object during the focus tracking process.

[0108] In one example, the mobile phone can intelligently determine the magnification based on the target detection processing results of each image (i.e., the detection frame and category information of each object) to ensure the integrity of the captured target object. Fig.11 Specifically explain the process of enlarging the target object:

[0109] Specifically, Fig.11 As shown in FIG. 11a , the camera is in the preview state of shooting or recording, and the mobile phone displays an image 1101, which includes a cat, a dog, and a teddy bear. If the user wants to focus on the teddy bear during the process of zooming in on the image, the teddy bear 1002 in the image 1101 that the user can click on is displayed. Fig.1111a in FIG. 11. In response to the user's click operation, the camera obtains the user's click position in the image 1101 and transmits it to the image processing module. When the camera is in the preview process, the camera transmits the captured image 1101 to the image processing module, and the image processing module transmits the image 1101 to the first model to obtain the detection frame of each object in the image 1101, such as the detection frame of the cat and the detection frame of the dog. The image processing module detects that the first model outputs 3 detection frames and determines that the image 1101 includes an object. The image processing module can crop the image 1101 according to the size of the detection frame of each object, and obtain the image of the cat (recorded as cat_3), the image of the dog (recorded as dog_3) and the image of the teddy bear (bear_3). The image processing module inputs the image of the cat (cat_3), the image of the dog (dog_3) and the image of the teddy bear (bear_3) into the second model, and the second model can extract the image features of the cat (cat_feature3), the image features of the dog (dog_feature3) and the image features of the teddy bear (bear_feature3). The second model calculates the similarity between the image features of the cat and each stored text feature, calculates the similarity between the image features of the dog and each stored text feature, and the similarity between the image features of the teddy bear and each stored text feature; obtains the text feature with the greatest similarity to the image feature of the teddy bear, and determines the category information corresponding to the image of the teddy bear. Similarly, the image processing module can determine the category information of the dog image and the category information of the cat image. The image processing module obtains the detection frame of each object and the location information of the user's click, and determines the object selected by the user as the target object (the target object is the teddy bear). The image processing module can add the detection frame of the target object in image 1101 and return to the camera application to display the focused object selected by the user, such as Fig.11 The detection frame 1104 of the teddy bear is shown in the image 1101 shown in 11b of FIG. The user performs a zoom-in operation on the interface, and the camera application can obtain the position and range of the detection frame of the target object (i.e., the teddy bear) in the current image, and determine the target focal length for shooting the next frame of the image, so that the image of the teddy bear does not exceed the range of the current display interface. For example, the camera application can determine the range and position of the magnified teddy bear and the size of the display screen according to each candidate focal length, and determine the target focal length. Fig.11 As shown in FIG. 11c , the determined target focal length is 2 times, and the teddy bear is fully displayed in the enlarged image 1105 , while the images of the cat and the dog are incompletely displayed.

[0110] In this example, when the electronic device receives a zoom focus operation from the user, it can determine the maximum magnification of the target object while displaying the target object in its entirety based on the detection frame of the object selected by the user. This method can reduce the number of operation steps for the zoom focus operation by the user and shorten the user's operation process.

[0111] In some embodiments, the target detection method of the present application can also be applied to other application scenarios. For example, the electronic device can determine the main object in the current image based on the category of the target object and the detection frame of each object in the captured image, and blur objects other than the main object in the current image. Fig.12 The process of blurring objects other than the main object in the image is described.

[0112] Specifically, Fig.12 As shown in 12a, the mobile phone is in a video chat state, and the display interface 1201 of the mobile phone displays image C1 and image D1, and the image C1 includes: landscape 1202, portrait 1203. Image D1 includes portrait 1204. Portrait 1203 is a portrait taken by this end, and portrait 1204 is a portrait taken by the opposite end device of the chat. When the mobile phone is in a video chat state, the mobile phone transmits the image C1 taken in real time by the local end to the image processing module. The image processing module transmits the image C1 to the first model to obtain the detection frame of each object in the image C1, such as the detection frame of the mountain, the detection frame of the water, and the detection frame of the person. The image processing module detects that the first model outputs 3 detection frames and determines that the image C1 includes an instance object. The image processing module can crop the image C1 according to the size of the detection frame of each object to obtain the image of the mountain, the image of the water, and the image of the person, wherein the detection frame of the image of the mountain includes the detection frame of the water.

[0113] The image processing module inputs the image of the mountain, the image of the water and the image of the person into the second model, and the second model can extract the image features of the mountain, the image features of the water and the image features of the portrait. The second model calculates the similarity between the image features of the mountain and each stored text feature, the similarity between the image features of the water and each stored text feature, and the similarity between the image features of the portrait and each stored text feature. The image processing module obtains the text feature with the greatest similarity to the image feature of the mountain based on the calculated similarities, and determines the category information corresponding to the image of the mountain; obtains the text feature with the greatest similarity to the image feature of the water, and determines the category information corresponding to the image of the water.

[0114] The image processing module can return the detection frame and category information of each object image to the chat application, and the chat application instructs the display screen to display the image C1. In response to the blur operation input by the user (such as clicking the blur button 1205 in 12a), the chat application can instruct to determine the main image (i.e., the target object) in the current image based on the detection frame and category information of each object image in the image C1. For example, the chat application can select an image classified as a portrait. If there are multiple portraits, the portrait with the largest area in the image C1 can be selected from the multiple portraits as the main image.

[0115] After determining the subject image, the chat application determines that the area outside the detection frame of the subject image in the image C1 is a blurred area, blurs the blurred area in the image C1, generates a new image and displays it. Fig.12 As shown in 12a in FIG. 1, the chat application can determine that the main image in the current image C1 is the portrait 1203 according to the category information of each object image, and perform a blur operation on the object image outside the detection frame of the portrait 1203, such as Fig.12 As shown in FIG. 12b, after the blurring process, the images other than the main image in the image C2 displayed on the display screen have been blurred, and the main image has not been changed.

[0116] In this example, when the mobile phone blurs the image in the video, if the image includes multiple people, the main image can be quickly and accurately determined based on the category information of each object in the image and the area of ​​the detection box in the image, avoiding the problem of incorrectly determining the main image and causing blurring errors.

[0117] In some embodiments, the electronic device may also perform different processing strategies on the image according to the category information of each object in the image. Fig.13 To explain the process in detail:

[0118] Specifically, Fig.13As shown in 13a, the camera is in a preview state of shooting or recording, and the mobile phone displays an image 1301, which includes a flower. When the camera is in the preview process, the camera transmits the captured image 1301 to the image processing module, and the image processing module transmits the image 1301 to the first model to obtain the detection frame of each object in the image 1301, such as the detection frame of each flower and the detection frame of each leaf. The image processing module detects that the first model outputs the detection frame of each object and determines that the image 1301 includes an object. The image processing module can crop the image 1301 according to the size of the detection frame of each object to obtain the image of each flower and the image of each leaf. The image processing module inputs the image of each flower and the image of each leaf into the second model, and the second model can extract the image features of the flower and the image features of the leaf. The second model calculates the similarity between the image features of the flower and each stored text feature, and calculates the similarity between the image features of the leaf and each stored text feature; obtains the text feature with the greatest similarity to the image feature of the flower, and determines the category information corresponding to the image of the flower. Similarly, the image processing module can determine the category information of the leaf image.

[0119] The image processing module can return the detection frame and category information of each object image in the image 1301 to the camera application. If the camera application detects that the scene mode of the camera is turned on (the default scene mode is turned on as an example in this example), the camera application can determine that the scene category of the current image 1301 is the flower mode based on the category of the flower in the image 1301 and the area occupied by the detection frame. Optionally, different scene modes and the recognition conditions of each scene mode can be pre-stored in the camera application. For example, the recognition condition of the flower mode is that the image is detected to include the category of flowers, and the area occupied by the flowers in the image exceeds the first threshold (such as 55%); for another example, the recognition condition of the portrait mode is: the image is detected to include people, and the area occupied by the single portrait is recognized to exceed the second threshold (such as 60%).

[0120] When the camera application recognizes that the current scene is a flower mode, it obtains a shooting strategy that matches the flower mode, for example, adjusting the brightness and saturation of the image. In this example, the brightness of the image is adjusted. Fig.13 As shown in 13a in FIG. 13 , the user can click on the screen to view the brightness 1302 of the image 1301, and the brightness of the image 1301 is d. The camera application obtains the shooting strategy of the flower mode, and the shooting strategy indicates to lower the brightness of the current image 1301, such as the brightness after adjustment is e; Fig.13As shown in FIG. 13b, the user can click the screen to view the brightness 1304 of image 1303. It can be seen that the brightness of image 1303 is lower than that of image 1301. The display interface also displays a current scene mode logo 1305, which is used to indicate that the current scene is in the flower mode.

[0121] In this example, the application can determine the category of the shooting scene based on the detection frame and category information of each object image in the image determined by the image processing module, and then adopt the shooting strategy corresponding to the scene category to shoot, so that the image quality of the captured image is more in line with the current scene. For example, adopting the shooting strategy corresponding to the flower mode makes the color of the captured image more vivid.

[0122] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0123] This embodiment also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the target detection method in the above-mentioned embodiment. The storage medium includes: a U disk, a mobile hard disk, a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0124] This embodiment also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the target detection method in the above-mentioned embodiment.

[0125] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0126] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0127] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for target detection, characterized in that: Applied to a first electronic device, the method includes: acquiring a first image; Inputting the first image into a first model, obtaining a first result output by the first model, wherein the first result includes a detection box of each object in the first image, wherein the detection box of each object is used to indicate a position and a range of the object, wherein the first model is generated by performing knowledge distillation on a first student network using a first teacher network and a first sample set, wherein the first teacher network is a SAM, and the first sample set includes a plurality of first sample images and a first verification image corresponding to the first sample image, wherein the first verification image includes a detection box of each object; Acquire an object image of each object in the first image according to a detection frame of each object in the first image; Determine the category information of each object in the first image according to each object image and the second model, wherein the second model is generated by performing knowledge distillation on the second student network using a second teacher network and a second sample set, wherein the second sample set includes a second sample image and text information of an object category corresponding to the second sample image, and the second teacher network includes an image-text model, and the image-text model is used to determine the correspondence between image features and text features; The method further comprises: Displaying a preview interface of the first image; In response to a first selection operation of the user on the preview interface of the first image, determining a first target object selected by the user in the first image according to the detection frames of the objects in the first image; Acquire and store the category information of the first target object so that the first electronic device can determine the object to be focused in the next frame of image; A detection frame of the first target object is displayed in a preview interface of the first image, and a focusing operation is performed with the first target object as a focusing focus.

2. The method according to claim 1, characterized in that Before determining the category information of each object in the first image according to each object image and the second model, the method further includes: storing the newly added text features sent by the second electronic device, where the newly added text features are features obtained by the second electronic device after feature extraction of text information of the newly added category; Determining category information of each object in the first image according to each object image and the second model includes: The second model obtains image features of each object image according to each object image input; The second model obtains the similarity between each object image and each text feature based on the stored text features and the image features of each object image; According to the text feature corresponding to the maximum similarity in each object image, the category information of the object in each object image is determined.

3. The method according to claim 1, characterized in that Before determining the category information of each object in the first image according to each object image and the second model, the method further includes: storing new object category text information sent by the second electronic device or new object category text information input by the user; Determining category information of each object in the first image according to each object image and the second model includes: The second model obtains the image features of each object image and the text features of each object category text information according to each object image input and the stored object category text information; The second model obtains the similarity between each object image and each text feature according to each text feature and the image feature of each object image; According to the text feature corresponding to the maximum similarity in each object image, the category information of the object in each object image is determined.

4. The method according to claim 1, characterized in that The method further comprises: When the first electronic device captures a second image, acquiring the second image, the first image being a previous frame image of the second image; Inputting the second image into the first model, obtaining a second result output by the first model, wherein the second result includes a detection box of each object in the second image; Acquire an image of each object in the second image according to a detection frame of each object in the second image; determining category information of each object in the second image according to the object image of each object in the second image and the second model; Determine a second target object in the second image according to the detection frame and category information of the first target object in the first image, and the detection frame of each object and category information of each object in the second image, wherein the second target object is the same object as the first target object; A detection frame of the second target object is displayed on the preview interface of the second image, and a focusing operation is performed with the second target object as a focusing focus.

5. The method according to claim 1 or 4, characterized in that: The method further comprises: Displaying a detection frame of a target object on a preview interface of a current frame image, wherein the target object is the first target object; In response to an operation of zooming in on the preview interface of the current frame image, a target focal length for shooting the next frame image is determined based on the position and range of the detection frame of the target object in the current frame image, so that the first electronic device can fully display the detection frame of the target object in the image shot according to the target focal length, and the target focal length is greater than the focal length for shooting the current frame image.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining a detection frame of each object and category information of each object in the first image; Determining a category of a scene in which the first image was taken according to category information and a detection frame of each object; A shooting strategy for the first image is determined according to the scene category, where the shooting strategy is used to indicate image processing for the first image, and the image processing includes: adjusting white balance and adjusting brightness.

7. The method according to claim 1, characterized in that The method further comprises: In response to a blur operation input by a user, determining a first target object in the first image according to each object detection frame and category information in the first image; According to the detection frame and category information of the first target object, an object image outside the detection frame of the first target object is blurred.

8. The method according to claim 4, characterized in that Determining a second target object in the second image according to a detection frame and category information of the first target object in the first image, and a detection frame of each object and category information of each object in the second image, includes: Searching for an object in the second image that has the same category as the first target object according to category information of the first target object in the first image and category information of each object in the second image; According to the detection frame of the first target object in the first image and the detection frame of the found object, a detection frame having the greatest similarity to the detection frame of the first target object is obtained as the detection frame of the second target object.

9. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the target detection method described in any one of claims 1 to 8.

10. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed on an electronic device, the electronic device executes the target detection method according to any one of claims 1 to 8.

Images