Training sample generation method, device, system and target object detection method

Abstract

The present application discloses a method, device, system and target object detection method for generating training samples. The training sample generation method includes: obtaining a point-annotated image and a frame-annotated image, each point-annotated image includes an object to be annotated and a target pixel corresponding to each object to be annotated, each frame-annotated image includes an annotated object and a target annotated frame corresponding to each annotated object, and the number of point-annotated images is greater than the number of frame-annotated images; determining a target prediction frame corresponding to the object to be annotated based on the target pixel; annotating the object to be annotated based on the target prediction frame to obtain an annotated point-annotated image; generating a target training sample based on the annotated point-annotated image and the frame-annotated image, and the target training sample is used to train a target detection model to detect the object in the image to be detected. The present application solves the technical problem of high annotation cost in the prior art when annotating training samples.

Description

Technical Field

[0001] The present application relates to the field of data processing, and more specifically, to a method, device, system for generating training samples and a method for detecting target objects. Background Art

[0002] In recent years, with the rapid development of deep learning object detection technology, the performance of detection models has been greatly improved. However, in the existing technology, the training of detection models heavily relies on a large amount of manually annotated training data. How to reduce the annotation cost required for detection model training is an important issue.

[0003] In the prior art, annotation is performed by generating pseudo-annotated frames on unlabeled data. In this method, the quality of the pseudo-annotated frames is difficult to control, and the problems of missed detection and false detection of the pseudo-annotated frames will affect the training effect of the detection model.

[0004] In addition, in the prior art, the image is annotated by using the pixel points of the object in the image as the annotation points. In this method, due to the lack of accurate object bounding box annotation, the training effect of the detection model is very poor, especially when there are many objects in the image.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present application provide a method, device, system for generating a training sample and a method for detecting a target object, so as to at least solve the technical problem of high labeling cost when labeling training samples in the prior art.

[0007] According to one aspect of an embodiment of the present application, a method for generating training samples is provided, comprising: obtaining at least one point-annotated image and at least one frame-annotated image, wherein each point-annotated image contains at least one object to be annotated and target pixel points corresponding to each object to be annotated, and each frame-annotated image contains at least one annotated object and a target annotation frame corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image; determining a target prediction frame corresponding to each object to be annotated based on the target pixel points; annotating the object to be annotated in each point-annotated image based on the target prediction frame to obtain an annotated point-annotated image; generating a target training sample based on the annotated point-annotated image and at least one frame-annotated image, wherein the target training sample is used to train an object detection model, and the target detection model is used to perform target detection on the object in the image to be detected.

[0008] According to another aspect of an embodiment of the present application, a method for generating training samples is also provided, including: displaying at least one point-annotated image, wherein each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated; in response to an annotation box prediction instruction, determining a target prediction box corresponding to each object to be annotated based on the target pixels, and displaying the target prediction box on at least one point-annotated image; in response to an annotation instruction, annotating the object to be annotated in each point-annotated image based on the target prediction box, and displaying the annotated point-annotated image; in response to a sample generation instruction, generating a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and displaying relevant information of the target training sample, wherein the target training sample is used to train an object detection model, and the target detection model is used to perform target detection on objects in the image to be detected, and each frame-annotated image contains at least one annotated object and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0009] According to another aspect of an embodiment of the present application, a method for generating training samples is also provided, including: a cloud server obtains at least one point-annotated image sent by a terminal device, wherein each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated; the cloud server determines a target prediction box corresponding to each object to be annotated based on the target pixels, and annotates the object to be annotated in each point-annotated image based on the target prediction box to obtain an annotated point-annotated image; the cloud server annotates the object to be annotated in each point-annotated image based on the target prediction box to obtain an annotated point-annotated image; the cloud server sends the annotated point-annotated image to the terminal device, so that the terminal device generates a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and trains a target detection model based on the target training sample, wherein the target detection model is used to perform target detection on an object in the image to be detected, each frame-annotated image contains at least one annotated object and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0010] According to another aspect of an embodiment of the present application, a method for detecting a target object is also provided, including: obtaining an image to be detected, wherein the image to be detected contains at least the object to be detected; inputting the image to be detected into a target detection model to obtain an output result of the target detection model, wherein the target detection model is trained based on annotated point-annotated images and at least one frame-annotated image, and the annotated point-annotated images are obtained by annotating the objects to be annotated corresponding to the target pixels based on target prediction frames determined by the target pixels, the objects to be annotated are objects contained in each point-annotated image, and each frame-annotated image contains at least one annotated object and a target annotated frame corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image; based on the output result, the object to be detected is identified to obtain an identification result.

[0011] According to another aspect of an embodiment of the present application, a training sample generation device is also provided, including: an acquisition module, used to acquire at least one point-annotated image and at least one frame-annotated image, wherein each point-annotated image contains at least one object to be annotated and target pixel points corresponding to each object to be annotated, and each frame-annotated image contains at least one annotated object and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image; a determination module, used to determine a target prediction box corresponding to each object to be annotated based on the target pixel points; a labeling module, used to label the object to be annotated in each point-annotated image based on the target prediction box to obtain an annotated point-annotated image; a generation module, used to generate a target training sample based on the annotated point-annotated image and at least one frame-annotated image, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on the object in the image to be detected.

[0012] According to another aspect of an embodiment of the present application, a training sample generation device is also provided, including: a first display module, used to display at least one point-annotated image, wherein each point-annotated image includes at least one object to be annotated and a target pixel corresponding to each object to be annotated; a second display module, used to respond to a labeling frame prediction instruction, determine a target prediction frame corresponding to each object to be annotated based on the target pixel, and display the target prediction frame on at least one point-annotated image; a third display module, used to respond to the labeling instruction, label the object to be annotated in each point-annotated image based on the target prediction frame, and display the annotated point-annotated image; a fourth display module, used to respond to the sample generation instruction, generate a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and display relevant information of the target training sample, wherein the target training sample is used to train a target detection model, the target detection model is used to perform target detection on an object in the image to be detected, each frame-annotated image includes at least one annotated object, and a target annotation frame corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0013] According to another aspect of an embodiment of the present application, a training sample generation device is also provided, including: a first acquisition module, used for a cloud server to acquire at least one point-annotated image sent by a terminal device, wherein each point-annotated image contains at least one object to be annotated and a target pixel corresponding to each object to be annotated; a first determination module, used for the cloud server to determine a target prediction box corresponding to each object to be annotated based on the target pixel, and annotate the object to be annotated in each point-annotated image based on the target prediction box to obtain an annotated point-annotated image; a first annotation module, used for the cloud server to annotate the object to be annotated in each point-annotated image based on the target prediction box to obtain an annotated point-annotated image; a sending module, used for the cloud server to send the annotated point-annotated image to the terminal device, so that the terminal device generates a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and trains a target detection model based on the target training sample, wherein the target detection model is used to perform target detection on an object in the image to be detected, each frame-annotated image contains at least one annotated object, and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0014] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for generating training samples and the above-mentioned method for detecting target objects when running.

[0015] According to another aspect of an embodiment of the present application, an electronic device is also provided, which includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the above-mentioned method for generating training samples and the above-mentioned method for detecting target objects when running.

[0016] According to another aspect of an embodiment of the present application, a system for generating training samples is also provided, the system comprising: a processor; and a memory connected to the processor and used to store executable instructions of the processor; wherein the processor is configured to execute instructions to implement the above-mentioned training sample generation method and the above-mentioned target object detection method.

[0017] In an embodiment of the present application, a method combining semi-supervised learning and weakly supervised learning is adopted. After acquiring at least one point-annotated image and at least one box-annotated image, a target prediction box of each object to be annotated contained in the point-annotated image is determined based on the target pixel points of the object to be annotated contained in the point-annotated image, and the object to be annotated in each point-annotated image is annotated based on the target prediction box to obtain an annotated point-annotated image. Then, a target training sample is generated based on the annotated point-annotated image and at least one box-annotated image, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on objects in the image to be detected, each point-annotated image contains at least one object to be annotated and target pixel points corresponding to each object to be annotated, each box-annotated image contains at least one annotated object and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one box-annotated image.

[0018] In the above process, the image is annotated based on the semi-supervised learning method, and the obtained image is a box-annotated image; the image is annotated based on the weakly supervised learning method, and the obtained image is a point-annotated image. Since the number of point-annotated images is greater than the number of box-annotated images, that is, in the present application, a small number of images in the target training samples are box-annotated images with higher annotation costs, while a large number of images are point-annotated images with lower annotation costs, that is, in the present application, only a small number of box-annotated images are needed to generate training samples, thereby reducing the annotation cost of training samples. In addition, in the present application, the prediction box corresponding to the object to be annotated is determined based on the target pixel points corresponding to the object to be annotated in the point-annotated image, so that a suitable prediction box can be selected as a pseudo-annotation box for each annotation point, thereby avoiding the problems of missed detection and false detection of pseudo-annotation boxes in semi-supervised learning in the prior art, improving the quality of training samples, and further improving the detection accuracy of the target detection model.

[0019] It can be seen that the solution provided in the present application achieves the purpose of accurately labeling training samples, thereby realizing the technical effect of reducing the labeling cost of training samples, and further solving the technical problem of high labeling cost when labeling training samples in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a hardware structure block diagram of an electronic device for implementing a method for generating training samples according to an embodiment of the present application;

[0022] Figure 2 is a flow chart of a method for generating a training sample according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an optional point-annotated image according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of an optional frame-annotated image according to an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of an optional point-annotated image according to an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of an optional point-annotated image according to an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of generating an optional training sample according to an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of the structure of an optional neural network according to an embodiment of the present application;

[0029] Fig. 9 is a flow chart of a method for generating a training sample according to an embodiment of the present application;

[0030] Fig.10 is a flow chart of a method for generating a training sample according to an embodiment of the present application;

[0031] Fig.11 is a flow chart of a method for detecting a target object according to an embodiment of the present application;

[0032] Fig.12 is a schematic diagram of a training sample generation device according to an embodiment of the present application;

[0033] Fig.13 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] Example 1

[0037] According to an embodiment of the present application, an embodiment of a method for generating training samples is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of an electronic device (or mobile device) for implementing a method for generating training samples. Figure 1As shown, the electronic device 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (BUS) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0039] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the electronic device 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for generating training samples in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for generating training samples described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the electronic device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0041] The transmission device 106 is used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0042] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the electronic device 10 (or mobile device).

[0043] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the above-described computer device (or mobile device).

[0044] Under the above operating environment, this application provides Figure 2 The method for generating training samples shown in FIG. Figure 2 is a flowchart of a method for generating training samples according to Embodiment 1 of the present application, comprising Figure 2 It can be seen that the method includes the following steps:

[0045] Step S202: acquiring at least one point-annotated image and at least one frame-annotated image.

[0046] In step S202, the point-annotated image is an image obtained by annotating the image based on weakly supervised learning, and the box-annotated image is an image obtained by annotating the image based on semi-supervised learning. The semi-supervised learning scheme generates pseudo-annotated boxes on unannotated data, while the weakly supervised learning scheme randomly determines target pixels as annotations from multiple pixels corresponding to each object in the image. That is, each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated, such as Figure 3 In the point-annotated image shown, pixel A is the target pixel corresponding to the athlete, and pixel B is the pixel corresponding to the football. Each frame-annotated image contains at least one annotated object and a target annotated frame corresponding to each annotated object. Figure 4In the box annotation image shown, box C is the target annotation box corresponding to the cartoon character.

[0047] It should be noted that, in the point-annotated image, the target pixel point is a pixel point obtained by randomly selecting a plurality of pixel points corresponding to the object to be annotated in the point-annotated image. In addition, the target pixel point can also be a pixel point obtained by selecting a plurality of pixel points corresponding to the object to be annotated in the point-annotated image by the terminal device through a certain algorithm.

[0048] In an optional embodiment, the terminal device may obtain point-annotated images and frame-annotated images from the deep learning image dataset. Optionally, the terminal device may obtain multiple images from the deep learning image dataset, and classify the obtained multiple images to obtain multiple sub-datasets, that is, identify point-annotated images and frame-annotated images from the multiple images, and obtain the data sets corresponding to the point-annotated images and the data sets corresponding to the frame-annotated images respectively. Among them, the number of point-annotated images in the multiple images obtained by the terminal device is greater than the number of frame-annotated images.

[0049] It should be noted that in step S202, the number of at least one point-annotated image is greater than the number of at least one frame-annotated image. It is easy to notice that in practical applications, the annotation cost of frame annotation is higher, while the annotation cost of point annotation is lower. In the present application, the training samples include a small number of frame-annotated images and a large number of point-annotated images, thereby avoiding the problem of high model training cost in the prior art of using frame-annotated images to train detection models.

[0050] Step S204: determining a target prediction box corresponding to each object to be labeled based on the target pixel points.

[0051] In step S204, each target pixel corresponds to at least one prediction box. Figure 5 In the point-annotated image shown, the target pixel point corresponding to the athlete is A, and the target pixel point A corresponds to three prediction boxes, namely prediction box 31, prediction box 32, and prediction box 33; the target pixel point corresponding to the football is B, and its corresponding prediction box is 34.

[0052] Optionally, after determining the target pixel point, when there are multiple prediction boxes corresponding to the target pixel point, the terminal device can obtain the confidence corresponding to each prediction box, and select the prediction box with the highest confidence among the multiple prediction boxes as the target prediction box, wherein the confidence of the prediction box represents the accuracy of the prediction box as the target prediction box.

[0053] Step S206 , annotating the object to be annotated in each point annotated image based on the target prediction frame to obtain an annotated point annotated image.

[0054] In step S206, after determining the target prediction box corresponding to each object to be labeled, the target prediction box corresponding to each object to be labeled is displayed in each point labeled image. Figure 6 In the point-annotated image shown, the prediction box corresponding to the operator's target pixel point A is the target prediction box 31 , and the prediction box corresponding to the football's target pixel point B is the target prediction box 34 .

[0055] Optionally, the terminal device identifies the object to be labeled in the target prediction box, determines the object type corresponding to the object to be labeled, and marks the object type at the target prediction box corresponding to the object to be labeled, where the object type may include but is not limited to people, animals, plants, daily necessities, office supplies, etc.

[0056] In another optional embodiment, after determining the target prediction box corresponding to each object to be labeled, the terminal device identifies the object to be labeled and uses different forms of prediction boxes to represent different object types. For example, a red prediction box is used to indicate that the object to be labeled is a person, a blue prediction box is used to indicate that the object to be labeled is an animal, and a red dotted box is used to indicate that the object to be labeled is a child, etc.

[0057] Step S208: generating a target training sample based on the annotated point annotated image and at least one frame annotated image.

[0058] In step S208, after obtaining the annotated point-annotated image, the terminal device mixes the annotated point-annotated image with the frame-annotated image to obtain a target training sample to train the target detection model, and then the user can use the target detection model to perform target detection on the object in the image to be detected. The image to be detected may be, but is not limited to, a natural image containing natural scenery, a remote sensing image, etc., that is, the terminal device may use the target detection model to detect pedestrians in natural images, and to detect vehicles, ships, buildings, etc. in remote sensing images.

[0059] It should be noted that, in actual applications, for different target detection scenarios, the corresponding target training samples may be different. For example, in a person detection scenario, the point-annotated image and the box-annotated image include at least a person, and the corresponding target training samples include at least an image after the person is annotated; and in a vehicle detection scenario, the point-annotated image and the box-annotated image include at least a vehicle, and the corresponding target training samples include at least an image after the vehicle is annotated. That is, in the present application, for different detection objects, the target detection models used are different, and the data sets used to train the target detection models are also different. It is easy to notice that using the target detection model corresponding to the application scenario for target detection can improve the accuracy of target detection.

[0060] Based on the scheme defined in the above steps S202 to S208, it can be known that in an embodiment of the present application, a combination of semi-supervised learning and weakly supervised learning is adopted. After acquiring at least one point-annotated image and at least one frame-annotated image, a target prediction frame of each object to be annotated contained in the point-annotated image is determined based on the target pixel points of the object to be annotated contained in the point-annotated image, and the object to be annotated in each point-annotated image is annotated based on the target prediction frame to obtain an annotated point-annotated image. Then, a target training sample is generated based on the annotated point-annotated image and the at least one frame-annotated image, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on objects in the image to be detected. Each point-annotated image contains at least one object to be annotated and target pixel points corresponding to each object to be annotated, and each frame-annotated image contains at least one annotated object and a target annotation frame corresponding to each annotated object. The number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0061] It is easy to notice that in the above process, the image is annotated based on the semi-supervised learning method, and the obtained image is a box-annotated image; the image is annotated based on the weakly supervised learning method, and the obtained image is a point-annotated image. Since the number of point-annotated images is greater than the number of box-annotated images, that is, in the present application, a small number of images in the target training samples are box-annotated images with higher annotation costs, while a large number of images are point-annotated images with lower annotation costs, that is, in the present application, only a small number of box-annotated images are needed to generate training samples, thereby reducing the annotation cost of training samples. In addition, in the present application, the prediction box corresponding to the object to be annotated is determined based on the target pixel points corresponding to the object to be annotated in the point-annotated image, so that a suitable prediction box can be selected as a pseudo-annotation box for each annotation point, avoiding the problems of missed detection and false detection of pseudo-annotation boxes in semi-supervised learning in the prior art, improving the quality of training samples, and further improving the detection accuracy of the target detection model.

[0062] It can be seen that the solution provided in the present application achieves the purpose of accurately labeling training samples, thereby realizing the technical effect of reducing the labeling cost of training samples, and further solving the technical problem of high labeling cost when labeling training samples in the prior art.

[0063] In an optional embodiment, after acquiring the point-annotated image, since a target pixel may correspond to multiple prediction boxes, the terminal device needs to determine the target prediction box from the multiple prediction boxes corresponding to the target pixel. Specifically, the terminal device may determine the multiple prediction boxes corresponding to the target pixel based on a preset model, calculate the confidence corresponding to each prediction box, and then determine the target prediction box from the multiple prediction boxes according to the confidence.

[0064] Optional, in Figure 7 In the schematic diagram of generating training samples shown in FIG. 1 , the preset model can determine multiple prediction boxes corresponding to each target pixel based on the target pixel, that is, after the terminal device inputs the point-annotated image into the preset model, the preset model can output the point-annotated image with multiple prediction boxes. For example, the point-annotated image is Figure 3 The image shown, the preset model Figure 3 After processing the image shown, we get Figure 5 An image with multiple predicted boxes is shown.

[0065] Further, such as Figure 7 As shown in FIG. 1 , after determining the multiple prediction boxes corresponding to the target pixel point, the terminal device performs a point matching operation, that is, the terminal device determines the target prediction box corresponding to each target pixel point, such as in FIG. Figure 6 In the figure, the target prediction box corresponding to each object to be labeled is displayed. The terminal device can calculate the confidence corresponding to each prediction box, and select the prediction box with the largest confidence from multiple prediction boxes as the target prediction box.

[0066] Furthermore, if Figure 7 As shown, after determining the target prediction box corresponding to each object to be labeled, the terminal device labels the point labeled image based on the target prediction box, and mixes the labeled point labeled image with the box labeled image to obtain a target training sample.

[0067] In an optional embodiment, the terminal device may determine multiple prediction boxes corresponding to the object to be labeled based on the object type corresponding to the object to be labeled. Specifically, the terminal device determines the target object type of the object to be labeled corresponding to the target pixel point based on a preset model, and determines multiple prediction boxes corresponding to the target object type based on the preset model, wherein the object type corresponding to the multiple prediction boxes is the target object type. That is, the terminal device uses the prediction boxes of other objects of the same object type as the object to be labeled as the prediction boxes corresponding to the labeled object. For example, for Figure 3 For athletes in the game, the terminal device can use the labeled boxes corresponding to all characters as prediction boxes.

[0068] In addition, in addition to considering the object type, the terminal device can also analyze the object to be labeled, and then determine multiple prediction boxes corresponding to the object to be labeled. For example, for a person, the terminal device can also consider the gender, posture, and scene of the object to be labeled to determine multiple prediction boxes of the object to be labeled.

[0069] After determining the multiple prediction boxes corresponding to each target pixel point, the terminal device calculates the confidence corresponding to each prediction box, wherein the confidence corresponding to each prediction box can be represented by the cost corresponding to each prediction box.

[0070] Specifically, the terminal device calculates the spatial cost and classification cost between each prediction box and each target pixel, and calculates the sum of the spatial cost and the classification cost to obtain the prediction cost between each prediction box and each target pixel. Then, a cost matrix is ​​formed based on the prediction cost, wherein each element in the cost matrix represents the confidence between each prediction box at each target pixel, the spatial cost represents the cost of determining the prediction box of the object to be labeled corresponding to the target pixel, and the classification cost represents the cost of classifying the object to be labeled corresponding to the target pixel.

[0071] Alternatively, the cost matrix can be expressed as follows:

[0072] L match (i, j) = L spatial +L classification

[0073] In the above formula, L match (i, j) represents the cost matrix, L spatial represents the space cost, L classification represents the classification cost. N p Indicates the number of target pixels, N b Represents the number of prediction boxes, i represents the index of the target pixel, and j represents the index of the prediction box.

[0074] in,

[0075] In the above formula, Represents pixel Tags, Represents the predicted box b j Tags, represents the output of the classifier in the neural network and the P branch in the neural network, C represents the number of categories that do not contain background, σ(·) represents the softmax operation performed by the classifier on the second dimension, σ P (·) represents the softmax operation performed by the P branch on the second dimension, where the P branch is used to detect the quality of the prediction box at the pixel level and supervised by the point-by-point MIL (Multiple Instance Learning) loss.

[0076] Optional, Figure 8A schematic diagram of an optional neural network structure is shown, consisting of Figure 8 It can be seen that the neural network model in this embodiment has an additional I branch and P branch at the head compared to the RPN (Region Proposal Network) in the prior art, where the I branch is used to suppress the probability of classification prediction that is inconsistent with the image-level annotation and is supervised by the image-level MIL loss. Figure 8 It can be seen that the head of the neural network model in this embodiment also includes a regression unit and a classification unit (ie, a classifier).

[0077] It should be noted that after the cost matrix is ​​determined, the matching problem of target pixels can be converted into a mathematical bipartite matching problem.

[0078] Furthermore, after obtaining the confidence corresponding to each prediction box, the terminal device can determine the target prediction box from multiple prediction boxes according to the confidence. Specifically, the terminal device determines the initial prediction box with the largest confidence from multiple prediction boxes, and when the confidence of the initial prediction box is greater than or equal to the preset confidence, determines the initial prediction box as the target prediction box.

[0079] That is, in this embodiment, the terminal device selects the prediction box with the highest confidence as the target prediction box. In addition, if the confidence corresponding to the prediction box with the highest confidence is relatively small, the prediction box with the highest confidence will also be filtered out, and there is no target prediction box corresponding to the target pixel point among the multiple prediction boxes.

[0080] It should be noted that in practical applications, due to the small size of the object to be labeled, there may be a phenomenon that the target pixel does not exist in the target prediction box, which may lead to the problem of missed detection of the object to be labeled. Figure 7 As shown, the terminal device avoids the above problem by pasting images of other objects of the same type at the target pixel point.

[0081] Specifically, when the confidence of the initial prediction box is less than the preset confidence, the terminal device determines the target object type of the object to be annotated corresponding to the target pixel point, obtains the target image from the preset image library, and then sets the target image at the target image position in the point annotated image, and finally determines the prediction box corresponding to the target image as the target prediction box. The target image contains the target object, the object type of the target object is the target object type, and the target image position is the position of the object to be annotated in the point annotated image.

[0082] For example, in Figure 5In the process of determining the target prediction box corresponding to the football, the terminal device filters out the prediction box corresponding to the football due to the small size of the football. In order to avoid missing the object to be labeled, the terminal device obtains the image of the target object belonging to the same category as the object to be labeled from the preset image library. For example, the terminal device first determines the object whose target prediction box has not been determined from the multiple objects to be labeled contained in the point-annotated image, and determines the target object type corresponding to the object, and then obtains the target image containing the object of the target object type from the prediction image library, and pastes the target image at the position of the object to be labeled in the point-annotated image, and uses the prediction box corresponding to the target image as the target prediction box of the object to be labeled, thereby obtaining Figure 6 Image shown.

[0083] Optionally, in the process of acquiring the target image from the preset image library, when the terminal device detects that there are multiple images corresponding to the target object type in the preset image library, the image size of each image is further detected, and an image whose image size is larger than the size of the object to be annotated is selected from the multiple images as the target image. Furthermore, when there are multiple images whose image size is larger than the size of the object to be annotated, the terminal device can determine the target image from the multiple images in combination with the overall image size of the image to be annotated.

[0084] It should be noted that by performing point-guided copy and paste operations on the objects to be annotated that are not matched to the prediction box, data enhancement of the training samples can be achieved, reducing the impact of unmatched target pixels on model training.

[0085] Furthermore, after determining the target prediction box corresponding to each object to be labeled, the terminal device labels the object to be labeled in each point-labeled image based on the target prediction box to obtain the labeled point-labeled image, and then generates a target training sample based on the labeled point-labeled image and at least one box-labeled image. After obtaining the target training sample, the terminal device may also optimize the preset model for determining the prediction box corresponding to the pixel point.

[0086] Specifically, the terminal device first obtains the first data set and the second data set, and calculates the cross entropy loss value of the first data set based on all the prediction box information to obtain the first loss value. Then, the confidence score corresponding to each sub-data set is calculated, and the confidence score is entropy calculated to obtain the second loss value corresponding to the second data set. Finally, the terminal device calculates the sum of the first loss value, the second loss value and the third loss value to obtain the target loss value, and optimizes the model parameters of the preset model based on the target loss value to obtain the optimized preset model. Among them, the first data set contains all the prediction box information corresponding to at least one point-annotated image, the second data set includes all the point annotation information corresponding to at least one point-annotated image, and the second data set is composed of multiple sub-data sets, each of which includes a positive annotation point and multiple negative annotation points, the positive annotation point is the target pixel point corresponding to the object to be annotated, and the negative annotation point is the non-target pixel point corresponding to the object to be annotated; the confidence score is positively correlated with the annotation cost of the point-annotated image; the third loss function is the initial loss value corresponding to the preset model.

[0087] Optional, such as Figure 7 As shown in Figure 1, the architecture of the student model is the same as that of the preset model, and the student model is optimized on box-marked images and point-marked images with strong enhancement capabilities. The target loss value can be expressed as follows:

[0088] L=L3+λ1L1+λ2L2

[0089] In the above formula, L is the target loss value; L3 is the third loss value; L1 is the first loss value; L2 is the second loss value; λ1 and λ2 are parameters used to balance these three loss values.

[0090] L1 satisfies the following formula:

[0091]

[0092] Where C represents the number of categories; Label indicating the image category; φ c Represents the score of the predicted image classification;

[0093]

[0094] In the above formula, s represents the output of the classifier; s I represents the branch of the I branch, where σ I (·) represents the softmax operation performed in the first dimension; σ(s) represents the matrix corresponding to the classifier; Represents the matrix corresponding to the I branch; Among them, X s =σ(s)⊙σ I(s I ).

[0095] L2 satisfies the following formula:

[0096]

[0097] Among them, N p is the number of pixels;

[0098]

[0099] In the above formula, Represents the confidence score of each sub-dataset for a stick; Ψ i Represents a sub-dataset; σ(·) and σP(·) represent the softmax operation.

[0100] In addition, Figure 7 As shown in the figure, the first data set is the prediction box information, where "+" indicates that the prediction box is the target prediction box, and "-" indicates that the prediction box does not match any pixel point; the second data set is the point annotation information, where "+" indicates a positive annotation point, that is, there is a target prediction box corresponding to the pixel point, and "-" indicates a negative annotation point, that is, a target prediction box that does not correspond to the pixel point. Figure 7 It can be seen that the second data set is composed of multiple sub-data sets A1 and A2, and each sub-data set has a positive annotation point.

[0101] Based on the above content, it can be seen that in this application, by matching the target pixel point with the target prediction box, it is ensured that a suitable prediction box is selected as a pseudo-annotation box for each annotation point in the point-annotated image, thereby avoiding the problem of missed detection and false detection of pseudo-annotation boxes in semi-supervised learning. In addition, in this application, multi-instance learning loss functions at the image level and point annotation level are added to ensure full utilization of point annotation information. Finally, this application also uses the copy-and-paste technology guided by annotation points to achieve data enhancement of training samples, thereby ensuring the training accuracy of the target detection model.

[0102] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method for generating training samples according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0104] Example 2

[0105] This application also provides Fig. 9 The method for generating training samples shown in FIG. Fig. 9 is a flowchart of a method for generating training samples according to an embodiment of the present application, comprising Fig. 9 It can be seen that the method includes the following steps:

[0106] Step S902, displaying at least one point-annotated image, wherein each point-annotated image includes at least one object to be annotated and target pixel points corresponding to each object to be annotated;

[0107] Step S904, responding to the annotation box prediction instruction, determining a target prediction box corresponding to each object to be annotated based on the target pixel point, and displaying the target prediction box on at least one point annotated image;

[0108] Step S906, responding to the labeling instruction, labeling the object to be labeled in each point-labeled image based on the target prediction box, and displaying the labeled point-labeled image;

[0109] Step S908, in response to the sample generation instruction, a target training sample is generated based on the annotated point annotated image and at least one frame annotated image, and relevant information of the target training sample is displayed, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on objects in the image to be detected, each frame annotated image contains at least one annotated object, and a target annotation box corresponding to each annotated object, and the number of at least one point annotated image is greater than the number of at least one frame annotated image.

[0110] Based on the scheme defined in the above steps S902 to S908, it can be known that in an embodiment of the present application, a combination of semi-supervised learning and weakly supervised learning is adopted. After acquiring at least one point-annotated image and at least one frame-annotated image, a target prediction frame of each object to be annotated contained in the point-annotated image is determined based on the target pixel points of the object to be annotated contained in the point-annotated image, and the object to be annotated in each point-annotated image is annotated based on the target prediction frame to obtain an annotated point-annotated image. Then, a target training sample is generated based on the annotated point-annotated image and the at least one frame-annotated image, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on objects in the image to be detected. Each point-annotated image contains at least one object to be annotated and target pixel points corresponding to each object to be annotated, and each frame-annotated image contains at least one annotated object and a target annotation frame corresponding to each annotated object. The number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0111] It is easy to notice that in the above process, the image is annotated based on the semi-supervised learning method, and the obtained image is a box-annotated image; the image is annotated based on the weakly supervised learning method, and the obtained image is a point-annotated image. Since the number of point-annotated images is greater than the number of box-annotated images, that is, in the present application, a small number of images in the target training samples are box-annotated images with higher annotation costs, while a large number of images are point-annotated images with lower annotation costs, that is, in the present application, only a small number of box-annotated images are needed to generate training samples, thereby reducing the annotation cost of training samples. In addition, in the present application, the prediction box corresponding to the object to be annotated is determined based on the target pixel points corresponding to the object to be annotated in the point-annotated image, so that a suitable prediction box can be selected as a pseudo-annotation box for each annotation point, avoiding the problems of missed detection and false detection of pseudo-annotation boxes in semi-supervised learning in the prior art, improving the quality of training samples, and further improving the detection accuracy of the target detection model.

[0112] It can be seen that the solution provided in the present application achieves the purpose of accurately labeling training samples, thereby realizing the technical effect of reducing the labeling cost of training samples, and further solving the technical problem of high labeling cost when labeling training samples in the prior art.

[0113] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method for generating training samples according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0115] Example 3

[0116] Under the above operating environment, this application provides Fig.10 The method for generating training samples shown in FIG. Fig.10 is a flow chart of a method for generating training samples according to an embodiment of the present application, such as Fig.10 As shown, the method comprises the following steps:

[0117] Step S1002: The cloud server obtains at least one point-annotated image sent by the terminal device, wherein each point-annotated image includes at least one object to be annotated and target pixel points corresponding to each object to be annotated.

[0118] In step S1004, the cloud server determines a target prediction box corresponding to each object to be labeled based on the target pixel point, and labels the object to be labeled in each point-labeled image based on the target prediction box to obtain a labeled point-labeled image.

[0119] Step S1006: The cloud server labels the object to be labeled in each point-labeled image based on the target prediction box to obtain a labeled point-labeled image.

[0120] Step S1008, the cloud server sends the annotated point annotated image to the terminal device, so that the terminal device generates a target training sample based on the annotated point annotated image and at least one frame annotated image, and trains a target detection model based on the target training sample, wherein the target detection model is used to perform target detection on objects in the image to be detected, each frame annotated image contains at least one annotated object, and a target annotation box corresponding to each annotated object, and the number of at least one point annotated image is greater than the number of at least one frame annotated image.

[0121] Based on the scheme defined in the above steps S1002 to S1008, it can be known that in an embodiment of the present application, a combination of semi-supervised learning and weakly supervised learning is adopted. After acquiring at least one point-annotated image and at least one frame-annotated image, a target prediction frame of each object to be annotated contained in the point-annotated image is determined based on the target pixel points of the object to be annotated contained in the point-annotated image, and the object to be annotated in each point-annotated image is annotated based on the target prediction frame to obtain an annotated point-annotated image. Then, a target training sample is generated based on the annotated point-annotated image and the at least one frame-annotated image, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on objects in the image to be detected. Each point-annotated image contains at least one object to be annotated and target pixel points corresponding to each object to be annotated, and each frame-annotated image contains at least one annotated object and a target annotation frame corresponding to each annotated object. The number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0122] It is easy to notice that in the above process, the image is annotated based on the semi-supervised learning method, and the obtained image is a box-annotated image; the image is annotated based on the weakly supervised learning method, and the obtained image is a point-annotated image. Since the number of point-annotated images is greater than the number of box-annotated images, that is, in the present application, a small number of images in the target training samples are box-annotated images with higher annotation costs, while a large number of images are point-annotated images with lower annotation costs, that is, in the present application, only a small number of box-annotated images are needed to generate training samples, thereby reducing the annotation cost of training samples. In addition, in the present application, the prediction box corresponding to the object to be annotated is determined based on the target pixel points corresponding to the object to be annotated in the point-annotated image, so that a suitable prediction box can be selected as a pseudo-annotation box for each annotation point, avoiding the problems of missed detection and false detection of pseudo-annotation boxes in semi-supervised learning in the prior art, improving the quality of training samples, and further improving the detection accuracy of the target detection model.

[0123] It can be seen that the solution provided in the present application achieves the purpose of accurately labeling training samples, thereby realizing the technical effect of reducing the labeling cost of training samples, and further solving the technical problem of high labeling cost when labeling training samples in the prior art.

[0124] It should be noted that, in the cloud server, the prediction box of each object to be annotated is determined based on the target pixel points in the point annotated image, and then the point annotated image is annotated based on the prediction box. Finally, the box annotated image and the annotated point annotated image are used to form the target training sample for training the target detection model. The specific method is the same as the method in Example 1 and will not be repeated here.

[0125] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method for generating training samples according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0127] Example 4

[0128] This application also provides Fig.11 The detection method of the target object shown in FIG. Fig.11 is a flow chart of a method for detecting a target object according to an embodiment of the present application. Fig.11 As shown, the method comprises the following steps:

[0129] Step S1102, obtaining an image to be detected, wherein the image to be detected at least contains an object to be detected;

[0130] Step S1104: input the image to be detected into the target detection model to obtain the output result of the target detection model, wherein the target detection model is trained based on the annotated point annotated image and at least one frame annotated image, the annotated point annotated image is obtained by annotating the object to be annotated corresponding to the target pixel point based on the target prediction frame determined by the target pixel point, the object to be annotated is the object contained in each point annotated image, each frame annotated image contains at least one annotated object and a target annotated frame corresponding to each annotated object, and the number of at least one point annotated image is greater than the number of at least one frame annotated image;

[0131] Step S1106: Identify the object to be detected based on the output result to obtain a recognition result.

[0132] Based on the scheme defined by the above steps S1102 to S1106, it can be known that in the embodiment of the present application, a combination of semi-supervised learning and weakly supervised learning is adopted. After obtaining the image to be detected containing the object to be detected, the image to be detected is input into the target detection model, and the output result of the target detection model is obtained. Then, based on the output result, the object to be detected is identified to obtain the identification result. Among them, the target detection model is trained based on the annotated point-annotated image and at least one frame-annotated image, and the annotated point-annotated image is obtained by annotating the object to be annotated corresponding to the target pixel point based on the target prediction frame determined by the target pixel point. The object to be annotated is the object contained in each point-annotated image, and each frame-annotated image contains at least one annotated object and a target annotated frame corresponding to each annotated object. The number of at least one point-annotated image is greater than the number of at least one frame-annotated image, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0133] It is easy to notice that in the above process, the image is annotated based on the semi-supervised learning method, and the obtained image is a box-annotated image; the image is annotated based on the weakly supervised learning method, and the obtained image is a point-annotated image. Since the number of point-annotated images is greater than the number of box-annotated images, that is, in the present application, a small number of images in the target training samples are box-annotated images with higher annotation costs, while a large number of images are point-annotated images with lower annotation costs, that is, in the present application, only a small number of box-annotated images are needed to generate training samples, thereby reducing the annotation cost of training samples. In addition, in the present application, the prediction box corresponding to the object to be annotated is determined based on the target pixel points corresponding to the object to be annotated in the point-annotated image, so that a suitable prediction box can be selected as a pseudo-annotation box for each annotation point, avoiding the problems of missed detection and false detection of pseudo-annotation boxes in semi-supervised learning in the prior art, improving the quality of training samples, and further improving the detection accuracy of the target detection model.

[0134] It can be seen that the solution provided in the present application achieves the purpose of accurately labeling training samples, thereby realizing the technical effect of reducing the labeling cost of training samples, and further solving the technical problem of high labeling cost when labeling training samples in the prior art.

[0135] Example 5

[0136] According to an embodiment of the present application, a device for implementing the above-mentioned method for generating training samples is also provided, such as Fig.12 As shown, the device 120 includes: an acquisition module 1201 , a determination module 1203 , a labeling module 1205 and a generation module 1207 .

[0137] Among them, the acquisition module 1201 is used to obtain at least one point-annotated image and at least one frame-annotated image, wherein each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated, and each frame-annotated image contains at least one annotated object and a target annotation frame corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image; the determination module 1203 is used to determine the target prediction frame corresponding to each object to be annotated based on the target pixels; the annotation module 1205 is used to annotate the object to be annotated in each point-annotated image based on the target prediction frame to obtain the annotated point-annotated image; the generation module 1207 is used to generate a target training sample based on the annotated point-annotated image and at least one frame-annotated image, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on the object in the image to be detected.

[0138] It should be noted that the acquisition module 1201, the determination module 1203, the annotation module 1205 and the generation module 1207 correspond to steps S202 to S208 in Example 1, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules, as part of the device, can be run in the electronic device 10 provided in Example 1.

[0139] Example 6

[0140] According to an embodiment of the present application, there is also provided an apparatus embodiment for implementing the method provided in the above-mentioned second embodiment, the apparatus comprising: a first display module, a second display module, a third display module and a fourth display module.

[0141] Among them, the first display module is used to display at least one point-annotated image, wherein each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated; the second display module is used to respond to the annotation box prediction instruction, determine the target prediction box corresponding to each object to be annotated based on the target pixels, and display the target prediction box on at least one point-annotated image; the third display module is used to respond to the annotation instruction, annotate the object to be annotated in each point-annotated image based on the target prediction box, and display the annotated point-annotated image; the fourth display module is used to respond to the sample generation instruction, generate target training samples based on the annotated point-annotated image and at least one frame-annotated image, and display relevant information of the target training samples, wherein the target training samples are used to train the target detection model, the target detection model is used to perform target detection on the object in the image to be detected, each frame-annotated image contains at least one annotated object, and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0142] It should be noted here that the above-mentioned first display module, second display module, third display module and fourth display module correspond to steps S902 to S908 in Example 2, and the four modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 2.

[0143] Example 7

[0144] According to an embodiment of the present application, there is also provided an apparatus embodiment for implementing the method provided in the above-mentioned embodiment 3, the apparatus comprising: a first acquisition module, a first determination module, a first marking module and a sending module.

[0145] Among them, the first acquisition module is used for the cloud server to acquire at least one point-annotated image sent by the terminal device, wherein each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated; the first determination module is used for the cloud server to determine the target prediction box corresponding to each object to be annotated based on the target pixels, and annotate the object to be annotated in each point-annotated image based on the target prediction box to obtain the annotated point-annotated image; the first annotation module is used for the cloud server to annotate the object to be annotated in each point-annotated image based on the target prediction box to obtain the annotated point-annotated image; the sending module is used for the cloud server to send the annotated point-annotated image to the terminal device, so that the terminal device generates a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and trains a target detection model based on the target training sample, wherein the target detection model is used to perform target detection on the object in the image to be detected, each frame-annotated image contains at least one annotated object, and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0146] It should be noted here that the above-mentioned first acquisition module, first determination module, first labeling module and sending module correspond to steps S1002 to S1008 in Example 3, and the four modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 3.

[0147] Example 8

[0148] According to an embodiment of the present application, a system for generating training samples is also provided. The system includes: a processor and a memory.

[0149] The memory is connected to the processor and is used to store executable instructions of the processor; the processor is configured to execute instructions to implement the training sample generation method and the target object detection method in the above embodiments.

[0150] Example 9

[0151] The embodiment of the present application may provide an electronic device, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0152] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0153] In this embodiment, the electronic device can execute the program code of the following steps in the method for generating training samples: obtaining at least one point-annotated image and at least one frame-annotated image, wherein each point-annotated image contains at least one object to be annotated and target pixel points corresponding to each object to be annotated, and each frame-annotated image contains at least one annotated object and a target annotation frame corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image; determining a target prediction frame corresponding to each object to be annotated based on the target pixel points; annotating the object to be annotated in each point-annotated image based on the target prediction frame to obtain an annotated point-annotated image; generating a target training sample based on the annotated point-annotated image and at least one frame-annotated image, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on the object in the image to be detected.

[0154] Optionally, Fig.13 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.13 As shown, the electronic device 10 may include: one or more (only one is shown in the figure) processors 1302 , a memory 1304 , and a peripheral interface 1306 .

[0155] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method for generating training samples and the method for detecting target objects in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method for generating training samples and the method for detecting target objects. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the electronic device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0156] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain at least one point-annotated image and at least one frame-annotated image, wherein each point-annotated image contains at least one object to be annotated and target pixel points corresponding to each object to be annotated, and each frame-annotated image contains at least one annotated object and a target annotation frame corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image; determine the target prediction frame corresponding to each object to be annotated based on the target pixel points; annotate the object to be annotated in each point-annotated image based on the target prediction frame to obtain an annotated point-annotated image; generate a target training sample based on the annotated point-annotated image and at least one frame-annotated image, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on the object in the image to be detected.

[0157] Optionally, the processor may also execute the program code of the following steps: determining multiple prediction boxes corresponding to the target pixel point based on a preset model; calculating the confidence level corresponding to each prediction box; and determining the target prediction box from the multiple prediction boxes based on the confidence level.

[0158] Optionally, the processor may also execute the program code of the following steps: determining the target object type of the object to be labeled corresponding to the target pixel point based on a preset model; and determining multiple prediction boxes corresponding to the target object type based on the preset model, wherein the object type corresponding to the multiple prediction boxes is the target object type.

[0159] Optionally, the processor may also execute program code of the following steps: determining an initial prediction box with the highest confidence from multiple prediction boxes; and determining the initial prediction box as a target prediction box when the confidence of the initial prediction box is greater than or equal to a preset confidence.

[0160] Optionally, the processor may also execute the program code of the following steps: when the confidence of the initial prediction box is less than a preset confidence, determining the target object type of the object to be labeled corresponding to the target pixel point; obtaining a target image from a preset image library, wherein the target image contains a target object, and the object type of the target object is a target object type; setting the target image at a target image position in a point-labeled image, wherein the target image position is a position of the object to be labeled in the point-labeled image; and determining that the prediction box corresponding to the target image is a target prediction box.

[0161] Optionally, the processor may also execute the following steps of program code: calculating the spatial cost and classification cost between each prediction box and each target pixel, wherein the spatial cost represents the cost of determining the prediction box of the object to be labeled corresponding to the target pixel, and the classification cost represents the cost of classifying the object to be labeled corresponding to the target pixel; calculating the sum of the spatial cost and the classification cost to obtain the prediction cost between each prediction box and each target pixel; and constructing a cost matrix based on the prediction cost, wherein each element in the cost matrix represents the confidence between each prediction box at each target pixel.

[0162] Optionally, the processor may also execute the following steps of program code: after generating a target training sample based on the annotated point annotated image and at least one box annotated image, obtaining a first data set and a second data set, wherein the first data set includes all predicted box information corresponding to at least one point annotated image, the second data set includes all point annotation information corresponding to at least one point annotated image, and the second data set is composed of multiple sub-data sets, each sub-data set includes a positive annotation point and multiple negative annotation points, the positive annotation point is a target pixel point corresponding to the object to be annotated, and the negative annotation point is a non-target pixel point corresponding to the object to be annotated; calculating the cross entropy loss value of the first data set based on all the predicted box information to obtain a first loss value; calculating the confidence score corresponding to each sub-data set, wherein the confidence score is positively correlated with the annotation cost of the point annotated image; performing entropy calculation on the confidence score to obtain a second loss value corresponding to the second data set; calculating the sum of the first loss value, the second loss value and the third loss value to obtain a target loss value, wherein the third loss function is the initial loss value corresponding to the preset model; optimizing the model parameters of the preset model based on the target loss value to obtain an optimized preset model.

[0163] Optionally, the processor may also execute program code for the following steps: displaying at least one point-annotated image, wherein each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated; in response to an annotation box prediction instruction, determining a target prediction box corresponding to each object to be annotated based on the target pixels, and displaying the target prediction box on at least one point-annotated image; in response to an annotation instruction, annotating the object to be annotated in each point-annotated image based on the target prediction box, and displaying the annotated point-annotated image; in response to a sample generation instruction, generating a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and displaying relevant information of the target training sample, wherein the target training sample is used to train an object detection model, and the target detection model is used to perform target detection on objects in the image to be detected, and each frame-annotated image contains at least one annotated object and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0164] Optionally, the processor may also execute program code of the following steps: the cloud server obtains at least one point-annotated image sent by the terminal device, wherein each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated; the cloud server determines a target prediction box corresponding to each object to be annotated based on the target pixels, and annotates the object to be annotated in each point-annotated image based on the target prediction box to obtain an annotated point-annotated image; the cloud server annotates the object to be annotated in each point-annotated image based on the target prediction box to obtain an annotated point-annotated image; the cloud server sends the annotated point-annotated image to the terminal device, so that the terminal device generates a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and trains a target detection model based on the target training sample, wherein the target detection model is used to perform target detection on objects in the image to be detected, each frame-annotated image contains at least one annotated object and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0165] Optionally, the processor may also execute the program code of the following steps: obtaining an image to be detected, wherein the image to be detected contains at least an object to be detected; inputting the image to be detected into a target detection model to obtain an output result of the target detection model, wherein the target detection model is trained based on annotated point-annotated images and at least one box-annotated image, the annotated point-annotated image is obtained by annotating the object to be annotated corresponding to the target pixel point based on a target prediction box determined by the target pixel point, the object to be annotated is an object contained in each point-annotated image, each box-annotated image contains at least one annotated object, and a target annotated box corresponding to each annotated object, the number of at least one point-annotated image is greater than the number of at least one box-annotated image, and the number of at least one point-annotated image is greater than the number of at least one box-annotated image; identifying the object to be detected based on the output result to obtain an identification result.

[0166] It can be understood by those skilled in the art that Fig.12 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.12 The structure of the electronic device is not limited. Fig.12 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.12 Different configurations are shown.

[0167] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0168] Example 10

[0169] The embodiment of the present application also provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the computer-readable storage medium method provided in the above embodiment.

[0170] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0171] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining at least one point-annotated image and at least one box-annotated image, wherein each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated, and each box-annotated image contains at least one annotated object and a target annotated box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one box-annotated image; determining a target prediction box corresponding to each object to be annotated based on the target pixels; annotating the object to be annotated in each point-annotated image based on the target prediction box to obtain an annotated point-annotated image; generating a target training sample based on the annotated point-annotated image and at least one box-annotated image, wherein the target training sample is used to train an object detection model, and the target detection model is used to perform target detection on the object in the image to be detected.

[0172] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining multiple prediction boxes corresponding to the target pixel point based on a preset model; calculating the confidence level corresponding to each prediction box; and determining the target prediction box from the multiple prediction boxes based on the confidence level.

[0173] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: determining a target object type of an object to be labeled corresponding to a target pixel point based on a preset model; and determining multiple prediction boxes corresponding to the target object type based on the preset model, wherein the object type corresponding to the multiple prediction boxes is the target object type.

[0174] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining an initial prediction box with the highest confidence from multiple prediction boxes; when the confidence of the initial prediction box is greater than or equal to a preset confidence, determining the initial prediction box as a target prediction box.

[0175] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: when the confidence of the initial prediction box is less than a preset confidence, determining the target object type of the object to be labeled corresponding to the target pixel point; obtaining a target image from a preset image library, wherein the target image contains a target object, and the object type of the target object is a target object type; setting the target image at a target image position in the point-labeled image, wherein the target image position is a position of the object to be labeled in the point-labeled image; and determining that the prediction box corresponding to the target image is the target prediction box.

[0176] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: calculating the spatial cost and the classification cost between each prediction box and each target pixel, wherein the spatial cost represents the cost of determining the prediction box of the object to be labeled corresponding to the target pixel, and the classification cost represents the cost of classifying the object to be labeled corresponding to the target pixel; calculating the sum of the spatial cost and the classification cost to obtain the prediction cost between each prediction box and each target pixel; and forming a cost matrix based on the prediction cost, wherein each element in the cost matrix represents the confidence between each prediction box under each target pixel.

[0177] Optionally, in this embodiment, the computer-readable storage medium is configured to store a program code for executing the following steps: after generating a target training sample based on the annotated point-annotated image and at least one box-annotated image, obtaining a first data set and a second data set, wherein the first data set includes all predicted box information corresponding to at least one point-annotated image, the second data set includes all point annotation information corresponding to at least one point-annotated image, and the second data set is composed of multiple sub-data sets, each of which includes a positive annotation point and multiple negative annotation points, the positive annotation point is a target pixel point corresponding to the object to be annotated, and the negative annotation point is a non-target pixel point corresponding to the object to be annotated; calculating the cross entropy loss value of the first data set based on all the predicted box information to obtain a first loss value; calculating the confidence score corresponding to each sub-data set, wherein the confidence score is positively correlated with the annotation cost of the point-annotated image; performing entropy calculation on the confidence score to obtain a second loss value corresponding to the second data set; calculating the sum of the first loss value, the second loss value and the third loss value to obtain a target loss value, wherein the third loss function is the initial loss value corresponding to the preset model; optimizing the model parameters of the preset model based on the target loss value to obtain an optimized preset model.

[0178] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: displaying at least one point-annotated image, wherein each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated; in response to an annotation box prediction instruction, determining a target prediction box corresponding to each object to be annotated based on the target pixels, and displaying the target prediction box on at least one point-annotated image; in response to an annotation instruction, annotating the object to be annotated in each point-annotated image based on the target prediction box, and displaying the annotated point-annotated image; in response to a sample generation instruction, generating a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and displaying relevant information of the target training sample, wherein the target training sample is used to train an object detection model, and the target detection model is used to perform target detection on objects in the image to be detected, and each frame-annotated image contains at least one annotated object and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0179] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: the cloud server obtains at least one point-annotated image sent by the terminal device, wherein each point-annotated image contains at least one object to be annotated and target pixels corresponding to each object to be annotated; the cloud server determines a target prediction box corresponding to each object to be annotated based on the target pixels, and annotates the object to be annotated in each point-annotated image based on the target prediction box to obtain an annotated point-annotated image; the cloud server annotates the object to be annotated in each point-annotated image based on the target prediction box to obtain an annotated point-annotated image; the cloud server sends the annotated point-annotated image to the terminal device, so that the terminal device generates a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and trains a target detection model based on the target training sample, wherein the target detection model is used to perform target detection on the object in the image to be detected, each frame-annotated image contains at least one annotated object and a target annotation box corresponding to each annotated object, and the number of at least one point-annotated image is greater than the number of at least one frame-annotated image.

[0180] Optionally, in this embodiment, a computer-readable storage medium is configured to store a program code for executing the following steps: obtaining an image to be detected, wherein the image to be detected contains at least an object to be detected; inputting the image to be detected into a target detection model to obtain an output result of the target detection model, wherein the target detection model is trained based on annotated point-annotated images and at least one box-annotated image, and the annotated point-annotated image is obtained by annotating the object to be annotated corresponding to the target pixel point based on a target prediction box determined by the target pixel point, the object to be annotated is an object contained in each point-annotated image, each box-annotated image contains at least one annotated object, and a target annotated box corresponding to each annotated object, the number of at least one point-annotated image is greater than the number of at least one box-annotated image, and the number of at least one point-annotated image is greater than the number of at least one box-annotated image; based on the output result, the object to be detected is identified to obtain an identification result.

[0181] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0182] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0184] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0186] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0187] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating training samples, characterized in that: include: Acquire at least one point-annotated image and at least one frame-annotated image, wherein each point-annotated image includes at least one object to be annotated and target pixel points corresponding to each object to be annotated, and each frame-annotated image includes at least one annotated object and a target annotated frame corresponding to each annotated object, and the number of the at least one point-annotated image is greater than the number of the at least one frame-annotated image; Determine a target prediction box corresponding to each object to be labeled based on the target pixel point; Annotating the object to be annotated in each point-annotated image based on the target prediction frame to obtain an annotated point-annotated image; Generating a target training sample based on the annotated point annotated image and the at least one frame annotated image, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on an object in the image to be detected; Wherein, determining the target prediction box corresponding to each to-be-annotated object based on the target pixel point includes: Determine a plurality of prediction boxes corresponding to the target pixel points based on a preset model; Calculate the confidence corresponding to each prediction box; Determining the target prediction box from the multiple prediction boxes according to the confidence level; Calculate the confidence corresponding to each prediction box, including: Calculating the spatial cost and classification cost between each prediction box and each target pixel point, wherein the spatial cost represents the cost of determining the prediction box of the object to be labeled corresponding to the target pixel point, and the classification cost represents the cost of classifying the object to be labeled corresponding to the target pixel point; Calculating the sum of the spatial cost and the classification cost to obtain a prediction cost between each of the prediction boxes and each of the target pixels; A cost matrix is ​​formed based on the prediction cost, wherein each element in the cost matrix represents the confidence between each prediction box at each target pixel point.

2. The method according to claim 1, characterized in that: Determining a plurality of prediction boxes corresponding to the target pixel point based on a preset model includes: Determine the target object type of the object to be labeled corresponding to the target pixel point based on the preset model; Based on the preset model, a plurality of prediction boxes corresponding to the target object type are determined, wherein the object type corresponding to the plurality of prediction boxes is the target object type.

3. The method according to claim 1, characterized in that Determining the target prediction box from the multiple prediction boxes according to the confidence level includes: Determine the initial prediction frame with the greatest confidence from the multiple prediction frames; When the confidence of the initial prediction box is greater than or equal to a preset confidence, the initial prediction box is determined to be the target prediction box.

4. The method according to claim 3, characterized in that The method further comprises: When the confidence of the initial prediction frame is less than the preset confidence, determining the target object type of the object to be labeled corresponding to the target pixel point; Acquire a target image from a preset image library, wherein the target image includes a target object, and the object type of the target object is the target object type; Setting the target image at the target image position in the point-annotated image, wherein the target image position is the position of the object to be annotated in the point-annotated image; A prediction box corresponding to the target image is determined as the target prediction box.

5. The method according to claim 1, characterized in that After generating a target training sample based on the annotated point annotated image and the at least one frame annotated image, the method further includes: Acquire a first data set and a second data set, wherein the first data set includes all prediction box information corresponding to the at least one point-annotated image, the second data set includes all point annotation information corresponding to the at least one point-annotated image, and the second data set is composed of a plurality of sub-data sets, each of which includes a positive annotation point and a plurality of negative annotation points, the positive annotation point is a target pixel point corresponding to the object to be annotated, and the negative annotation point is a non-target pixel point corresponding to the object to be annotated; Calculate the cross entropy loss value of the first data set based on all the prediction box information to obtain a first loss value; Calculating a confidence score corresponding to each sub-dataset, wherein the confidence score is positively correlated with the annotation cost of the point annotation image; Performing entropy calculation on the confidence score to obtain a second loss value corresponding to the second data set; Calculating the sum of the first loss value, the second loss value and the third loss value to obtain a target loss value, wherein the third loss value is an initial loss value corresponding to the preset model; The model parameters of the preset model are optimized based on the target loss value to obtain an optimized preset model.

6. A method for generating training samples, characterized in that: include: Displaying at least one point-annotated image, wherein each point-annotated image includes at least one object to be annotated and target pixel points corresponding to each object to be annotated; In response to the annotation box prediction instruction, determine a target prediction box corresponding to each of the to-be-annotated objects based on the target pixel points, and display the target prediction box on the at least one point-annotated image; In response to the labeling instruction, label the object to be labeled in each point-labeled image based on the target prediction frame, and display the labeled point-labeled image; In response to the sample generation instruction, a target training sample is generated based on the annotated point annotated image and at least one frame annotated image, and relevant information of the target training sample is displayed, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on an object in an image to be detected, each frame annotated image includes at least one annotated object and a target annotated frame corresponding to each annotated object, and the number of the at least one point annotated image is greater than the number of the at least one frame annotated image; Wherein, determining the target prediction box corresponding to each to-be-annotated object based on the target pixel point includes: Determine a plurality of prediction boxes corresponding to the target pixel points based on a preset model; Calculate the confidence corresponding to each prediction box; Determining the target prediction box from the multiple prediction boxes according to the confidence level; Calculate the confidence corresponding to each prediction box, including: Calculating the spatial cost and classification cost between each prediction box and each target pixel point, wherein the spatial cost represents the cost of determining the prediction box of the object to be labeled corresponding to the target pixel point, and the classification cost represents the cost of classifying the object to be labeled corresponding to the target pixel point; Calculating the sum of the spatial cost and the classification cost to obtain a prediction cost between each of the prediction boxes and each of the target pixels; A cost matrix is ​​formed based on the prediction cost, wherein each element in the cost matrix represents the confidence between each prediction box at each target pixel point.

7. A method for generating training samples, characterized in that: include: The cloud server obtains at least one point-annotated image sent by the terminal device, wherein each point-annotated image includes at least one object to be annotated and target pixel points corresponding to each object to be annotated; The cloud server determines a target prediction box corresponding to each object to be labeled based on the target pixel point, and labels the object to be labeled in each point-labeled image based on the target prediction box to obtain a labeled point-labeled image; The cloud server annotates the object to be annotated in each point annotated image based on the target prediction frame to obtain an annotated point annotated image; The cloud server sends the annotated point-annotated image to the terminal device, so that the terminal device generates a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and trains a target detection model based on the target training sample, wherein the target detection model is used to perform target detection on an object in the image to be detected, each frame-annotated image includes at least one annotated object and a target annotation frame corresponding to each annotated object, and the number of the at least one point-annotated image is greater than the number of the at least one frame-annotated image; Wherein, determining the target prediction box corresponding to each to-be-annotated object based on the target pixel point includes: Determine a plurality of prediction boxes corresponding to the target pixel points based on a preset model; Calculate the confidence corresponding to each prediction box; Determining the target prediction box from the multiple prediction boxes according to the confidence level; Calculate the confidence corresponding to each prediction box, including: Calculating the spatial cost and classification cost between each prediction box and each target pixel point, wherein the spatial cost represents the cost of determining the prediction box of the object to be labeled corresponding to the target pixel point, and the classification cost represents the cost of classifying the object to be labeled corresponding to the target pixel point; Calculating the sum of the spatial cost and the classification cost to obtain a prediction cost between each of the prediction boxes and each of the target pixels; A cost matrix is ​​formed based on the prediction cost, wherein each element in the cost matrix represents the confidence between each prediction box at each target pixel point.

8. A method for detecting a target object, characterized in that: include: Acquire an image to be detected, wherein the image to be detected at least contains an object to be detected; Input the image to be detected into the target detection model described in any one of claims 1 to 5, and obtain the output result of the target detection model, wherein the target detection model is trained based on the annotated point annotated image and at least one frame annotated image, and the annotated point annotated image is obtained by annotating the object to be annotated corresponding to the target pixel point based on the target prediction frame determined by the target pixel point, the object to be annotated is the object contained in each point annotated image, each frame annotated image contains at least one annotated object and a target annotated frame corresponding to each annotated object, and the number of at least one point annotated image is greater than the number of the at least one frame annotated image; The object to be detected is identified based on the output result to obtain an identification result.

9. A training sample generation device, characterized in that: include: an acquisition module, configured to acquire at least one point-annotated image and at least one frame-annotated image, wherein each point-annotated image includes at least one object to be annotated and target pixel points corresponding to each object to be annotated, and each frame-annotated image includes at least one annotated object and a target annotated frame corresponding to each annotated object, and the number of the at least one point-annotated image is greater than the number of the at least one frame-annotated image; A determination module, used to determine a target prediction box corresponding to each object to be labeled based on the target pixel point; A labeling module, used for labeling the object to be labeled in each point-labeled image based on the target prediction frame to obtain a labeled point-labeled image; A generating module, configured to generate a target training sample based on the annotated point annotated image and the at least one frame annotated image, wherein the target training sample is used to train a target detection model, and the target detection model is used to perform target detection on an object in an image to be detected; The determination module is further used to determine multiple prediction boxes corresponding to the target pixel point based on a preset model; calculate the confidence level corresponding to each prediction box; and determine the target prediction box from the multiple prediction boxes according to the confidence level; The determination module is also used to calculate the spatial cost and classification cost between each prediction box and each target pixel point, wherein the spatial cost represents the cost of determining the prediction box of the object to be labeled corresponding to the target pixel point, and the classification cost represents the cost of classifying the object to be labeled corresponding to the target pixel point; the sum of the spatial cost and the classification cost is calculated to obtain the prediction cost between each prediction box and each target pixel point; and a cost matrix is ​​formed based on the prediction cost, wherein each element in the cost matrix represents the confidence between each prediction box at each target pixel point.

10. A training sample generation device, characterized in that: include: A first display module is used to display at least one point-annotated image, wherein each point-annotated image includes at least one object to be annotated and target pixels corresponding to each object to be annotated; A second display module is used to respond to the annotation box prediction instruction, determine the target prediction box corresponding to each object to be annotated based on the target pixel point, and display the target prediction box on the at least one point annotated image; A third display module is used to respond to the labeling instruction, label the object to be labeled in each point-labeled image based on the target prediction frame, and display the labeled point-labeled image; a fourth display module, configured to respond to a sample generation instruction, generate a target training sample based on the annotated point annotated image and at least one frame annotated image, and display relevant information of the target training sample, wherein the target training sample is used to train a target detection model, the target detection model is used to perform target detection on an object in an image to be detected, each frame annotated image includes at least one annotated object and a target annotated frame corresponding to each annotated object, and the number of the at least one point annotated image is greater than the number of the at least one frame annotated image; The second display module is further used to determine multiple prediction boxes corresponding to the target pixel point based on a preset model; calculate the confidence level corresponding to each prediction box; and determine the target prediction box from the multiple prediction boxes according to the confidence level; The second display module is also used to calculate the spatial cost and classification cost between each prediction box and each target pixel point, wherein the spatial cost represents the cost of determining the prediction box of the object to be labeled corresponding to the target pixel point, and the classification cost represents the cost of classifying the object to be labeled corresponding to the target pixel point; the sum of the spatial cost and the classification cost is calculated to obtain the prediction cost between each prediction box and each target pixel point; and a cost matrix is ​​formed based on the prediction cost, wherein each element in the cost matrix represents the confidence between each prediction box at each target pixel point.

11. A training sample generation device, characterized in that: include: A first acquisition module is used for the cloud server to acquire at least one point-annotated image sent by the terminal device, wherein each point-annotated image includes at least one object to be annotated and a target pixel point corresponding to each object to be annotated; A first determination module is used for the cloud server to determine a target prediction box corresponding to each object to be labeled based on the target pixel point, and to label the object to be labeled in each point-labeled image based on the target prediction box to obtain a labeled point-labeled image; A first labeling module, configured for the cloud server to label the object to be labeled in each point-labeled image based on the target prediction frame to obtain a labeled point-labeled image; A sending module, used for the cloud server to send the annotated point-annotated image to the terminal device, so that the terminal device generates a target training sample based on the annotated point-annotated image and at least one frame-annotated image, and trains a target detection model based on the target training sample, wherein the target detection model is used to perform target detection on an object in an image to be detected, each frame-annotated image contains at least one annotated object and a target annotation frame corresponding to each annotated object, and the number of the at least one point-annotated image is greater than the number of the at least one frame-annotated image; The first determination module is further used to determine a plurality of prediction boxes corresponding to the target pixel point based on a preset model; calculate the confidence level corresponding to each prediction box; and determine the target prediction box from the plurality of prediction boxes according to the confidence level; The first determination module is also used to calculate the spatial cost and classification cost between each prediction box and each target pixel point, wherein the spatial cost represents the cost of determining the prediction box of the object to be labeled corresponding to the target pixel point, and the classification cost represents the cost of classifying the object to be labeled corresponding to the target pixel point; the sum of the spatial cost and the classification cost is calculated to obtain the prediction cost between each prediction box and each target pixel point; and a cost matrix is ​​formed based on the prediction cost, wherein each element in the cost matrix represents the confidence between each prediction box at each target pixel point.

12. A computer-readable storage medium, characterized in that: A computer program is stored in a computer-readable storage medium, wherein the computer program is configured to execute the method for generating training samples as described in any one of claims 1 to 7 and the method for detecting target objects as described in claim 8 when running.

13. An electronic device, characterized in that: The electronic device includes one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the method for generating training samples described in any one of claims 1 to 7 and the method for detecting target objects described in claim 8 when running.

14. A training sample generation system, characterized in that: include: processor; as well as A memory connected to the processor, for storing executable instructions of the processor; wherein the processor is configured to execute the instructions to implement the method for generating training samples as described in any one of claims 1 to 7 and the method for detecting a target object as described in claim 8.

Images