Object detection model training method, object detection method and device

Abstract

The present application relates to an object detection model training method, an object detection method, an apparatus, a computer device, a storage medium and a computer program product. The method comprises: inputting each first image in a first image set into an initial object detection model to obtain a set of initial detection frames corresponding to each first image; for the same set of initial detection frames, based on the detection frame similarity between the initial detection frames belonging to the same object category, matching the initial detection frames output by the current object detection network with the initial detection frames output by each other object detection network to obtain a matching detection frame corresponding to the initial detection frame output by the current object detection network; fusing the initial detection frame and the corresponding matching detection frames to obtain a virtual detection frame; based on the first image set and the corresponding virtual detection frame sets, training the initial object detection model to obtain a target object detection model. The method can reduce the annotation cost of training samples.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an object detection model training method, an object detection method, an apparatus, a computer device, a storage medium, and a computer program product. Background Art

[0002] With the development of image processing technology, object detection tasks have emerged. Object detection tasks use boxes to mark the location of objects in images and give the categories of objects. For object detection tasks, they are usually processed through object detection models. The image is input into a trained object detection model, and the model outputs a predicted box and the corresponding object category.

[0003] In traditional technology, objects in images are usually manually labeled to obtain training data for the model, and then the model is trained based on the training data. However, pure manual labeling is time-consuming and labor-intensive, and has the problem of high labeling costs. Summary of the invention

[0004] Based on this, it is necessary to provide an object detection model training method, object detection method, device, computer equipment, computer-readable storage medium and computer program product that can reduce the labeling cost in response to the above technical problems.

[0005] The present application provides an object detection model training method. The method comprises:

[0006] Acquire a first image set; the first image set is an unlabeled image set;

[0007] Inputting each first image in the first image set into an initial object detection model to obtain a set of initial detection frames corresponding to each first image; the set of initial detection frames corresponding to the first images includes initial detection frames outputted by each object detection network in the initial object detection model and object categories corresponding to the initial detection frames;

[0008] For the same set of initial detection frames, based on the detection frame similarity between the initial detection frames belonging to the same object category, the initial detection frames output by the current object detection network are matched with the initial detection frames output by each other object detection network to obtain the matching detection frame corresponding to the initial detection frame output by the current object detection network;

[0009] The initial detection frame and the corresponding matching detection frames are fused to obtain a virtual detection frame, and a set of virtual detection frames corresponding to the first image is obtained based on the virtual detection frames corresponding to the same first image;

[0010] Based on the first image set and the corresponding virtual detection frame sets, the initial object detection model is trained to obtain a target object detection model.

[0011] In one embodiment, the object detection network includes an extraction layer and a prediction layer connected in series, the extraction layer is shared between the object detection networks, the prediction layer includes a prediction branch of at least one scale, and the output data of the object detection network is obtained by fusing the output data of the prediction branches of each scale.

[0012] In one embodiment, the method further comprises:

[0013] Get a test image set;

[0014] Based on the test image set, each object detection network of the target object detection model is tested to obtain the detection accuracy corresponding to each object detection network;

[0015] Based on the detection accuracy, determining a key object detection network from each object detection network of the target object detection model;

[0016] The other object detection networks in the target object detection model except the key object detection network are filtered to obtain an updated target object detection model; the updated target object detection model is used to process the object detection task for the image.

[0017] The present application also provides an object detection model training device. The device comprises:

[0018] An image set acquisition module, used to acquire a first image set; the first image set is an unlabeled image set;

[0019] an initial detection frame determination module, configured to input each first image in the first image set into an initial object detection model to obtain a set of initial detection frames corresponding to each first image; the set of initial detection frames corresponding to the first images includes initial detection frames outputted by each object detection network in the initial object detection model and object categories corresponding to the initial detection frames;

[0020] A detection frame matching module is used to match the initial detection frame output by the current object detection network with the initial detection frames output by each other object detection network for the same initial detection frame set based on the detection frame similarity between the initial detection frames belonging to the same object category, so as to obtain a matching detection frame corresponding to the initial detection frame output by the current object detection network;

[0021] A virtual detection frame determination module is used to fuse the initial detection frame and the corresponding matching detection frames to obtain a virtual detection frame, and obtain a set of virtual detection frames corresponding to the first image based on the virtual detection frames corresponding to the same first image;

[0022] The model training module is used to perform model training on the initial object detection model based on the first image set and the corresponding virtual detection frame sets to obtain a target object detection model.

[0023] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the above-mentioned object detection model training method when executing the computer program.

[0024] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps described in the above-mentioned object detection model training method.

[0025] A computer program product includes a computer program, and when the computer program is executed by a processor, the steps described in the above-mentioned object detection model training method are implemented.

[0026] The object detection model training method, apparatus, computer equipment, storage medium and computer program product obtain a first image set, which is an unlabeled image set, input each first image in the first image set into an initial object detection model, and obtain a set of initial detection frames corresponding to each first image. The initial detection frame set corresponding to the first image includes the initial detection frames output by each object detection network in the initial object detection model and the object category corresponding to the initial detection frame. For the same initial detection frame set, based on the detection frame similarity between the initial detection frames belonging to the same object category, the initial detection frame output by the current object detection network is matched with the initial detection frames output by each other object detection network to obtain the matching detection frame corresponding to the initial detection frame output by the current object detection network, the initial detection frame and the corresponding matching detection frames are fused to obtain a virtual detection frame, based on the virtual detection frames corresponding to the same first image, a set of virtual detection frames corresponding to the first image is obtained, and based on the first image set and the corresponding virtual detection frame sets, the initial object detection model is trained to obtain a target object detection model. In this way, the unlabeled image is input into the model for object detection, and the unlabeled image is intelligently marked with a virtual detection frame based on the initial detection frame set obtained by detection. The virtual detection frame is a pseudo-label, and the unlabeled image and the pseudo-label are used as training data for the model, so that there is no need for manual labeling, and images with labeled information can be obtained to train the model, which effectively reduces the image labeling cost. Furthermore, when generating a virtual detection frame, the initial detection frames output by different object detection networks in the model for the same image are matched. The matching initial detection frames belong to the same object category and are relatively similar. The matching detection frames corresponding to the same initial detection frame are fused to obtain a virtual detection frame with higher accuracy, which helps to improve the quality of model training.

[0027] The present application provides an object detection method. The method comprises:

[0028] Acquire the image to be detected;

[0029] Inputting the image to be detected into a target object detection model to obtain a set of object detection frames corresponding to the image to be detected;

[0030] The training process of the target object detection model includes the following steps:

[0031] Acquire a first image set; the first image set is an unlabeled image set; input each first image in the first image set into an initial object detection model to obtain a set of initial detection frames corresponding to each first image; the set of initial detection frames corresponding to the first image includes the initial detection frames output by each object detection network in the initial object detection model and the object category corresponding to the initial detection frame; for the same initial detection frame set, based on the detection frame similarity between the initial detection frames belonging to the same object category, match the initial detection frame output by the current object detection network with the initial detection frame output by each other object detection network to obtain the matching detection frame corresponding to the initial detection frame output by the current object detection network; fuse the initial detection frame and the corresponding matching detection frames to obtain a virtual detection frame, and based on the virtual detection frames corresponding to the same first image, obtain a set of virtual detection frames corresponding to the first image; based on the first image set and the corresponding virtual detection frame sets, perform model training on the initial object detection model to obtain a target object detection model.

[0032] In one embodiment, the image to be detected is a video frame; the method further includes:

[0033] Obtain a set of video frames corresponding to the target video, input each video frame in the set of video frames into the target object detection model, and obtain a set of object detection frames corresponding to each video frame;

[0034] Extracting features of each object sub-image corresponding to the object detection frame set to obtain multiple object features corresponding to the target video;

[0035] Performing clustering processing on each object feature corresponding to the target video to obtain multiple central features;

[0036] Based on the feature distance between the object feature and the central feature, determining each target subsidiary feature corresponding to the central feature from the object features;

[0037] Based on each central feature and each corresponding target subsidiary feature, video representation information corresponding to the target video is obtained.

[0038] In one embodiment, the determining, from the various object features, various target subsidiary features corresponding to the central feature based on the feature distance between the object feature and the central feature includes:

[0039] Obtain the central feature with the smallest feature distance from the object feature as the target central feature corresponding to the object feature, and obtain the target central features corresponding to each object feature respectively;

[0040] Based on the feature distances between each object feature corresponding to the current target center feature and the current target center feature, a distance variance value corresponding to the feature distance is calculated;

[0041] From each object feature corresponding to the current target center feature, delete the object features of a first preset proportion according to the distance variance values ​​from large to small, to obtain each initial subsidiary feature corresponding to the current target center feature;

[0042] Based on the initial subsidiary features corresponding to the current target central feature, the target subsidiary features corresponding to the current target central feature are obtained.

[0043] In one embodiment, obtaining each target subsidiary feature corresponding to the current target center feature based on each initial subsidiary feature corresponding to the current target center feature includes:

[0044] Based on the feature distances between each initial subsidiary feature corresponding to the current target center feature and the current target center feature, obtaining the initial subsidiary features of a second preset proportion as intermediate subsidiary features in ascending order of feature distances;

[0045] An intermediate subsidiary feature whose feature distance is less than the first preset distance is acquired as a target subsidiary feature, and each target subsidiary feature corresponding to the current target center feature is obtained.

[0046] In one embodiment, the target video includes a query video and a candidate video; the method further includes:

[0047] Determining a recalled video corresponding to the query video from each candidate video based on a feature distance between a reference feature corresponding to the query video and a reference feature corresponding to the candidate video; the reference feature includes at least one of a central feature or a target-attached feature;

[0048] Matching the video representation information corresponding to the query video with the video representation information corresponding to the recalled video to determine a matching degree between the query video and the recalled video;

[0049] A query result corresponding to the query video is determined based on the matching degree.

[0050] In one embodiment, matching the video representation information corresponding to the query video with the video representation information corresponding to the recalled video to determine the matching degree between the query video and the recalled video includes:

[0051] The central feature corresponding to the query video is used as the query central feature, and the central feature corresponding to the recall video is used as the recall central feature;

[0052] From each recall center feature, determine the matching center feature corresponding to the query center feature, and form a center feature pair with the query center feature and the corresponding matching center feature;

[0053] Among the target subsidiary features corresponding to the recalled videos, the target subsidiary features whose feature distance with the query central feature in the central feature pair is less than a second preset distance are counted to obtain the number of subsidiary feature matches corresponding to each central feature pair;

[0054] Based on the number of subsidiary feature matches corresponding to each of the central feature pairs, the degree of match between the query video and the recalled video is determined.

[0055] In one embodiment, when the query task corresponding to the query video is a duplicate elimination task, determining the matching degree between the query video and the recalled video based on the number of subsidiary feature matches corresponding to each of the central feature pairs includes:

[0056] Based on the proportion of the number of subsidiary feature matches in the total number of subsidiary features corresponding to the query video and the number of central features corresponding to the query video, first matching scores corresponding to the central feature pairs are calculated to obtain first matching scores corresponding to each central feature pair;

[0057] Based on the first matching scores corresponding to the respective central feature pairs, obtaining a matching degree between the query video and the recalled video;

[0058] The determining, based on the matching degree, a query result corresponding to the query video includes:

[0059] Determine a target matching degree from each matching degree, and determine the innovation degree corresponding to the query video based on the target matching degree; the innovation degree is negatively correlated with the target matching degree;

[0060] A query result corresponding to the query video is determined based on the novelty.

[0061] In one embodiment, when the query task corresponding to the query video is a duplicate checking task, determining the matching degree between the query video and the recalled video based on the number of matching subsidiary features corresponding to each of the central feature pairs includes:

[0062] Based on the proportion of the number of subsidiary feature matches in the total number of subsidiary features corresponding to the recalled video and the number of central features corresponding to the recalled video, the second matching scores corresponding to the central feature pairs are calculated to obtain the second matching scores corresponding to the respective central feature pairs;

[0063] Obtaining a degree of match between the query video and the recalled video based on the second matching scores corresponding to the respective central feature pairs;

[0064] The determining, based on the matching degree, a query result corresponding to the query video includes:

[0065] Sort the recalled videos according to the matching degree from large to small to obtain a recalled video sequence;

[0066] The recalled video sequence is used as the query result corresponding to the query video.

[0067] The present application also provides an object detection device. The device comprises:

[0068] An image acquisition module, used for acquiring an image to be detected;

[0069] An object detection module is used to input the image to be detected into a target object detection model to obtain a set of object detection frames corresponding to the image to be detected;

[0070] The training process of the target object detection model includes the following steps:

[0071] Acquire a first image set; the first image set is an unlabeled image set; input each first image in the first image set into an initial object detection model to obtain a set of initial detection frames corresponding to each first image; the set of initial detection frames corresponding to the first image includes the initial detection frames output by each object detection network in the initial object detection model and the object category corresponding to the initial detection frame; for the same initial detection frame set, based on the detection frame similarity between the initial detection frames belonging to the same object category, match the initial detection frame output by the current object detection network with the initial detection frame output by each other object detection network to obtain the matching detection frame corresponding to the initial detection frame output by the current object detection network; fuse the initial detection frame and the corresponding matching detection frames to obtain a virtual detection frame, and based on the virtual detection frames corresponding to the same first image, obtain a set of virtual detection frames corresponding to the first image; based on the first image set and the corresponding virtual detection frame sets, perform model training on the initial object detection model to obtain a target object detection model.

[0072] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the object detection method when executing the computer program.

[0073] A computer-readable storage medium stores a computer program, which implements the steps of the object detection method when executed by a processor.

[0074] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the steps of the object detection method are implemented.

[0075] The object detection method, device, computer equipment, storage medium and computer program product described above obtain an image to be detected, input the image to be detected into a target object detection model, and obtain a set of object detection frames corresponding to the image to be detected. In the training process of the model, an unlabeled image is input into the model for object detection, and a virtual detection frame is intelligently added to the unlabeled image based on the initial detection frame set obtained by the detection. The virtual detection frame is a pseudo-label, and the unlabeled image and the pseudo-label are used as training data for the model, so that there is no need for manual labeling, and images with labeling information can be obtained to train the model, which effectively reduces the image labeling cost. Furthermore, when generating a virtual detection frame, the initial detection frames output by different object detection networks in the model for the same image are matched. The mutually matched initial detection frames belong to the same object category and are relatively similar. The matching detection frames corresponding to the same initial detection frame are fused to obtain a virtual detection frame with higher accuracy, which helps to improve the quality of model training. The target object detection model is trained based on non-manually annotated images. The training process effectively reduces the image annotation cost and has a certain accuracy. The image to be detected is input into the target object detection model. The model can accurately predict the area where the object in the image to be detected is located and obtain the object detection box set corresponding to the image to be detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A diagram of an application environment of an object detection model training method and an object detection method in one embodiment;

[0077] Figure 2 A schematic diagram of a flow chart of an object detection model training method in one embodiment;

[0078] Figure 3 A schematic diagram of a process for updating model parameters in one embodiment;

[0079] Figure 4 is a flowchart of an object detection model training method in another embodiment;

[0080] Figure 5Ais a schematic diagram of a model structure in one embodiment;

[0081] Figure 5B A schematic diagram of the main structural parameters of the model in one embodiment;

[0082] Figure 5C A schematic diagram of model training in one embodiment;

[0083] Figure 6 A schematic diagram of a model application in one embodiment;

[0084] Figure 7 is a schematic diagram of an object detection model used when applying a model in one embodiment;

[0085] Figure 8 is a flow chart of an object detection method in one embodiment;

[0086] Fig. 9 A schematic diagram of a video detection process in one embodiment;

[0087] Fig.10 is a structural block diagram of an object detection model training device in one embodiment;

[0088] Fig.11 is a structural block diagram of an object detection model training device in another embodiment;

[0089] Fig.12 is a structural block diagram of an object detection device in one embodiment;

[0090] Fig.13 is an internal structure diagram of a computer device in one embodiment;

[0091] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0092] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0093] The solution provided in the embodiments of the present application involves artificial intelligence computer vision technology, machine learning and other technologies, which are specifically described by the following embodiments:

[0094] The object detection model training method and object detection method provided in the embodiments of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. Among them, the terminal 102 can be but is not limited to various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers or a cloud server.

[0095] Both the terminal and the server can be used independently to execute the object detection model training method and the object detection method provided in the embodiments of the present application.

[0096] For example, the server obtains a first image set, inputs each first image in the first image set into an initial object detection model, and obtains a set of initial detection frames corresponding to each first image, wherein the first image set is an unlabeled image set, and the set of initial detection frames corresponding to the first images includes the initial detection frames output by each object detection network in the initial object detection model and the object categories corresponding to the initial detection frames. For the same set of initial detection frames, the server matches the initial detection frames output by the current object detection network with the initial detection frames output by each other object detection network based on the detection frame similarity between the initial detection frames belonging to the same object category, and obtains the matching detection frames corresponding to the initial detection frames output by the current object detection network. The server fuses the initial detection frame and the corresponding matching detection frames to obtain a virtual detection frame, and obtains a set of virtual detection frames corresponding to the first image based on the virtual detection frames corresponding to the same first image. The server performs model training on the initial object detection model based on the first image set and the corresponding virtual detection frame sets to obtain a target object detection model.

[0097] The server obtains the image to be detected, inputs the image to be detected into the target object detection model, and obtains a set of object detection frames corresponding to the image to be detected.

[0098] The terminal and the server can also be used together to execute the object detection model training method and the object detection method provided in the embodiments of the present application.

[0099] For example, the server obtains a first image set from a terminal, inputs each first image in the first image set into an initial object detection model, and obtains a set of initial detection frames corresponding to each first image. For the same set of initial detection frames, the server matches the initial detection frames output by the current object detection network with the initial detection frames output by each other object detection network based on the detection frame similarity between the initial detection frames belonging to the same object category, and obtains a matching detection frame corresponding to the initial detection frame output by the current object detection network. The server fuses the initial detection frame and the corresponding matching detection frames to obtain a virtual detection frame, and obtains a set of virtual detection frames corresponding to the first image based on the virtual detection frames corresponding to the same first image. The server performs model training on the initial object detection model based on the first image set and the corresponding virtual detection frame sets to obtain a target object detection model.

[0100] The server sends the target object detection model to the terminal. The terminal obtains the image to be detected, inputs the image to be detected into the target object detection model, and obtains the object detection frame set corresponding to the image to be detected.

[0101] In one embodiment, Figure 2 As shown, a method for training an object detection model is provided. The method is applied to a computer device as an example. The computer device can be a terminal or a server. The method can be executed by the terminal or the server itself, or can be implemented through interaction between the terminal and the server. Figure 2 ,The object detection model training method includes the following steps:

[0102] Step S202, obtaining a first image set; the first image set is an unlabeled image set.

[0103] The unlabeled image set includes multiple unlabeled images. An unlabeled image refers to an image in which the object in the image is not labeled, that is, an unlabeled image refers to an image in which the area where the object is located and the category to which it belongs are unknown. It can be understood that an unlabeled image may contain an object or may not contain an object.

[0104] An object may be a living thing, such as an animal, a plant, etc., or an object may be an inanimate object, such as an article, a product, etc.

[0105] Specifically, a computer device obtains a first image set locally or from other devices, where the first image set is an unlabeled image set. Pseudo-annotation information corresponding to the unlabeled images is generated based on an initial object detection model. The initial object detection model is trained based on the unlabeled images and the corresponding pseudo-annotation information, thereby obtaining a target object detection model that can be put into use.

[0106] In one embodiment, a computer device obtains an unlabeled video, performs frame extraction processing on the unlabeled video to obtain a plurality of unlabeled images, uses the unlabeled images as first images, and groups the first images into a first image set.

[0107] Step S204: input each first image in the first image set into the initial object detection model to obtain a set of initial detection frames corresponding to each first image; the set of initial detection frames corresponding to the first images includes the initial detection frames output by each object detection network in the initial object detection model and the object categories corresponding to the initial detection frames.

[0108] Among them, the object detection model is a machine learning model for detecting objects in an image. The input data of the object detection model is an image, and the output data includes a detection frame and an object category corresponding to the detection frame. Furthermore, the output data may also include a confidence corresponding to the detection frame. A detection frame refers to a positioning frame for locating an object in an image, for example, a rectangular frame is used to circle the image area where the object is located. The object category corresponding to the detection frame is used to identify the category to which the object circled by the detection frame belongs, that is, to determine what specific object the object circled by the detection frame is, for example, the object category corresponding to a certain detection frame is a cat or a dog or a table or a car, etc. The confidence corresponding to the detection frame is used to indicate the category probability corresponding to the object circled by the detection frame. It can be understood that the confidence corresponding to the detection frame may include the category probability that the object circled by the detection frame belongs to each candidate object category, and the confidence corresponding to the detection frame may also include only the maximum category probability, and the object category corresponding to the detection frame is the candidate object category corresponding to the maximum category probability. The model can identify whether there is an object in the image by performing data processing on the image information of the input image, predict the area where the object is located in the image, output the detection box, predict the category of the object in the image, and output the object category and confidence corresponding to the detection box.

[0109] The object detection model includes at least two object detection networks, and each object detection network can output its own detection results and prediction results respectively. The input data of the object detection network is an image, and the output data includes a detection box and an object category corresponding to the detection box. Furthermore, the output data can also include the confidence corresponding to the detection box. The output data of the object detection model includes the output data corresponding to each object detection network. It can be understood that the network parameters of each object detection network are different.

[0110] The initial object detection model refers to an object detection model to be trained. The initial object detection model can be an initialized object detection model or an object detection model obtained through pre-training.

[0111] The image is input into the initial object detection model, and the initial object detection model outputs the initial detection frame set corresponding to the image. The initial detection frame set includes the initial detection frames output by each object detection network in the initial object detection model and the object categories corresponding to the initial detection frames. Furthermore, the initial detection frame set may also include the confidences corresponding to the initial detection frames output by each object detection network in the initial object detection model. It can be understood that if the object detection network detects that there is no object in the image, a detection result representing that there is no object in the image can be output; if the object detection network detects that there is an object in the image, a detection frame representing the area where the object is located in the image can be output; if the object detection network detects that there are at least two objects in the image, a detection frame corresponding to each object can be output.

[0112] Specifically, the computer device inputs each first image in the first image set into the initial object detection model, and after the model processes the image data, the model outputs the initial detection frame set corresponding to each first image. In the model, each object detection network performs data processing respectively, and each object detection network outputs its own detection result respectively, and the detection results corresponding to each object detection network are integrated to obtain the initial detection frame set.

[0113] Step S206, for the same initial detection frame set, based on the detection frame similarity between the initial detection frames belonging to the same object category, the initial detection frame output by the current object detection network is matched with the initial detection frames output by each other object detection network to obtain the matching detection frame corresponding to the initial detection frame output by the current object detection network.

[0114] The detection frame similarity is used to indicate the similarity between different detection frames. The greater the detection frame similarity, the more similar the different detection frames are. The detection frame similarity between different detection frames is calculated based on the difference between the detection frame positions corresponding to different detection frames.

[0115] The current object detection network refers to the object detection network currently being processed, and can be any object detection network in the initial object detection model. For example, at least one object detection network is selected from each object detection network as the current object detection network; each object detection network is used as the current object detection network; and so on.

[0116] The matching detection frame corresponding to the initial detection frame output by the current object detection network refers to the initial detection frame that successfully matches the initial detection frame output by the current object detection network among the initial detection frames output by other object detection networks. The initial detection frame and the corresponding matching detection frame are output by different object detection networks, have the same object category, and correspond to the same image.

[0117] Specifically, after obtaining the initial detection frame set corresponding to the first image, the computer device matches the detection results output by different object detection networks to determine initial detection frames output by different object detection networks that match each other. The computer device determines the current object detection network from each object detection network, and for the initial detection frame set corresponding to an image, matches the initial detection frame output by the current object detection network with the initial detection frames output by each other object detection network to determine the matching detection frame corresponding to the initial detection frame output by the current object detection network. When matching, the initial detection frames output by different object detection networks are matched based on the detection frame similarity between the initial detection frames belonging to the same object category, that is, from the initial detection frames corresponding to the same object category as the initial detection frame output by the current object detection network and output by other object detection networks, based on the detection frame similarity between the initial detection frame output by the current object detection network and the initial detection frame output by the other object detection networks, an initial detection frame matching the initial detection frame output by the current object detection network is determined. For example, the initial detection frame output by other object detection networks with a detection frame similarity greater than a preset similarity is used as the matching detection frame corresponding to the initial detection frame output by the current object detection network; the initial detection frame corresponding to the maximum detection frame similarity is obtained from the initial detection frames output by other object detection networks with a detection frame similarity greater than a preset similarity, as the matching detection frame corresponding to the initial detection frame output by the current object detection network; and so on. If no initial detection frame matching the initial detection frame output by the current object detection network can be found from other object detection networks, then there is no corresponding matching detection frame for the initial detection frame output by the current object detection network. It can be understood that the initial detection frame sets corresponding to different images are matched separately.

[0118] For example, assume that the initial object detection model includes three object detection networks, namely object detection network 1, object detection network 2 and object detection network 3. Image A is input into the initial object detection model, and each object detection network outputs a detection result. Taking object detection network 1 as an example, an initial detection frame output by object detection network 1 is used as the target detection frame, and the matching detection frame corresponding to the target detection frame is determined from the initial detection frames output by object detection network 2 and object detection network 3. From each initial detection frame output by object detection network 2, the initial detection frame corresponding to the maximum value of the detection frame similarity between the target detection frame and the target detection frame is used as the matching detection frame, and the maximum value of the detection frame similarity is greater than the preset similarity. From each initial detection frame output by object detection network 3, the initial detection frame corresponding to the same object category as the target detection frame and the maximum value of the detection frame similarity between the target detection frame and the target detection frame is used as the matching detection frame, and the maximum value of the detection frame similarity is greater than the preset similarity. If there is no initial detection frame that meets the conditions in each initial detection frame output by object detection network 2 and object detection network 3, then there is no corresponding matching detection frame for the target detection frame. Another initial detection frame output by the object detection network 1 is obtained as the target detection frame, and a matching detection frame corresponding to the target detection frame is determined, and so on, until all initial detection frames output by the object detection network 1 are matched.

[0119] Step S208, fusing the initial detection frame and the corresponding matching detection frames to obtain a virtual detection frame, and obtaining a set of virtual detection frames corresponding to the first image based on the virtual detection frames corresponding to the same first image.

[0120] The virtual detection frame refers to a detection frame that is not manually annotated and is used as a pseudo label. The virtual detection frame set includes each virtual detection frame corresponding to the same image.

[0121] Specifically, the computer device fuses the initial detection frame and its corresponding matching detection frames to obtain a virtual detection frame. For example, the mean of the detection frame coordinates of the initial detection frame and its corresponding matching detection frames is calculated as the detection frame coordinates of the virtual detection frame; the initial detection frame and its corresponding matching detection frame coordinates are weighted summed to obtain the detection frame coordinates of the virtual detection frame. The greater the confidence corresponding to the detection frame, the greater the weight corresponding to the detection frame, and the sum of the weights corresponding to each detection frame is 1; and so on. The initial detection frame and its corresponding matching detection frame are both output by the model for data processing of the same image. Therefore, the virtual detection frame generated based on the initial detection frame and the corresponding matching detection frame belongs to the same image as the initial detection frame and the matching detection frame. If there are multiple initial detection frames corresponding to an image with matching detection frames, multiple virtual detection frames can be obtained. Finally, the computer device combines the virtual detection frames corresponding to the same first image to obtain a set of virtual detection frames corresponding to the first image.

[0122] Furthermore, the virtual detection frame may also have a corresponding object category and confidence. The object category corresponding to the virtual detection frame obtained by fusing the initial detection frame and the corresponding matching detection frame is the object category corresponding to the initial detection frame. The confidence corresponding to the virtual detection frame obtained by fusing the initial detection frame and the corresponding matching detection frame is the fusion value of the confidences corresponding to the initial detection frame and the matching detection frame, for example, the mean of each confidence is used as the fusion value; the median value of each confidence is used as the fusion value; and so on.

[0123] Step S210: Based on the first image set and the corresponding virtual detection frame sets, the initial object detection model is trained to obtain a target object detection model.

[0124] Among them, the target object detection model refers to the trained object detection model.

[0125] Specifically, after obtaining the virtual detection frame set, the computer device performs model training on the initial object detection model based on the first image set and the corresponding virtual detection frame sets to obtain the target object detection model. During model training, the virtual detection frame set is used as a training label, and the first image is used as input data of the model. Based on the first image set and the corresponding virtual detection frame sets, the initial object detection model is supervised trained. After the training is completed, the target object detection model is obtained.

[0126] In the above-mentioned object detection model training method, an unlabeled image is input into the model for object detection, and a virtual detection frame is intelligently added to the unlabeled image based on the initial detection frame set obtained by the detection. The virtual detection frame is a pseudo-label, and the unlabeled image and the pseudo-label are used as the training data of the model, so that there is no need for manual labeling, and images with labeling information can be obtained to train the model, which effectively reduces the image labeling cost. Furthermore, when generating a virtual detection frame, the initial detection frames output by different object detection networks in the model for the same image are matched. The matching initial detection frames belong to the same object category and are relatively similar. The matching detection frames corresponding to the same initial detection frame are fused to obtain a virtual detection frame with higher accuracy, which helps to improve the quality of model training.

[0127] In one embodiment, the object detection model training method further includes:

[0128] A second image set is obtained; the second image set is a labeled image set; each second image in the second image set is input into a candidate object detection model to obtain a set of predicted detection boxes corresponding to each second image; the set of predicted detection boxes includes a subset of predicted detection boxes output by each object detection network in the candidate object detection model, and the predicted detection box subset includes a predicted detection box output by the same object detection network, an object category and a confidence corresponding to the predicted detection box; based on the difference between the labeled detection box set corresponding to the same second image and the corresponding predicted detection box subsets, the sub-loss corresponding to the second image on each object detection network is obtained; the sub-loss includes a detection box loss, a category loss and a confidence loss; based on the sub-loss corresponding to each second image on the same object detection network, the network loss corresponding to each object detection network is obtained; based on the network loss, the network parameters of the corresponding object detection network in the candidate object detection model are adjusted until the first convergence condition is met to obtain an initial object detection model.

[0129] Among them, the annotated image set includes multiple annotated images, and annotated images refer to images in which objects in the images have been annotated, that is, annotated images refer to images in which the area where the objects in the images are located and the categories to which they belong are known. There is a corresponding annotated detection frame set for the annotated images, and the annotated detection frame set includes the annotated detection frames corresponding to the objects in the images and the object categories corresponding to the annotated detection frames. The annotated detection frames refer to the detection frames annotated in advance, and the annotated detection frame set is used as a training label. For the annotated images obtained through manual annotation, the corresponding annotated detection frames are accurate detection frames that can correctly locate the area where the objects in the images are located, and the object categories corresponding to the annotated detection frames are the correct categories corresponding to the objects in the images.

[0130] The candidate object detection model refers to the object detection model to be trained, and the candidate object detection model can be an initialized object detection model. The candidate object detection model is pre-trained to obtain the initial object detection model. The image is input into the candidate object detection model, and the candidate object detection model outputs a set of predicted detection frames corresponding to the image. The set of predicted detection frames includes the predicted detection frames output by each object detection network in the candidate object detection model, the object category and confidence corresponding to the predicted detection frame, which is equivalent to the predicted detection frame set including the predicted detection frame subsets output by each object detection network in the candidate object detection model, and the predicted detection frame subsets include the predicted detection frames output by the same object detection network, the object category and confidence corresponding to the predicted detection frame.

[0131] The candidate object detection model includes at least two object detection networks. When the candidate object detection model is trained, each object detection network calculates its own network loss and adjusts its own network parameters based on its own network loss. The image is input into the candidate object detection model, and the candidate object detection model outputs the detection results corresponding to each object detection network. For any object detection network, based on the set of annotated detection frames corresponding to the image and the detection results corresponding to the object detection network, the sub-loss corresponding to the image on the object detection network is calculated. The annotated detection frame set includes an annotated detection frame, an object category and a confidence corresponding to the annotated detection frame, and the predicted detection frame subset includes a predicted detection frame, an object category and a confidence corresponding to the predicted detection frame. When calculating the sub-loss, the detection frame loss is calculated based on the difference between the annotated detection frame and the predicted detection frame, the category loss is calculated based on the difference between the object category corresponding to the annotated detection frame and the object category corresponding to the predicted detection frame, and the confidence loss is calculated based on the difference between the confidence corresponding to the annotated detection frame and the confidence corresponding to the predicted detection frame. It can be understood that the confidence corresponding to the annotated detection frame can be pre-set as needed. For example, if the confidence is expressed as a probability between 0 and 1, the confidence corresponding to the annotated detection frame can be set to the maximum probability of 1, indicating that the annotated detection frame is the most correct and reliable detection frame. When calculating the sub-loss, the detection frame loss, category loss and confidence loss can be obtained based on the differences in various data between the predicted detection frame and the annotated detection frame with the greatest detection frame similarity to the predicted detection frame; the detection frame loss, category loss and confidence loss can also be obtained based on the differences in various data between the predicted detection frame and the annotated detection frame with the greatest detection frame similarity to the predicted detection frame greater than a certain threshold. In one embodiment, if the maximum detection frame similarity is less than a certain threshold, the sub-loss corresponding to the predicted detection frame can be not calculated, or it can be determined as a preset fixed value, which is used to characterize that the accuracy of the predicted detection frame is low. The second image set includes multiple second images, and the network loss corresponding to the object detection network is obtained by combining the sub-losses corresponding to each second image on the same object detection network.

[0132] The first convergence condition may be at least one of the following: the number of model iterations is greater than a first preset number of iterations, each network loss is less than its corresponding first preset loss, etc.

[0133] Specifically, the initial object detection model may be obtained through pre-training. The computer device obtains a second image set, and pre-trains the candidate object detection model based on the second image set to obtain the initial object detection model. The second image set is a labeled image set, for example, the second image set may be an image set obtained through manual labeling.

[0134] During pre-training, the computer device inputs each second image in the second image set into the candidate object detection model. After data processing by the model, the model outputs a set of predicted detection frames corresponding to each second image. The set of predicted detection frames includes a subset of predicted detection frames output by each object detection network in the candidate object detection model. Based on the difference between the set of labeled detection frames corresponding to the same second image and the corresponding subset of predicted detection frames, the computer device obtains the sub-losses corresponding to the second image on each object detection network. The sub-losses specifically include detection frame loss, category loss, and confidence loss. Based on the sub-losses corresponding to each second image on the same object detection network, the computer device obtains the network loss corresponding to the object detection network. For example, the sum of the same seed loss corresponding to each second image on the same object detection network is calculated to obtain the total detection frame loss, the total category loss, and the total confidence loss. The sum of the total detection frame loss, the total category loss, and the total confidence loss is calculated to obtain the network loss. The total detection frame loss, the total category loss, and the total confidence loss are weighted and summed to obtain the network loss. The weight can be set as needed. And so on. Each object detection network performs loss calculations to obtain the network loss corresponding to each object detection network. Finally, the network parameters of the corresponding object detection network in the candidate object detection model are adjusted based on the network loss until the first convergence condition is met to obtain the initial object detection model.

[0135] In one embodiment, in the pre-training stage, the network parameters are updated based on the stochastic gradient descent algorithm (SGD). In each model iteration process, the network loss is calculated and the network loss is fed back to the object detection network to update the network parameters through the stochastic gradient descent algorithm.

[0136] In the above embodiment, the initial object detection model is obtained by pre-training based on the labeled image set, and has a certain accuracy, so the pseudo-labels generated based on the initial object detection model also have a certain accuracy. Further training of the initial object detection model based on the images with pseudo-labels can further improve the accuracy of the initial object detection model. During pre-training, each object detection network in the model calculates its own network loss, and adjusts its own network parameters based on its own network loss to avoid mutual influence between different object detection networks.

[0137] In one embodiment, step S206 includes:

[0138] For the current initial detection frame set, a first detection frame is determined from the initial detection frame output by the current object detection network, and a second detection frame is determined from the initial detection frame output by other object detection networks; the object categories corresponding to the first detection frame and the second detection frame are the same; based on the detection frame overlap between the first detection frame and the second detection frame, the detection frame similarity between the first detection frame and the second detection frame is obtained; the second detection frame corresponding to the detection frame similarity greater than the preset similarity is obtained as a candidate detection frame, and the candidate detection frame corresponding to the maximum detection frame similarity is obtained as the matching detection frame corresponding to the first detection frame.

[0139] The current initial detection frame set refers to the initial detection frame set currently being processed, and may be any initial detection frame set among the initial detection frame sets corresponding to each first image.

[0140] The detection frame overlap is used to indicate the degree of overlap between different detection frames. The intersection-over-union ratio between different detection frames is calculated as the detection frame overlap. Specifically, the area union and area intersection between different detection frames are calculated. Based on the ratio of the area union and area intersection, the detection frame overlap between different detection frames is obtained.

[0141] The preset similarity refers to a preset similarity threshold, which can be set according to actual needs.

[0142] Specifically, in the initial detection frame sets corresponding to the first images, any initial detection frame set is obtained as the current initial detection frame set, and the detection frame matching process is explained by taking the current initial detection frame set as an example. For the current initial detection frame set, an initial detection frame is selected from the initial detection frames output by the current object detection network as the first detection frame, and the initial detection frames output by other object detection networks and belonging to the same object category as the first detection frame are respectively used as second detection frames, and the detection frame overlap between the first detection frame and each second detection frame is calculated, and the detection frame overlap is used as the detection frame similarity. From each second detection frame, a second detection frame whose detection frame similarity is greater than a preset similarity is obtained as a candidate detection frame, and a matching detection frame corresponding to the first detection frame is further determined from each candidate detection frame, and the candidate detection frame corresponding to the maximum detection frame similarity is obtained as the matching detection frame corresponding to the first detection frame.

[0143] It can be understood that, for the current initial detection frame set, each initial detection frame output by the current object detection network is used as a first detection frame, and the matching detection frame corresponding to each first detection frame is determined respectively.

[0144] In the above embodiment, the detection frame overlap is used as the detection frame similarity, and the initial detection frame whose detection frame similarity with the first detection frame is greater than the preset similarity and is most similar to the first detection frame is used as the matching detection frame corresponding to the first detection frame, which can ensure that the first detection frame and the corresponding matching detection frame not only have sufficient similarity, but are also the most similar. The first detection frame and the corresponding matching detection frame are output by different object detection networks and correspond to the same object category, which is similar to the first detection frame and the corresponding matching detection frame being obtained by different models for the same object detection on the same image. Generating a virtual detection frame based on the first detection frame and the corresponding matching detection frame is equivalent to integrating the prediction results of different models, which can effectively ensure the accuracy of the virtual detection frame.

[0145] In one embodiment, an unlabeled image is input into a pre-trained model for object detection, and pseudo-label information (i.e., pseudo-label) is added to the unlabeled image according to the detection result. The object detection model includes three object detection networks as an example to illustrate the pseudo-label generation process. The three object detection networks can also be called three prediction heads or detection heads, which are represented by header0, header1, and header2 respectively.

[0146] First, extract video frames from unlabeled videos as unlabeled images. For example, in order to reduce data processing pressure, extract one frame every 2 seconds as an unlabeled image. Then, input the unlabeled image into the initial object detection model obtained through pre-training for inference, and obtain the detection results of three prediction heads. Each detection result contains the coordinates of the detection box (x1, y1, x2, y2) and the object category corresponding to the detection box. Next, initialize the pseudo-label information list Pdata to be empty. For any detection box output by one of the three prediction heads for an image, search for a matching detection box from the detection results output by other detection heads, record the coordinates of the found detection box in Pdata, and finally generate a pseudo detection box based on the detection box and the Pdata corresponding to the detection box, and use the pseudo detection box as a pseudo label.

[0147] Taking header0 as an example, for a detection box i output by header0, in the detection box output by header1, query the detection box whose IOU (Intersection over Union) is greater than 0.5, retain the detection box of the same category as i, and record it as j. If multiple detection boxes meet the requirements, take the detection box with the largest IOU with i. For a detection box i output by header0, in the detection box output by header2, query the detection box whose IOU is greater than 0.5, retain the detection box of the same category as i, and record it as k. If multiple detection results meet the requirements, take the detection box with the largest IOU with i. When neither j nor k exists, that is, no detection box similar to i is found in header1 or header2, i is discarded. When any one of j and k exists or both exist, the mean of the coordinates of the three detection boxes of i, j, and k is calculated as the coordinates of the pseudo detection box, and a pseudo label with a strong confidence is obtained and saved in Pdata. Each detection box output by header0 is matched separately to generate a pseudo label. Similarly, for a detection box i output by header1, find a detection box similar to i in header0 and header2 to generate a pseudo label. For a detection box i output by header2, find a detection box similar to i in header0 and header1 to generate a pseudo label.

[0148] In one embodiment, Figure 3 As shown, step S210 includes:

[0149] Step S302, based on the difference between the initial detection frame subset and the corresponding virtual detection frame set output by the same target object detection network for the same first image, calculate the target loss corresponding to the target object detection network, and obtain the target losses corresponding to each target object detection network.

[0150] The target object detection network is at least one object detection network selected from each object detection network. For example, the object detection model includes four object detection networks, namely object detection network 1, object detection network 2, object detection network 3, and object detection network 4. Object detection network 1, object detection network 2, and object detection network 3 can be used as target object detection networks.

[0151] The first image is input into an initial object detection model, and the initial object detection model outputs an initial detection frame set, which includes an initial detection frame subset output by each object detection network in the initial object detection model, and the initial detection frame subset includes an initial detection frame output by the same object detection network and an object category corresponding to the initial detection frame. The initial detection frame subset may also include a confidence level corresponding to the initial detection frame.

[0152] Specifically, when training the initial object detection model, different object detection networks in the model can use different network parameter adjustment methods. For the target object detection network in the model, adjust its own network parameters based on its own network loss. For other object detection networks in the model except the target object detection network, update their own network parameters based on the update results of the target object detection network. Pseudo-labels may have noise, and accordingly, the network parameter adjustment method of the target object detection network may have noise. The network parameter adjustment method of other object detection networks helps to smooth the noise of the target object detection network.

[0153] Step S304: Based on the target loss corresponding to the target object detection network, the initial network parameters corresponding to the target object detection network in the initial object detection model are adjusted to obtain updated network parameters corresponding to the target object detection network.

[0154] Specifically, after obtaining the initial detection frame set corresponding to the first image, the computer device calculates the target loss corresponding to a single target object detection network based on the difference between the initial detection frame subset output by the same target object detection network and the corresponding virtual detection frame set for the same first image, and finally obtains the target loss corresponding to each target object detection network. Then, the computer device back-propagates the target loss corresponding to the target object detection network, adjusts the initial network parameters corresponding to the target object detection network, and uses the adjusted initial network parameters as the updated network parameters corresponding to the target object detection network, and finally obtains the updated network parameters corresponding to each target object detection network.

[0155] In one embodiment, the target loss may be calculated by referring to the network loss calculation method in the above-mentioned embodiment.

[0156] Step S306, based on the updated network parameters corresponding to each target object detection network, the initial network parameters of other object detection networks except the target object detection network in the initial object detection model are adjusted to obtain updated network parameters corresponding to other object detection networks.

[0157] Specifically, for other object detection networks except the target object detection network in the initial object detection model, the computer device adjusts the initial network parameters of other object detection networks based on the updated network parameters corresponding to each target object detection network, and uses the adjusted initial network parameters as the updated network parameters corresponding to the other object detection networks.

[0158] Step S308, obtaining a target object detection model based on the updated network parameters corresponding to each object detection network.

[0159] Specifically, after the network parameters of each object detection network in the initial object detection model are adjusted, the computer device obtains the target object detection model based on the updated network parameters corresponding to each object detection network. For example, each object detection network with updated network parameters is used to form the target object detection model.

[0160] It can be understood that if the model training adopts an iterative training method, an intermediate object detection model is obtained based on the updated network parameters corresponding to each object detection network, and the intermediate object detection model is used as the initial object detection model. The step of obtaining the first image set is returned to execute until the convergence condition is met to obtain the target object detection model.

[0161] In the above embodiment, for the target object detection network in the model, its own network parameters are adjusted based on its own network loss, and for other object detection networks in the model except the target object detection network, their own network parameters are updated based on the update results of the target object detection network. This training method can enable other object detection networks to smooth out noise glitches in the target object detection network and improve training quality.

[0162] In one embodiment, step S302 includes:

[0163] For the same initial detection frame set, based on the detection frame similarity between the initial detection frames belonging to the same object category, the initial detection frame output by the current target object detection network is matched with the initial detection frames output by each other target object detection network to obtain the matching result corresponding to the initial detection frame output by the current target object detection network; for the initial detection frame subset output by the current target object detection network for the current first image, based on the difference between the initial detection frame in the initial detection frame subset and the corresponding supervised detection frame, the initial sub-loss corresponding to each initial detection frame in the initial detection frame subset is obtained; when the matching result is a successful match, the supervised detection frame corresponding to the initial detection frame is a matched frame. The matched detection frame is matched successfully. When the matching result is a matching failure, the supervised detection frame corresponding to the initial detection frame is the corresponding virtual detection frame in the virtual detection frame set; the loss weight corresponding to the initial detection frame is determined based on the matching result corresponding to the initial detection frame; the loss weight corresponding to the successful match is greater than the loss weight corresponding to the failed match; based on the loss weight corresponding to the initial detection frame, the initial sub-losses corresponding to each initial detection frame in the initial detection frame subset are fused to obtain the target sub-loss corresponding to the current first image on the current target object detection network; based on the target sub-losses corresponding to each first image on the current target object detection network, the target loss corresponding to the current target object detection network is obtained.

[0164] The current target object detection network refers to the target object detection network currently being processed, and may be any one of the target object detection networks. The current first image refers to the first image currently being processed, and may be any one of the first images.

[0165] Specifically, when calculating the target loss, the initial detection frames output by different target object detection networks are first matched. Referring to the detection frame matching method in the aforementioned embodiment, the initial detection frame output by the current target object detection network is matched with the initial detection frames output by each other target object detection network, and the matching result corresponding to the initial detection frame output by the current target object detection network is obtained. For the same initial detection frame set, the computer device matches the initial detection frame output by the current target object detection network with the initial detection frames output by each other target object detection network based on the detection frame similarity between the initial detection frames belonging to the same object category, and obtains the matching result corresponding to the initial detection frame output by the current target object detection network. The matching result includes a successful match and a failed match. A successful match indicates that the initial detection frame finds the corresponding matching detection frame, and a failed match indicates that the initial detection frame does not find the corresponding matching detection frame. It can be understood that the preset similarity used for detection frame matching in different links can be adaptively set according to the actual needs of each link. For example, the preset similarity used for detection frame matching in the process of determining the virtual detection frame can be different from the preset similarity used for detection frame matching in the process of calculating the target loss.

[0166] Next, the sub-loss corresponding to the initial detection frame is calculated according to the matching result. For the initial detection frame subset output by the current target object detection network of the current first image, the initial sub-loss corresponding to the initial detection frame is calculated based on the difference between the initial detection frame in the initial detection frame subset and the corresponding supervisory detection frame. The sub-loss corresponding to each initial detection frame is calculated respectively, and the initial sub-loss corresponding to each initial detection frame in the initial detection frame subset is obtained. The supervisory detection frame is used as the supervisory information in the training process. The goal of model training is to make the initial detection frame and the corresponding supervisory detection frame closer and closer and more similar. When the matching result of the detection frame matching is a successful match, the supervisory detection frame corresponding to the initial detection frame is a successfully matched matching detection frame. Based on the difference between the initial detection frame and the corresponding matching detection frame, the initial sub-loss corresponding to the initial detection frame is calculated. When the matching result of the detection frame matching is a failed match, the supervisory detection frame corresponding to the initial detection frame is the corresponding virtual detection frame in the virtual detection frame set. Based on the difference between the initial detection frame and the corresponding virtual detection frame, the initial sub-loss corresponding to the initial detection frame is calculated. The virtual detection frame corresponding to the initial detection frame may be a virtual detection frame with the greatest detection frame similarity to the initial detection frame, or may be a virtual detection frame with a detection frame similarity to the initial detection frame greater than a certain threshold and with the greatest detection frame similarity. It is understood that if no corresponding virtual detection frame is found, the sub-loss corresponding to the initial detection frame may not be calculated, or may be determined as a preset fixed value.

[0167] In addition, the loss weight corresponding to the initial detection frame is determined based on the matching result corresponding to the initial detection frame. The detection frame that failed to match may be noise, but the noise data is helpful to the generalization performance of the model and cannot be completely discarded. Therefore, the loss weight corresponding to the successful match is greater than the loss weight corresponding to the failed match. For the detection frame that failed to match, a lower weight is used when calculating the total loss to avoid excessive influence on the model parameter update.

[0168] Then, based on the loss weight corresponding to the initial detection frame, the initial sub-losses corresponding to each initial detection frame in the initial detection frame subset are weighted fused, and the weighted average is calculated to obtain the target sub-loss corresponding to the current first image on the current target object detection network. The target sub-losses corresponding to each first image on the current target object detection network are calculated in the same way. Finally, based on the target sub-losses corresponding to each first image on the current target object detection network, the target loss corresponding to the current target object detection network is obtained, for example, the average of the target sub-losses is calculated as the target loss.

[0169] In the above embodiment, when calculating the target loss, different matching results use different supervisory detection frames to improve the accuracy of the target loss. When the matching result is a successful match, the supervisory detection frame corresponding to the initial detection frame is a successfully matched matching detection frame, and the initial detection frame and the matching detection frame are output by different object detection frame networks, so that different object detection networks can learn from each other. When the matching result is a failed match, the supervisory detection frame corresponding to the initial detection frame is a corresponding virtual detection frame in the virtual detection frame set, and the virtual detection frame is a training label, so that the object detection network outputs a detection frame that is similar or consistent with the training label. Furthermore, different matching results use different loss weights. Even if the initial detection frame fails to match, it is used to calculate the loss. However, a lower loss weight is used when calculating the loss, which can improve the generalization performance of the model while avoiding excessive impact on the parameters of the model.

[0170] In one embodiment, step S306 includes:

[0171] The network weights corresponding to each object detection network are obtained; the network weight corresponding to the target object detection network is less than the network weights corresponding to other object detection networks; based on the network weights corresponding to each object detection network, the updated network parameters corresponding to each target object detection network and the initial network parameters corresponding to other object detection networks are fused to obtain the updated network parameters corresponding to other object detection networks.

[0172] Specifically, the computer device obtains the network weights corresponding to each object detection network, and based on the network weights corresponding to each object detection network, performs weighted summation of the updated network parameters corresponding to each target object detection network and the initial network parameters corresponding to other object detection networks to obtain updated network parameters corresponding to other object detection networks. The network weight corresponding to the target object detection network is smaller than the network weights corresponding to other object detection networks, so as to avoid the updated network parameters of the target object detection network from excessively affecting the initial network parameters of other object detection networks, so that the network parameters of other object detection networks are steadily adjusted on the basis of the initial network parameters.

[0173] In one embodiment, Figure 4 As shown, step S210 includes:

[0174] Step S402: Based on the first image set and the corresponding virtual detection frame sets, adjust the model parameters of the initial object detection model to obtain an intermediate object detection model.

[0175] Step S404: Use the intermediate object detection model as the initial object detection model, and return to the step of acquiring the first image set until the second convergence condition is met to obtain the target object detection model.

[0176] Among them, model parameter adjustment refers to calculating the model loss, and back-propagating the model loss to adjust the model parameters of the model. For example, based on the difference between the virtual detection frame sets corresponding to the same first image and the corresponding initial detection frame subsets, the network losses corresponding to each object detection network are calculated, and the network losses are back-propagated to adjust the network parameters of the object detection network. For another example, referring to the aforementioned embodiment, the target losses corresponding to each target object detection network are calculated, and the target losses corresponding to the target object detection network are back-propagated to adjust the network parameters corresponding to the target object detection network, and based on the adjusted network parameters corresponding to each target object detection network, the network parameters of other object detection networks are adjusted.

[0177] The second convergence condition may be at least one of the number of model iterations greater than a preset number of iterations, the model loss less than a preset loss, the model accuracy greater than a preset accuracy, or the labeling cost greater than a preset cost. Execution. The model accuracy is obtained by performing an accuracy test on the intermediate object detection model based on the test set, and is used to characterize the prediction accuracy of the intermediate object detection model obtained through training. If the model accuracy corresponding to the intermediate object detection model obtained through a certain training is greater than the preset accuracy, the intermediate object detection model is used as the target object detection model. The labeling cost refers to the labeling cost spent on the labeled image set.

[0178] Specifically, the model training adopts an iterative training method. In the first model iteration, the first image set and the initial object detection model are obtained, the first image in the first image set is input into the initial object detection model, the virtual detection frame set corresponding to the first image is determined based on the output data of the initial object detection model, and the model parameters of the initial object detection model are adjusted based on the first image set and the corresponding virtual detection frame sets to obtain an intermediate object detection model. In the second model iteration, the intermediate object detection model obtained in the previous model iteration is used as the initial object detection model, the first image set is re-acquired, and the first image in the new first image set is input into the new initial object detection model to determine the virtual detection frame set corresponding to the new first image. Based on the new first image set and the corresponding virtual detection frame sets, the model parameters of the new initial object detection model are adjusted to obtain a new intermediate object detection model. In the third model iteration, the intermediate object detection model obtained in the previous model iteration is used as the initial object detection model, and the data processing process of a single model iteration is repeated to obtain a new intermediate object detection model. And so on, until the second convergence condition is met, the target object detection model is obtained. For example, the second convergence condition is that the number of model iterations is greater than the preset number of iterations. If the preset number of iterations is 200, the intermediate object detection model obtained by the 201st model iteration is used as the target object detection model. It can be understood that the first image sets acquired at different times can be completely different image sets or image sets with repeated images.

[0179] In the above embodiment, the model training adopts an iterative training method, which can improve the quality of model training and obtain a target object detection model with higher accuracy.

[0180] In one embodiment, step S402 includes:

[0181] A third image set is obtained; the third image set is a labeled image set; based on the first image set and the corresponding virtual detection frame sets, and the third image set, the model parameters of the initial object detection model are adjusted to obtain an intermediate object detection model.

[0182] The third image set is a labeled image set, and the third image set may specifically be an image set obtained through manual labeling.

[0183] Specifically, in order to further improve the training quality of the model, the model training is performed by combining the labeled image set and the unlabeled image set. The computer device obtains a third image set, which is a labeled image set, and adjusts the model parameters of the initial object detection model based on the first image set and the corresponding virtual detection frame sets, and the third image set to obtain an intermediate object detection model. When adjusting the model parameters, the images in the first image set and the third image set are used as input data of the model, the virtual detection frame corresponding to the first image is used as a training label, and the labeled detection frame corresponding to the third image is used as a training label.

[0184] In one embodiment, the images in the first image set and the third image set are input into the initial object detection model to obtain an initial detection frame set. Based on the difference between the initial detection frame subset and the corresponding reference detection frame set output by the same target object detection network for the same image, the target loss corresponding to the target object detection network is calculated to obtain the target loss corresponding to each target object detection network. If the image is the first image, the reference detection frame set is a virtual detection frame set; if the image is the third image, the reference detection frame set is a labeled detection frame set. Based on the target loss corresponding to the target object detection network, the initial network parameters corresponding to the target object detection network in the initial object detection model are adjusted to obtain the updated network parameters corresponding to the target object detection network. Based on the updated network parameters corresponding to each target object detection network, the initial network parameters of other object detection networks other than the target object detection network in the initial object detection model are adjusted to obtain the updated network parameters corresponding to the other object detection networks. Based on the updated network parameters corresponding to each object detection network, an intermediate object detection model is obtained.

[0185] In the above embodiment, the initial object detection model is trained based on pseudo-annotated samples and manually annotated samples, which can further improve the training quality of the model and obtain a target object detection model with higher accuracy through training.

[0186] In one embodiment, an object detection model including three object detection networks is used as an example to illustrate the process of model training based on pseudo-labeled samples and labeled samples. The three object detection networks are represented by header0, header1 and header2 respectively.

[0187] Input all labeled samples into the model for training. Each training of all data completes one epoch. Due to limited training resources, it is usually difficult to train all samples at once. It is necessary to train the model by batches of bs training samples and update the model based on a batch of data. Among them, the manually labeled samples are recorded as A and the pseudo-labeled samples are recorded as B. Take bs / 2 A samples and bs / 2 B samples for training in each batch.

[0188] Input a batch of bs samples into the model to get the prediction results of header1 and 2. For the same sample, for a certain prediction box of header1, find a prediction box with an IOU greater than 0.3 and the same predicted category from all prediction boxes of header2. If there are multiple matching prediction boxes, keep all matching prediction boxes. For the same sample, for a certain prediction box of header2, find a prediction box with an IOU greater than 0.3 and the same predicted category from all prediction boxes of header1. If there are multiple matching prediction boxes, keep all matching prediction boxes. Calculate the total loss of header1 and 2. For the successfully matched detection box, calculate the loss according to the normal loss calculation method. For the successfully matched detection box, multiply the normal loss calculation result by a weight of 0.1. Each detection box needs to calculate the average of three losses (i.e., detection box loss, category loss, and confidence loss). The total loss = (L all matching boxes + 0.1*L all unmatched) / total number of boxes.

[0189] For header1 and 2, the SGD method is used to update the model parameters of header1 and 2 based on their respective total losses. For header0, the momentum update method is used to update header0, for example, w0 = 0.8w0 + 0.1w1 + 0.1w2, that is, the weights of header1 and 2 are updated each time as w1 and w2, and the weights of the two are updated to the final w0 each time to avoid the influence of single learning noise on w0. At the same time, the momentum update method can also make the prediction of header0 more consistent in multiple iterations before and after.

[0190] In one embodiment, acquiring a third image set includes:

[0191] A fourth image set is obtained; the fourth image set is an unlabeled image set; each fourth image in the fourth image set is input into an initial object detection model to obtain an initial detection frame set corresponding to each fourth image; the initial detection frame set corresponding to the fourth image includes an initial detection frame subset output by each object detection network in the initial object detection model, and the initial detection frame subset includes an initial detection frame output by the same object detection network and a confidence corresponding to the initial detection frame; information entropy statistics are performed on the confidence corresponding to each initial detection frame in the same initial detection frame subset to obtain a predicted entropy corresponding to each initial detection frame subset, and each predicted entropy corresponding to the same fourth image is fused to obtain a fused entropy corresponding to each fourth image; information entropy division is performed based on the fused entropy corresponding to each fourth image to obtain a plurality of information entropy intervals, and a target interval is determined from the plurality of information entropy intervals; a target image to be labeled is determined from the fourth image whose fused entropy is located in the target interval, and annotates the target image to obtain a labeled detection frame set corresponding to the target image; a third image set is obtained based on each target image and the corresponding labeled detection frame set.

[0192] Among them, information entropy statistics refers to the accumulation of information entropies corresponding to each initial detection frame output by the same object detection network for the same image. The image is input into the initial object detection model, and the model outputs the initial detection frame set corresponding to the image. The information entropy corresponding to the initial detection frame in the initial detection frame set is obtained by substituting the confidence corresponding to the initial detection frame into the information entropy calculation formula. Information entropy calculation formula: H i =-p i *log2p i .H i represents the information entropy corresponding to the i-th initial detection box, p i It represents the confidence level corresponding to the i-th initial detection box. Information entropy can be referred to as entropy.

[0193] Prediction entropy refers to the statistical result of information entropy corresponding to a single image on a single object detection network. The confidence corresponding to each initial detection box in the initial detection box subset corresponding to an image is accumulated to obtain a prediction entropy corresponding to the image. The initial detection box set corresponding to the image includes the initial detection box subsets output by each object detection network in the initial object detection model. Therefore, the prediction entropy corresponding to a single image on each object detection network can be calculated.

[0194] The fusion entropy is obtained by fusing the predicted entropies corresponding to the same image. For example, the average value of the predicted entropies can be calculated as the fusion entropy. Each image has its own corresponding fusion entropy.

[0195] Information entropy division refers to dividing the value range of fusion entropy into multiple information entropy intervals. For example, from the fusion entropies corresponding to each fourth image, the minimum value and the maximum value are obtained, and the information entropy range determined by the minimum value and the maximum value is evenly divided into multiple information entropy intervals. The target interval is at least one information entropy interval selected from each information entropy interval, for example, an information entropy interval whose information entropy is greater than a preset entropy is selected as the target interval; each information entropy interval is sorted from small to large according to the information entropy, and at least one information entropy interval with a lower sorting is obtained as the target interval; and so on.

[0196] Object annotation refers to manually annotating objects in an image to obtain the annotation detection box corresponding to the object in the image and the object category corresponding to the annotation detection box. Each image has its own corresponding set of landmark detection boxes.

[0197] Specifically, the third image set includes at least one of labeled images obtained by directly acquiring unlabeled images to perform object labeling, and labeled images obtained by performing object labeling on difficult samples screened out from unlabeled images.

[0198] The computer device can filter out difficult samples from unlabeled images based on information entropy. First, obtain a fourth image set, which is an unlabeled image set, input each fourth image in the fourth image set into the initial object detection model, and obtain the initial detection frame set corresponding to each fourth image. Then, calculate the fusion entropy corresponding to the image based on the initial detection frame set corresponding to the image, and calculate the fusion entropy corresponding to each fourth image respectively. For a single fourth image, calculate the entropy corresponding to the confidence of each initial detection frame corresponding to the image, and accumulate the entropies corresponding to each initial detection frame in the same initial detection frame subset to obtain the predicted entropy corresponding to each initial detection frame subset. For example, if an image predicts two detection frames through a certain detection frame head, the confidence of detection frame 1 is 0.6, and the confidence of detection frame 2 is 0.7, then the predicted entropy corresponding to the image is -0.6log2(0.6)-0.7log2(0.7). The larger the value of the predicted entropy, the more uncertain the prediction frame corresponding to the image. Fusion the predicted entropies corresponding to the single fourth image to obtain the fusion entropy corresponding to the fourth image. For example, the predicted entropies corresponding to the image are averaged to obtain the fused entropy. Further, information entropy is divided based on the fused entropies corresponding to each fourth image to obtain multiple information entropy intervals, a target interval is determined from the multiple information entropy intervals, and a fourth image whose fused entropy is in the target interval is randomly sampled to obtain a target image to be labeled. The target image is labeled with an object to obtain a set of labeled detection frames corresponding to the target image, so that the target image becomes a labeled image. Finally, a third image set is obtained based on each target image and the corresponding set of labeled detection frames.

[0199] In one embodiment, for the detection box output by each prediction head of each image, the entropy of the detection box is counted respectively to obtain the prediction entropy corresponding to each prediction head of each image. The average value of the prediction entropy of the three prediction heads of an image is calculated to obtain the fusion entropy representing the overall prediction entropy of the image. For all images, they are sorted from small to large according to the overall prediction entropy of the image, and are evenly divided into 5 sections according to the minimum and maximum entropy ranges. From the 2nd to 5th sections, K images are randomly selected from each section as stratified difficult samples and sent for manual labeling. Among them, the first section is the section with the smallest entropy, indicating that the model has a greater grasp of the samples in this section, so no additional manual labeling is required, and from the 2nd to the 5th sections, it is increasingly difficult to sample and label them. In order to avoid the model falling into a bad parameter space due to overfitting of difficult samples, difficult samples cannot be sampled only for the 5th section (the most difficult samples). Through stratified sampling, the difficulty of the samples is balanced, which is conducive to improving the quality of model training.

[0200] In one embodiment, a first image set is obtained, a virtual detection frame set corresponding to the first image in the first image set is generated based on an initial object detection model, a common annotated image set is obtained, model parameters of the initial object detection model are adjusted based on the first image set and the corresponding virtual detection frame sets, and the common annotated image set, to obtain an updated object detection model, a difficult sample annotated image set is obtained, model parameters of the updated object detection model are adjusted based on the difficult sample annotated image set, an intermediate object detection model is obtained, the intermediate object detection model is used as the initial object detection model, and the step of obtaining the first image set is returned to execute until the second convergence condition is met to obtain a target object detection model. The images in the common annotated image set are obtained by directly obtaining and annotating objects, while the images in the difficult sample annotated image set are obtained by screening and then annotating objects.

[0201] In the above embodiment, the fusion entropy corresponding to the image can screen out difficult samples from the unlabeled images, and the difficult samples are manually labeled, which can ensure the accuracy of the labeling information of the difficult samples, thereby helping to improve the quality of model training.

[0202] In one embodiment, the object detection network includes an extraction layer and a prediction layer connected in series, the extraction layer is shared between the object detection networks, the prediction layer includes a prediction branch of at least one scale, and the output data of the object detection network is obtained by fusing the output data of the prediction branches of each scale.

[0203] The extraction layer is used to extract image features. The input data of the extraction layer is the image, and the output data is the image features, which can also be called feature maps. Although the parameters of each object detection network are different, the structure is the same. By sharing the extraction layer, the model training speed can be effectively improved.

[0204] The prediction layer is used to output prediction results and detection results. The input data of the prediction layer is image features, and the output data is a set of detection boxes. The prediction layer includes at least one prediction branch of different scales. Prediction branches of different scales obtain detection results based on features of different depths in the input feature map. Prediction branches of different scales have different corresponding receptive fields and are used to detect objects of different sizes. Detection branches with larger receptive fields are used to detect larger objects.

[0205] In the above embodiment, sharing the extraction layer between the various object detection networks can improve the model training speed, and the object detection network includes at least one scale prediction branch, which can expand the object detection range and improve the detection accuracy.

[0206] In one embodiment, Figure 5A As shown in the figure, refer to the yolov3 network structure, and replicate the predicted structure of yolov3 three times to obtain the final structure of the object detection model. Among them, yolov3 is a target detection network based on the yolo algorithm. The network structure of yolov3 is referenced Figure 5B The three object detection networks are called prediction head 0 (i.e., header0), prediction head 1 (i.e., header1), and prediction head 2 (i.e., header2). Each object detection network shares the underlying parameters of the multi-model, only predicting the structure design multiple heads and implementing the multi-model through different parameters. In the initialization stage, standard Gaussian distribution is used to initialize header1 and 2, and mean distribution is used to initialize header0.

[0207] During model training, header0 is used as the teacher branch to save the best parameters of each training, and header1 and 2 are used as student branches for joint learning of noise. Figure 5C When training the model based on pseudo samples, the prediction results of header1 and header2 are matched to calculate the loss Loss1 of header1, and back propagation is performed based on Loss1 to update the parameters of header1. Similarly, the prediction results of header2 and header1 are matched to calculate the loss Loss2 of header2, and back propagation is performed based on Loss2 to update the parameters of header2. The parameters of header0 are updated based on the momentum of the parameters of header1 and 2 combined with their own. Since the pseudo samples in the training samples are noisy, the prediction differences between header1 and 2 on the noisy samples may be large, and the parameter update method of header0 is equivalent to smoothing out the noise glitches in header1 and 2, so its saved parameters are optimal.

[0208] In one embodiment, the object detection model training method further includes:

[0209] Acquire a test image set; based on the test image set, test each object detection network of the target object detection model to obtain the detection accuracy corresponding to each object detection network; based on the detection accuracy, determine the key object detection network from each object detection network of the target object detection model; filter other object detection networks in the target object detection model except the key object detection network to obtain an updated target object detection model; the updated target object detection model is used to process the object detection task for the image.

[0210] The test image set refers to the test set of the model, which is used to test the detection accuracy of each object detection network in the model. The test image set is a labeled image set, which can be a labeled image set obtained through manual labeling. The key object detection network refers to the object detection network with the highest detection accuracy.

[0211] Specifically, in order to improve the detection efficiency of the model when it is applied, the computer device can test each object detection network of the target object detection model, select the object detection network with the best test result as the final model, and discard the remaining object detection networks.

[0212] The computer device obtains a test image set, tests each object detection network of the target object detection model based on the test image set, and obtains the detection accuracy corresponding to each object detection network. From the various object detection networks of the target object detection model, the object detection network with the highest detection accuracy is used as the key object detection network. The other object detection networks in the target object detection model except the key object detection network are filtered and discarded to obtain an updated target object detection model. Subsequently, the object detection task for the image is processed based on the updated target object detection model, that is, the object in the image is detected based on the updated target object detection model.

[0213] In one embodiment, in order to improve detection efficiency, the object detection task can also be split into at least two object detection subtasks, and a target object detection model is trained for each object detection subtask. When the model is applied, the same image is input into each target object detection model to obtain each detection frame set, and each detection frame set is merged to obtain the final detection result corresponding to the image.

[0214] For example, if the detection target of the object detection task has 100,000 objects, divide the 100,000 detection targets into 10 groups of 10,000, and each 10,000 detection targets corresponds to an object detection subtask. Through training, the target object detection model corresponding to each object detection subtask is obtained. Figure 6, extract frames from the video, perform 10 model detections on the extracted video frames, each model returns a set of detection frames, and then merge the 10 detected detection frame sets to get the final detection result. For further information, refer to Figure 7 ,When the model is applied, for each prediction head in a single model, the prediction head with the best prediction result on the test set is selected as the final model, and the remaining prediction heads are discarded.

[0215] In the above embodiment, after the model training is completed, the object detection network with the best test result is selected as the final model, and the remaining object detection networks are discarded, which can improve the detection efficiency while ensuring the detection accuracy.

[0216] In one embodiment, Figure 8 As shown, an object detection method is provided, and the method is applied to a computer device as an example. The computer device can be a terminal or a server, and the method can be executed by the terminal or the server itself alone, or through the interaction between the terminal and the server. Figure 8 , the object detection method includes the following steps:

[0217] Step S802: Acquire the image to be detected.

[0218] Step S804: input the image to be detected into the target object detection model to obtain a set of object detection frames corresponding to the image to be detected.

[0219] The image to be detected refers to an image to be subjected to object detection. The image to be detected may be any image. The target object detection model refers to a trained object detection model. The specific training process may refer to the various embodiments of the aforementioned object detection model training method, which will not be described in detail here.

[0220] The object detection frame set refers to the output data of the target object detection model. The object detection frame set may include the object detection frames output by each object detection network in the target object detection model, the object category corresponding to the object detection frame, and even the confidence corresponding to the object detection frame. It can be understood that the target object detection model includes at least one object detection network. For example, the initial object detection model includes at least two object detection networks. The trained initial object detection model can be directly used as the target object detection model for the final application, or the trained initial object detection model can be tested, and the object detection network with the best test result can be used as the target object detection model for the final application.

[0221] Specifically, the computer device obtains the image to be detected locally or from other devices, inputs the image to be detected into the target object detection model, and after data processing by the model, the model outputs a set of object detection frames corresponding to the image to be detected.

[0222] In the above object detection method, for the training process of the model, the unlabeled image is input into the model for object detection, and the unlabeled image is intelligently marked with a virtual detection frame based on the initial detection frame set obtained by the detection. The virtual detection frame is a pseudo label, and the unlabeled image and the pseudo label are used as the training data of the model, so that there is no need for manual labeling, and images with labeling information can also be obtained to train the model, which effectively reduces the image labeling cost. Further, when generating the virtual detection frame, the initial detection frames output by different object detection networks in the model for the same image are matched, and the matching initial detection frames belong to the same object category and are relatively similar. The matching detection frames corresponding to the same initial detection frame are fused to obtain a virtual detection frame with high accuracy, which helps to improve the quality of model training. The target object detection model is obtained by training based on non-manually labeled images. The training process effectively reduces the image labeling cost and has a certain accuracy. The image to be detected is input into the target object detection model, and the model can accurately predict the area where the object in the image to be detected is located, and obtain the object detection frame set corresponding to the image to be detected.

[0223] In one embodiment, step S804 includes:

[0224] The image to be detected is input into at least two target object detection models to obtain at least two object detection frame sets corresponding to the image to be detected; wherein different target object detection models correspond to different candidate object category sets, and a single target object detection model is used to determine the target object category corresponding to the image to be detected from the corresponding candidate object category set.

[0225] The candidate object category set includes multiple candidate object categories. The candidate object category refers to the candidate object category, and the target object category refers to the detected object category. The target object detection model is used to identify whether the object in the image belongs to the object category in the corresponding candidate object category set. It can be understood that one candidate object category set corresponds to one object detection subtask.

[0226] Specifically, in order to improve the detection efficiency, a large number of candidate object categories can be divided into at least two candidate object category sets, and the corresponding models can be trained based on the candidate object category sets to obtain at least two target object detection models. When the model is applied, the image to be detected is input into at least two target object detection models respectively to obtain at least two object detection frame sets corresponding to the image to be detected, and each object detection frame set constitutes the final detection result corresponding to the image to be detected.

[0227] In the above embodiment, different target object detection models correspond to different candidate object category sets, and inputting the images to be detected into at least two target object detection models respectively can improve the detection efficiency while ensuring the detection accuracy.

[0228] In one embodiment, the image to be detected is a video frame. The object detection method further includes:

[0229] A set of video frames corresponding to the target video is obtained, and each video frame in the set of video frames is input into the target object detection model to obtain a set of object detection frames corresponding to each video frame; feature extraction is performed on each object sub-image corresponding to the set of object detection frames to obtain multiple object features corresponding to the target video; clustering is performed on each object feature corresponding to the target video to obtain multiple central features; based on the feature distance between the object feature and the central feature, each target subsidiary feature corresponding to the central feature is determined from each object feature; based on each central feature and each corresponding target subsidiary feature, video representation information corresponding to the target video is obtained.

[0230] The target video refers to a video to be subjected to object detection. The target video may be any video. The video frame set corresponding to the target video includes multiple video frames in the target video. For example, the video frame set may include each video frame in the target video. The target video is subjected to frame extraction to obtain a corresponding video frame set.

[0231] An object detection box corresponds to an object sub-image. The image area circled by the object detection box on the image is the object sub-image. Feature extraction is performed on the object sub-image to obtain the corresponding object features. Various feature extraction algorithms for images can be used to extract features. For example, the object sub-image is input into the image feature extraction model, and the model outputs the object features.

[0232] The central feature refers to the cluster center obtained by clustering the central features corresponding to the same video. Various clustering algorithms can be used for clustering, for example, the kmeans clustering algorithm. The target subsidiary feature corresponding to the central feature refers to the object feature whose feature distance to the central feature is less than a preset distance.

[0233] The video representation information includes each central feature corresponding to the video and target subsidiary features corresponding to each central feature. The video representation information is used as a feature representation of the video.

[0234] Specifically, the computer device can generate video representation information corresponding to the target video based on the target object detection model. First, obtain a set of video frames corresponding to the target video, input each video frame in the video frame set into the target object detection model, and obtain a set of object detection frames corresponding to each video frame. Perform feature extraction on each object sub-image corresponding to the object detection frame set to obtain object features corresponding to each object sub-image, and each object feature belongs to the target video. Then, perform clustering processing on each object feature corresponding to the target video to obtain multiple central features. From each object feature, the target subsidiary feature corresponding to the central feature is obtained based on the object feature whose feature distance from the central feature is less than a preset distance. Each central feature has its own corresponding target subsidiary feature. Finally, each central feature and each corresponding target subsidiary feature form the video representation information corresponding to the target video.

[0235] In one embodiment, the number of central features may be determined based on the total number of object detection frames corresponding to the target video. For example, the number of central features = the total number of object detection frames / 20; the number of central features = the total number of object detection frames / 30.

[0236] In one embodiment, the object sub-image can be input into an image feature extraction model based on resnet101 (residual network) to obtain object features. The object sub-image can be input into the model composed of the network layers shown in Table 1 below, and the model outputs a 1x2048 representation embedding as the object feature. Among them, conv1~5 are network parameters based on ImageNet training. ImageNet is a large visualization database used for visual object recognition software research.

[0237] Table 1

[0238]

[0239] In one embodiment, the central feature may be labeled with a category to obtain the object category corresponding to the central feature. For example, the central feature may be labeled with a category manually. Accordingly, the video representation information includes the object category corresponding to the central feature.

[0240] In the above embodiment, the target object detection model is used to identify the area where the object is located in the video, feature extraction is performed on the identified object sub-image to obtain object features, and video representation information is generated based on the object features, so that accurate video representation can be quickly obtained.

[0241] In one embodiment, based on the feature distance between the object feature and the central feature, determining each target subsidiary feature corresponding to the central feature from each object feature includes:

[0242] The central feature with the smallest characteristic distance to the object feature is obtained as the target central feature corresponding to the object feature, and the target central features corresponding to each object feature are obtained; based on the characteristic distances between each object feature corresponding to the current target central feature and the current target central feature, the distance variance value corresponding to the characteristic distance is calculated; from each object feature corresponding to the current target central feature, a first preset proportion of object features are deleted according to the distance variance values ​​from large to small, and each initial subsidiary feature corresponding to the current target central feature is obtained; based on each initial subsidiary feature corresponding to the current target central feature, each target subsidiary feature corresponding to the current target central feature is obtained.

[0243] The current target center feature refers to any target center feature. The distance variance value refers to the variance value calculated according to the feature distance. The first preset ratio can be set according to actual needs, for example, 10%, 15%.

[0244] Specifically, for any object feature, the nearest central feature is found from each central feature as the target central feature corresponding to the object feature, indicating that the object feature may belong to the object represented by the target central feature, so that each central feature may find the object feature it may contain. Based on the feature distances between each object feature corresponding to the current target central feature and the current target central feature, the distance variance values ​​corresponding to each feature distance are calculated. The smaller the distance variance value, the more likely the object feature belongs to the object represented by the target central feature, and the more accurate the clustering. From each object feature corresponding to the current target central feature, the object features of the first preset proportion are deleted from large to small according to the distance variance value, and each initial subsidiary feature corresponding to the current target central feature is obtained. Finally, based on each initial subsidiary feature corresponding to the current target central feature, each target subsidiary feature corresponding to the current target central feature is obtained, for example, the initial subsidiary feature is used as the target subsidiary feature; the initial subsidiary feature is further screened to obtain the target subsidiary feature.

[0245] In the above embodiment, from the various object features corresponding to the target center feature, the object features with smaller distance variance values ​​are retained, and the object features with larger distance variance values ​​are deleted, so as to obtain the various target subsidiary features corresponding to the target center feature, which can improve the accuracy of the target subsidiary features belonging to the object represented by the target center feature.

[0246] In one embodiment, based on each initial subsidiary feature corresponding to the current target center feature, each target subsidiary feature corresponding to the current target center feature is obtained, including:

[0247] Based on the feature distances between each initial subsidiary feature corresponding to the current target center feature and the current target center feature, a second preset proportion of initial subsidiary features are obtained as intermediate subsidiary features according to the feature distance from small to large; the intermediate subsidiary features whose feature distance is less than the first preset distance are obtained as target subsidiary features, and each target subsidiary feature corresponding to the current target center feature is obtained.

[0248] The second preset ratio can be set according to actual needs, for example, 40%, 50%. The first preset distance can be set according to actual needs, for example, 0.2, 0.3.

[0249] Specifically, the initial subsidiary features are further screened, and based on the feature distances between each initial subsidiary feature corresponding to the current target center feature and the current target center feature, the initial subsidiary features of a second preset ratio are obtained as intermediate subsidiary features in ascending order of feature distances. The intermediate subsidiary features are further screened, and intermediate subsidiary features whose feature distances are less than the first preset distance are obtained as target subsidiary features, thereby obtaining each target subsidiary feature corresponding to the current target center feature.

[0250] In the above embodiment, the closer the initial subsidiary feature is to the target center feature, the more the initial subsidiary feature belongs to the target center feature. The initial subsidiary feature with a smaller feature distance to the target center feature and less than the first preset distance is used as the target subsidiary feature, which can ensure the accuracy of the target subsidiary feature belonging to the object represented by the target center feature.

[0251] In one embodiment, the specific process of generating video representation information of a video is described. First, the object detection result is obtained. For the video, one frame is extracted every 1s (second) to obtain multiple frames of global images, and each image is input into the target object detection model for object detection to obtain multiple object sub-images. Then, the object representation (i.e., object features, embedding) is obtained, and the image features of the intercepted object sub-image are extracted to obtain object features. Finally, the video representation information of the video is generated. Since the number of objects contained in the video is large and the object features generated are large, in order to abstract the "object" from these object features, the following clustering compression is required.

[0252] First, use kmeans clustering (the number of clusters is k, k = the number of detected object frames / 20, if the number of detected object frames is less than 10, k = 1) to obtain k centers (i.e., center features). It can be understood that the object centers here may be repeated. For example, when a car appears repeatedly, there may be 2 centers in k that represent the car, but represent two different states. Then, for all object representations (assuming that for a certain object representation j), find the closest center x from the k centers, indicating that the object representation j may belong to the object represented by the center x, so that each center can find the object representation it may contain. Further, remove invalid object representations from the object representations that may be contained in each center. For each center, count the distance between the object representations clustered to the center and the center, and calculate the variance of these distances. The smaller the variance, the more accurate the clustering, and the larger the variance, the inaccurate the clustering. The variance values ​​are sorted from small to large, and the top 90% of the object representations are retained, and the remaining object representations are discarded. In addition, for the k centers, among the object representations that each center may contain, the first 40% of the object representations that are closest to the center and whose feature distance is less than the preset threshold are selected as the object representations of the center, and the remaining object representations are discarded because they are too far from the center. Finally, k centers are obtained, each center represents an object, and each object contains a certain number of object representations.

[0253] The k centers and the corresponding object representations are stored. The storage format of each object representation is embedding->(embedding ID, object ID, video ID, total number of objects in the video, total number of embeddings in the video). The associated information after embedding is used for subsequent retrieval. The object ID is the cluster ID, that is, the center ID.

[0254] In one embodiment, the target video includes a query video and a candidate video. The object detection method further includes:

[0255] Based on the feature distance between the reference feature corresponding to the query video and the reference feature corresponding to the candidate video, the recalled video corresponding to the query video is determined from each candidate video; the reference feature includes at least one of the central feature or the target subsidiary feature; the video representation information corresponding to the query video and the video representation information corresponding to the recalled video are matched to determine the matching degree between the query video and the recalled video; and the query result corresponding to the query video is determined based on the matching degree.

[0256] The query video refers to a video used as a query condition for querying information related thereto.

[0257] Specifically, the video representation information can be applied to video query. The computer device obtains the query video and generates the video representation information corresponding to the query video based on the target object detection model. The video representation information corresponding to the candidate video can be generated in real time or in advance. Based on the feature distance between the reference feature corresponding to the query video and the reference feature corresponding to the candidate video, the recall video corresponding to the query video is determined from each candidate video. For example, if the feature distance between the reference feature corresponding to the query video and the reference feature corresponding to the candidate video is less than a certain threshold, the candidate video is used as the recall video corresponding to the query video; if the number of feature distances less than a certain threshold is greater than a preset number, the candidate video is used as the recall video corresponding to the query video; and so on. Further, the video representation information corresponding to the query video and the video representation information corresponding to the recall video are matched to determine the matching degree between the query video and the recall video, for example, the similarity between the video representation information is calculated as the matching degree. Finally, the query result corresponding to the query video is determined based on the matching degree, for example, the recall video with a matching degree greater than a preset threshold is used as the query result.

[0258] In the above embodiment, the recalled video corresponding to the query video is first determined from the candidate videos, where the recalled video is a video in which the same object as the query video appears, and then the query result corresponding to the query video is determined from the recalled video, thereby ensuring the accuracy of the query result.

[0259] In one embodiment, matching the video representation information corresponding to the query video with the video representation information corresponding to the recalled video to determine the matching degree between the query video and the recalled video includes:

[0260] The central feature corresponding to the query video is used as the query central feature, and the central feature corresponding to the recalled video is used as the recalled central feature; from each recalled central feature, the matching central feature corresponding to the query central feature is determined, and the query central feature and the corresponding matching central feature are combined into a central feature pair; among each target subsidiary feature corresponding to the recalled video, the target subsidiary features whose feature distance with the query central feature in the central feature pair is less than a second preset distance are counted to obtain the number of subsidiary feature matches corresponding to each central feature pair; based on the number of subsidiary feature matches corresponding to each central feature pair, the matching degree between the query video and the recalled video is determined.

[0261] Among them, the number of subsidiary feature matches corresponding to the central feature pair refers to the number of target subsidiary features whose feature distance from the query central feature in the central feature pair is less than a second preset distance, and the target subsidiary feature is the target subsidiary feature corresponding to the recalled video.

[0262] Specifically, the computer device determines the matching central feature corresponding to the query central feature from each recall central feature, for example, the recall central feature whose feature distance with the query central feature is less than the third preset distance is used as the matching central feature corresponding to the query central feature; the recall central feature whose object category is consistent with the query central feature is used as the matching central feature corresponding to the query central feature; the recall central feature whose object category is consistent with the query central feature and whose feature distance is less than the third preset distance is used as the matching central feature corresponding to the query central feature. The query central feature and the corresponding matching central feature form a central feature pair. The feature distances between the query central feature in the central feature pair and each target subsidiary feature corresponding to the recalled video are calculated, and among each target subsidiary feature corresponding to the recalled video, the target subsidiary features whose feature distance with the query central feature in the central feature pair is less than the second preset distance are counted to obtain the number of subsidiary feature matches corresponding to the central feature pair. The number of subsidiary feature matches corresponding to each central feature pair is counted respectively. Finally, based on the number of subsidiary feature matches corresponding to each central feature pair, the matching degree between the query video and the recalled video is determined, for example, the greater the number of subsidiary feature matches, the greater the matching degree.

[0263] In the above embodiment, it is not necessary to match all the features of the query video and the recalled video separately. Since the central feature is the most representative, the central features of the query video and the recalled video are first matched to establish a central feature pair. Among the target subsidiary features corresponding to the recalled video, the number of target subsidiary features that are close to the query central feature in the central feature pair is counted to obtain the number of subsidiary feature matches corresponding to each central feature pair. Based on the number of subsidiary feature matches corresponding to each central feature pair, the matching degree between the query video and the recalled video is determined. In this way, the accurate matching degree between the query video and the recalled video can be quickly calculated by relying on the central feature of the query video.

[0264] In one embodiment, when the query task corresponding to the query video is a duplicate removal task, the degree of match between the query video and the recalled video is determined based on the number of subsidiary feature matches corresponding to each central feature pair, including: calculating the first matching score corresponding to the central feature pair based on the proportion of the number of subsidiary feature matches in the total number of subsidiary features corresponding to the query video and the number of central features corresponding to the query video, and obtaining the first matching score corresponding to each central feature pair; obtaining the degree of match between the query video and the recalled video based on the first matching score corresponding to each central feature pair.

[0265] Determining the query result corresponding to the query video based on the matching degree includes: determining a target matching degree from each matching degree, determining the innovation degree corresponding to the query video based on the target matching degree; the innovation degree and the target matching degree are negatively correlated; and determining the query result corresponding to the query video based on the innovation degree.

[0266] The deduplication task is used to determine the degree of non-repetition between the query video and the candidate videos, that is, to evaluate the degree of innovation of the query video. The total number of subsidiary features corresponding to the query video refers to the total number of target subsidiary features corresponding to the query video.

[0267] Specifically, for the duplicate elimination task, when calculating the matching degree, the computer device calculates the first matching score corresponding to the central feature pair based on the proportion of the number of subsidiary feature matches in the total number of subsidiary features corresponding to the query video and the number of central features corresponding to the query video. For example, the ratio of the proportion to the number of central features is used as the first matching score. The first matching scores corresponding to each central feature pair are calculated respectively. Finally, the first matching scores corresponding to each central feature pair are fused to obtain the matching degree between the query video and the recalled video. For example, the sum of each first matching score is used as the matching degree.

[0268] The computer device determines the target matching degree from each matching degree, for example, obtains the maximum matching degree as the target matching degree; sorts the matching degrees from large to small, obtains several matching degrees with the highest sorting, and takes the average value as the target matching degree. The matching degree is used to indicate the matching degree and similarity between the query video and the recalled video. The higher the matching degree, the more similar the query video and the recalled video are. The innovation degree is used to indicate the innovation degree of the video. The more unique the video, the higher the innovation degree. Therefore, when determining the innovation degree corresponding to the query video based on the target matching degree, the innovation degree and the target matching degree are negatively correlated. The smaller the target matching degree, the greater the innovation degree. Finally, the innovation degree is used as the query result corresponding to the query video.

[0269] In one embodiment, for the duplicate elimination task, the calculation formula of the matching degree is as follows:

[0270]

[0271] Where n1 represents the number of central feature pairs. Nq3 represents the total number of central features of the query video. n0_i represents the number of subsidiary feature matches corresponding to the ith central feature pair. Nq4 represents the total number of subsidiary features corresponding to the query video.

[0272] In the above embodiment, for the deduplication task, the matching degree of the recalled video for the query video is calculated, and the matching degree can reflect the degree of repetition between the recalled video and the query video. The innovation degree is generated based on the matching degree, and the innovation degree can reflect the degree of non-repetition between the recalled video and the query video. The innovation degree is returned as the query result so as to quickly know the innovation degree of the query video.

[0273] In one embodiment, when the query task corresponding to the query video is a duplicate checking task, the degree of match between the query video and the recalled video is determined based on the number of subsidiary feature matches corresponding to each central feature pair, including: calculating the second matching score corresponding to the central feature pair based on the proportion of the number of subsidiary feature matches in the total number of subsidiary features corresponding to the recalled video, and the number of central features corresponding to the recalled video, to obtain the second matching score corresponding to each central feature pair; obtaining the degree of match between the query video and the recalled video based on the second matching score corresponding to each central feature pair.

[0274] Determining the query result corresponding to the query video based on the matching degree includes: sorting the recalled videos in descending order according to the matching degree to obtain a recalled video sequence; and using the recalled video sequence as the query result corresponding to the query video.

[0275] The duplication checking task is used to determine the duplication degree between the query video and the candidate videos, that is, to search for videos that are the same or similar to the query video from the candidate videos. The total number of subsidiary features corresponding to the recalled videos refers to the total number of target subsidiary features corresponding to the recalled videos.

[0276] Specifically, for the duplicate checking task, when calculating the matching degree, the computer device calculates the second matching score corresponding to the central feature pair based on the proportion of the number of subsidiary feature matches in the total number of subsidiary features corresponding to the recalled video, and the number of central features corresponding to the recalled video. For example, the ratio of the proportion and the number of central features is used as the second matching score. The second matching scores corresponding to each central feature pair are calculated respectively. Finally, based on the second matching scores corresponding to each central feature pair, the matching degree between the query video and the recalled video is obtained. For example, the sum of each first matching score is used as the matching degree.

[0277] The higher the matching degree, the more similar the query video and the recalled video are. The computer device sorts the recalled videos in descending order according to the matching degree to obtain a recalled video sequence, and uses the recalled video sequence as the query result corresponding to the query video. On the query terminal corresponding to the query video, the recalled videos can be displayed in sequence according to the video sorting in the recalled video sequence.

[0278] Subsequently, corresponding measures can be taken for the query video with reference to the duplicate checking results of the query video. For example, if the stock video is an uploaded video and the query video is an unuploaded video, if there are more than a preset number of recalled videos in the recalled video sequence whose matching degree is greater than a preset threshold, the query video is prohibited from being uploaded, otherwise the query video is allowed to be uploaded.

[0279] In one embodiment, for the duplicate checking task, the calculation formula of the matching degree is as follows:

[0280]

[0281] Where n1 represents the number of central feature pairs. N3 represents the total number of central features of the recalled video. n0_i represents the number of subsidiary feature matches corresponding to the ith central feature pair. N4 represents the total number of subsidiary features corresponding to the recalled video.

[0282] In the above embodiment, for the duplicate checking task, the matching degree of the query video to the recalled video is calculated. The matching degree can reflect the degree of repetition between the query video and the recalled video. The recalled videos are sorted from large to small according to the matching degree and returned as the query result, so as to quickly obtain similar videos of the query video.

[0283] In a specific embodiment, the method of the present application can be applied to rare object detection scenarios, such as rare bird detection scenarios and rare plant detection scenarios. In rare object detection scenarios, rare objects rarely appear in real videos and images, and it will be very difficult to manually collect data to annotate them in large quantities. The model training method of the present application can mine pseudo-labels and difficult samples that are both diverse and valuable for training to improve the model accuracy and model recall.

[0284] First, build a multi-head detection model. The multi-head detection model includes multiple prediction heads (i.e., object detection networks). Different prediction heads share underlying parameters, and different prediction heads have different initialization parameters. The input data of the multi-head detection model is an image, and the output data is a detection box, the object category corresponding to the detection box, and the output data can also include the confidence corresponding to the detection box. The multi-head detection model is used to detect rare objects from images.

[0285] The first stage of model training: obtain a small amount of manually labeled samples to pre-train the initialized multi-head detection model.

[0286] The second stage of model training: predict the video frames of unlabeled videos based on the trained model, label the video frames with pseudo labels according to the prediction results, obtain pseudo samples, and further train the trained model based on pseudo samples and manually labeled samples. In this stage, some of the multiple detection heads adjust parameters through a joint learning method to improve training efficiency. For example, Figure 5CThe prediction heads 1 and 2 in the other part of the detection head adjust the parameters by the average teacher method to control the impact of noise on model learning. For example, refer to Figure 5C The prediction head 0 in .

[0287] The third stage of model training: predict the video frames of unlabeled videos based on the trained model, and mine difficult samples based on the prediction results. Difficult samples help improve the accuracy of the model. Manually label the difficult samples, and further train the trained model based on the manually labeled difficult samples.

[0288] The second and third stages of model training are repeated repeatedly until the model reaches the target recall rate or the manual labeling budget is exhausted.

[0289] In the above embodiments, active learning and weak supervision methods are integrated and applied to object detection learning, and better results are obtained by taking the advantages of both methods than a single method. Active learning refers to a cyclic training method in which a small amount of data is first labeled for model training, and then the unlabeled data is predicted based on the existing model, and a second labeling is performed according to the prediction results, and then the model is trained, and so on. Weak supervision refers to mining pseudo-samples and difficult samples. Each detection head in the multi-head detection model improves detection accuracy through joint learning and average teacher methods. After the model training is completed, the best detection head can be extracted for use when the model is applied, so no additional reasoning resources will be generated.

[0290] In one embodiment, the method of the present application can be applied to video duplication elimination and duplication detection scenarios. Fig. 9 For the stock video (i.e., candidate video), video frames are extracted from it and input into the detection model trained by this application for object detection, multiple object sub-graphs are obtained, object features are extracted from the object sub-graphs to establish video representation information, and the video representation information is stored in the video stock. For the query video, video frames are extracted from it and input into the detection model trained by this application for object detection, multiple object sub-graphs are obtained, object features are extracted from the object sub-graphs to establish video representation information, and based on the video representation information, it is searched from the video stock whether there is a video with the same object.

[0291] For duplicate elimination scenarios, the original score of the query video is returned. The original score is used to reflect the non-duplicate parts of the query video and the stock videos. The recalled video corresponding to the query video is determined from the stock videos, and the matching score score_query between the query video and the recalled video is calculated. The matching scores are sorted from large to small, the top 1 matching score is taken, and 1-score_query is calculated to obtain the original score of the query video.

[0292] For duplicate checking scenarios, similar videos of the query video are returned. The recalled video corresponding to the query video is determined from the stock videos, the matching score score_db between the query video and the recalled video is calculated, and the recalled videos are sorted from large to small according to the matching score to obtain the duplicate checking result of the query video.

[0293] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0294] Based on the same inventive concept, the embodiment of the present application also provides an object detection model training device for implementing the object detection model training method involved above, and an object detection device for implementing the object detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more object detection model training device embodiments provided below can refer to the limitations on the object detection model training method above, and the specific limitations in one or more object detection device embodiments can refer to the limitations on the object detection method above, which will not be repeated here.

[0295] In one embodiment, Fig.10 As shown, an object detection model training device is provided, including: an image set acquisition module 1002, an initial detection frame determination module 1004, a detection frame matching module 1006, a virtual detection frame determination module 1008 and a model training module 1010, wherein:

[0296] The image set acquisition module 1002 is used to acquire a first image set; the first image set is an unlabeled image set.

[0297] The initial detection frame determination module 1004 is used to input each first image in the first image set into the initial object detection model to obtain an initial detection frame set corresponding to each first image; the initial detection frame set corresponding to the first image includes the initial detection frames output by each object detection network in the initial object detection model and the object category corresponding to the initial detection frame.

[0298] The detection frame matching module 1006 is used to match the initial detection frame output by the current object detection network with the initial detection frames output by each other object detection network for the same initial detection frame set based on the detection frame similarity between the initial detection frames belonging to the same object category, so as to obtain the matching detection frame corresponding to the initial detection frame output by the current object detection network.

[0299] The virtual detection frame determination module 1008 is used to fuse the initial detection frame and the corresponding matching detection frames to obtain a virtual detection frame, and obtain a set of virtual detection frames corresponding to the first image based on the virtual detection frames corresponding to the same first image.

[0300] The model training module 1010 is used to perform model training on the initial object detection model based on the first image set and the corresponding virtual detection frame sets to obtain a target object detection model.

[0301] In one embodiment, Fig.11 As shown, the object detection model training device also includes:

[0302] The pre-training module 1102 is used to obtain a second image set; the second image set is a labeled image set; each second image in the second image set is input into the candidate object detection model to obtain a set of predicted detection boxes corresponding to each second image; the set of predicted detection boxes includes a subset of predicted detection boxes output by each object detection network in the candidate object detection model, and the predicted detection box subset includes a predicted detection box output by the same object detection network, an object category and a confidence corresponding to the predicted detection box; based on the difference between the labeled detection box set corresponding to the same second image and the corresponding predicted detection box subsets, the sub-loss corresponding to the second image on each object detection network is obtained; the sub-loss includes a detection box loss, a category loss and a confidence loss; based on the sub-loss corresponding to each second image on the same object detection network, the network loss corresponding to each object detection network is obtained; based on the network loss, the network parameters of the corresponding object detection network in the candidate object detection model are adjusted until the first convergence condition is met to obtain an initial object detection model.

[0303] In one embodiment, the detection frame matching module 1006 is further configured to:

[0304] For the current initial detection frame set, a first detection frame is determined from the initial detection frame output by the current object detection network, and a second detection frame is determined from the initial detection frame output by other object detection networks; the object categories corresponding to the first detection frame and the second detection frame are the same; based on the detection frame overlap between the first detection frame and the second detection frame, the detection frame similarity between the first detection frame and the second detection frame is obtained; the second detection frame corresponding to the detection frame similarity greater than the preset similarity is obtained as a candidate detection frame, and the candidate detection frame corresponding to the maximum detection frame similarity is obtained as the matching detection frame corresponding to the first detection frame.

[0305] In one embodiment, the model training module 1010 is further used to:

[0306] Based on the difference between the initial detection frame subset and the corresponding virtual detection frame set output by the same target object detection network for the same first image, the target loss corresponding to the target object detection network is calculated to obtain the target losses corresponding to each target object detection network; based on the target loss corresponding to the target object detection network, the initial network parameters corresponding to the target object detection network in the initial object detection model are adjusted to obtain the updated network parameters corresponding to the target object detection network; based on the updated network parameters corresponding to each target object detection network, the initial network parameters of other object detection networks in the initial object detection model except the target object detection network are adjusted to obtain the updated network parameters corresponding to the other object detection networks; based on the updated network parameters corresponding to each object detection network, the target object detection model is obtained.

[0307] In one embodiment, the model training module 1010 is further used to:

[0308] For the same initial detection frame set, based on the detection frame similarity between the initial detection frames belonging to the same object category, the initial detection frame output by the current target object detection network is matched with the initial detection frames output by each other target object detection network to obtain the matching result corresponding to the initial detection frame output by the current target object detection network; for the initial detection frame subset output by the current target object detection network for the current first image, based on the difference between the initial detection frame in the initial detection frame subset and the corresponding supervised detection frame, the initial sub-loss corresponding to each initial detection frame in the initial detection frame subset is obtained; when the matching result is a successful match, the supervised detection frame corresponding to the initial detection frame is a matched frame. The matched detection frame is matched successfully. When the matching result is a matching failure, the supervised detection frame corresponding to the initial detection frame is the corresponding virtual detection frame in the virtual detection frame set; the loss weight corresponding to the initial detection frame is determined based on the matching result corresponding to the initial detection frame; the loss weight corresponding to the successful match is greater than the loss weight corresponding to the failed match; based on the loss weight corresponding to the initial detection frame, the initial sub-losses corresponding to each initial detection frame in the initial detection frame subset are fused to obtain the target sub-loss corresponding to the current first image on the current target object detection network; based on the target sub-losses corresponding to each first image on the current target object detection network, the target loss corresponding to the current target object detection network is obtained.

[0309] In one embodiment, the model training module 1010 is further used to:

[0310] The network weights corresponding to each object detection network are obtained; the network weight corresponding to the target object detection network is less than the network weights corresponding to other object detection networks; based on the network weights corresponding to each object detection network, the updated network parameters corresponding to each target object detection network and the initial network parameters corresponding to other object detection networks are fused to obtain the updated network parameters corresponding to other object detection networks.

[0311] In one embodiment, the model training module 1010 is further used to:

[0312] Based on the first image set and the corresponding virtual detection frame sets, the model parameters of the initial object detection model are adjusted to obtain an intermediate object detection model; the intermediate object detection model is used as the initial object detection model, and the step of obtaining the first image set is returned to execute until the second convergence condition is met to obtain the target object detection model.

[0313] In one embodiment, the model training module 1010 is further used to:

[0314] A third image set is obtained; the third image set is a labeled image set; based on the first image set and the corresponding virtual detection frame sets, and the third image set, the model parameters of the initial object detection model are adjusted to obtain an intermediate object detection model.

[0315] In one embodiment, the model training module 1010 is further used to:

[0316] A fourth image set is obtained; the fourth image set is an unlabeled image set; each fourth image in the fourth image set is input into an initial object detection model to obtain an initial detection frame set corresponding to each fourth image; the initial detection frame set corresponding to the fourth image includes an initial detection frame subset output by each object detection network in the initial object detection model, and the initial detection frame subset includes an initial detection frame output by the same object detection network and a confidence corresponding to the initial detection frame; information entropy statistics are performed on the confidence corresponding to each initial detection frame in the same initial detection frame subset to obtain a predicted entropy corresponding to each initial detection frame subset, and each predicted entropy corresponding to the same fourth image is fused to obtain a fused entropy corresponding to each fourth image; information entropy division is performed based on the fused entropy corresponding to each fourth image to obtain a plurality of information entropy intervals, and a target interval is determined from the plurality of information entropy intervals; a target image to be labeled is determined from the fourth image whose fused entropy is located in the target interval, and annotates the target image to obtain a labeled detection frame set corresponding to the target image; a third image set is obtained based on each target image and the corresponding labeled detection frame set.

[0317] In one embodiment, the object detection network includes an extraction layer and a prediction layer connected in series, the extraction layer is shared between the object detection networks, the prediction layer includes a prediction branch of at least one scale, and the output data of the object detection network is obtained by fusing the output data of the prediction branches of each scale.

[0318] In one embodiment, Fig.11 As shown, the object detection model training device also includes:

[0319] The model testing module 1104 is used to obtain a test image set; based on the test image set, each object detection network of the target object detection model is tested to obtain the detection accuracy corresponding to each object detection network; based on the detection accuracy, a key object detection network is determined from each object detection network of the target object detection model; other object detection networks except the key object detection network in the target object detection model are filtered to obtain an updated target object detection model; the updated target object detection model is used to process object detection tasks for images.

[0320] The above-mentioned object detection model training device can obtain images with annotation information to train the model without manual annotation, which effectively reduces the image annotation cost. In addition, the annotation information has a certain accuracy.

[0321] In one embodiment, Fig.12 As shown, an object detection device is provided, including: an image acquisition module 1202 and an object detection module 1204, wherein:

[0322] The image acquisition module 1202 is used to acquire the image to be detected.

[0323] The object detection module 1204 is used to input the image to be detected into the target object detection model to obtain a set of object detection frames corresponding to the image to be detected;

[0324] Among them, the training process of the target object detection model can refer to the content of the aforementioned object detection model training method.

[0325] In one embodiment, the object detection module 1204 is further configured to:

[0326] The image to be detected is input into at least two target object detection models to obtain at least two object detection frame sets corresponding to the image to be detected; wherein different target object detection models correspond to different candidate object category sets, and a single target object detection model is used to determine the target object category corresponding to the image to be detected from the corresponding candidate object category set.

[0327] In one embodiment, the image to be detected is a video frame. The object detection module 1204 is also used for:

[0328] A set of video frames corresponding to the target video is obtained, and each video frame in the set of video frames is input into the target object detection model to obtain a set of object detection frames corresponding to each video frame; feature extraction is performed on each object sub-image corresponding to the set of object detection frames to obtain multiple object features corresponding to the target video; clustering is performed on each object feature corresponding to the target video to obtain multiple central features; based on the feature distance between the object feature and the central feature, each target subsidiary feature corresponding to the central feature is determined from each object feature; based on each central feature and each corresponding target subsidiary feature, video representation information corresponding to the target video is obtained.

[0329] In one embodiment, the object detection module 1204 is further configured to:

[0330] The central feature with the smallest characteristic distance to the object feature is obtained as the target central feature corresponding to the object feature, and the target central features corresponding to each object feature are obtained; based on the characteristic distances between each object feature corresponding to the current target central feature and the current target central feature, the distance variance value corresponding to the characteristic distance is calculated; from each object feature corresponding to the current target central feature, a first preset proportion of object features are deleted according to the distance variance values ​​from large to small, and each initial subsidiary feature corresponding to the current target central feature is obtained; based on each initial subsidiary feature corresponding to the current target central feature, each target subsidiary feature corresponding to the current target central feature is obtained.

[0331] In one embodiment, the object detection module 1204 is further configured to:

[0332] Based on the feature distances between each initial subsidiary feature corresponding to the current target center feature and the current target center feature, a second preset proportion of initial subsidiary features are obtained as intermediate subsidiary features according to the feature distance from small to large; the intermediate subsidiary features whose feature distance is less than the first preset distance are obtained as target subsidiary features, and each target subsidiary feature corresponding to the current target center feature is obtained.

[0333] In one embodiment, the target video includes a query video and a candidate video. The object detection module 1204 is also used to:

[0334] Based on the feature distance between the reference feature corresponding to the query video and the reference feature corresponding to the candidate video, the recalled video corresponding to the query video is determined from each candidate video; the reference feature includes at least one of the central feature or the target subsidiary feature; the video representation information corresponding to the query video and the video representation information corresponding to the recalled video are matched to determine the matching degree between the query video and the recalled video; and the query result corresponding to the query video is determined based on the matching degree.

[0335] In one embodiment, the object detection module 1204 is further configured to:

[0336] The central feature corresponding to the query video is used as the query central feature, and the central feature corresponding to the recalled video is used as the recalled central feature; from each recalled central feature, the matching central feature corresponding to the query central feature is determined, and the query central feature and the corresponding matching central feature are combined into a central feature pair; among each target subsidiary feature corresponding to the recalled video, the target subsidiary features whose feature distance with the query central feature in the central feature pair is less than a second preset distance are counted to obtain the number of subsidiary feature matches corresponding to each central feature pair; based on the number of subsidiary feature matches corresponding to each central feature pair, the matching degree between the query video and the recalled video is determined.

[0337] In one embodiment, when the query task corresponding to the query video is a duplicate elimination task, the object detection module 1204 is further configured to:

[0338] Based on the proportion of the number of subsidiary feature matches in the total number of subsidiary features corresponding to the query video, and the number of central features corresponding to the query video, the first matching scores corresponding to the central feature pairs are calculated to obtain the first matching scores corresponding to each central feature pair; based on the first matching scores corresponding to each central feature pair, the matching degree between the query video and the recalled video is obtained; the target matching degree is determined from each matching degree, and the innovation degree corresponding to the query video is determined based on the target matching degree; the innovation degree and the target matching degree are negatively correlated; and the query result corresponding to the query video is determined based on the innovation degree.

[0339] In one embodiment, when the query task corresponding to the query video is a duplicate check task, the object detection module 1204 is further used to:

[0340] Based on the proportion of the number of matching auxiliary features in the total number of auxiliary features corresponding to the recalled video, and the number of central features corresponding to the recalled video, the second matching scores corresponding to the central feature pairs are calculated to obtain the second matching scores corresponding to each central feature pair; based on the second matching scores corresponding to each central feature pair, the matching degree between the query video and the recalled video is obtained; the recalled videos are sorted from large to small according to the matching degree to obtain a recalled video sequence; the recalled video sequence is used as the query result corresponding to the query video.

[0341] In the above-mentioned object detection device, the target object detection model is obtained by training based on non-artificially annotated images. The training process effectively reduces the image annotation cost and has a certain accuracy. The image to be detected is input into the target object detection model. The model can accurately predict the area where the object in the image to be detected is located and obtain the object detection frame set corresponding to the image to be detected.

[0342] Each module in the above-mentioned object detection model training device and object detection device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0343] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.13As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as training sets, test sets, models, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an object detection model training method and an object detection method are implemented.

[0344] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.14 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an object detection model training method and an object detection method are implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0345] Those skilled in the art will understand that Fig.13 , 14The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0346] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0347] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0348] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0349] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0350] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0351] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0352] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for training an object detection model, characterized in that: The method comprises: Acquire a first image set; the first image set is an unlabeled image set; Inputting each first image in the first image set into an initial object detection model to obtain a set of initial detection frames corresponding to each first image; the set of initial detection frames corresponding to the first images includes initial detection frames outputted by each object detection network in the initial object detection model and object categories corresponding to the initial detection frames; For the same set of initial detection frames, based on the detection frame similarity between the initial detection frames belonging to the same object category, the initial detection frames output by the current object detection network are matched with the initial detection frames output by each other object detection network to obtain the matching detection frame corresponding to the initial detection frame output by the current object detection network; The initial detection frame and the corresponding matching detection frames are fused to obtain a virtual detection frame, and a set of virtual detection frames corresponding to the first image is obtained based on the virtual detection frames corresponding to the same first image; Based on the first image set and the corresponding virtual detection frame sets, the initial object detection model is trained to obtain a target object detection model, including: based on the difference between the initial detection frame subset and the corresponding virtual detection frame set output by the same target object detection network for the same first image, the target loss corresponding to the target object detection network is calculated to obtain the target losses corresponding to each target object detection network; based on the target loss corresponding to the target object detection network, the initial network parameters corresponding to the target object detection network in the initial object detection model are adjusted to obtain the updated network parameters corresponding to the target object detection network; based on the updated network parameters corresponding to each target object detection network, the initial network parameters of other object detection networks in the initial object detection model except the target object detection network are adjusted to obtain the updated network parameters corresponding to the other object detection networks; based on the updated network parameters corresponding to each object detection network, the target object detection model is obtained.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a second image set; the second image set is a labeled image set; Inputting each second image in the second image set into a candidate object detection model to obtain a set of predicted detection boxes corresponding to each second image; the set of predicted detection boxes includes a subset of predicted detection boxes output by each object detection network in the candidate object detection model, and the subset of predicted detection boxes includes a predicted detection box output by the same object detection network, an object category corresponding to the predicted detection box, and a confidence level; Based on the differences between the annotated detection box sets corresponding to the same second image and the corresponding predicted detection box subsets, the sub-losses corresponding to the second image on each object detection network are obtained; the sub-losses include detection box loss, category loss and confidence loss; Based on the sub-losses corresponding to each second image on the same object detection network, the network losses corresponding to each object detection network are obtained; Based on the network loss, a network parameter of a corresponding object detection network in the candidate object detection model is adjusted until a first convergence condition is met, thereby obtaining an initial object detection model.

3. The method according to claim 1, characterized in that The method of matching the initial detection frame output by the current object detection network with the initial detection frames output by each other object detection network based on the detection frame similarity between the initial detection frames belonging to the same object category for the same initial detection frame set to obtain the matching detection frame corresponding to the initial detection frame output by the current object detection network includes: For the current initial detection frame set, determine a first detection frame from the initial detection frame output by the current object detection network, and determine a second detection frame from the initial detection frames output by other object detection networks; the first detection frame and the second detection frame correspond to the same object category; Obtaining a detection frame similarity between the first detection frame and the second detection frame based on a detection frame overlap between the first detection frame and the second detection frame; A second detection frame corresponding to a detection frame similarity greater than a preset similarity is obtained as a candidate detection frame, and a candidate detection frame corresponding to a maximum detection frame similarity is obtained as a matching detection frame corresponding to the first detection frame.

4. The method according to claim 1, characterized in that: The method of calculating the target loss corresponding to the target object detection network based on the difference between the initial detection frame subset output by the same target object detection network and the corresponding virtual detection frame set based on the same first image, and obtaining the target loss corresponding to each target object detection network, respectively, includes: For the same set of initial detection frames, based on the detection frame similarity between the initial detection frames belonging to the same object category, the initial detection frame output by the current target object detection network is matched with the initial detection frames output by each other target object detection network to obtain the matching result corresponding to the initial detection frame output by the current target object detection network; For the initial detection frame subset output by the current target object detection network of the current first image, based on the difference between the initial detection frame in the initial detection frame subset and the corresponding supervised detection frame, the initial sub-loss corresponding to each initial detection frame in the initial detection frame subset is obtained; when the matching result is a successful match, the supervised detection frame corresponding to the initial detection frame is a successfully matched matching detection frame, and when the matching result is a failed match, the supervised detection frame corresponding to the initial detection frame is a corresponding virtual detection frame in the virtual detection frame set; Determine a loss weight corresponding to the initial detection frame based on a matching result corresponding to the initial detection frame; the loss weight corresponding to a successful match is greater than the loss weight corresponding to a failed match; Based on the loss weight corresponding to the initial detection frame, the initial sub-losses corresponding to each initial detection frame in the initial detection frame subset are fused to obtain the target sub-loss corresponding to the current first image on the current target object detection network; Based on the target sub-losses corresponding to each first image on the current target object detection network, the target loss corresponding to the current target object detection network is obtained.

5. The method according to claim 1, characterized in that The adjusting the initial network parameters of other object detection networks except the target object detection network in the initial object detection model based on the updated network parameters respectively corresponding to each target object detection network to obtain the updated network parameters corresponding to the other object detection networks includes: Obtaining network weights corresponding to each object detection network; the network weight corresponding to the target object detection network is less than the network weights corresponding to the other object detection networks; Based on the network weights corresponding to the respective object detection networks, the updated network parameters corresponding to the respective target object detection networks and the initial network parameters corresponding to the other object detection networks are fused to obtain the updated network parameters corresponding to the other object detection networks.

6. The method according to any one of claims 1 to 5, characterized in that The performing model training on the initial object detection model based on the first image set and the corresponding virtual detection frame sets to obtain a target object detection model includes: Based on the first image set and the corresponding virtual detection frame sets, adjusting model parameters of the initial object detection model to obtain an intermediate object detection model; The intermediate object detection model is used as the initial object detection model, and the step of obtaining the first image set is returned to be executed until the second convergence condition is met to obtain the target object detection model.

7. The method according to claim 6, characterized in that The adjusting model parameters of the initial object detection model based on the first image set and the corresponding virtual detection frame sets to obtain an intermediate object detection model includes: Acquire a third image set; the third image set is a labeled image set; Based on the first image set and the corresponding virtual detection frame sets, and the third image set, the model parameters of the initial object detection model are adjusted to obtain an intermediate object detection model.

8. The method according to claim 7, characterized in that The acquiring of the third image set comprises: Acquire a fourth image set; the fourth image set is an unlabeled image set; Inputting each fourth image in the fourth image set into the initial object detection model to obtain a set of initial detection frames corresponding to each fourth image; the set of initial detection frames corresponding to the fourth images includes a subset of initial detection frames output by each object detection network in the initial object detection model, and the subset of initial detection frames includes an initial detection frame output by the same object detection network and a confidence level corresponding to the initial detection frame; Perform information entropy statistics on the confidences corresponding to each initial detection frame in the same initial detection frame subset to obtain the predicted entropies corresponding to each initial detection frame subset, and fuse the predicted entropies corresponding to the same fourth image to obtain the fused entropies corresponding to each fourth image; Performing information entropy division based on the fusion entropies corresponding to each fourth image to obtain a plurality of information entropy intervals, and determining a target interval from the plurality of information entropy intervals; Determine a target image to be labeled from the fourth image whose fusion entropy is located in the target interval, perform object labeling on the target image, and obtain a set of labeled detection frames corresponding to the target image; A third image set is obtained based on each target image and the corresponding set of annotated detection frames.

9. The method according to any one of claims 1 to 5, characterized in that The object detection network includes an extraction layer and a prediction layer connected in series, the extraction layer is shared between the object detection networks, the prediction layer includes a prediction branch of at least one scale, and the output data of the object detection network is obtained by fusing the output data of the prediction branches of each scale.

10. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Get a test image set; Based on the test image set, each object detection network of the target object detection model is tested to obtain the detection accuracy corresponding to each object detection network; Based on the detection accuracy, determining a key object detection network from each object detection network of the target object detection model; The other object detection networks in the target object detection model except the key object detection network are filtered to obtain an updated target object detection model; the updated target object detection model is used to process the object detection task for the image.

11. An object detection method, characterized in that: The method comprises: Acquire the image to be detected; Inputting the image to be detected into a target object detection model to obtain a set of object detection frames corresponding to the image to be detected; The training process of the target object detection model includes the following steps: Acquire a first image set; the first image set is an unlabeled image set; Inputting each first image in the first image set into an initial object detection model to obtain a set of initial detection frames corresponding to each first image; the set of initial detection frames corresponding to the first images includes initial detection frames outputted by each object detection network in the initial object detection model and object categories corresponding to the initial detection frames; For the same set of initial detection frames, based on the detection frame similarity between the initial detection frames belonging to the same object category, the initial detection frames output by the current object detection network are matched with the initial detection frames output by each other object detection network to obtain the matching detection frame corresponding to the initial detection frame output by the current object detection network; The initial detection frame and the corresponding matching detection frames are fused to obtain a virtual detection frame, and a set of virtual detection frames corresponding to the first image is obtained based on the virtual detection frames corresponding to the same first image; Based on the first image set and the corresponding virtual detection frame sets, the initial object detection model is trained to obtain a target object detection model, including: based on the difference between the initial detection frame subset and the corresponding virtual detection frame set output by the same target object detection network for the same first image, the target loss corresponding to the target object detection network is calculated to obtain the target losses corresponding to each target object detection network; based on the target loss corresponding to the target object detection network, the initial network parameters corresponding to the target object detection network in the initial object detection model are adjusted to obtain the updated network parameters corresponding to the target object detection network; based on the updated network parameters corresponding to each target object detection network, the initial network parameters of other object detection networks in the initial object detection model except the target object detection network are adjusted to obtain the updated network parameters corresponding to the other object detection networks; based on the updated network parameters corresponding to each object detection network, the target object detection model is obtained.

12. The method according to claim 11, characterized in that The step of inputting the image to be detected into a target object detection model to obtain a set of object detection frames corresponding to the image to be detected includes: Inputting the image to be detected into at least two target object detection models to obtain at least two object detection frame sets corresponding to the image to be detected; Among them, different target object detection models correspond to different candidate object category sets, and a single target object detection model is used to determine the target object category corresponding to the image to be detected from the corresponding candidate object category set.

13. The method according to claim 11 or 12, characterized in that: The image to be detected is a video frame; The method further comprises: Obtain a set of video frames corresponding to the target video, input each video frame in the set of video frames into the target object detection model, and obtain a set of object detection frames corresponding to each video frame; Extracting features of each object sub-image corresponding to the object detection frame set to obtain multiple object features corresponding to the target video; Performing clustering processing on each object feature corresponding to the target video to obtain multiple central features; Based on the feature distance between the object feature and the central feature, determining each target subsidiary feature corresponding to the central feature from the object features; Based on each central feature and each corresponding target subsidiary feature, video representation information corresponding to the target video is obtained.

14. The method according to claim 13, characterized in that The determining, based on the feature distance between the object feature and the central feature, from the object features, each target subsidiary feature corresponding to the central feature includes: Obtain the central feature with the smallest feature distance from the object feature as the target central feature corresponding to the object feature, and obtain the target central features corresponding to each object feature respectively; Based on the feature distances between each object feature corresponding to the current target center feature and the current target center feature, a distance variance value corresponding to the feature distance is calculated; From each object feature corresponding to the current target center feature, delete a first preset proportion of object features according to distance variance values ​​from large to small, to obtain each initial subsidiary feature corresponding to the current target center feature; Based on the initial subsidiary features corresponding to the current target central feature, the target subsidiary features corresponding to the current target central feature are obtained.

15. The method according to claim 14, characterized in that The obtaining, based on each initial subsidiary feature corresponding to the current target center feature, each target subsidiary feature corresponding to the current target center feature includes: Based on the feature distances between each initial subsidiary feature corresponding to the current target center feature and the current target center feature, obtaining the initial subsidiary features of a second preset proportion as intermediate subsidiary features in ascending order of feature distances; An intermediate subsidiary feature whose feature distance is less than the first preset distance is acquired as a target subsidiary feature, and each target subsidiary feature corresponding to the current target center feature is obtained.

16. The method according to claim 13, characterized in that The target video includes a query video and a candidate video; The method further comprises: Determining a recalled video corresponding to the query video from each candidate video based on a feature distance between a reference feature corresponding to the query video and a reference feature corresponding to the candidate video; the reference feature includes at least one of a central feature or a target-attached feature; Matching the video representation information corresponding to the query video with the video representation information corresponding to the recalled video to determine a matching degree between the query video and the recalled video; A query result corresponding to the query video is determined based on the matching degree.

17. The method according to claim 16, characterized in that The matching of the video representation information corresponding to the query video with the video representation information corresponding to the recalled video to determine the matching degree between the query video and the recalled video includes: The central feature corresponding to the query video is used as the query central feature, and the central feature corresponding to the recall video is used as the recall central feature; From each recall center feature, determine the matching center feature corresponding to the query center feature, and form a center feature pair with the query center feature and the corresponding matching center feature; Among the target subsidiary features corresponding to the recalled videos, the target subsidiary features whose feature distance with the query central feature in the central feature pair is less than a second preset distance are counted to obtain the number of subsidiary feature matches corresponding to each central feature pair; Based on the number of subsidiary feature matches corresponding to each of the central feature pairs, the degree of match between the query video and the recalled video is determined.

18. The method according to claim 17, characterized in that When the query task corresponding to the query video is a duplicate elimination task, determining the matching degree between the query video and the recalled video based on the number of subsidiary feature matches corresponding to each of the central feature pairs includes: Based on the proportion of the number of subsidiary feature matches in the total number of subsidiary features corresponding to the query video and the number of central features corresponding to the query video, first matching scores corresponding to the central feature pairs are calculated to obtain first matching scores corresponding to each central feature pair; Based on the first matching scores corresponding to the respective central feature pairs, obtaining a matching degree between the query video and the recalled video; The determining, based on the matching degree, a query result corresponding to the query video includes: Determine a target matching degree from each matching degree, and determine the innovation degree corresponding to the query video based on the target matching degree; the innovation degree is negatively correlated with the target matching degree; A query result corresponding to the query video is determined based on the novelty.

19. The method according to claim 17, characterized in that When the query task corresponding to the query video is a duplicate checking task, determining the matching degree between the query video and the recalled video based on the number of matching subsidiary features corresponding to each of the central feature pairs includes: Based on the proportion of the number of subsidiary feature matches in the total number of subsidiary features corresponding to the recalled video and the number of central features corresponding to the recalled video, the second matching scores corresponding to the central feature pairs are calculated to obtain the second matching scores corresponding to the respective central feature pairs; Obtaining a degree of match between the query video and the recalled video based on the second matching scores corresponding to the respective central feature pairs; The determining, based on the matching degree, a query result corresponding to the query video includes: Sort the recalled videos according to the matching degree from large to small to obtain a recalled video sequence; The recalled video sequence is used as the query result corresponding to the query video.

20. An object detection model training device, characterized in that: The device comprises: An image set acquisition module, used to acquire a first image set; the first image set is an unlabeled image set; an initial detection frame determination module, configured to input each first image in the first image set into an initial object detection model to obtain a set of initial detection frames corresponding to each first image; the set of initial detection frames corresponding to the first images includes initial detection frames outputted by each object detection network in the initial object detection model and object categories corresponding to the initial detection frames; A detection frame matching module is used to match the initial detection frame output by the current object detection network with the initial detection frames output by each other object detection network for the same initial detection frame set based on the detection frame similarity between the initial detection frames belonging to the same object category, so as to obtain a matching detection frame corresponding to the initial detection frame output by the current object detection network; A virtual detection frame determination module is used to fuse the initial detection frame and the corresponding matching detection frames to obtain a virtual detection frame, and obtain a set of virtual detection frames corresponding to the first image based on the virtual detection frames corresponding to the same first image; The model training module is used to perform model training on the initial object detection model based on the first image set and the corresponding virtual detection frame sets to obtain a target object detection model, including: calculating the target loss corresponding to the target object detection network based on the difference between the initial detection frame subset and the corresponding virtual detection frame set output by the same first image through the same target object detection network, and obtaining the target losses corresponding to each target object detection network; adjusting the initial network parameters corresponding to the target object detection network in the initial object detection model based on the target loss corresponding to the target object detection network, and obtaining the updated network parameters corresponding to the target object detection network; adjusting the initial network parameters of other object detection networks in the initial object detection model except the target object detection network, and obtaining the updated network parameters corresponding to the other object detection networks based on the updated network parameters corresponding to each target object detection network, and obtaining the target object detection model.

21. An object detection device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be detected; An object detection module is used to input the image to be detected into a target object detection model to obtain a set of object detection frames corresponding to the image to be detected; The training process of the target object detection model includes the following steps: Acquire a first image set; the first image set is an unlabeled image set; Inputting each first image in the first image set into an initial object detection model to obtain a set of initial detection frames corresponding to each first image; the set of initial detection frames corresponding to the first images includes initial detection frames outputted by each object detection network in the initial object detection model and object categories corresponding to the initial detection frames; For the same set of initial detection frames, based on the detection frame similarity between the initial detection frames belonging to the same object category, the initial detection frames output by the current object detection network are matched with the initial detection frames output by each other object detection network to obtain the matching detection frame corresponding to the initial detection frame output by the current object detection network; The initial detection frame and the corresponding matching detection frames are fused to obtain a virtual detection frame, and a set of virtual detection frames corresponding to the first image is obtained based on the virtual detection frames corresponding to the same first image; Based on the first image set and the corresponding virtual detection frame sets, the initial object detection model is trained to obtain a target object detection model, including: based on the difference between the initial detection frame subset and the corresponding virtual detection frame set output by the same target object detection network for the same first image, the target loss corresponding to the target object detection network is calculated to obtain the target losses corresponding to each target object detection network; based on the target loss corresponding to the target object detection network, the initial network parameters corresponding to the target object detection network in the initial object detection model are adjusted to obtain the updated network parameters corresponding to the target object detection network; based on the updated network parameters corresponding to each target object detection network, the initial network parameters of other object detection networks in the initial object detection model except the target object detection network are adjusted to obtain the updated network parameters corresponding to the other object detection networks; based on the updated network parameters corresponding to each object detection network, the target object detection model is obtained.

22. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 19 are implemented.

23. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 19 are implemented.

24. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 19 are implemented.

Images