Object detection method, device, computer-readable storage medium and processor

Abstract

The application discloses an object detection method and device, a computer readable storage medium and a processor. The method comprises the following steps: obtaining a target image of an object to be detected; detecting the target image based on a detector to obtain at least one target sub-object in the object to be detected, wherein the detector is obtained by training a target detection data set; extracting a sub-object feature of the target sub-object; and searching in a search library based on the sub-object feature to obtain a type of the target sub-object, wherein the search library comprises types of objects from which different object features come. The application solves the technical problem of low accuracy of object detection.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically, to an object detection method, apparatus, computer-readable storage medium, and processor. Background Technology

[0002] Currently, the types of objects that need to be detected are numerous and diverse. Simply using detection algorithms cannot encompass all the objects that need to be detected. For example, if an image of an object is input and then the score of the objects contained in the image is determined, the detection algorithm needs to predefine the types of all object categories. Moreover, the recognition effect of small target objects is not good, and the accuracy of object classification is not high, thus resulting in the technical problem of low accuracy in object detection.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides an object detection method, apparatus, computer-readable storage medium, and processor to at least solve the technical problem of low accuracy in object detection.

[0005] According to one aspect of the present invention, an object detection method is provided. The method may include: acquiring a target image of an object to be detected; detecting the target image based on a detector to obtain at least one target sub-object among the objects to be detected, wherein the detector is trained using an object detection dataset; extracting sub-object features of the target sub-objects; and searching a retrieval library based on the sub-object features to obtain the type of the target sub-object, wherein the retrieval library includes the types of objects from which different object features originate.

[0006] According to another aspect of the present invention, another object detection method is also provided. The method may include: acquiring a target image of an object to be detected; detecting the target image based on a detector to obtain at least one target sub-item among the objects to be detected, wherein the detector is trained using an object detection dataset; extracting sub-item features of the target sub-item; and searching a retrieval database based on the sub-item features to determine that the target sub-item is a prohibited item and to determine the type of the target sub-item, wherein the retrieval database includes types of prohibited items from which different item features originate.

[0007] According to another aspect of the present invention, another object detection method is also provided. The method may include: acquiring a target image of the content to be detected; detecting the target image based on a detector to obtain at least one target sub-content in the content to be detected, wherein the detector is trained using an object detection dataset; extracting sub-content features of the target sub-content; and searching a retrieval database based on the sub-content features to determine that the target sub-content is prohibited content and to determine the type of the target sub-content, wherein the retrieval database includes types of prohibited content from which different content features originate.

[0008] According to another aspect of the present invention, an object detection apparatus is also provided. The apparatus may include: a first acquisition unit for acquiring a target image of an object to be detected; a first detection unit for detecting the target image based on a detector to obtain at least one target sub-object among the objects to be detected, wherein the detector is trained using an object detection dataset; a first extraction unit for extracting sub-object features of the target sub-objects; and a first retrieval unit for retrieving the target sub-objects from a retrieval database based on the sub-object features to obtain the type of the target sub-object, wherein the retrieval database includes the types of objects from which different object features originate.

[0009] According to another aspect of the present invention, another object detection apparatus is also provided. The apparatus may include: a second acquisition unit for acquiring a target image of an object to be detected; a second detection unit for detecting the target image based on a detector to obtain at least one target sub-item among the objects to be detected, wherein the detector is trained using an object detection dataset; a second extraction unit for extracting sub-item features of the target sub-item; and a second retrieval unit for searching a retrieval database based on the sub-item features to determine that the target sub-item is a prohibited item and to determine the type of the target sub-item, wherein the retrieval database includes types of prohibited items from which different item features originate.

[0010] According to another aspect of the present invention, another object detection apparatus is also provided. The apparatus may include: a third acquisition unit for acquiring a target image of the content to be detected; a third detection unit for detecting the target image based on a detector to obtain at least one target sub-content in the content to be detected, wherein the detector is trained using a target detection dataset; a third extraction unit for extracting sub-content features of the target sub-content; and a third retrieval unit for searching a retrieval database based on the sub-content features to determine that the target sub-content is prohibited content and to determine the type of the target sub-content, wherein the retrieval database includes types of prohibited content from which different content features originate.

[0011] According to another aspect of the present invention, a computer-readable storage medium is also provided. This computer-readable storage medium may include a stored program, wherein, when the program is run by a processor, it controls the device where the computer-readable storage medium is located to execute the object detection method of the present invention.

[0012] According to another aspect of the present invention, a processor is also provided. The processor is used to run a program, wherein the program executes the object detection method of the present invention during runtime.

[0013] According to another aspect of the present invention, an object detection system is also provided. The object detection system may include: a processor; and a memory connected to the processor, configured to provide the processor with instructions to perform the following processing steps: acquiring a target image of an object to be detected; detecting the target image based on a detector to obtain at least one target sub-object among the objects to be detected, wherein the detector is trained using an object detection dataset; extracting sub-object features of the target sub-objects; and searching a retrieval library based on the sub-object features to obtain the type of the target sub-object, wherein the retrieval library includes the types of objects from which different object features originate.

[0014] In this embodiment of the invention, a target image of the object to be detected is acquired; the target image is detected based on a detector to obtain at least one target sub-object among the objects to be detected, wherein the detector is trained using a target detection dataset; sub-object features of the target sub-objects are extracted; and a search is performed in a retrieval library based on the sub-object features to obtain the type of the target sub-object, wherein the retrieval library includes the types of objects from which different object features originate. In related technologies, the types of objects to be detected are numerous and diverse. Purely using detection algorithms cannot encompass all the objects to be detected, and the aforementioned detection algorithms require pre-defining the types of all objects, which results in poor recognition of objects without pre-defined types, leading to low accuracy in object classification. Furthermore, when there is a need to identify new types of objects, the classification model needs to collect data again and retrain the classification model, resulting in slow response time. Re-collecting data also cannot guarantee the accuracy of object classification. However, in this application, since the detector trained using a target detection dataset has general detection capabilities, the aforementioned detector can be used to detect at least one target sub-object among the objects to be detected. If one target sub-object is missing, the features of the target sub-object are further searched in the retrieval database. Since the retrieval database includes the types of objects from which different object features come, the type of the target sub-object can be accurately determined in the retrieval database. That is, this application adopts a data retrieval combined with a sample set to determine the type of the target sub-object. This avoids the problem of using a detection algorithm alone, which cannot cover all the objects that need to be detected. It also avoids the need to collect data again and retrain the classification model, which makes it difficult to guarantee the accuracy of object detection. Thus, it solves the technical problem of low accuracy in object detection and achieves the technical effect of improving the accuracy of object detection. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0016] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an object detection method according to an embodiment of the present invention;

[0017] Figure 2 This is a flowchart of an object detection method according to an embodiment of the present invention;

[0018] Figure 3 This is a flowchart of another object detection method according to an embodiment of the present invention;

[0019] Figure 4 This is a flowchart of another object detection method according to an embodiment of the present invention;

[0020] Figure 5 This is a schematic diagram of the network structure of a detector according to an embodiment of the present invention;

[0021] Figure 6 This is a schematic diagram of a feature extraction network for fine-grained feature extraction according to an embodiment of the present invention;

[0022] Figure 7 This is a schematic diagram of an object detection scenario according to an embodiment of the present invention.

[0023] Figure 8 This is a schematic diagram of an object detection device according to an embodiment of the present invention;

[0024] Figure 9 This is a schematic diagram of another object detection device according to an embodiment of the present invention;

[0025] Figure 10 This is a schematic diagram of another object detection device according to an embodiment of the present invention; and

[0026] Figure 11 This is a structural block diagram of a mobile terminal according to an embodiment of the present invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0030] A convolutional neural network (CNN) is a type of feedforward neural network whose artificial neurons can respond to a portion of the surrounding units within their coverage area.

[0031] Ground truth boxes are detection boxes that are manually labeled with information (such as location) of existing objects from pre-collected samples.

[0032] Non-maximum suppression (NMS) suppresses elements that are not maxima, eliminating redundant detection boxes.

[0033] Example 1

[0034] According to an embodiment of the present invention, an embodiment of an object detection method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an object detection method according to an embodiment of the present invention. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0036] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0037] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the object detection method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the object detection method described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0039] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0040] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1This is only one instance of a specific particular instance and is intended to illustrate the types of components that may exist in the aforementioned computer device (or mobile device).

[0041] exist Figure 1 Under the operating environment shown, this application provides the following: Figure 2 The object detection method shown is illustrated. It should be noted that the data processing method in this embodiment is based on... Figure 1 The mobile terminal of the illustrated embodiment is executed.

[0042] Figure 2 This is a flowchart of an object detection method according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0043] Step S202: Obtain the target image of the object to be detected.

[0044] In the technical solution provided by step S202 of the present invention, the object to be detected can be an object that needs to be detected to include a sub-object that meets the target condition. For example, the sub-object of the target condition is an unqualified sub-object, or it can be a prohibited object, etc.

[0045] In this embodiment, the target image of the object to be detected can be obtained by taking a picture of the object to be detected using an image acquisition device. The obtained image can be a picture, an animation, or an image or image frame from a video. No specific limitations are made here.

[0046] Step S204: Detect the target image based on the detector to obtain at least one target sub-object among the objects to be detected.

[0047] In the technical solution provided in step S204 of the present invention, after obtaining the target image of the object to be detected, the target image can be input into the detector, and the detector can detect the target image to obtain at least one target sub-object in the object to be detected. The detector is trained using a target detection dataset.

[0048] In this embodiment, the detector can be a general detector, which can be a model with a certain network structure. It is trained on a target detection dataset with a data volume greater than the target threshold. The largest open-source image dataset (OpenImagesV4) can be used as the training data for the detector. The detector trained in this way has general target detection capabilities and can capture almost all objects that need to be detected.

[0049] This embodiment detects the target image based on a detector and outputs at least one target sub-object among the objects to be detected. This at least one target sub-object can be any object in the target image that meets the target criteria, including potentially compliant objects, such as unqualified objects or prohibited objects. Optionally, each target sub-object in this embodiment can correspond to a detection box.

[0050] Optionally, the target size of at least one target sub-object in this embodiment may occupy a very small proportion of the target image size, the at least one target sub-object may be very similar to the background of the target image, and the type of the at least one target sub-object may be diverse, without specific requirements here.

[0051] Step S206: Extract the sub-object features of the target sub-object.

[0052] In the technical solution provided by step S206 of the present invention, after detecting the target image based on the detector and obtaining at least one target sub-object among the objects to be detected, the sub-object features of the target sub-object are extracted.

[0053] In this embodiment, the target sub-object includes sub-object features, which are used to determine the type of the target sub-object. Optionally, this embodiment can use a pre-trained target feature extraction network to extract the sub-object features of the target sub-object and then output the sub-object features.

[0054] It should be noted that, in order to accurately determine the type of the target sub-object, the sub-object feature can be a fine-grained feature of the target sub-object, which can be used to characterize the detailed features of the target sub-object so as to accurately retrieve the type of the target sub-object in the retrieval database. The target feature extraction network mentioned above is a model trained for the need to extract fine-grained features, and can therefore be used to extract fine-grained features of the target sub-object.

[0055] Step S208: Search the retrieval database based on the sub-object features to obtain the type of the target sub-object.

[0056] In the technical solution provided by step S208 of the present invention, after extracting the sub-object features of the target sub-object, a search is performed in the search library based on the sub-object features to obtain the type of the target sub-object. The search library includes the types of objects from which different object features originate.

[0057] In this embodiment, a retrieval database can be pre-established. This database is a database of objects that meet the target criteria, such as a database of prohibited objects. It includes the types of objects from which different object features originate. This database can be established by mining high-uncertainty samples and high-confidence samples using information entropy. In this embodiment, the retrieval is performed in the database based on the sub-object features of the target sub-object. Optionally, if sub-object features exist in the database, the type corresponding to the sub-object features in the database can be determined as the type of the target sub-object.

[0058] For example, given an image to be detected, the system first uses a general detector to obtain all prohibited items in the image. Then, fine-grained feature extraction is performed on each prohibited item. Finally, the features of the prohibited items are retrieved from the search database to obtain the specific type of prohibited item.

[0059] In related technologies, the types of objects that need to be detected are numerous and diverse. Simply using detection algorithms cannot encompass all potential objects, and their classification accuracy is not high, resulting in low accuracy in object detection.

[0060] Through steps S202 to S208 of this application, a target image of the object to be detected is obtained; the target image is detected based on the detector to obtain at least one target sub-object in the object to be detected, wherein the detector is trained using a target detection dataset; sub-object features of the target sub-object are extracted; and a search is performed in the search library based on the sub-object features to obtain the type of the target sub-object, wherein the search library includes the types of objects from which different object features originate. In other words, since the detector trained on the object detection dataset has general detection capabilities, this embodiment can first use the detector to detect at least one target sub-object of the object to be detected, and then further search the features of the target sub-object in the retrieval library. Since the retrieval library includes the types of objects from which different object features come, the type of the target sub-object can be accurately determined in the retrieval library. That is, this application uses a combination of data retrieval and sample set to determine the type of the target sub-object, avoiding the inability to encompass all objects to be detected by simply using a detection algorithm, and also avoiding the need to collect data again and retrain the classification model, which makes it difficult to guarantee the accuracy of object detection. This solves the technical problem of low accuracy in object detection and achieves the technical effect of improving the accuracy of object detection.

[0061] The method described in this embodiment will be further explained below.

[0062] As an optional implementation, after obtaining the target image of the object to be detected in step S202, the method further includes: performing data augmentation on the target image; and detecting the target image based on the detector to obtain at least one target sub-object in the object to be detected, including: detecting the data-augmented target image based on the detector to obtain at least one target sub-object.

[0063] In this embodiment, after acquiring the target image of the object to be detected, data augmentation can be performed on the target image. This data augmentation is a method of enhancing data for the target detection task. In this embodiment, data augmentation of the target image can increase the number of small target sub-objects and a few target sub-objects in the target image, thereby alleviating the data imbalance problem. It can also enrich the scale information of the target sub-objects and allow the detector to learn more data information with limited data, thereby generating stronger robustness.

[0064] After data augmentation of the target image, the augmented target image can be detected based on the detector to obtain at least one target sub-object.

[0065] As an optional implementation, data augmentation of the target image includes: data augmentation of the target image; and / or data transformation of the target image.

[0066] In this embodiment, when performing data augmentation on the target image, two methods can be used. One method is to perform data augmentation (instance-balanced) on the target image. This method can achieve the purpose of increasing the number of small target sub-objects and a few target sub-objects in the target image, alleviating the data imbalance problem, and enriching the scale information of the target sub-objects. The other method is to perform data transformation on the target image. This method can achieve the purpose of allowing the detector to learn more data information with limited data, thereby generating stronger robustness.

[0067] The method for augmenting the target image data in this embodiment will be further described below.

[0068] As an optional implementation, data augmentation of the target image includes: scaling the target image according to a target scaling ratio; determining multiple sliding window regions in the scaled target image based on the target sliding window; and determining at least one target sliding window region including the target sub-object among the multiple sliding window regions.

[0069] In this embodiment, a target scaling ratio can be determined first. This target scaling ratio is a specified scaling scale used to scale the target image. The target image is then scaled according to the target scaling ratio. A fixed-size target sliding window is then used to slide across the scaled target image, thereby generating a large number of sliding window regions. Finally, at least one target sliding window region containing small target sub-objects and a few target sub-objects is selected from these large number of sliding window regions. This at least one target sliding window region is a preferred sliding window set, which can be added to the training data used to train the detector.

[0070] The method for data transformation of the target image in this embodiment will be further described below.

[0071] As an optional implementation, data transformation of the target image data includes: transforming the image data of the target image based on multiple sets of random transformation parameters.

[0072] In this embodiment, multiple sets of random transformation parameters can be obtained. These multiple sets of random transformation parameters are used to perform image transformations on the target image. They can control the target image to perform image transformations such as shearing, rotating, equalizing, and cutting out. No specific limitations are imposed here.

[0073] As an optional implementation, detecting a target image based on a detector to obtain at least one target sub-object among the objects to be detected includes: extracting image features of the target image based on the feature extraction network layer in the detector; classifying and regressing the image features based on the first convolutional layer in the detector to obtain the target region of the target image, wherein the first convolutional layer is connected to the feature extraction network layer, the target region includes the target sub-object; and detecting the target sub-object in the target region.

[0074] In this embodiment, the detector may include a feature extraction network layer, which can be a deep neural network, a CNN neural network, a deep learning convolutional neural network (VGG16 network), an initial network model V1-v4 (InceptionV1-V4 network), a deep residual network (ResNet network), a model for grouped convolution (ResNeXt), etc., without specific limitations. This embodiment can input the data-enhanced target image into the aforementioned feature extraction network layer, and then use the feature extraction network layer to extract image features from the data-enhanced target image. These image features can be located in the feature layer of the detector. Optionally, the detector in this embodiment may include a first convolutional layer, for example, a 3*3 convolution, which can connect to several layers of image features in the aforementioned feature layer. The first convolutional layer performs classification and regression on the image features to obtain the initial target region of the target image. Here, the classification can only distinguish between target sub-objects and non-target sub-objects to obtain the initial target region. Optionally, this embodiment can also perform more refined regression and classification of specific target categories on the initial target region to obtain the final target region, and then detect target sub-objects in it.

[0075] As an optional implementation, detecting a target sub-object in a target region includes: performing data distillation and overlapping soft sampling on the target region using a loss function in the detector to obtain multiple target detection boxes; determining at least one first target detection box from the multiple target detection boxes, wherein the cross-union ratio between the first target detection box and a predetermined detection box is greater than a first threshold; and determining a target sub-object based on at least one first target detection box.

[0076] In this embodiment, the detector also has a coloss function, which can be used to perform data distillation and overlap soft sampling on the target region of the target image to obtain multiple target detection boxes. The data distillation of the target region can be performed on some existing detection types, such as existing knife types, human types, etc., to predict the target region and obtain multiple predicted detection boxes. The loss of the predicted detection boxes is weighted by the prediction confidence corresponding to each predicted detection box to obtain a first weighted result. In addition, when predicting the target region, since multiple predicted detection boxes are obtained, the overlap soft sampling method is used to determine the overlap between each predicted detection box and the predetermined detection box to obtain multiple overlaps. Then, the multiple overlaps are weighted to obtain a second weighted result. Multiple target detection boxes are determined from the multiple predicted detection boxes using the first weighted result and the second weighted result. For example, the detection boxes corresponding to the first weighted result and the second weighted result that meet a certain threshold range are determined as target detection boxes. Among them, the predetermined detection box can be a target annotation box (Ground Truth), which is a detection box that is manually annotated by the author from pre-collected samples for information (such as location) of existing objects, while the multiple target detection boxes are detection boxes determined from the above multiple predicted detection boxes based on the first weighted result and the second weighted result.

[0077] This embodiment uses a loss function in the detector to perform data distillation and overlapping soft sampling on the target region to obtain multiple target detection boxes, minimizing the impact of missed ground truth boxes (missed samples) on the model. This embodiment then applies Soft Non-maximum Suppression (SoftNMS) to the obtained multiple target detection boxes. Optionally, at least one first target detection box is determined if the intersection-union ratio (IUU) between the multiple target detection boxes and a predetermined detection box is greater than a first threshold, where the first threshold can be 0.5. After determining at least one first target detection box from the multiple target detection boxes, target sub-objects can be determined based on at least one first target detection box. This method is further described below.

[0078] As an optional implementation, determining a target sub-object based on at least one first target detection box includes: obtaining the second target detection box with the highest corresponding value in at least one first target detection box; attenuating the values ​​corresponding to third target detection boxes in multiple target detection boxes whose overlap with the second target detection box is greater than a second threshold; removing third target detection boxes in multiple target detection boxes whose attenuated values ​​are lower than a third threshold to obtain at least one fourth target detection box; and determining the target sub-object in the fourth target detection box.

[0079] In this embodiment, at least one first target detection box has a corresponding numerical value, which can be the score of the first target detection box. This score can be used to characterize the confidence that the first target detection box includes a target sub-object of a specific type. The at least one first target detection box can be sorted in descending order according to the score of each detection box. Then, the second target detection box with the highest score is selected. Then, a third target detection box with an overlap greater than a second threshold with the second target detection box is identified among multiple target detection boxes. The value corresponding to the third target detection box is attenuated, that is, the score corresponding to the third target detection box is attenuated. A third threshold is set, and third target detection boxes with attenuated values ​​lower than the third threshold can be removed. The above method is continued until all fourth target detection boxes that meet the conditions are finally output, and the target sub-object is then identified in the fourth target detection box.

[0080] It should be noted that the Soft NMS processing of all target detection boxes in this embodiment is only an example of the present invention, and it can also be non-maximum suppression (NMS), which is not specifically limited here.

[0081] As an optional implementation, multiple target detection boxes are obtained by performing data distillation and overlapping soft sampling on the target region using a loss function in the detector. This includes: predicting the data in the target region using a target prediction model to obtain multiple predicted detection boxes and their confidence scores, wherein the target prediction model is trained on a public dataset that includes data on objects of a known type; performing overlapping soft sampling on the predicted detection boxes to obtain their loss scores; weighting the loss scores based on the confidence scores to obtain the target values ​​of the predicted detection boxes; and determining multiple target detection boxes from the multiple predicted detection boxes based on the target values.

[0082] In this embodiment, when performing data distillation on the target region, a target prediction model can be used to predict the data in the target region, obtaining multiple predicted detection boxes and their confidence scores. The target prediction model is a complex model trained using a complex algorithm on a public dataset, used to predict on an existing dataset. The public dataset can include data on objects of known types, such as existing detection types like "knife" or "person." This embodiment can use the predicted target detection boxes as pseudo-target ground truths. These pseudo-target ground truths differ from actual target ground truths; they are not manually labeled from pre-collected samples based on existing object information (e.g., location). They are essentially pseudo-labels that can be directly determined by the algorithm, thus eliminating manual labeling and reducing labeling costs. The prediction confidence scores are used to weight the loss generated by the pseudo-target ground truths.

[0083] This embodiment performs overlapping soft sampling on the target region to obtain the loss of the predicted detection boxes. During detection, many predicted detection boxes are generated, and each predicted detection box incurs a certain loss. This loss can be weighted according to its intersection-union ratio (IU) with the predetermined detection boxes. Optionally, this embodiment weights the loss based on the confidence level of each predicted detection box to obtain the target value of the predicted detection box. Then, multiple target detection boxes are determined from the multiple predicted detection boxes based on the target value.

[0084] The method for extracting object features of the target sub-object in this embodiment will be further described below.

[0085] As an optional implementation, step S206, extracting sub-object features of the target sub-object, includes: extracting sub-object features of the target sub-object based on a target feature extraction network model.

[0086] In this embodiment, when extracting sub-object features from a target sub-object, the sub-object features can be input into a target feature extraction network model, which is generated through a specific network structure. This embodiment processes the target sub-object based on the target feature extraction network model to obtain sub-object features, and then outputs these features. The method for extracting sub-object features from a target sub-object based on a target feature extraction network model in this embodiment will be further explained below.

[0087] As an optional implementation, extracting sub-object features of a target sub-object based on a target feature extraction network model includes: extracting first sub-object features of the target sub-object based on a sub-feature extraction network model within the target feature extraction network model, wherein the sub-feature extraction network is followed by a second convolutional layer after the feature layer, and the dimension of the sub-object features is the dimension of the second convolutional layer; performing auxiliary recognition on the feature map of the target sub-object based on a local auxiliary model within the target feature extraction network model to obtain second sub-object features, wherein the local auxiliary model includes at least a third convolutional layer, a first pooling layer, a second pooling layer, and a classification layer, wherein the first pooling layer is used to perform global max pooling on the heatmap obtained after convolution processing of the feature map by the third convolutional layer, and the second pooling layer is used to perform average pooling on the feature map of the target size; and determining sub-object features based on the first and second sub-object features.

[0088] In this embodiment, the target feature extraction network model includes a sub-feature extraction network model, which can be a deep residual network 50 model (ResNet50 network model). ResNet50 uses residual learning in convolutional neural networks, which can solve the problems of gradient vanishing and accuracy degradation in the target feature extraction network model, thereby improving the accuracy of the target feature extraction network model and controlling its training speed. A second convolutional layer of 2048->512 can be added after its 2048-dimensional feature layer to reduce the feature dimension. In this embodiment, the dimension of the sub-object feature is the dimension of the second convolutional layer. For example, the above-mentioned 512-dimensional feature is used as the final first sub-object feature of the target object.

[0089] The target feature extraction network model in this embodiment may include a Local Assistant Module, which can be used to capture local information of the target sub-object in order to extract more discriminative features of the second sub-object.

[0090] The local auxiliary model in this embodiment includes at least a third convolutional layer, a first pooling layer, a second pooling layer, and a classification layer. The third convolutional layer can be a 1*1 convolution, with each 1*1 convolution kernel having a size of C*1*1, where C can be the number of channels in the Conv4_3 feature map of the target feature extraction network model. A 1*1 convolution can be performed on the Conv4_3 feature map. In this embodiment, the third convolutional layer acts as a discriminator for a specific region of the feature map of the target sub-object. After the feature map of the target sub-object undergoes the aforementioned 1*1 convolution processing, a heatmap is generated. The regions with high response values ​​on this heatmap represent a specific type of patch to be found. Then, global max pooling is performed on the heatmap in the first pooling layer to find the aforementioned type of patch. Optionally, the local auxiliary model in this embodiment has a convolutional layer Conv4_3 with a feature map size of C*H*W, where H represents the height of the feature map and W represents the width of the feature map. The number of types to be identified is N, and k local discriminators can be learned for each type. Therefore, the number of 1x1 filters is kN, and the feature map of the convolutional layer Conv6 of the local auxiliary model can be kN*H*W.

[0091] In this embodiment, the heatmap of the global max pooling operation can be processed by GMP in the first pooling layer. The feature map obtained after FMP processing can be KN*1*1 in size. Then, the feature vector of KN is connected to an fc layer (FC-layer) to output a classification layer of N classes. It can also be connected to a normalized exponent (softmax) layer.

[0092] In this embodiment, since it cannot be guaranteed that the P-stream learns a class-specific discriminative detector, a side branch can be added after the first pooling layer as supervision information for the 1x1 discriminative filter. The main module is the second pooling layer, which can be a cross-channel pooling layer. It can perform N sets of average pooling operations on the kN*1*1 feature map. That is, it performs average pooling operations on the feature values ​​of k channels of each class, and finally obtains an N-dimensional feature vector. Then, it directly classifies it into N classes. This can guide the 1x1 discriminative filter to learn the class-specific features of each class to obtain the features of the second sub-object.

[0093] In this embodiment, the sub-feature extraction network model in the target feature extraction network model is used to obtain the main first sub-object features of the target sub-object, and the local auxiliary model is used to obtain the more discriminative second sub-object features of the target sub-object. The sub-object features of the target sub-object are generated by combining the first sub-object features and the second sub-object features.

[0094] As an alternative implementation, the loss weight of the local auxiliary model is less than the loss weight of the sub-feature extraction network model.

[0095] In this embodiment, the loss of the path containing the sub-feature extraction network model, the loss of the path containing the first pooling layer, and the loss of the path containing the second pooling layer can be weighted and summed. The loss weight of the local auxiliary module is relatively small and is only used for auxiliary recognition. Optionally, in this embodiment, when performing forward prediction, only the output of the sub-feature extraction network model from the previous path can be used, without adding any extra cost during prediction.

[0096] As an optional implementation, the method further includes: clustering the first sub-object features and the second sub-object features to obtain cluster centers; obtaining the distance between the first sub-object features and the cluster centers and the sum of the distances between the second sub-object features and the cluster centers; and determining the sum as the center distance loss, wherein the center distance loss is used to aggregate similar features in the sub-object features.

[0097] In this embodiment, the center distance loss can also be used to constrain the sub-object features of the extracted target sub-object. In this embodiment, each type of feature in the first sub-object feature and the second sub-object feature can be clustered to obtain cluster centers. Then, the sum of the distance between the first sub-object feature and the cluster center and the distance between the second sub-object feature and the cluster center is calculated. This sum is the center distance loss, which is used to make similar features in the sub-object features more aggregated.

[0098] In this embodiment, when training the target feature extraction network, the center distance loss can be used to constrain the extracted features. When training with batch samples, each feature class of all samples in the batch can be clustered, and then the distance between each sample and the cluster center is calculated. The sum of all distances is the center distance loss. Optionally, in this embodiment, the weighted center distance loss is summed with the softmax loss of the target feature extraction network.

[0099] The method for establishing the search database in this embodiment will be further described below.

[0100] As an optional implementation, before performing a search in the retrieval library based on sub-object features in step S208, the method further includes: obtaining target object samples; establishing a sample library based on the target object samples; extracting the object features to be identified from the sample library based on the target feature extraction network; quantizing the object features to obtain a feature codebook; and establishing a retrieval library based on the feature codebook.

[0101] In this embodiment, a target object sample can be obtained first. This target object sample can be relevant data of objects that meet the target conditions mined from massive unlabeled data using active learning technology. A sample library can be established based on the target object sample. Then, the target feature extraction network mentioned above is used to extract the object features to be identified from the sample library. The object features are quantized, for example, PQ quantization is performed on the object to obtain a feature codebook. This feature codebook can be a contraband feature codebook. Then, a retrieval library is established based on the feature codebook.

[0102] As an optional implementation, before establishing a sample library based on the target object samples, the method further includes: training the target feature extraction network based on a periodic learning rate to obtain multiple network models; predicting multiple sub-samples in the target object samples multiple times based on the dropout layers of the multiple network models to obtain multiple prediction results; and averaging the multiple prediction results to obtain the target prediction result.

[0103] In this embodiment, before building a sample library based on the target object samples, a snapshot ensemble method can be used for training. Specifically, a periodic learning rate can be used to generate multiple network models. Optionally, for unlabeled samples, this embodiment uses the dropout layers of multiple network models to predict multiple sub-samples within the target object samples multiple times, resulting in multiple prediction results. These prediction results can be softmax results. The multiple prediction results are then averaged to obtain the target prediction result, which can be the final softmax result.

[0104] The method for establishing a sample library based on target object samples in this embodiment will be described below.

[0105] As an optional implementation, a sample library is established based on target object samples, including: determining the information entropy of multiple sub-samples based on target prediction results; obtaining a first target sub-sample of the target quantity from the multiple sub-samples, wherein the information entropy of the first target sub-sample is greater than the information entropy of the sub-samples other than the first target sub-sample of the target quantity; labeling the types of objects in the first target sub-samples; adding the labeled first target sub-samples to the sample library; labeling the types of objects in second target sub-samples whose information entropy is less than a fourth threshold with pseudo-labels; correcting the pseudo-labeled second target sub-samples by responding to correction operation instructions; and adding the corrected second target sub-samples to the sample library.

[0106] In this embodiment, the information entropy of multiple sub-samples can be determined based on the target prediction results. Then, the information entropy of the multiple sub-samples can be sorted in descending order. The first target sub-sample with the largest number of targets can be selected from the multiple sub-samples. The type of the object in the first target sub-sample is labeled, and the labeled first target sub-sample is added to the sample library.

[0107] Optionally, in this embodiment, the type of the object of the second target sub-sample whose information entropy is less than the fourth threshold is pseudo-labeled, and the pseudo-labeled second target sub-sample is corrected by responding to the correction operation command. That is, the pseudo-labeled second target sub-sample is manually corrected, and then the corrected second target sub-sample is added to the sample library.

[0108] This embodiment uses a search library combined with a sample library, which can quickly cover new types of objects. When a new type of object to be detected appears, it is only necessary to add the sample corresponding to the new type to the sample library to quickly complete the coverage.

[0109] This invention also provides another object detection method applicable to item detection scenarios.

[0110] Figure 3 This is a flowchart of another object detection method according to an embodiment of the present invention. Figure 3 As shown, the method may include the following steps:

[0111] Step S302: Obtain the target image of the object to be detected.

[0112] In the technical solution provided by step S302 of the present invention, the item to be tested can be an item that needs to be tested to see if it includes sub-items that meet the target conditions. For example, the sub-items that meet the target conditions are unqualified sub-items, which can be prohibited items, such as prohibited adult products, political flags, controlled implements, etc. No specific restrictions are made here.

[0113] This embodiment obtains the target image of the item to be detected by taking a picture of the item with an image acquisition device.

[0114] Step S304: Detect the target image based on the detector to obtain at least one target sub-item among the items to be detected.

[0115] In the technical solution provided in step S304 of the present invention, after obtaining the target image of the item to be detected, the target image can be input into the detector, and the detector can detect the target image to obtain at least one target sub-item in the item to be detected. The detector is trained using a target detection dataset.

[0116] In this embodiment, the detector can be a general detector, which can be a model with a certain network structure. It is trained on a target detection dataset with a data volume greater than the target threshold. The largest dataset, OpenImagesV4, can be used as the training data for the detector. The detector trained in this way has general target detection capabilities and can capture almost all the items that need to be detected.

[0117] This embodiment detects a target image using a detector and outputs at least one target sub-item from the object. This at least one target sub-item can be any item in the target image that meets the target criteria, including potentially matching items, such as unqualified items or prohibited items. Optionally, each target sub-item in this embodiment can correspond to a detection box.

[0118] Optionally, the target size of at least one target sub-item in this embodiment may occupy a very small proportion of the target image size, the at least one target sub-item may be very similar to the background of the target image, and the type of the at least one target sub-object may be diverse, without specific requirements here.

[0119] Step S306: Extract the sub-item features of the target sub-item.

[0120] In the technical solution provided by step S306 of the present invention, after detecting the target image based on the detector and obtaining at least one target sub-item among the items to be detected, the sub-item features of the target sub-item are extracted.

[0121] In this embodiment, the target sub-item includes sub-item features, which are used to determine the type of the target sub-item. Optionally, this embodiment can use a pre-trained target feature extraction network to extract the sub-item features of the target sub-item and then output the sub-item features, which can be fine-grained features.

[0122] Step S308: Search the search database based on the characteristics of the sub-item to determine that the target sub-item is a prohibited item and to determine the type of the target sub-item.

[0123] In the technical solution provided by step S308 of the present invention, after extracting the sub-item features of the target sub-item, a search is performed in the search library based on the sub-item features to obtain the type of the target sub-item, wherein the search library includes the types of items from which different item features originate.

[0124] The method described in this embodiment will be further described below.

[0125] As an optional implementation, after acquiring the target image of the item to be detected in step S302, the method further includes: performing data augmentation on the target image; detecting the target image based on a detector to obtain at least one target sub-item among the items to be detected, including: detecting the data-enhanced target image based on a detector to obtain at least one target sub-item. Wherein, performing data augmentation on the target image includes: data expansion of the target image; and / or data transformation of the target image.

[0126] In this embodiment, after acquiring the target image of the object to be detected, data augmentation can be performed on the target image. This data augmentation is a method of enhancing data for the target detection task. In this embodiment, data augmentation of the target image can increase the number of small target sub-objects and a few target sub-objects in the target image, thereby alleviating the data imbalance problem. It can also enrich the scale information of the target sub-objects and allow the detector to learn more data information with limited data, thereby generating stronger robustness.

[0127] After data augmentation of the target image, the augmented target image can be detected based on the detector to obtain at least one target sub-item.

[0128] As an optional implementation, step S306, extracting sub-item features of the target sub-item, includes: extracting sub-item features of the target sub-item based on a target feature extraction network model.

[0129] In this embodiment, when extracting the sub-item features of the target sub-item, the sub-item features can be input into the target feature extraction network model. This embodiment processes the target sub-item based on the target feature extraction network model to obtain the sub-item features, and then outputs the sub-item features.

[0130] As an optional implementation, before step S308, which involves searching the retrieval database based on the sub-item features to determine that the target sub-item is a prohibited item, the method further includes: obtaining a target item sample; establishing a sample database based on the target item sample; extracting the item features to be identified from the sample database based on a target feature extraction network; quantifying the item features to obtain a feature codebook; and establishing a retrieval database based on the feature codebook.

[0131] In this embodiment, a target item sample can be obtained first. This target item sample can be relevant data of items that meet the target conditions, which can be mined from massive unlabeled data using active learning technology. A sample library can be established based on the target item sample. Then, the target feature extraction network mentioned above is used to extract the item features to be identified from the sample library. The item features are quantified, for example, PQ quantization is performed on the items to obtain a feature codebook. This feature codebook can be a contraband feature codebook. Then, a retrieval library is established based on the feature codebook.

[0132] This invention also provides another object detection method in content moderation scenarios.

[0133] Figure 4 This is a flowchart of another object detection method according to an embodiment of the present invention. Figure 4 As shown, the method may include the following steps:

[0134] Step S402: Obtain the target image of the content to be detected.

[0135] In the technical solution provided by step S402 of the present invention, the content to be detected can be the content that needs to be detected to include sub-content that meets the target conditions. For example, the sub-content of the target conditions is unqualified sub-content, which can be prohibited content, such as prohibited adult products, political flags, controlled implements, etc. No specific restrictions are made here.

[0136] In this embodiment, the target image of the content to be detected can be obtained by taking a picture of the content to be detected using an image acquisition device.

[0137] Step S404: Detect the target image based on the detector to obtain at least one target sub-content in the content to be detected.

[0138] In the technical solution provided by step S404 of the present invention, after obtaining the target image of the content to be detected, the target image can be input into the detector, and the detector can detect the target content to obtain at least one target sub-content in the content to be detected. The detector is trained by the target detection dataset, and the detector trained in this way has a general target detection capability and can capture almost all items that need to be detected.

[0139] This embodiment detects the target image based on a detector and outputs at least one target sub-content in the object. This at least one target sub-content can be all content in the target image that meets the target conditions, including potentially compliant content, such as unqualified content or prohibited content. Optionally, the target sub-content in this embodiment can correspond to a detection box.

[0140] Optionally, the target size of at least one target sub-content in this embodiment may occupy a very small proportion of the target image size, the at least one target sub-content may be very similar to the background of the target image, and the type of the at least one target sub-content may be diverse, without specific requirements here.

[0141] Step S406: Extract the sub-content features of the target sub-content.

[0142] In the technical solution provided by step S406 of the present invention, after detecting the target image based on the detector and obtaining at least one target sub-content in the item to be detected, the sub-content features of the target sub-content are extracted.

[0143] In this embodiment, the target sub-content includes sub-content features, which are used to determine the type of the target sub-content. Optionally, this embodiment can use a pre-trained target feature extraction network to extract the sub-content features of the target sub-content and output the sub-content features, which can be fine-grained features.

[0144] Step S408: Based on the sub-content features, a search is performed in the search database to determine that the target sub-content is prohibited content and to determine the type of the target sub-content.

[0145] In the technical solution provided by step S408 of the present invention, after extracting the sub-content features of the target sub-content, a search is performed in the search library based on the sub-content features to obtain the type of the target sub-content. The search library includes the types of content from which different content features originate.

[0146] The method described in this embodiment will be further described below.

[0147] As an optional implementation, in S402, after acquiring the target image of the content to be detected, the method further includes: performing data augmentation on the target image; detecting the target image based on a detector to obtain at least one target sub-content in the content to be detected, including: detecting the data-augmented target image based on a detector to obtain at least one target sub-content. Wherein, performing data augmentation on the target image includes: data expansion of the target image; and / or data transformation of the target image.

[0148] In this embodiment, after acquiring the target image of the content to be detected, data augmentation can be performed on the target image to increase the number of small target sub-contents and a few target sub-contents in the target image, thereby alleviating the data imbalance problem, enriching the scale information of the target sub-objects, and allowing the detector to learn more data information with limited data, thereby generating stronger robustness.

[0149] After data augmentation of the target image, the augmented target image can be detected based on the detector to obtain at least one target sub-content.

[0150] As an optional implementation, S406, extracting sub-content features of the target sub-content includes: extracting sub-content features of the target sub-content based on a target feature extraction network model.

[0151] In this embodiment, when extracting sub-content features from the target sub-content, the sub-content features can be input into a target feature extraction network model. This embodiment processes the target sub-content based on the target feature extraction network model to obtain sub-content features, and then outputs these sub-content features.

[0152] As an optional implementation, in S408, before searching the retrieval database based on sub-content features to determine that the target sub-content is prohibited content, the method further includes: obtaining a target content sample; establishing a sample database based on the target content sample; extracting the content features to be identified from the sample database based on a target feature extraction network; quantifying the content features to obtain a feature codebook; and establishing a retrieval database based on the feature codebook.

[0153] In this embodiment, a target content sample can be obtained first. This target content sample can be relevant data of content that meets the target conditions, which can be mined from massive unlabeled data using active learning technology. A sample library can be established based on the target content sample. Then, the target feature extraction network mentioned above is used to extract the content features to be identified from the sample library and quantize the content features, for example, by performing PQ quantization on the content, thereby obtaining a feature codebook. This feature codebook can be a codebook of prohibited content features, and then a retrieval library can be established based on the feature codebook.

[0154] In related technologies, algorithms for classifying objects require pre-defining all categories, and their performance is poor for recognizing small objects. For example, some objects occupy a small proportion of the entire image, some objects are similar to the background, some objects have indistinct features, and there are many types and styles of objects, making it impossible to exhaustively enumerate them in advance. As a result, there is a problem with insufficient accuracy in object detection. In this embodiment, a detector is first used to detect the bounding boxes of potential objects, and then a target feature extraction network is used to extract object features. The above-mentioned object detection method can achieve good detection results for small objects, thereby improving the accuracy of object detection.

[0155] Furthermore, in related technologies, there are numerous types and styles of objects. The features used in detection algorithms for classification are not highly accurate and cannot quickly cover new object types, resulting in slow detection speeds. This embodiment, however, utilizes a fine-grained feature extraction network with local auxiliary modules to extract object features from detected targets, thus achieving strong recognition capabilities. Simultaneously, this embodiment employs center distance loss to constrain the features of target objects, making features of similar samples more aggregated and improving accuracy. This embodiment uses the largest object detection dataset currently available as training data for the detector, resulting in a model with general object detection capabilities, capable of capturing almost all illegal target objects. When a new type of object to be detected appears, simply adding the corresponding sample to the sample library quickly completes coverage, solving the technical problem of low object detection accuracy and achieving the technical effect of improving object detection accuracy.

[0156] Example 2

[0157] The object detection method of the present invention will be further described below with reference to preferred embodiments, specifically taking prohibited items as the objects to be detected.

[0158] Identifying whether a contraband target is present in an image or video still faces the following challenges:

[0159] (1) The size of some prohibited items occupies a very small proportion of the entire map;

[0160] (2) Some of the contraband items have very similar targets and backgrounds;

[0161] (3) Some prohibited items do not have obvious target characteristics;

[0162] (4) There are many types and styles of prohibited items, making it impossible to exhaustively list them all.

[0163] In related technologies, an input image can be used to determine the score of the prohibited items contained within it, and the type of prohibited item can be determined based on this score. However, this classification algorithm requires pre-defining all categories and performs poorly in recognizing small targets.

[0164] Another related technique involves an algorithm that takes an image as input and detects the specific location and confidence level of any contraband within it to determine the type of contraband. However, since contraband comes in a wide variety of types and styles, using features from the detection algorithm for classification results in relatively low accuracy.

[0165] To address the aforementioned issues, this embodiment employs a general contraband detection algorithm that combines general object detection with fine-grained feature extraction and vector retrieval methods. Because contraband is diverse and varied, a simple detection algorithm cannot encompass all potential targets, and its classification accuracy is low. This embodiment first uses a detector to identify potential target boxes, and then uses a target feature extraction network to extract features.

[0166] Furthermore, if a classification model is used directly for classification, there may be a problem: when there is a need to identify new types of targets, the classification model needs to collect data and be retrained, resulting in a slow response time. This embodiment, however, uses vector retrieval combined with a sample database, which provides rapid coverage of new target types.

[0167] Figure 5 This is a schematic diagram of the network structure of a detector according to an embodiment of the present invention. Figure 5 As shown, this embodiment can use the largest open-source image dataset (OpenImagesV4) as the training data for the network structure of the detector. The detector trained in this way has general object detection capabilities and can capture almost all illegal targets.

[0168] S1, perform data augmentation on the input image.

[0169] This embodiment can use two methods for data augmentation, data expansion, and data transformation.

[0170] In this embodiment, when performing data augmentation on an image, the image can first be scaled up according to a specified scale; then, a fixed-size sliding window can be used to slide across the scaled image to generate a large number of sliding window regions; finally, a preferred set of sliding windows containing small targets and a few targets can be selected from the large number of sliding window regions and added to the training data.

[0171] The above methods can, on the one hand, increase the number of small and a few targets, thus alleviating the problem of data imbalance; on the other hand, they can also enrich the scale information of the targets.

[0172] In this embodiment, when performing data augmentation on images using data transformation, multiple sets of random parameters can be used to control image transformations such as shearing, rotating, equalizing, and cutting, thereby allowing the detector to learn more data information from limited data and produce stronger robustness.

[0173] S2, deep neural network for feature extraction.

[0174] This embodiment can input a data-enhanced image and use a CNN deep neural network to extract features from it, wherein the deep neural network can be a VGG16 network.

[0175] S3, Initial target area generation.

[0176] In this embodiment, several feature layers in the feature layer of S2 can be selected and subjected to 3*3 convolutions for classification and regression. In this embodiment, the classification only distinguishes between target and non-target categories to obtain the initial target region.

[0177] S4, fine-tuning of the target area.

[0178] In this embodiment, the initial target region detected in S3 is subjected to more refined regression and classification of specific target categories.

[0179] S5 utilizes the loss function to refine the target region onto the existing dataset, employing data distillation and overlap soft sampling to minimize the impact of missed ground truth boxes in the dataset on the detector.

[0180] In this embodiment, when performing data distillation, for some existing detection types in public datasets, such as knives and people, a complex model trained on the public dataset can be used to predict the existing dataset. Then, the predicted target detection boxes are used as Pseduo Ground Truth, and the prediction confidence is used to weight the resulting loss.

[0181] When implementing overlap soft sampling, many detection boxes are generated, and each detection box generates a loss, which can be weighted according to the degree of overlap between it and the target annotation box.

[0182] Optionally, the samples in this embodiment include positive samples that intersect with real objects, negative samples that are judged as being due to missing labels, and positive samples generated by complex models. Overlap soft sampling is performed on these samples to obtain weights w1, w2, and w3 respectively. Then, the resulting losses loss1, loss2, and loss3 are summed in a weighted manner, i.e., Loss = w1loss1 + w2loss2 + w3loss3.

[0183] S6. All generated detection boxes are processed using Soft NMS.

[0184] In this embodiment, for all generated detection boxes, detection boxes with an IoU greater than 0.5 with the Ground Truth can be extracted, sorted according to their corresponding scores, and the detection box with the highest score can be selected. Then, the scores of target detection boxes whose overlap with the highest-scoring detection box is greater than a threshold are attenuated. Then, a threshold is set, and target detection boxes with scores lower than the threshold after attenuation are removed. This process is continued until all target detection boxes that meet the conditions are finally output, and the prohibited items are detected in the target detection boxes.

[0185] Optionally, this embodiment can actively learn from the unlabeled data pool to obtain valuable samples, which are then manually labeled. A CNN is used to extract features from these labeled valuable samples to obtain a violation sample library. When an input image is fed into another CNN, the other CNN can extract features of object 1, object 2, object 3, and object 4. This CNN is then trained and learned according to the features in the violation sample library to obtain object information 11 for object 1, object information 21 for object 2, and object information 31 for object 3. This object information may include the object's score and label, etc., without specific limitations.

[0186] Figure 6 This is a schematic diagram of a feature extraction network for fine-grained feature extraction according to an embodiment of the present invention. Figure 6 As shown, the feature extraction network in this embodiment involves the following steps:

[0187] S1, the target features of the contraband are extracted using the first path network structure of the feature extraction network. In this embodiment, the image of the target area where each detected contraband is located can be used as the input of the first path network structure consisting of Conv1_x-Conv4_x, Conv5_x, FC-layer, two FC-layers, and Loss-layer, from which the depth features of the contraband can be extracted.

[0188] This embodiment can use the ResNet50 network, which can be followed by a 2048->512 convolutional layer after its 2048-dimensional feature layer to reduce the feature dimension and use the 512-dimensional feature as the final feature of the contraband.

[0189] S2, the second network structure of the feature extraction network in this embodiment is a local auxiliary model.

[0190] In this embodiment, a 1x1 convolution is performed on the Conv4_3 feature map of the feature extraction network. The size of each 1x1 convolution kernel can be C*1*1, where C is the number of channels in Conv4_3. This embodiment uses it as a discriminator for a certain region of the feature map. After the 1x1 convolution processing, a heatmap is generated. The regions with high response values ​​on this heatmap are the patches of a certain type that need to be found. Global Max Pooling can then be applied to this heatmap to find the patches of that type. Assuming that the size of the Conv4_3 feature map is C*H*W and the number of categories to be identified is N, this embodiment learns k local discriminators for each category. Therefore, the number of 1x1 filters is kN, and the feature map of Conv6 is kN*H*W. After GMP processing in Pool6, the feature map size is KN*1*1. Then, the KN feature vectors are connected to an FC-layer to output a classification layer of N classes, followed by a softmax layer.

[0191] Since it cannot be guaranteed that the P-stream learns a class-specific discriminative detector, this embodiment adds a second network structure (bypass) after the pool6 layer as supervision information for the 1x1 discriminative filter. The main module is a cross-channel pooling layer, which is used to perform N sets of average pooling on the kN*1*1 feature map, that is, average pooling the feature values ​​of k channels of each class, and finally obtaining an N-dimensional vector. Then, it directly performs N-class classification, which can guide the 1x1 discriminative filter to learn the class-specific features of each class.

[0192] In this embodiment, during training, the losses of the three network structures in the diagram are weighted and summed to obtain the final loss (Loss). The loss weight of the local auxiliary modules is relatively small, serving only as an auxiliary recognition function. When the network performs forward prediction, only the output of the top-level classification network is used, without adding any extra cost during prediction.

[0193] S3, determine the center distance loss.

[0194] This embodiment uses center distance loss to constrain the extracted features. The specific calculation steps for center distance loss are as follows: cluster each feature class of the samples, then calculate the distance between each sample and the cluster center. The sum of all distances is the center distance loss. The weighted center distance loss is then summed with the softmax loss of the classification network.

[0195] In this embodiment, it is necessary to establish a database for retrieval of prohibited items.

[0196] In this embodiment, active learning technology can be used to mine data related to contraband from massive amounts of unlabeled data. Then, features are extracted through a feature extraction network, and the extracted features are PQ quantized to generate a contraband feature codebook. A contraband retrieval database is then established using the feature codebook. The details are as follows:

[0197] S1 utilizes the feature extraction network and trains it using the Snapshot Ensemble method. The specific strategy is to use a periodic learning rate to generate multiple network models.

[0198] S2, perform Monte Carlo dropout prediction.

[0199] This embodiment performs the following operations on all unlabeled samples: Dropout layers are enabled in multiple network models, and each model makes multiple predictions for each sample. The final softmax result is obtained by averaging all the predicted softmax results.

[0200] S3 calculates the information entropy of the final softmax result in S2, then sorts the information entropy of all samples, selects the M samples with the largest information entropy for labeling, and adds the labeled samples to the sample library; selects all samples with information entropy less than a certain threshold, uses an algorithm to label them with pseudo-labels, then manually corrects whether they are correct, and adds the corrected samples to the sample library.

[0201] S4. All samples are processed through a feature extraction network to extract features, and PQ quantization is performed to generate a contraband feature codebook. Then, a contraband retrieval database is established using the feature codebook.

[0202] In practical applications, this embodiment can involve inputting an image, first obtaining all prohibited items through a general detector, then performing fine-grained feature extraction on each prohibited item, and finally retrieving the features of the prohibited item from the retrieval database to obtain the specific category of the prohibited item.

[0203] It should be noted that the feature extraction network in this embodiment can also be replaced by InceptionV1-V4 networks, ResNet networks, ResNeXt networks, etc.; the Soft NMS method in this embodiment can be replaced by ordinary NMS methods.

[0204] In this embodiment, instance-balanced data augmentation and dynamic data augmentation methods can be used to augment data for the target detection task; data distillation and overlap soft sampling methods can reduce the impact of missed samples on the model; a local auxiliary module is used to capture local information of contraband targets and extract more discriminative features; center distance loss is used to constrain features, making features of similar samples more aggregated; a combination of Snapshot Ensemble and Monte Carlo dropout is used to mine samples; and information entropy is used to mine high-uncertainty samples and high-confidence samples to establish a retrieval database.

[0205] Figure 7 This is a schematic diagram of an object detection scenario according to an embodiment of the present invention. Figure 7 As shown, the computing device acquires a target image of the object to be detected. This object can be an object whose sub-objects satisfy target conditions, such as unqualified sub-objects or prohibited objects. In this embodiment, an image acquisition device can capture an image of the object to be detected, which can be a picture, an animated image, or an image or frame from a video; no specific limitations are imposed here. This image is then input into the computing device.

[0206] In this embodiment, the computing device is equipped with a detector, which can be a general detector or a model with a certain network structure. It is trained on a target detection dataset with a data volume greater than the target threshold. OpenImagesV4 can be used as the training data for the detector. The detector trained in this way has a general target detection capability and can capture almost all objects that need to be detected.

[0207] In this embodiment, a detector in a computing device detects a target image and outputs at least one target sub-object among the detected objects. This at least one target sub-object can be any object in the target image that meets the target criteria, including potentially compliant objects, such as unqualified objects or prohibited objects. Optionally, each target sub-object in this embodiment can correspond to a detection box. Optionally, the target size of the at least one target sub-object in this embodiment can occupy a very small proportion of the target image size, and the at least one target sub-object can also be very similar to the background of the target image. The type of the at least one target sub-object can also be diverse; no specific requirements are specified here.

[0208] In this embodiment, the target sub-object includes sub-object features, which are used to determine the type of the target sub-object. Optionally, the computing device in this embodiment can use a pre-trained target feature extraction network to extract the sub-object features of the target sub-object and then output the sub-object features.

[0209] In this embodiment, a retrieval database can be pre-established in the computing device. This database is a database of objects that meet the target criteria, such as a database of prohibited objects. It includes the types of objects from which different object characteristics originate, and can be established by mining high-uncertainty samples and high-confidence samples using information entropy. In this embodiment, the retrieval is performed in the database based on the sub-object characteristics of the target sub-object. Optionally, if sub-object characteristics exist in the database, the type corresponding to the sub-object characteristics in the database can be determined as the type of the target sub-object, and then output to the display interface of the computing device for display.

[0210] In existing classification algorithms, all categories need to be predefined, and the recognition performance for small targets is poor, with no ability to quickly cover new categories. This embodiment, however, uses a detector to first identify potential target boxes, and then uses a feature extraction network to extract features, resulting in a target detection method that achieves excellent detection performance for small targets.

[0211] Contraband comes in a wide variety of types and styles, and detection algorithms that use features for classification often lack accuracy and cannot quickly cover new categories. This embodiment, however, utilizes a fine-grained feature extraction network with added local auxiliary modules to extract features from detected contraband targets, resulting in strong identification capabilities. Furthermore, this embodiment employs a center distance loss to constrain features, leading to greater feature aggregation among similar samples, thus improving the accuracy of contraband feature identification.

[0212] This embodiment can use OpenImagesV4, the largest object detection dataset currently available, as the training data for the detector. The detector trained in this way has general object detection capabilities and can capture almost all illegal objects. When a new type of object to be detected appears, it is only necessary to add the sample of the new type to the sample library to quickly complete the coverage.

[0213] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0214] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0215] Example 3

[0216] According to an embodiment of the present invention, a data processing apparatus for implementing the above-described object detection method is also provided. It should be noted that the object detection apparatus of this embodiment can be used to execute embodiments of the present invention. Figure 2 The object detection method shown.

[0217] Figure 8 This is a schematic diagram of an object detection device according to an embodiment of the present invention. Figure 8 As shown, the object detection device 80 may include: a first acquisition unit 81, a first detection unit 82, a first extraction unit 83, and a first retrieval unit 84.

[0218] The first acquisition unit 81 is used to acquire the target image of the object to be detected.

[0219] The first detection unit 82 is used to detect the target image based on the detector to obtain at least one target sub-object in the object to be detected, wherein the detector is trained using the target detection dataset.

[0220] The first extraction unit 83 is used to extract the sub-object features of the target sub-object.

[0221] The first retrieval unit 84 is used to perform a retrieval in the retrieval database based on the characteristics of the sub-object to obtain the type of the target sub-object. The retrieval database includes the types of objects from which different object characteristics originate.

[0222] It should be noted that the first acquisition unit 81, the first detection unit 82, the first extraction unit 83, and the first retrieval unit 84 mentioned above correspond to steps S202 to S208 in Embodiment 1. The four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above units, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0223] According to an embodiment of the present invention, another data processing apparatus for implementing the above-described object detection method is also provided. It should be noted that the object detection apparatus of this embodiment can be used to execute embodiments of the present invention. Figure 3 The object detection method shown.

[0224] Figure 9 This is a schematic diagram of another object detection device according to an embodiment of the present invention. Figure 9 As shown, the object detection device 90 may include: a second acquisition unit 91, a second detection unit 92, a second extraction unit 93, and a second retrieval unit 94.

[0225] The second acquisition unit 91 is used to acquire the target image of the object to be detected.

[0226] The second detection unit 92 is used to detect the target image based on the detector to obtain at least one target sub-item among the items to be detected, wherein the detector is trained using the target detection dataset.

[0227] The second extraction unit 93 is used to extract the sub-item features of the target sub-item.

[0228] The second retrieval unit 94 is used to search the retrieval database based on the characteristics of the sub-item, determine that the target sub-item is a prohibited item, and determine the type of the target sub-item. The retrieval database includes the types of prohibited items from which different item characteristics originate.

[0229] It should be noted that the second acquisition unit 91, the second detection unit 92, the second extraction unit 93, and the second retrieval unit 94 mentioned above correspond to steps S302 to S308 in Embodiment 1. The four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above units, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0230] According to an embodiment of the present invention, another data processing apparatus for implementing the above-described object detection method is also provided. It should be noted that the object detection apparatus of this embodiment can be used to execute embodiments of the present invention. Figure 4 The object detection method shown.

[0231] Figure 10 This is a schematic diagram of another object detection device according to an embodiment of the present invention. Figure 10 As shown, the object detection device 100 may include: a third acquisition unit 101, a third detection unit 102, a third extraction unit 103, and a third retrieval unit 104.

[0232] The third acquisition unit 101 is used to acquire the target image of the content to be detected.

[0233] The third detection unit 102 is used to detect the target image based on the detector to obtain at least one target sub-content in the content to be detected, wherein the detector is trained using a target detection dataset.

[0234] The third extraction unit 103 is used to extract the sub-content features of the target sub-content.

[0235] The third retrieval unit 104 is used to perform a retrieval in the retrieval database based on sub-content features, determine that the target sub-content is prohibited content, and determine the type of the target sub-content. The retrieval database includes types of prohibited content from different content features.

[0236] It should be noted that the third acquisition unit 101, the third detection unit 102, the third extraction unit 103, and the third retrieval unit 104 mentioned above correspond to steps S402 to S408 in Embodiment 1. The four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above units, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0237] In the object detection device of this embodiment, since the detector trained on the object detection dataset has general detection capabilities, it can first use the detector to detect at least one target sub-object of the object to be detected, and then further search the features of the target sub-object in the retrieval library. Since the retrieval library includes the types of objects from which different object features come, the type of the target sub-object can be accurately determined in the retrieval library. That is, this application adopts a data retrieval combined with a sample set to determine the type of the target sub-object, avoiding the inability to encompass all objects to be detected by using a detection algorithm alone, and avoiding the need to collect data again and retrain the classification model, which makes it difficult to guarantee the accuracy of object detection. Thus, it solves the technical problem of low accuracy of object detection and achieves the technical effect of improving the accuracy of object detection.

[0238] Example 4

[0239] Embodiments of the present invention may provide a computer terminal, which can be installed in the object detection system of the present invention. The computer terminal can be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0240] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0241] In this embodiment, the computer terminal described above can execute the program code for the following steps in the object detection method: acquiring a target image of the object to be detected; detecting the target image based on a detector to obtain at least one target sub-object in the object to be detected, wherein the detector is trained using a target detection dataset; extracting sub-object features of the target sub-object; and searching in a retrieval library based on the sub-object features to obtain the type of the target sub-object, wherein the retrieval library includes the types of objects from which different object features originate.

[0242] Optionally, Figure 11 This is a structural block diagram of a mobile terminal according to an embodiment of the present invention. Figure 11 As shown, the mobile terminal A may include one or more (only one is shown in the figure) processors 1102, memory 1104 and transmission devices 1106.

[0243] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the object detection method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the object detection method described above. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the mobile terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0244] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring a target image of the object to be detected; detecting the target image based on a detector to obtain at least one target sub-object in the object to be detected, wherein the detector is trained using a target detection dataset; extracting sub-object features of the target sub-object; and searching a retrieval library based on the sub-object features to obtain the type of the target sub-object, wherein the retrieval library includes the types of objects from which different object features originate.

[0245] Optionally, the processor may also execute program code that performs the following steps: after acquiring the target image of the object to be detected, perform data augmentation on the target image; and detect the data-augmented target image based on the detector to obtain at least one target sub-object.

[0246] Optionally, the processor may also execute program code that performs the following steps: data augmentation on the target image; and / or data transformation on the target image.

[0247] Optionally, the processor may also execute program code that performs the following steps: scaling the target image according to the target scaling ratio; determining multiple sliding window regions in the scaled target image based on the target sliding window; and determining at least one target sliding window region that includes the target sub-object among the multiple sliding window regions.

[0248] Optionally, the processor may also execute program code that performs the following steps: transforming the image data of the target image based on multiple sets of random transformation parameters.

[0249] Optionally, the processor may also execute program code that performs the following steps: extracting image features of the target image based on the feature extraction network layer in the detector; classifying and regressing the image features based on the first convolutional layer in the detector to obtain the target region of the target image, wherein the first convolutional layer is connected to the feature extraction network layer, and the target region includes target sub-objects; and detecting the target sub-objects in the target region.

[0250] Optionally, the processor may also execute program code that performs the following steps: performing data distillation and overlapping soft sampling on the target region using a loss function in the detector to obtain multiple target detection boxes; determining at least one first target detection box from the multiple target detection boxes, wherein the cross-union ratio between the first target detection box and a predetermined detection box is greater than a first threshold; and determining a target sub-object based on at least one first target detection box.

[0251] Optionally, the processor may also execute program code that performs the following steps: obtaining the second target detection box with the highest corresponding value in at least one first target detection box; attenuating the values ​​corresponding to the third target detection boxes in multiple target detection boxes whose overlap with the second target detection box is greater than a second threshold; removing the third target detection boxes in multiple target detection boxes whose attenuated values ​​are lower than a third threshold to obtain at least one fourth target detection box; and determining the target sub-object in the fourth target detection box.

[0252] Optionally, the processor may also execute program code for the following steps: predicting data in the target region using a target prediction model to obtain multiple predicted detection boxes and their confidence scores, wherein the target prediction model is trained on a public dataset that includes data of objects of a known type; performing overlapping soft sampling on the predicted detection boxes to obtain their loss scores; weighting the loss scores based on the confidence scores to obtain the target values ​​of the predicted detection boxes; and determining multiple target detection boxes from the multiple predicted detection boxes based on the target values.

[0253] Optionally, the processor may also execute program code that performs the following steps: extracting sub-object features of the target sub-object based on the target feature extraction network model.

[0254] Optionally, the processor may also execute program code for the following steps: extracting first sub-object features of the target sub-object based on the sub-feature extraction network model in the target feature extraction network model, wherein the sub-feature extraction network is followed by a second convolutional layer after the feature layer, and the dimension of the sub-object features is the dimension of the second convolutional layer; performing auxiliary recognition on the feature map of the target sub-object based on the local auxiliary model in the target feature extraction network model to obtain second sub-object features, wherein the local auxiliary model includes at least a third convolutional layer, a first pooling layer, a second pooling layer, and a classification layer, wherein the first pooling layer is used to perform global max pooling on the heatmap obtained after convolution processing of the feature map by the third convolutional layer, and the second pooling layer is used to perform average pooling on the feature map of the target size; and determining sub-object features based on the first sub-object features and the second sub-object features.

[0255] Optionally, the processor may also execute program code that performs the following steps: clustering the first sub-object features and the second sub-object features to obtain cluster centers; obtaining the distance between the first sub-object features and the cluster centers and the sum of the distances between the second sub-object features and the cluster centers; and determining the sum as the center distance loss, wherein the center distance loss is used to aggregate similar features in the sub-object features.

[0256] Optionally, the processor may also execute program code that performs the following steps: obtaining target object samples before searching the retrieval library based on sub-object features; establishing a sample library based on the target object samples; extracting the object features to be identified from the sample library based on the target feature extraction network; quantizing the object features to obtain a feature codebook; and establishing a retrieval library based on the feature codebook.

[0257] Optionally, the processor may also execute program code that performs the following steps: before establishing a sample library based on the target object sample, training the target feature extraction network based on a periodic learning rate to obtain multiple network models; predicting multiple sub-samples in the target object sample multiple times based on the dropout layers of the multiple network models to obtain multiple prediction results; averaging the multiple prediction results to obtain the target prediction result.

[0258] Optionally, the processor may also execute program code that performs the following steps: determining the information entropy of multiple sub-samples based on the target prediction results; obtaining a first target sub-sample of the target quantity from the multiple sub-samples, wherein the information entropy of the first target sub-sample is greater than the information entropy of the sub-samples other than the first target sub-sample of the target quantity; labeling the types of objects in the first target sub-sample; adding the labeled first target sub-sample to the sample library; labeling the types of objects in the second target sub-sample of the multiple sub-samples whose information entropy is less than a fourth threshold with pseudo-labels; correcting the pseudo-labeled second target sub-samples by responding to a correction operation command; and adding the corrected second target sub-sample to the sample library.

[0259] As an alternative example, the processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring a target image of the item to be detected; detecting the target image based on a detector to obtain at least one target sub-item among the items to be detected, wherein the detector is trained using a target detection dataset; extracting sub-item features of the target sub-item; and searching a retrieval database based on the sub-item features to determine that the target sub-item is a prohibited item and to determine the type of the target sub-item, wherein the retrieval database includes types of prohibited items from which different item features originate.

[0260] Optionally, the processor may also execute program code for the following steps: after acquiring the target image of the object to be detected, performing data augmentation on the target image; and detecting the target image based on the detector to obtain at least one target sub-item among the objects to be detected, including: detecting the data-augmented target image based on the detector to obtain at least one target sub-item.

[0261] Optionally, the processor may also execute program code that performs the following steps: extracting sub-item features of the target sub-item based on the target feature extraction network model.

[0262] Optionally, the processor may also execute program code for the following steps: before searching the retrieval database based on sub-item features to determine that the target sub-item is a prohibited item, obtain a sample of the target item; establish a sample database based on the target item sample; extract the item features to be identified from the sample database based on the target feature extraction network; quantify the item features to obtain a feature codebook; and establish a retrieval database based on the feature codebook.

[0263] As another alternative example, the processor can invoke information and applications stored in memory via a transmission device to perform the following steps: acquiring a target image of the content to be detected; detecting the target image based on a detector to obtain at least one target sub-content in the content to be detected, wherein the detector is trained using a target detection dataset; extracting sub-content features of the target sub-content; and searching a retrieval database based on the sub-content features to determine that the target sub-content is prohibited content and to determine the type of the target sub-content, wherein the retrieval database includes types of prohibited content from which different content features originate.

[0264] Optionally, the processor may also execute program code for the following steps: after acquiring the target image of the content to be detected, performing data augmentation on the target image; and detecting the target image based on the detector to obtain at least one target sub-content in the content to be detected, including: detecting the data-augmented target image based on the detector to obtain at least one target sub-content.

[0265] Optionally, the processor may also execute program code that performs the following steps: extracting sub-content features of the target sub-content based on the target feature extraction network model.

[0266] Optionally, the processor may also execute program code that performs the following steps: before searching the retrieval database based on sub-content features to determine that the target sub-content is prohibited content, obtain a target content sample; establish a sample database based on the target content sample; extract the content features to be identified from the sample database based on the target feature extraction network; quantify the content features to obtain a feature codebook; and establish a retrieval database based on the feature codebook.

[0267] This invention provides a method for object detection. The method involves: acquiring a target image of the object to be detected; detecting the target image using a detector trained on a target detection dataset to obtain at least one target sub-object; extracting sub-object features from the target sub-objects; and retrieving the target sub-object type from a retrieval database based on these features. The retrieval database includes object types from which different object features originate. In other words, because the detector trained on the target detection dataset has general detection capabilities, it can first detect at least one target sub-object of the object to be detected. Then, the features of the target sub-object are further retrieved from the retrieval database. Since the retrieval database includes object types from which different object features originate, the type of the target sub-object can be accurately determined from the database. This application uses a combination of data retrieval and a sample set to determine the type of the target sub-object, avoiding the limitations of purely using detection algorithms that cannot encompass all objects to be detected, and avoiding the need to re-collect data and retrain the classification model, which makes it difficult to guarantee the accuracy of object detection. This solves the technical problem of low accuracy in object detection and achieves the technical effect of improving the accuracy of object detection.

[0268] Those skilled in the art will understand that Figure 11 The structure shown is for illustrative purposes only. Mobile terminal A can also be a smartphone (such as an Android phone, iOS phone, etc.), tablet computer, mobile internet device (MID), PAD, and other terminal devices. Figure 11 This does not limit the structure of the aforementioned mobile terminal. For example, mobile terminal 11 may also include components that are more... Figure 11 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 11 The different configurations shown.

[0269] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0270] Example 5

[0271] Embodiments of the present invention also provide a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the object detection method provided in Embodiment 1.

[0272] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0273] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring a target image of an object to be detected; detecting the target image based on a detector to obtain at least one target sub-object in the object to be detected, wherein the detector is trained using an object detection dataset; extracting sub-object features of the target sub-object; and searching a retrieval library based on the sub-object features to obtain the type of the target sub-object, wherein the retrieval library includes the types of objects from which different object features originate.

[0274] Optionally, the storage medium is also configured to store program code for performing the following steps: after acquiring the target image of the object to be detected, performing data augmentation on the target image; and detecting the data-augmented target image based on the detector to obtain at least one target sub-object.

[0275] Optionally, the storage medium is also configured to store program code for performing the following steps: data augmentation of the target image; and / or data transformation of the target image.

[0276] Optionally, the storage medium is also configured to store program code for performing the following steps: scaling the target image according to a target scaling ratio; determining multiple sliding window regions in the scaled target image based on the target sliding window; and determining at least one target sliding window region including the target sub-object among the multiple sliding window regions.

[0277] Optionally, the storage medium is also configured to store program code for performing the following steps: transforming image data of the target image based on multiple sets of random transformation parameters.

[0278] Optionally, the storage medium is also configured to store program code for performing the following steps: extracting image features of the target image based on the feature extraction network layer in the detector; classifying and regressing the image features based on the first convolutional layer in the detector to obtain the target region of the target image, wherein the first convolutional layer is connected to the feature extraction network layer, and the target region includes target sub-objects; detecting the target sub-objects in the target region.

[0279] Optionally, the storage medium is also configured to store program code for performing the following steps: performing data distillation and overlapping soft sampling on the target region using a loss function in the detector to obtain multiple target detection boxes; determining at least one first target detection box from the multiple target detection boxes, wherein the cross-union ratio between the first target detection box and a predetermined detection box is greater than a first threshold; and determining a target sub-object based on at least one first target detection box.

[0280] Optionally, the storage medium is further configured to store program code for performing the following steps: obtaining the second target detection box with the highest corresponding value in at least one first target detection box; attenuating the values ​​corresponding to third target detection boxes in multiple target detection boxes whose overlap with the second target detection box is greater than a second threshold; removing third target detection boxes in multiple target detection boxes whose attenuated values ​​are lower than a third threshold to obtain at least one fourth target detection box; and determining the target sub-object in the fourth target detection box.

[0281] Optionally, the storage medium is also configured to store program code for performing the following steps: predicting data in a target region using a target prediction model to obtain multiple predicted detection boxes and their confidence scores, wherein the target prediction model is trained on a public dataset that includes data of objects of a known type; performing overlapping soft sampling on the predicted detection boxes to obtain their loss values; weighting the loss values ​​based on the confidence scores to obtain the target values ​​of the predicted detection boxes; and determining multiple target detection boxes from the multiple predicted detection boxes based on the target values.

[0282] Optionally, the storage medium is also configured to store program code for performing the following steps: extracting sub-object features of the target sub-object based on the target feature extraction network model.

[0283] Optionally, the storage medium is further configured to store program code for performing the following steps: extracting first sub-object features of the target sub-object based on a sub-feature extraction network model in the target feature extraction network model, wherein the sub-feature extraction network is followed by a second convolutional layer after the feature layer, and the dimension of the sub-object features is the dimension of the second convolutional layer; performing auxiliary recognition on the feature map of the target sub-object based on a local auxiliary model in the target feature extraction network model to obtain second sub-object features, wherein the local auxiliary model includes at least a third convolutional layer, a first pooling layer, a second pooling layer, and a classification layer, wherein the first pooling layer is used to perform global max pooling on the heatmap obtained after convolution processing of the feature map by the third convolutional layer, and the second pooling layer is used to perform average pooling on the feature map of the target size; and determining sub-object features based on the first sub-object features and the second sub-object features.

[0284] Optionally, the storage medium is further configured to store program code for performing the following steps: clustering the first sub-object features and the second sub-object features to obtain cluster centers; obtaining the distance between the first sub-object features and the cluster centers and the sum of the distances between the second sub-object features and the cluster centers; and determining the sum as the center distance loss, wherein the center distance loss is used to aggregate similar features among the sub-object features.

[0285] Optionally, the storage medium is also configured to store program code for performing the following steps: obtaining target object samples before performing a retrieval in the retrieval library based on sub-object features; establishing a sample library based on the target object samples; extracting the object features to be identified from the sample library based on the target feature extraction network; quantizing the object features to obtain a feature codebook; and establishing a retrieval library based on the feature codebook.

[0286] Optionally, the storage medium is also configured to store program code for performing the following steps: before establishing a sample library based on the target object samples, training the target feature extraction network based on a periodic learning rate to obtain multiple network models; predicting multiple sub-samples in the target object samples multiple times based on the dropout layers of the multiple network models to obtain multiple prediction results; averaging the multiple prediction results to obtain the target prediction result.

[0287] Optionally, the storage medium is also configured to store program code for performing the following steps: determining the information entropy of multiple subsamples based on the target prediction results; obtaining a first target subsample of the target quantity from the multiple subsamples, wherein the information entropy of the first target subsample is greater than the information entropy of the subsamples other than the first target subsample of the target quantity; labeling the types of objects in the first target subsample; adding the labeled first target subsample to the sample library; labeling the types of objects in the second target subsample of the multiple subsamples whose information entropy is less than a fourth threshold with pseudo-labels; correcting the pseudo-labeled second target subsamples by responding to a correction operation command; and adding the corrected second target subsample to the sample library.

[0288] As an optional example, the storage medium is also configured to store program code for performing the following steps: acquiring a target image of an item to be detected; detecting the target image based on a detector to obtain at least one target sub-item among the items to be detected, wherein the detector is trained using an object detection dataset; extracting sub-item features of the target sub-item; and searching a retrieval library based on the sub-item features to determine that the target sub-item is a prohibited item and to determine the type of the target sub-item, wherein the retrieval library includes the types of prohibited items from which different item features originate.

[0289] Optionally, the storage medium is further configured to store program code for performing the following steps: after acquiring a target image of an item to be detected, performing data augmentation on the target image; and detecting the target image based on a detector to obtain at least one target sub-item among the items to be detected, including: detecting the data-augmented target image based on a detector to obtain at least one target sub-item.

[0290] Optionally, the storage medium is also configured to store program code for performing the following steps: extracting sub-item features of the target sub-item based on the target feature extraction network model.

[0291] Optionally, the storage medium is also configured to store program code for performing the following steps: obtaining a target item sample before searching the retrieval library based on sub-item features to determine that the target sub-item is a prohibited item; establishing a sample library based on the target item sample; extracting the item features to be identified from the sample library based on a target feature extraction network; quantifying the item features to obtain a feature codebook; and establishing a retrieval library based on the feature codebook.

[0292] As another alternative example, the storage medium is also configured to store program code for performing the following steps: acquiring a target image of the content to be detected; detecting the target image based on a detector to obtain at least one target sub-content in the content to be detected, wherein the detector is trained using a target detection dataset; extracting sub-content features of the target sub-content; and searching a retrieval library based on the sub-content features to determine that the target sub-content is prohibited content and to determine the type of the target sub-content, wherein the retrieval library includes types of prohibited content from which different content features originate.

[0293] Optionally, the storage medium is further configured to store program code for performing the following steps: after acquiring a target image of the content to be detected, performing data augmentation on the target image; and detecting the target image based on a detector to obtain at least one target sub-content in the content to be detected, including: detecting the data-augmented target image based on a detector to obtain at least one target sub-content.

[0294] Optionally, the storage medium is also configured to store program code for performing the following steps: extracting sub-content features of the target sub-content based on the target feature extraction network model.

[0295] Optionally, the storage medium is also configured to store program code for performing the following steps: obtaining a target content sample before searching the retrieval library based on sub-content features to determine that the target sub-content is prohibited content; establishing a sample library based on the target content sample; extracting the content features to be identified from the sample library based on a target feature extraction network; quantifying the content features to obtain a feature codebook; and establishing a retrieval library based on the feature codebook.

[0296] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0297] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0298] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0299] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0300] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0301] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0302] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An object detection method, characterized in that, include: Acquire the target image of the object to be detected; The target image is detected based on the detector to obtain at least one target sub-object among the objects to be detected, wherein the detector is trained on the target detection dataset and the detector includes a loss function; Extract the sub-object features of the target sub-object; Based on the sub-object features, a search is performed in the search library to obtain the type of the target sub-object, wherein the search library includes the types of objects from which different object features originate; The method further includes: using the loss function to predict the target region of the target image to obtain multiple predicted detection boxes; weighting the loss of the predicted detection boxes based on the prediction confidence corresponding to the predicted detection boxes to obtain a first weighted result; and weighting multiple overlaps between the multiple predicted detection boxes and predetermined detection boxes to obtain a second weighted result; and using the first weighted result and the second weighted result, determining multiple target detection boxes from the multiple predicted detection boxes, wherein the target detection boxes are used to determine the target sub-object.

2. The method according to claim 1, characterized in that, After acquiring the target image of the object to be detected, the method further includes: performing data augmentation on the target image; Detecting the target image based on the detector to obtain at least one target sub-object among the objects to be detected includes: detecting the data-enhanced target image based on the detector to obtain the at least one target sub-object.

3. The method according to claim 2, characterized in that, Data augmentation of the target image includes: Data augmentation of the target image; and / or The target image undergoes data transformation.

4. The method according to claim 3, characterized in that, Data augmentation of the target image data includes: The target image is scaled according to the target scaling ratio; Multiple sliding window regions are determined in the scaled target image based on the target sliding window; At least one target sliding window region is determined among the plurality of sliding window regions, which includes the target sub-object.

5. The method according to claim 3, characterized in that, Data transformation of the target image data includes: The image data of the target image is transformed based on multiple sets of random transformation parameters.

6. The method according to claim 1, characterized in that, Based on the detector, the target image is detected to obtain at least one target sub-object among the objects to be detected, including: Image features of the target image are extracted based on the feature extraction network layer in the detector; The image features are classified and regressed based on the first convolutional layer in the detector to obtain the target region of the target image, wherein the first convolutional layer is connected to the feature extraction network layer, and the target region includes the target sub-object; The target sub-object was detected in the target area.

7. The method according to claim 6, characterized in that, Detecting the target sub-object in the target region includes: At least one first target detection box is determined from the plurality of target detection boxes, wherein the intersection-union ratio between the first target detection box and the predetermined detection box is greater than a first threshold; The target sub-object is determined based on the at least one first target detection box.

8. The method according to claim 7, characterized in that, Determining the target sub-object based on the at least one first target detection box includes: Obtain the second target detection box with the highest corresponding value from at least one first target detection box; The values ​​corresponding to the third target detection box whose overlap with the second target detection box is greater than the second threshold are attenuated; Among the plurality of target detection boxes, remove the third target detection boxes whose attenuated values ​​are lower than the third threshold to obtain at least one fourth target detection box; The target sub-object is determined in the fourth target detection box.

9. The method according to claim 7, characterized in that, The method further includes: The target prediction model is used to predict the data in the target area to obtain multiple predicted detection boxes and the confidence scores of the predicted detection boxes. The target prediction model is trained on a public dataset, which includes data of objects of a known type. Overlap soft sampling is performed on the predicted detection box to obtain the loss of the predicted detection box; The loss is weighted based on the confidence level to obtain the target value of the predicted detection box; The plurality of target detection boxes are determined from the plurality of predicted detection boxes based on the target value.

10. The method according to claim 1, characterized in that, Extracting the sub-object features of the target sub-object includes: The sub-object features of the target sub-object are extracted based on the target feature extraction network model.

11. The method according to claim 10, characterized in that, Extracting the sub-object features of the target sub-object based on the target feature extraction network model includes: The first sub-object feature of the target sub-object is extracted based on the sub-feature extraction network model in the target feature extraction network model, wherein the sub-feature extraction network is followed by a second convolutional layer after the feature layer, and the dimension of the first sub-object feature is the dimension of the second convolutional layer. Based on the local auxiliary model in the target feature extraction network model, the feature map of the target sub-object is assisted in recognition to obtain the features of the second sub-object. The local auxiliary model includes at least a third convolutional layer, a first pooling layer, a second pooling layer and a classification layer. The first pooling layer is used to perform global max pooling on the heat map obtained after the feature map is convolved by the third convolutional layer. The second pooling layer is used to perform average pooling on the feature map of the target size. The sub-object features are determined based on the first sub-object features and the second sub-object features.

12. The method according to claim 11, characterized in that, The loss weight of the local auxiliary model is less than the loss weight of the sub-feature extraction network model.

13. The method according to claim 11, characterized in that, The method further includes: Cluster the features of the first sub-object and the features of the second sub-object to obtain cluster centers; Obtain the sum of the distance between the first sub-object feature and the cluster center and the distance between the second sub-object feature and the cluster center; The sum is determined as the center distance loss, wherein the center distance loss is used to aggregate similar features in the sub-object features.

14. The method according to claim 10, characterized in that, Before performing a search in the retrieval database based on the sub-object features, the method further includes: Obtain a sample of the target object; A sample library will be established based on the target object samples; Based on the target feature extraction network, the target feature extraction network extracts the object features to be identified from the sample database; The features of the object are quantized to obtain a feature codebook; The retrieval database is established based on the aforementioned feature codebook.

15. The method according to claim 14, characterized in that, Before establishing a sample library based on the target object samples, the method further includes: The target feature extraction network is trained based on a periodic learning rate to obtain multiple network models; The dropout layer of the multiple network models makes multiple predictions on multiple sub-samples in the target object sample to obtain multiple prediction results. The target prediction result is obtained by averaging the multiple prediction results.

16. The method according to claim 15, characterized in that, Establishing a sample library based on the target object samples includes: The information entropy of the multiple sub-samples is determined based on the target prediction results; Obtain a target number of first target sub-samples from the plurality of sub-samples, wherein the information entropy of the first target sub-samples is greater than the information entropy of the sub-samples other than the target number of first target sub-samples in the plurality of sub-samples; Label the types of objects in the first target subsample; The first target sub-sample, after being labeled, is added to the sample library; For the objects in the second target subsample whose information entropy is less than the fourth threshold among the multiple subsamples, pseudo-labels are applied to their types; The second target sub-sample with the pseudo-label is corrected by responding to a correction operation command. The corrected second target subsample is added to the sample library.

17. An object detection method, characterized in that, include: Acquire the target image of the item to be detected; The target image is detected based on the detector to obtain at least one target sub-item among the items to be detected, wherein the detector is trained on the target detection dataset and the detector includes a loss function; Extract the sub-item features of the target sub-item; Based on the sub-item characteristics, a search is performed in the search database to determine that the target sub-item is a prohibited item and to determine the type of the target sub-item. The search database includes types of prohibited items from which different item characteristics originate. The method further includes: using the loss function to predict the target region of the target image to obtain multiple predicted detection boxes; weighting the loss of the predicted detection boxes based on the prediction confidence corresponding to the predicted detection boxes to obtain a first weighted result; and weighting multiple overlaps between the multiple predicted detection boxes and predetermined detection boxes to obtain a second weighted result; and using the first weighted result and the second weighted result, determining multiple target detection boxes from the multiple predicted detection boxes, wherein the target detection boxes are used to determine the target sub-items.

18. The method according to claim 17, characterized in that, After acquiring the target image of the item to be detected, the method further includes: performing data augmentation on the target image; Detecting the target image based on the detector to obtain at least one target sub-item among the items to be detected includes: detecting the data-enhanced target image based on the detector to obtain the at least one target sub-item.

19. The method according to claim 18, characterized in that, Extracting the sub-item features of the target sub-item includes: The sub-item features of the target sub-item are extracted based on the target feature extraction network model.

20. The method according to claim 19, characterized in that, Before performing a search in the retrieval database based on the sub-item characteristics to determine that the target sub-item is a prohibited item, the method further includes: Obtain a sample of the target item; Establish a sample library based on the target item samples; Based on the target feature extraction network, the feature of the item to be identified is extracted from the sample database; The features of the items are quantified to obtain a feature codebook; The retrieval database is established based on the aforementioned feature codebook.

21. An object detection method, characterized in that, include: Acquire the target image of the content to be detected; The target image is detected based on the detector to obtain at least one target sub-content in the content to be detected, wherein the detector is trained on a target detection dataset and includes a loss function; Extract the sub-content features of the target sub-content; Based on the sub-content features, a search is performed in the search database to determine that the target sub-content is prohibited content and to determine the type of the target sub-content, wherein the search database includes types of prohibited content from different content features; The method further includes: using the loss function to predict the target region of the target image to obtain multiple predicted detection boxes; weighting the loss of the predicted detection boxes based on the prediction confidence corresponding to the predicted detection boxes to obtain a first weighted result; and weighting multiple overlaps between the multiple predicted detection boxes and predetermined detection boxes to obtain a second weighted result; and using the first weighted result and the second weighted result, determining multiple target detection boxes from the multiple predicted detection boxes, wherein the target detection boxes are used to determine the target sub-content.

22. The method according to claim 21, characterized in that, After acquiring the target image of the content to be detected, the method further includes: performing data augmentation on the target image; Detecting the target image based on the detector to obtain at least one target sub-content in the content to be detected includes: detecting the data-enhanced target image based on the detector to obtain the at least one target sub-content.

23. The method according to claim 21, characterized in that, Extracting sub-content features of the target sub-content, including: The sub-content features of the target sub-content are extracted based on the target feature extraction network model.

24. The method according to claim 23, characterized in that, Before performing a search in the retrieval database based on the sub-content features to determine that the target sub-content is prohibited content, the method further includes: Obtain a sample of the target content; A sample library will be established based on the target content samples; Based on the target feature extraction network, the content features to be identified are extracted from the sample database; The content features are quantified to obtain a feature codebook; The retrieval database is established based on the aforementioned feature codebook.

25. An object detection device, characterized in that, include: The first acquisition unit is used to acquire the target image of the object to be detected; The first detection unit is used to detect the target image based on the detector to obtain at least one target sub-object among the objects to be detected, wherein the detector is trained through a target detection dataset and the detector includes a loss function; The first extraction unit is used to extract the sub-object features of the target sub-object; The first retrieval unit is used to perform a retrieval in the retrieval database based on the sub-object features to obtain the type of the target sub-object, wherein the retrieval database includes the types of objects from which different object features originate; The device is further configured to perform the following steps: predicting the target region of the target image using the loss function to obtain multiple predicted detection boxes; weighting the loss of the predicted detection boxes using the prediction confidence corresponding to the predicted detection boxes to obtain a first weighted result; and weighting multiple overlaps between the multiple predicted detection boxes and predetermined detection boxes to obtain a second weighted result; and determining multiple target detection boxes from the multiple predicted detection boxes using the first weighted result and the second weighted result, wherein the target detection boxes are used to determine the target sub-object.

26. An object detection device, characterized in that, include: The second acquisition unit is used to acquire the target image of the item to be detected; The second detection unit is used to detect the target image based on the detector to obtain at least one target sub-item among the items to be detected, wherein the detector is trained through a target detection dataset and includes a loss function; The second extraction unit is used to extract the sub-item features of the target sub-item; The second retrieval unit is used to search the retrieval database based on the sub-item characteristics, determine that the target sub-item is a prohibited item, and determine the type of the target sub-item, wherein the retrieval database includes the types of prohibited items from which different item characteristics originate; The device is further configured to perform the following steps: predicting the target region of the target image using the loss function to obtain multiple predicted detection boxes; weighting the loss of the predicted detection boxes using the prediction confidence corresponding to the predicted detection boxes to obtain a first weighted result; and weighting multiple overlaps between the multiple predicted detection boxes and predetermined detection boxes to obtain a second weighted result; and determining multiple target detection boxes from the multiple predicted detection boxes using the first weighted result and the second weighted result, wherein the target detection boxes are used to determine the target sub-items.

27. An object detection device, characterized in that, include: The third acquisition unit is used to acquire the target image of the content to be detected; The third detection unit is used to detect the target image based on the detector to obtain at least one target sub-content in the content to be detected, wherein the detector is trained through a target detection dataset and the detector includes a loss function; The third extraction unit is used to extract the sub-content features of the target sub-content; The third retrieval unit is used to search the retrieval database based on the sub-content features, determine that the target sub-content is prohibited content, and determine the type of the target sub-content, wherein the retrieval database includes types of prohibited content from different content features; The device is further configured to perform the following steps: predicting the target region of the target image using the loss function to obtain multiple predicted detection boxes; weighting the loss of the predicted detection boxes using the prediction confidence corresponding to the predicted detection boxes to obtain a first weighted result; and weighting multiple overlaps between the multiple predicted detection boxes and predetermined detection boxes to obtain a second weighted result; and determining multiple target detection boxes from the multiple predicted detection boxes using the first weighted result and the second weighted result, wherein the target detection boxes are used to determine the target sub-content.

28. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program is run by a processor, it controls the device on which the computer-readable storage medium resides to perform the method according to any one of claims 1 to 24.

29. A processor, characterized in that, The processor is used to run a program, wherein the program, when running, performs the method according to any one of claims 1 to 24.

30. An object detection system, characterized in that, include: processor; A memory, connected to the processor, is used to provide the processor with instructions to perform the following processing steps: acquiring a target image of the object to be detected; The target image is detected using a detector to obtain at least one target sub-object among the objects to be detected. The detector is trained using a target detection dataset and includes a loss function. Sub-object features of the target sub-objects are extracted. Based on the sub-object features, a search is performed in a search library to obtain the type of the target sub-object. The search library includes the types of objects from which different object features originate. The memory is further configured to perform the following steps: predicting the target region of the target image using the loss function to obtain multiple predicted detection boxes; weighting the loss of the predicted detection boxes using the prediction confidence corresponding to the predicted detection boxes to obtain a first weighted result; and weighting multiple overlaps between the multiple predicted detection boxes and predetermined detection boxes to obtain a second weighted result; and determining multiple target detection boxes from the multiple predicted detection boxes using the first weighted result and the second weighted result, wherein the target detection boxes are used to determine the target sub-object.

Images