Model training method, target detection method and corresponding device, equipment and medium

Abstract

The application provides a model training method, a target detection method, and corresponding devices, equipment, and media. The model training method includes: obtaining a plurality of sample images; performing multiple iterative training on a target detection model based on the plurality of sample images; starting from a specified training node in the multiple iterative training, performing contrast learning on confusing category information in the sample images by using the target detection model; and obtaining a trained target detection model. The technical solution of the application can obtain a target detection model with high precision, and can accurately detect a to-be-detected image.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a model training method, an object detection method, and corresponding devices, equipment, and media. Background Technology

[0002] Vision-based object detection typically employs deep learning methods. However, in many scenarios requiring object detection, such as industrial defect detection and complex medical image detection, the number of samples is usually small, and the similarity between samples is high. When using deep learning methods for object detection, the classification of the target object can be confused. This confusion can lead to many limitations in practical applications, such as inaccurate classification. These limitations can cause tedious rework and review, which is not conducive to efficient production. Summary of the Invention

[0003] This application provides a model training method, an object detection method, and corresponding devices, equipment, and media to solve the problems existing in related technologies. The technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a model training method, including:

[0005] Acquire multiple sample images;

[0006] The object detection model is trained iteratively multiple times based on multiple sample images. At a specified training node in the multiple iterations, the object detection model is used to compare and learn the easily confused category information in the sample images to obtain the trained object detection model.

[0007] Secondly, embodiments of this application provide a target detection method, including:

[0008] Acquire the image to be detected;

[0009] The image to be detected is input into the target detection model, and the target detection model is used to detect the category of the target object in the image to be detected; the target detection model is trained by the model training method provided in the first aspect of the embodiments of this application.

[0010] Thirdly, embodiments of this application provide a model training apparatus, including:

[0011] The image acquisition module is used to acquire multiple sample images;

[0012] The model training module is used to train the object detection model iteratively multiple times based on multiple sample images. Starting from a specified training node in the multiple iterations, the object detection model is used to compare and learn the easily confused category information in the sample images to obtain the trained object detection model.

[0013] Fourthly, embodiments of this application provide a target detection device, comprising:

[0014] The image acquisition module is used to acquire the image to be detected;

[0015] The target detection module is used to input the image to be detected into the target detection model and use the target detection model to detect the category of the target object in the image to be detected; the target detection model is trained by the model training device provided in the third aspect of the present application.

[0016] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the method provided in the first or second aspect of embodiments of this application.

[0017] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the first or second aspect of embodiments of this application.

[0018] The advantages or beneficial effects of the above technical solutions include at least the following:

[0019] The object detection model can be trained based on sample images. During training, comparative learning can be performed on easily confused category information in the sample images. Targeted comparative learning on easily confused category information can reduce the training burden of the object detection model and make the model more focused on easily confused category information, thereby improving the object detection model's ability to distinguish easily confused categories, improving classification accuracy, and ultimately improving the detection accuracy of the object detection model in the image to be detected. When applied to production scenarios, this can improve production efficiency. The training of the object detection model is iterative, which can improve training accuracy. Comparative learning based on specified training nodes in multiple iterations of training helps to control the amount of iterative training and the training accuracy.

[0020] Targeted comparative learning of easily confused category information can also make the trained object detection model more balanced in detection accuracy across categories, with smaller variance, avoiding polarization. The training method for object detection models has low requirements for the number of training samples, and can still achieve good training results with a small number of training samples, making it suitable for training in small sample scenarios.

[0021] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0022] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0023] Figure 1 A schematic flowchart illustrating a model training method provided in an embodiment of this application;

[0024] Figure 2 This is a schematic diagram illustrating the principle of automatically identifying easily confused category information in the embodiments of this application;

[0025] Figure 3 A structural framework and schematic diagram of a target detection model provided in this application embodiment;

[0026] Figure 4 This application provides a schematic diagram of the structural framework and principle of the target classification module in the target detection model.

[0027] Figure 5 This application provides a schematic diagram of the structural framework and principle of the contrastive learning module in the target detection model.

[0028] Figure 6 A schematic flowchart of a target detection method provided in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the structural framework of a model training device provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the structural framework of a target detection device provided in an embodiment of this application;

[0031] Figure 9 This is a schematic diagram of the structural framework of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0033] The inventors of this application discovered in their research that traditional object detection schemes train deep learning models, but during the training process, all samples are learned indiscriminately. This learning method increases a lot of meaningless learning. For example, indiscriminate learning can increase the feature distance between different categories, but some categories have low similarity and are not easily confused, such as chairs and buses, industrial defects and sofas. The indiscriminate learning method has little effect on categories with low similarity, while for categories with high similarity, such as chairs and sofas, buses and trains, the indiscriminate learning method lacks sufficient contrast and cannot accurately distinguish highly similar categories. Therefore, the indiscriminate learning method has limited ability to improve the model's detection capability and instead increases the complexity of the model.

[0034] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments.

[0035] This application provides a model training method, such as... Figure 1 As shown, the method includes:

[0036] S101, acquire multiple sample images.

[0037] Each sample image may include at least one target object. Depending on the actual application scenario, the target object may also be different. For example, in daily life scenarios, the sample image may be an image of household items, and the target object may be a chair, sofa, or other household items. In transportation scenarios, the sample image may be an image of a vehicle, and the target object may be a bus, train, or other vehicle. In industrial inspection scenarios, the sample image may be an image of an industrial defect, and the target object may be an industrial defect.

[0038] In step S101, sample images of multiple different application scenarios can be acquired to provide rich scene information.

[0039] S102, the target detection model is trained iteratively multiple times based on multiple sample images. Starting from a specified training node in the multiple iterations of training, the target detection model is used to compare and learn the easily confused category information in the sample images to obtain the trained target detection model.

[0040] The aforementioned easily confused category information can include categories with high similarity and easy confusion among the category information of the target object. A similarity threshold can be set as a basis to determine the categories with high similarity. For example, chairs and sofas are two categories with high similarity and can be regarded as a group of easily confused category information. Buses and trains are two categories with high similarity and can be regarded as a group of easily confused category information. Industrial defect 1 and industrial defect 2 are two categories with high similarity and can be regarded as a group of easily confused category information.

[0041] The aforementioned contrastive learning can be a learning method that compares and learns the features of easily confused category information to improve the ability to distinguish easily confused category information. Contrastive learning can increase the inter-class distance (i.e. the feature distance between different categories) and decrease the intra-class distance (i.e. the feature distance between the same category).

[0042] The model training method provided in this application embodiment can train the target detection model based on sample images. During the training process, it can perform comparative learning on easily confused category information. Targeted comparative learning on easily confused category information can reduce the training burden of the target detection model and make the target training model more focused on easily confused category information, thereby improving the target detection model's ability to distinguish easily confused category information, improving the accuracy of classification, and thus improving the detection accuracy of the target detection model on the image to be detected. When applied in production scenarios, it can improve production efficiency.

[0043] In multiple iterations of training, each iteration can input at least one sample image into the object detection model. Training is then performed based on this input sample image, with each iteration serving as a training node. Multiple iterations result in multiple training nodes, which improves training accuracy. Comparative learning based on specified training nodes across multiple iterations helps control the computational cost and training accuracy of the iterations.

[0044] Targeted comparative learning of easily confused category information can also make the trained object detection model more balanced in detection accuracy across categories, with smaller variance, avoiding polarization. The above training method has low requirements for the number of training samples, and can still achieve good training results with a small number of training samples, making it suitable for training in scenarios with few samples.

[0045] In an optional implementation, step S102 above, which involves using an object detection model to perform comparative learning on easily confused category information in the sample image, may include: using the object detection model to identify easily confused category information in the sample image, performing comparative learning on the identified easily confused category information, and automatically obtaining a list of parameters for the easily confused category information after the object detection module identifies the easily confused category information in the sample image. Based on this approach, the object detection model can be used to automatically identify easily confused category information in the sample image, enabling end-to-end training.

[0046] In another optional implementation, in step S102 above, the comparative learning of easily confused category information in the sample image using the object detection model can include: using the object detection model to compare and learn easily confused category information input by the user. This easily confused category information can be obtained by the user based on common sense, experience, etc., when identifying the easily confused category information in the sample image. After the user identifies the easily confused category information in the sample image, the hyperparameters of the object detection model can be specified as a list of easily confused category information, enabling the object detection model to obtain the list of easily confused category information and compare and learn each group of easily confused category information. Alternatively, the identified easily confused category information can be labeled on the sample image, and the object detection model can obtain the easily confused category information from the sample image. Based on this method, easily confused category information in the sample image can be manually identified, enabling end-to-end training.

[0047] In an optional implementation, in step S102 above, the object detection model is trained iteratively multiple times based on multiple sample images. Starting from a specified training node in the multiple iterations of training, the object detection model is used to perform comparative learning on easily confused category information in the sample images. This may include: at each training node before the specified training node (excluding the specified training node), comparative learning is performed on all category information in the sample images. All category information may include the true category information and predicted category information of each target object in the sample images; at the specified training node in the iterative training, easily confused category information in the sample images is identified. For example, the object detection model can be used to automatically identify easily confused category information in the sample images; at each training node after the specified training node in the iterative training (excluding the specified training node), comparative learning is performed on the identified easily confused category information.

[0048] In the initial training node of multiple iterations, i.e., the first iteration, the object detection model has not yet developed the ability to detect new datasets. If easily confused category information is identified and compared at the initial training node, the identification accuracy will be low, leading to low accuracy in comparative learning and affecting the overall training accuracy. Therefore, this embodiment adopts the above-described implementation method. In iterative training, all category information is first compared and learned indiscriminately, enabling the object detection model to develop a certain detection capability for new datasets. Then, easily confused category information is identified and compared and learned from a specified training node, which can improve the identification accuracy of easily confused category information and thus improve the overall training accuracy. Easily confused category information can be identified once. After easily confused category information in the sample image is identified at a specified training node, all subsequent training nodes can compare and learn from this easily confused category information, which is beneficial to improving the overall training efficiency.

[0049] In one alternative implementation, the sample image may include pre-labeled true category information for each target object, which may be pre-labeled by the user.

[0050] Correspondingly, easily confused category information can be identified by the object detection model in the following way: predicting the category of each target object in the sample image to obtain the predicted category information of each target object; determining whether the predicted category information and the true category information of the same target object are the same; when the predicted category information and the true category information of the same target object are different, the predicted category information and the true category information of the target object are regarded as a set of easily confused category information. Based on the comparison results of the true category information and the predicted category information, easily confused category information can be quickly identified.

[0051] The true category information and the predicted category information can take the form of, for example: Figure 2 The target boxes shown can be categorized as follows: the target box containing the true category information can be called the ground truth box, and the target box containing the predicted category information can be called the proposal box.

[0052] The following reference Figure 2 The following example illustrates a principle for automatically identifying easily confused categories: Figure 2 In the example, the sample image includes two target objects, target object one and target object two. Target object one has a pre-labeled ground truth bounding box R1, and target object two has a pre-labeled ground truth bounding box R2. The object detection model can predict the category of target object one and target object two, and at least one prediction result can be obtained for each target object. Figure 2The example uses two prediction results: the two prediction results for target object one are suggested target boxes P11 and P12, and the two prediction results for target object two are suggested target boxes P21 and P2. Figure 2 To clearly show each target bounding box, different line types (solid line, dashed line, dotted line) are used to represent different target bounding boxes. This does not mean that there are multiple line types of target bounding boxes in the same image in the actual scene. In the actual scene, the target bounding boxes in the same image can be of only one line type.

[0053] For target object one, refer to Figure 2 The target box information in the image shows that the predicted category information for the suggested target box P11 is sofa, the predicted category information for the suggested target box P12 is table, and the true category information for the ground truth target box R1 is chair. By comparing the information of the suggested target boxes (including suggested target boxes P11 and P12) and the true target box R1, it can be determined that the predicted category information for the suggested target boxes (including suggested target boxes P11 and P12) is different from the true category information for the ground truth target box R1. The predicted categories (sofa and table) for the suggested target boxes and the true category information (chair) for the ground truth target boxes can be considered as a set of easily confused category information.

[0054] For target object two, refer to Figure 2 The target box information in the image shows that the predicted category information for the suggested target box P21 is "chair" and the predicted category information for the suggested target box P22 is "sofa". The true category information for the ground truth target box R2 is "sofa". Comparing the information of the suggested target boxes (including suggested target boxes P21 and P22) and the true target box R2, it can be determined that the predicted category information for suggested target box P21 is different from the true category information for the true target box R2, while the predicted category information for suggested target box P22 is the same as the true category information for the true target box R2, both being "sofa". Therefore, the predicted category information for suggested target box P21 (chair) and the true category information for the true target box R2 (sofa) can be considered as a set of easily confused category information.

[0055] In one optional implementation, determining whether the predicted category information and the true category information of the same target object are the same, and if the predicted category information and the true category information of the target object are different, treating the predicted category information and the true category information of the target object as a set of easily confused category information, may include: selecting the predicted category information with a confidence level greater than a preset confidence threshold from multiple predicted category information of the same target object, and using it as the designated predicted category information of the target object; determining whether the designated predicted category information and the true category information of the target object are the same; and if the designated predicted category information and the true category information of the target object are different, treating the designated predicted category information and the true category information of the target object as a set of easily confused category information. The confidence threshold can be set according to actual needs or empirical values.

[0056] Based on the above approach, when there are multiple predicted category information for the same target object, the predicted category information with higher confidence can be selected first, and then compared with the true category information to determine the easily confused category information. This method can filter out predicted category information that deviates significantly from the true category information and identify information with high similarity, thereby improving the reliability of subsequent comparative learning.

[0057] Reference Figure 2 For example, for target object one, refer to Figure 2 If the confidence level of the suggested target box P11 is greater than the preset confidence threshold, and the confidence level of the suggested target box P12 is less than the preset confidence threshold, then the suggested target box P12 can be discarded, and only the predicted category information (sofa) corresponding to the suggested target box P11 and the true category information (chair) corresponding to the ground truth target box R1 can be regarded as a set of easily confused category information.

[0058] When performing comparative learning on easily confused category information, the predicted category information and the true category information contained in each group of easily confused category information can be compared and learned in a targeted manner. When performing comparative learning on easily confused category information, the learning of other category information can be abandoned, so that the model can focus more on learning the easily confused category information, reduce the training burden of the model, and improve the overall efficiency of model training.

[0059] In one optional implementation, predicting the category of each target object in the sample image to obtain the predicted category information of each target object may include: predicting the category of each target object in multiple sample images to obtain the predicted category information of each target object in each sample image.

[0060] In one implementation, corresponding to category prediction of multiple sample images, determining whether the predicted category information and the true category information of the same target object are the same may include: for the same target object in multiple sample images, determining whether the predicted category information of the target object in each sample image and the true category information of the target object in each sample image are the same; if they are the same, it can be determined that the predicted category information and the true category information of the target object are the same; otherwise, it is considered that the predicted category information and the true category information of the target object are different.

[0061] For example, when 100 sample images are input, and each sample image contains the same target object A, the target object A in each of the 100 sample images can be predicted to obtain 100 predicted category information for the target object A. These 100 predicted category information are then compared with the true category information of the target object A. If every one of the 100 predicted category information is the same as the true category information of the target object A, then the predicted category information and the true category information of the target object A can be considered to be the same. If at least one of the 100 predicted category information is different from the true category information, then the predicted category information and the true category information of the target object A can be considered to be different.

[0062] Corresponding to category prediction for multiple sample images, determining whether the predicted category information and the true category information of the same target object are the same can include: for the same target object in multiple sample images, determining whether the predicted category information and the true category information of the target object are the same in all sample images; when the predicted category information and the true category information of the target object are different in each sample image, determining whether the number of differences is greater than a preset threshold; if the number of differences is greater than the threshold, determining that the predicted category information and the true category information of the target object are the same; if the number of differences is less than or equal to the threshold, determining that the predicted category information and the true category information of the target object are different. The threshold can be determined based on actual needs or empirical values.

[0063] In one example, when 100 sample images are input, and each sample image contains the same target object A, the number of predictions threshold can be set to 30. Predictions are then made for target object A in each of the 100 sample images, resulting in 100 predicted category information entries for target object A. These 100 predicted category information entries are compared with the true category information of target object A. If the number of predicted category information entries that differs from the true category information is 50 (i.e., the number of differences is 50), which is greater than the number of predictions threshold 30, then the predicted category information of target object A is considered to be different from the true category information. If the number of predicted category information entries that differs from the true category information is 20 (i.e., the number of differences is 20), which is less than the number of predictions threshold 30, then the predicted category information of target object A is considered to be the same as the true category information.

[0064] In one optional implementation, the specified training node is the m-th training node in multiple iterations of training, where m is a times the total number of training nodes in multiple iterations of training, and a is a value between 0.3 and 0.7 (inclusive).

[0065] In one example, if the specified training node is the m-th training node in the iterative training, it can be determined as follows: m = a × n, where n is the total number of training nodes included in the iterative training, and a is the coefficient used to determine the specified training node. If the m determined in this way is a decimal, it can be rounded up or down as needed.

[0066] The optimal range of values ​​for 'a' varies depending on the object detection model. This can be determined based on the model's convergence trend, such as during the training phase when the model is converging normally but not completely, or based on actual needs or empirical values. When 'a' is greater than or equal to 0.3 and less than or equal to 0.7, a suitable designated training node can be identified. At this designated training node, the model begins to identify easily confused category information and perform comparative learning. Choosing this designated training node allows the model to have strong detection capabilities on new datasets after training on previous nodes, identifying easily confused category information with relatively high reliability. Simultaneously, it allows the model sufficient subsequent training nodes for more thorough comparative learning, effectively improving the trained model's ability to distinguish easily confused category information.

[0067] In an optional implementation, the sample images may include pre-labeled true category information for each target object. Correspondingly, in step S102 above, the target detection model is trained iteratively multiple times based on multiple sample images, which may include: predicting the category of each target object in the sample images during training to obtain predicted category information for each target object; determining the loss of the predicted category information of each target object relative to the true predicted information of each target object based on a loss function; and adjusting the model parameters of the target detection model based on the loss. The loss function may include a contrastive learning loss function, which can be constructed based on the weight information of the predicted category information of each target object, the number of current training iterations (i.e., the current iteration number), and the information of a specified training node. This contrastive learning loss can be used to determine the contrastive learning loss of the predicted category information in the easily confused category information relative to the true category information in the easily confused category information.

[0068] In one example, the contrastive learning loss function can be expressed as follows:

[0069]

[0070]

[0071] In expression (1), L RCL N represents the contrastive learning loss. RCL u represents the number of suggested target boxes participating in the contrastive learning. i This represents the intersection-union ratio (IoU) between the i-th suggested bounding box and its corresponding ground truth bounding box. Denotes the cross-entropy loss, w(u) i ) indicates the use of intersection-union ratio u i As the weight w of the i-th suggested target box, w(u i It can balance the performance of each suggested target box. Value. In expression (2), N all This represents the total number of suggested bounding boxes in a given image. C represents the number of suggested bounding boxes corresponding to easily confused category information in an image. I This indicates the current iteration number, i.e., the current iteration is the Cth iteration. I The training iteration, M I This indicates the index of the specified training node, that is, specifying the M-th training node in multiple training iterations. I There are 1 training node.

[0072] Referring to expressions (1) and (2), in the current iteration number C I The index M less than the specified training node IWhen N is at this time, it indicates that the current iteration of training has not yet reached the specified training node where it can begin to identify easily confused category information and perform comparative learning. RCL The value is N all This means that all proposed bounding boxes in the sample image are included in the scope of contrastive learning, and the contrastive learning loss of all proposed bounding boxes is calculated using expression (1); in the current iteration number C I Greater than or equal to the index M of the specified training node I When N reaches a specified training node, it indicates that the current iteration of training has reached a point where it can begin to identify easily confused category information and perform contrastive learning. RCL The value is That is, only the suggested target boxes corresponding to easily confused category information in the sample image are included in the scope of contrastive learning, while the suggested target boxes corresponding to other category information are discarded. The remaining suggested target boxes are combined into a new list, and the contrastive learning loss of the suggested target boxes in the new list is calculated by expression (1), so that other suggested target boxes cannot participate in subsequent training.

[0073] In the above expression (1), This can be expressed as follows:

[0074]

[0075] In expression (3), y i y represents the ground truth class information corresponding to the i-th proposed bounding box among all proposed bounding boxes of a given image. j This represents the true category information corresponding to the j-th proposed bounding box among all proposed bounding boxes of a given image. Indicates y i The number of suggested bounding boxes with the same category, N. This represents the regularization feature of the i-th suggested bounding box. This represents the regularization feature of the j-th suggested bounding box. This represents the regularization feature of the k-th suggested bounding box. Let cosine similarity be the regularized features of the i-th and j-th proposed bounding boxes. Let τ represent the cosine similarity of the regularized features of the i-th and k-th proposed bounding boxes, where τ is a hyperparameter. i =y j} is a type of operation, representing the operation in y. i =y j When I is active, I takes the value 1; otherwise, I takes the value 0.

[0076] In an optional implementation, based on the contrastive learning loss function, the aforementioned loss function may further include: a classification loss function, a bounding box regression loss function, and a target object presence / absence judgment loss function (i.e., a loss function used to determine whether a target object's bounding box exists), which are respectively used to determine the classification loss, bounding box regression loss, and target object presence / absence judgment loss (i.e., the loss for determining whether a target object's bounding box exists) relative to the predicted category information of each target object and the true category information of each target object. Correspondingly, the aforementioned losses can be determined based on the classification loss, bounding box regression loss, target object presence / absence judgment loss, contrastive learning loss, and the weight information of the contrastive learning loss.

[0077] The aforementioned loss can be expressed as follows:

[0078] L FSRC =L cls +L box +L objectness +λL RCL Expression (4)

[0079] In expression (4), L FSRC L represents the aforementioned loss. cls L represents the classification loss, which is the loss in determining the predicted class information of the target object. box L represents the target bounding box regression loss. objectness L represents the loss indicating whether the target object exists or not. RCL L represents the contrastive learning loss. RCL Substituting into the above expression (1), λ represents L RCL The weighting coefficient can be determined based on empirical values, such as 0.5.

[0080] In one implementation, such as Figure 3 As shown, the object detection model may include an object classification module 310 and a contrastive learning module 320.

[0081] The target classification module 310 can be used to: predict the category of each target object in the sample image and obtain the predicted category information of each target object.

[0082] The contrastive learning module 320 can be used to: perform contrastive learning on all category information in the sample image at each training node before the specified training node; identify easily confused category information in the sample image at the specified training node; and perform contrastive learning on the identified easily confused category information at each training node after the specified training node.

[0083] The target classification module 310 described above can be constructed and implemented by a convolutional neural network, such as Faster R-CNN (Fast Region Convolutional Neural Network). Based on the target classification module 310, the above implementation adds a contrastive learning module 320 to achieve targeted contrastive learning of easily confused category information.

[0084] Reference Figure 4 For example, the target classification module 310 may include: a backbone network 311, a proposed target box search network 312, a region of interest pooling (ROI Pooling) unit 313, a region of interest feature extraction unit 314, a target classification unit 315, and a target box regression unit 316. The network structure of the backbone network 311 can be selected according to actual needs; for example, a ResNet101 (fast training residual) network with feature pyramids can be used.

[0085] A batch of training sample images can be input into the backbone network 311 to obtain feature maps of different dimensions. These feature maps are then searched by the Region Proposal Network (RPN) to obtain all proposed bounding boxes in the sample images. The feature maps and proposed bounding boxes of different dimensions are then processed by the region of interest pooling unit 313 and the region of interest feature extraction unit 314 to obtain the features of the proposed bounding boxes. Figure 4 Four suggested target box features are shown as examples: feature f(p1), feature f(p2), feature f(p3), and feature f(p4). The features of the suggested target boxes can be input into the contrastive learning module 320, the target classification unit 315, and the target box regression unit 316, respectively. After processing by the target classification unit and the target box regression unit, the classification result (i.e., predicted category information) and regression result of the suggested target boxes can be obtained. Both the classification result and the regression result can be input into the contrastive learning module 320.

[0086] In one implementation, reference is made to Figure 5 For example, the contrastive learning module 320 includes: a node judgment unit 321, an easily confused category filter 322, and a contrastive learning unit 323.

[0087] The node determination unit 321 can be used to: determine whether the current training node is one of the specified training node, the training node before the specified training node, or the training node after the specified training node. When the current training node is determined to be the specified training node, the easily confused category filter 322 can be triggered to filter easily confused category information. When the current training node is determined to be the training node before the specified training node, the contrast learning unit 323 can be triggered to perform contrast learning on all category information. When the current training node is determined to be the training node after the specified training node, the contrast learning unit 323 can be triggered to perform contrast learning on easily confused category information.

[0088] The easily confused category filter 322 can be used to: identify easily confused category information in sample images when the current training node is a specified training node, for example, in Figure 5 In the example, the features of the proposed target boxes that are easily confused with the real target boxes can be determined from the features of multiple proposed target boxes. The predicted category information and the real category information corresponding to the proposed target box are taken as a set of easily confused category information and output to the contrast learning unit 322.

[0089] The contrastive learning unit 323 can be used to: perform contrastive learning on all category information in the sample image when the current training node is a training node before the specified training node, and perform contrastive learning on the easily confused category information identified by the easily confused category filter 322 when the current training node is a training node after the specified training node.

[0090] In one embodiment, the contrast learning module 320 may further include a classification encoder. The classification encoder can be used to encode the features of the easily confused category information identified by the easily confused category filter 322 to obtain the feature vector of the easily confused category information. The contrast learning unit 323 can perform contrast learning based on the feature vector. The number of classification encoders can be one or more. In actual use, the number of classification encoders can be set according to the requirements. For example, one classification encoder can be set, and each group of easily confused category information can be encoded by the classification encoder. Alternatively, a classification encoder can be set for each group of easily confused category information, and each classification encoder encodes a group of easily confused category information.

[0091] Based on the same technical concept, embodiments of this application also provide a target detection method, such as... Figure 4 As shown, the method may include the following steps S601-S602:

[0092] S601, acquire the image to be detected.

[0093] The image to be detected can be obtained in a variety of ways. For example, an image pre-stored in an image database can be used as the image to be detected, an image can be downloaded from the network, or an image can be acquired in real time by an image acquisition device.

[0094] S602, input the image to be detected into the target detection model, and use the target detection model to detect the category of the target object in the image to be detected.

[0095] The target detection model can be trained using any of the model training methods provided in the embodiments of this application. Using the trained target detection model, the image to be detected can be accurately detected.

[0096] Based on the same technical concept, embodiments of this application also provide a target detection device, such as... Figure 7 As shown, the device may include an image acquisition module 701 and a model training module 702.

[0097] The image acquisition module 701 can be used to acquire multiple sample images;

[0098] The model training module 702 can be used to train the object detection model iteratively multiple times based on multiple sample images. Starting from a specified training node in the multiple iterations of training, the object detection model is used to compare and learn the easily confused category information in the sample images to obtain the trained object detection model.

[0099] In one optional implementation, the model training module 702 can be used to: perform comparative learning on all category information in the sample image at each training node before the specified training node; identify easily confused category information in the sample image at the specified training node; and perform comparative learning on the identified easily confused category information at each training node after the specified training node.

[0100] In one optional implementation, the sample image includes pre-labeled true category information for each target object. Correspondingly, the model training module 702 can be used to: predict the category of each target object in the sample image to obtain predicted category information for each target object; determine whether the predicted category information and the true category information of the same target object are the same; when the predicted category information and the true category information of the same target object are not the same, treat the predicted category information and the true category information of the target object as a set of easily confused category information.

[0101] In an optional implementation, the model training module 702 can be used to: select from multiple predicted category information of the same target object a predicted category information with a confidence level greater than a preset confidence level threshold, and use it as the specified predicted category information of the target object; determine whether the specified predicted category information of the target object and the true category information of the target object are the same; when the specified predicted category information of the target object and the true category information of the target object are not the same, use the specified predicted category information of the target object and the true category information of the target object as a set of easily confused category information.

[0102] In an optional implementation, the model training module 702 can be used to: predict the category of each target object in multiple sample images to obtain the predicted category information of each target object in each sample image; for the same target object in multiple sample images, determine whether the predicted category information of the target object in each sample image and the true category information of the target object in each sample image are the same; when the predicted category information of the target object in each sample image and the true category information of the target object in each sample image are different, determine whether the number of differences is greater than a preset number threshold; when the number of differences is greater than the number threshold, determine that the predicted category information and the true category information of the target object are the same; when the number of differences is less than or equal to the number threshold, determine that the predicted category information and the true category information of the target object are different.

[0103] In one alternative implementation, the specified training node can be the m-th training node in multiple iterations of training, where m can be a times the total number of training nodes in multiple iterations of training, and a is a value between 0.3 and 0.7.

[0104] In one optional implementation, the sample images may include pre-labeled true category information for each target object. Correspondingly, the model training module 702 described above can be used to: predict the category of each target object in the sample images during training to obtain predicted category information for each target object; determine the loss of the predicted category information for each target object relative to the true category information for each target object based on a loss function; and adjust the model parameters of the target detection model based on the loss. The loss function includes a contrastive learning loss function, which is constructed based on the weight information of the predicted category information for each target object, the number of current training iterations, and the information of a specified training node, and is used to determine the contrastive learning loss of the predicted category information in the easily confused category information relative to the true category information in the easily confused category information.

[0105] In an optional implementation, the loss function further includes: a classification loss function, a bounding box regression loss function, and a target object presence / absence judgment loss function, which are used to determine the classification loss, bounding box regression loss, and target object presence / absence judgment loss of the predicted category information of each target object relative to the true category information of each target object, respectively. The loss is determined based on the classification loss, bounding box regression loss, target object presence / absence judgment loss, contrastive learning loss, and the weight information of the contrastive learning loss.

[0106] In one optional implementation, the object detection model includes an object classification module and a contrastive learning module. The object classification module is used to: predict the category of each object in the sample image to obtain the predicted category information of each object; the contrastive learning module is used to: perform contrastive learning on all category information in the sample image at each training node before the specified training node, identify easily confused category information in the sample image at the specified training node, and perform contrastive learning on the identified easily confused category information at each training node after the specified training node.

[0107] In one optional implementation, the contrastive learning module includes: a node determination unit, a confounding category filter, and a contrastive learning unit. The node determination unit is used to: determine whether the current training node is one of the following: a specified training node, a training node preceding the specified training node, or a training node following the specified training node. The confounding category filter is used to: identify confounding category information in the sample image when the current training node is the specified training node. The contrastive learning unit is used to: perform contrastive learning on all category information in the sample image when the current training node is a training node preceding the specified training node, and perform contrastive learning on the confounding category information identified by the confounding category filter when the current training node is a training node following the specified training node.

[0108] Based on the same technical concept, embodiments of this application also provide a target detection device, such as... Figure 8 As shown, the device includes an image acquisition module 801 and a target detection module 802.

[0109] Image acquisition module 801 is used to acquire the image to be detected.

[0110] The object detection module 802 is used to input the image to be detected into the object detection model, and use the object detection model to detect the category of the target object in the image to be detected. The object detection model is trained by any of the model training devices provided in the embodiments of this application.

[0111] The functions of each module in the devices provided in this application embodiment can be referred to the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0112] Based on the same technical concept, embodiments of this application also provide an electronic device, such as... Figure 9 As shown, the electronic device includes a memory 901 and a processor 902. The memory 901 stores a computer program, which is loaded and executed by the processor 902 to implement any of the methods provided in the embodiments of this application. The number of memories 901 and processors 902 can be one or more.

[0113] Optionally, the electronic device may also include:

[0114] The communication interface 903 is used to communicate with external devices and exchange and transmit data.

[0115] If the memory 901, processor 902, and communication interface 903 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The symbol is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0116] Optionally, in a specific implementation, if the memory 901, processor 902, and communication interface 903 are integrated on a single chip, then the memory 901, processor 902, and communication interface 903 can communicate with each other through an internal interface.

[0117] The electronic device that implements the model training method and the electronic device that implements the object detection method can be the same electronic device or different electronic devices.

[0118] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the target detection methods provided in embodiments of this application.

[0119] Based on the same inventive concept, this application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to execute the method provided in this application.

[0120] Based on the same inventive concept, this application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0121] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0122] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0123] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0124] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0125] It should be further understood that the term "comprising" as used in the specification of this application means the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The term "and / or" as used herein includes all or any unit and all combination of one or more associated listed items.

[0126] Any process or method description in the flowchart or otherwise described herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process.

[0127] It is understood that the steps, measures, and solutions in the various operations, methods, and processes discussed in this application can be alternated, modified, combined, or deleted. Furthermore, other steps, measures, and solutions in the various operations, methods, and processes discussed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and solutions in the prior art that are similar to those disclosed in this application can also be alternated, modified, rearranged, decomposed, combined, or deleted.

[0128] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method, characterized in that, The method comprises: obtaining a plurality of sample images; based on the plurality of sample images, iteratively training a target detection model a plurality of times, starting from a specified training node in the plurality of iterations, using the target detection model to perform contrastive learning on confusing class information in the sample images, obtaining a trained target detection model; the target detection model comprises a contrastive learning module, wherein the contrastive learning module comprises a node judgment unit, a confusing class filter and a contrastive learning unit; the node judgment unit is used to determine whether the current training node is one of the specified training node, the training node before the specified training node and the training node after the specified training node; the confusing class filter is used to identify the confusing class information in the sample images when the current training node is the specified training node; the contrastive learning unit is used to perform contrastive learning on all class information in the sample images when the current training node is the training node before the specified training node, and perform contrastive learning on the confusing class information identified by the confusing class filter when the current training node is the training node after the specified training node.

2. The model training method of claim 1, wherein, the method based on a plurality of sample images to iteratively train a target detection model a plurality of times, starting from a specified training node in the plurality of iterations, using the target detection model to perform contrastive learning on confusing class information in the sample images, comprises: at each training node before the specified training node, contrastively learning all class information in the sample images; at the specified training node, identifying the confusing class information in the sample images; at each training node after the specified training node, contrastively learning the identified confusing class information. 3.The model training method of claim 1 or 2, wherein, the sample images include pre-labeled real class information of each target object; the confusing class information is identified by the target detection module in the following way: predicting the class of each target object in the sample images to obtain predicted class information of each target object; determining whether the predicted class information of the same target object and the real class information are the same; when the predicted class information of the same target object and the real class information are not the same, the predicted class information of the target object and the real class information of the target object are taken as a set of confusing class information.

4. The model training method of claim 3, wherein, the method of determining whether the predicted class information of the target object and the real class information of the target object are the same, when the predicted class information of the target object and the real class information of the target object are not the same, the predicted class information of the target object and the real class information of the target object are taken as a set of confusing class information, comprises: selecting, from a plurality of predicted class information of the same target object, a predicted class information with a confidence greater than a preset confidence threshold as a specified predicted class information of the target object; determining whether the specified predicted class information of the target object and the real class information of the target object are the same; When the designated prediction category information of the target object is different from the real category information of the target object, the designated prediction category information of the target object and the real category information of the target object are determined as a set of easily confused category information.

5. The model training method of claim 3, wherein, The prediction of the category of each target object in the sample image comprises: The prediction of the category of each target object in the sample image comprises: The determination of whether the prediction category information and the real category information of the same target object are the same comprises: For the same target object in the plurality of sample images, it is determined whether the prediction category information of the target object in each sample image and the real category information of the target object in each sample image are the same; When the prediction category information of the target object in each sample image and the real category information of the target object in each sample image are different, it is determined whether the number of differences is greater than a preset number threshold; When the number of differences is greater than the number threshold, it is determined that the prediction category information and the real category information of the target object are different. When the number of differences is less than or equal to the number threshold, it is determined that the prediction category information and the real category information of the target object are the same. 6.The model training method of claim 1 or 2, wherein, The mth training node in the plurality of iterative training, m is a multiple of the total number of training nodes in the plurality of iterative training, and a is a value between 0.3 and 0.

7. 7.The model training method of claim 1 or 2, wherein, The sample image comprises pre-labeled real category information of each target object; The plurality of sample images comprises: The prediction of the category of each target object in the sample image comprises: Based on the loss function, the loss of the prediction category information of each target object relative to the real category information of each target object is determined; the loss function comprises a contrast learning loss function, the contrast learning loss function is constructed based on the weight information of the prediction category information of each target object, the number of current iterative training and the information of the designated training node, and is used to determine the contrast learning loss of the prediction category information in the easily confused category information relative to the real category information in the easily confused category information. The model parameters of the target detection model are adjusted based on the loss.

8. The model training method according to claim 7, characterized in that, The loss function further comprises a classification loss function, a target box regression loss function and a target object existence judgment loss function, which are respectively used to determine the classification loss, the target box regression loss and the target object existence judgment loss of the prediction category information of each target object relative to the real category information of each target object. The loss is determined based on the classification loss, the target box regression loss, the target object existence judgment loss, the contrast learning loss and the weight information of the contrast learning loss. 9.The model training method of claim 1 or 2, wherein, The target detection model further comprises a target classification module; The target classification module is used to predict the category of each target object in the sample image to obtain the prediction category information of each target object.

10. A target detection method characterized by, Comprise: Acquire the image to be detected; The image to be detected is input into the target detection model, and the target detection model is used to detect the category of the target object in the image to be detected; The target detection model is trained using the model training method described in any one of claims 1-9.

11. A model training apparatus, comprising: include: The image acquisition module is used to acquire multiple sample images; The model training module is used to perform multiple iterations of training on the target detection model based on the multiple sample images. Starting from a specified training node in the multiple iterations of training, the target detection model is used to compare and learn the easily confused category information in the sample images to obtain the trained target detection model. The target detection model includes a contrastive learning module, wherein the contrastive learning module includes: a node determination unit, an easily confused category filter, and a contrastive learning unit; the node determination unit is used to: determine whether the current training node is one of the specified training node, the training node before the specified training node, and the training node after the specified training node; The easily confused category filter is used to: identify easily confused category information in the sample image when the current training node is the specified training node; the contrastive learning unit is used to: perform contrastive learning on all category information in the sample image when the current training node is a training node before the specified training node, and perform contrastive learning on the easily confused category information identified by the easily confused category filter when the current training node is a training node after the specified training node.

12. A target detection apparatus characterized by comprising: include: The image acquisition module is used to acquire the image to be detected; The target detection module is used to input the image to be detected into the target detection model and use the target detection model to detect the category of the target object in the image to be detected; the target detection model is trained by the model training device as described in claim 11.

13. An electronic device, comprising: include: A memory and a processor, wherein the memory stores a computer program, which is loaded and executed by the processor to implement the method as described in any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the method as described in any one of claims 1-10.

Images