An object detection method and a training method for a classification model

Abstract

This application relates to the field of machine vision technology, providing an object detection method and a training method for a classification model. The object detection method includes acquiring an image to be detected and determining a region of interest (ROI) within the image; inputting the ROI into multiple base learners in a classification model, and using the base learners to output predicted classification scores for each candidate label corresponding to the ROI; training each base learner based on sample features of different feature types; determining the classification label corresponding to the ROI from the candidate labels based on the predicted classification scores; determining the location information corresponding to the ROI; and outputting the detection result of the image to be detected based on the classification label and location information. This method can accurately determine the corresponding predicted classification score based on the features of the image to be detected using each base learner, thereby accurately determining the classification label, thus improving the accuracy of object detection.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to an object detection method, apparatus, a training method for a classification model, a terminal device, and a computer-readable storage medium. Background Technology

[0002] In recent years, object detection technology has been widely used in various industries. Object detection technology refers to the technique of identifying and locating objects in an image using a pre-trained detection model. With the rapid development of deep learning technology and the support of a large number of training samples, object detection technology has made great progress.

[0003] In practice, target detection can be performed on images in complex traffic scenarios to determine the detection results; or target detection can be performed on PCB (Printed Circuit Board) images in complex work scenarios to identify faulty or damaged PCBs. The images to be detected may be acquired in scenarios with complex interference factors. Complex traffic scenarios include extreme lighting conditions (strong light), diverse traffic scenarios (highways or rural roads, etc.), and adverse weather conditions (rain or fog, etc.); complex work scenarios include concentrated lighting on the PCB, the PCB being in motion, and limitations imposed by the shooting angle. Because the occurrence of interference factors is relatively rare, the number of training samples corresponding to complex interference factors is also limited. This leads to an imbalance in the distribution of sample categories when training the detection model, resulting in poor generalization of the trained detection model and low accuracy in target detection of images with complex interference factors.

[0004] Therefore, how to improve the accuracy of target detection is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a target detection method, apparatus, a training method for a classification model, a terminal device, and a computer-readable storage medium, with the aim of improving the accuracy of target detection.

[0006] Firstly, this application provides a target detection method. The method includes:

[0007] Acquire the image to be detected and determine the region of interest in the image to be detected;

[0008] The regions of interest are respectively input into multiple base learners in the classification model, and the multiple base learners respectively output the predicted classification scores of each candidate label corresponding to the regions of interest; each base learner is trained based on sample features of different feature types;

[0009] Based on the predicted classification scores, determine the classification label corresponding to the region of interest from each of the candidate labels;

[0010] Determine the location information corresponding to the region of interest;

[0011] The detection result of the image to be detected is output based on the classification label and the location information.

[0012] In one embodiment, the inputs and outputs of the multiple base learners are sequentially connected; the step of inputting the region of interest into the multiple base learners in the classification model, and using the multiple base learners to output the predicted classification scores of each candidate label corresponding to the region of interest, includes:

[0013] For any non-first base learner among the multiple base learners in the classification model, the region of interest and the output of the previous base learner are input into the base learner, and the predicted classification score corresponding to each candidate label is output; the output result is the predicted classification score output by the previous base learner for each candidate label.

[0014] In one embodiment, determining the location information corresponding to the region of interest includes:

[0015] Determine the initial coordinates of the reference box corresponding to the region of interest;

[0016] Multiple regressors are used to iteratively calculate based on the region of interest and the initial coordinates to obtain the location information corresponding to the region of interest.

[0017] In one embodiment, acquiring the image to be detected and determining the region of interest in the image to be detected includes:

[0018] Acquire the image to be detected;

[0019] The multi-scale feature map of the image to be detected is determined using a feature extraction network;

[0020] The multi-scale feature maps are fused to obtain a fused feature map;

[0021] The region of interest in the image to be detected is determined based on the fused feature map.

[0022] In one embodiment, the step of fusing the multi-scale feature maps to obtain a fused feature map includes:

[0023] For each feature map in the multi-scale feature map, an intermediate feature map corresponding to the feature map is determined based on the feature map and a preset multiple upsampling layer;

[0024] For each intermediate feature map, a fused feature map corresponding to the intermediate feature map is determined based on the intermediate feature map and a preset downsampling layer.

[0025] In one embodiment, determining the multi-scale feature map of the image to be detected using a feature extraction network includes:

[0026] Determine the scene type of the image to be detected;

[0027] Determine the feature extraction network corresponding to the scene type;

[0028] The image to be detected is input into the feature extraction network, and features are extracted using the feature extraction network to obtain a multi-scale feature map.

[0029] Secondly, this application also provides a method for training a classification model. The classification model includes multiple base learners, and the method includes:

[0030] Each base learner is trained based on training samples of different feature types to obtain a corresponding base learner; the output of the base learner is the predicted classification score of each candidate label corresponding to the region of interest of the image to be detected;

[0031] The classification model is determined based on each of the base learners.

[0032] In one embodiment, multiple base learners are sequentially connected, and the training samples based on different feature types are used to train each base learner to obtain a corresponding base learner, including:

[0033] For any base learner that is not the first one, the base learner is trained based on the training samples of different feature types and the sample output results of the previous base learner to obtain the corresponding base learner; the training samples include regions of interest and sample labels; the previous base learner is the base learner connected to the previous base learner; the sample output results are the predicted classification scores output by the previous base learner for each sample label.

[0034] In one embodiment, the step of training the base learner based on training samples of different feature types and the sample output results of the previous base learner for any base learner other than the first one, to obtain the corresponding base learner, includes:

[0035] For any base learner that is not the first one, the classification weight of the base learner for the training sample is determined based on the sample features of the training sample and the sample output of the previous base learner.

[0036] The base learner is obtained by learning and training based on the region of interest and the classification weights.

[0037] In one embodiment, the step of learning and training based on the region of interest and the classification weights to obtain a corresponding base learner includes:

[0038] The classification error of the base learner is determined based on the region of interest, the sample labels corresponding to the predicted classification scores, and the classification weights.

[0039] The model parameters of the base learner are adjusted based on the classification error to determine the corresponding base learner.

[0040] Thirdly, this application also provides a target detection device. The device includes:

[0041] An acquisition module is used to acquire an image to be detected and determine the region of interest in the image to be detected;

[0042] The score determination module is used to input the region of interest into multiple base learners in the classification model, and use the multiple base learners to output the predicted classification scores of each candidate label corresponding to the region of interest; each base learner is trained based on sample features of different feature types.

[0043] The classification module is used to determine the classification label corresponding to the region of interest from the candidate labels based on the predicted classification scores.

[0044] The location determination module is used to determine the location information corresponding to the region of interest;

[0045] The output module is used to output the detection result of the image to be detected based on the classification label and the location information.

[0046] Fourthly, this application also provides a terminal device. The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0047] Fifthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described above.

[0048] This application provides an object detection method. After determining the region of interest (ROI) of an image to be detected, the ROI is input into multiple base learners in a classification model. Each base learner outputs a predicted classification score for a candidate label corresponding to the ROI. Based on these predicted scores, the corresponding classification label is determined from the candidate labels. Since each base learner is trained on sample features of different feature types, this method can accurately determine the corresponding predicted classification score based on the features of the image to be detected, and then accurately determine the classification label based on the predicted classification score. Therefore, this method can improve the accuracy of object detection.

[0049] It is understood that the target detection device, the classification model training method, the terminal device, and the computer-readable storage medium provided in the embodiments of this application have the same beneficial effects as the target detection method described above, and will not be repeated here. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0051] Figure 1 A flowchart of a target detection method provided in an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of the structure of a regressor provided in an embodiment of this application;

[0053] Figure 3 This is a schematic diagram of another target detection method provided in an embodiment of this application;

[0054] Figure 4 A flowchart illustrating a training method for a classification model provided in an embodiment of this application;

[0055] Figure 5 This is a schematic diagram of the structure of a base learner provided in an embodiment of this application;

[0056] Figure 6 This is a schematic diagram of the structure of a target detection device provided in an embodiment of this application;

[0057] Figure 7 A schematic diagram of the structure of a training device for a classification model provided in an embodiment of this application;

[0058] Figure 8 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0059] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0060] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0061] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0062] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0063] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0064] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. "A plurality" means "two or more."

[0065] The target detection method provided in this application embodiment can be executed by the processor of a terminal device when running a corresponding computer program.

[0066] Figure 1 The flowchart illustrates a target detection method provided in this embodiment. For ease of explanation, only the parts relevant to this embodiment are shown. The method provided in this embodiment includes the following steps:

[0067] S110: Acquire the image to be detected and determine the region of interest in the image to be detected.

[0068] The image to be detected is the image that needs to be detected; the image to be detected can be an image corresponding to a traffic scene, or a PCB image collected in an industrial operation scene, etc. This embodiment does not limit the specific type of the image to be detected.

[0069] In practice, the image to be detected can be acquired directly, or the image frame to be detected can be determined from the video data corresponding to the monitoring scene after acquisition, thus determining the corresponding image to be detected.

[0070] In image processing, the region of interest (ROI) refers to the area in the image to be detected, delineated using shapes such as rectangles, circles, ellipses, or irregular polygons. In other words, the ROI is a selected region within the image to be detected, and this region is the focus of object detection. In practice, one or more ROIs can be determined from the image to be detected; this embodiment does not limit this and can be set according to actual needs.

[0071] S120: Input the region of interest into multiple base learners in the classification model, and use the multiple base learners to output the predicted classification scores of each candidate label corresponding to the region of interest; each base learner is trained according to the sample features of different feature types.

[0072] In this embodiment, multiple base learners (individual learners) are pre-trained. Each base learner learns different feature types of sample features of the training samples during the learning and training process. That is, each pre-trained base learner is "good but different". Then, the base learners are combined according to a certain combination strategy to form a strong learner, which is the classification model.

[0073] After identifying the region of interest (ROI), the ROI of the image to be detected is input into each base learner in the classification model. Each base learner then outputs a predicted classification score for each candidate label corresponding to the ROI. In other words, for each classification label corresponding to the ROI, base learners of different feature types output corresponding predicted classification scores based on their pre-trained sample features.

[0074] S130: Determine the classification label corresponding to the region of interest from each candidate label based on each predicted classification score.

[0075] Specifically, after each base learner outputs the predicted classification score corresponding to each candidate label, a weighted average is calculated for each type of candidate label using the corresponding predicted classification scores to obtain the calculation result; the calculation results corresponding to each candidate label are compared to determine the classification label corresponding to the region of interest from each candidate label.

[0076] S140: Determine the location information corresponding to the region of interest.

[0077] The location information refers to the coordinates of the reference box corresponding to the region of interest (ROI); that is, the location information of the detected target within the ROI in the image to be detected. Specifically, after determining the ROI in the image to be detected, the ROI is input into a bounding-box regression unit. The regression unit outputs the regression coordinates corresponding to the ROI, thus determining the location information of the ROI. In practice, a four-dimensional vector (x, y, w, h) is generally used to represent the regression coordinates, where x, y, w, and h represent the coordinates of the center point, width, and height of the reference box, respectively.

[0078] S150: Output the detection results of the image to be detected based on the classification label and location information.

[0079] Specifically, after determining the classification label and location information corresponding to the region of interest, the corresponding markers can be directly set on the image to be detected, and the marked image can be output to obtain the detection result of the target detection in the image to be detected; alternatively, the classification label and location information can be encoded into other formats for output to determine the target detection result; this embodiment does not limit the specific method of outputting the detection result.

[0080] This application provides an object detection method. After determining the region of interest (ROI) of an image to be detected, the ROI is input into multiple base learners in a classification model. Each base learner outputs a predicted classification score for a candidate label corresponding to the ROI, and the classification label corresponding to the ROI is determined from the candidate labels based on the predicted classification scores. Since each base learner is trained based on sample features of different feature types, this method can accurately determine the corresponding predicted classification score based on the features of the image to be detected, and then accurately determine the classification label based on the predicted classification score. Therefore, this method can improve the accuracy of object detection.

[0081] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, the inputs and outputs of multiple base learners are connected sequentially; the region of interest is input into multiple base learners in the classification model, and the multiple base learners output the predicted classification scores of each candidate label corresponding to the region of interest, including:

[0082] For any non-first base learner among multiple base learners in the classification model, the region of interest and the output of the previous base learner are input into the base learner, and the predicted classification score corresponding to each candidate label is output; the output result is the predicted classification score output by the previous base learner for each candidate label.

[0083] In this embodiment, the inputs and outputs of multiple base learners are sequentially connected, meaning the output of the previous base learner is connected to the input of the next base learner. For any base learner that is not the first one in the classification model, the process of outputting a predicted classification score includes: outputting the region of interest and the output of the previous base learner (the one sequentially connected to this base learner) to this base learner; the previous base learner outputting the predicted classification score for each candidate label; and the base learner outputting the predicted classification score corresponding to each candidate label based on the region of interest and the output. After determining the predicted classification score corresponding to each candidate label, the base learner determines the output of the next base learner sequentially connected to it and inputs this output to the next base learner.

[0084] As can be seen, the method of this embodiment can accurately determine the predicted classification score corresponding to each candidate label using a classification model, thereby improving the accuracy of the determined classification label corresponding to the region of interest.

[0085] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, determining the location information corresponding to the region of interest includes:

[0086] Determine the initial coordinates of the reference box corresponding to the region of interest;

[0087] Multiple regressors are used to perform iterative calculations based on the region of interest and initial coordinates to obtain the location information corresponding to the region of interest.

[0088] Specifically, in this embodiment, multiple regressors are pre-set, and the regressors are connected sequentially. Figure 2 This is a schematic diagram of a regressor structure provided in an embodiment of this application; as shown, each regressor includes a small number of convolutional layers; after determining the initial coordinates of the reference box corresponding to the region of interest, multiple regressors are used to perform iterative calculations based on the region of interest and the initial coordinates to obtain the regression coordinates (position information) corresponding to the region of interest; the iterative calculation process is as follows:

[0089]

[0090] Where R(·) represents the nonlinear computation in the neural network, b represents the initial coordinates of the reference box corresponding to the region of interest, M is the number of all regressors, and x represents the feature map of the region of interest being regressed. Each regressor R... m Optimize from the previous regressor R m-1 The coordinates b of the reference frame m-1 .

[0091] For each regressor, the corresponding nonlinear computation includes: passing the features of the cropped region of interest through a small number of convolutional layers, a normalization layer (using normalization methods such as Batch Normalization flexibly according to the specific situation), and an activation layer, then flattening them, and finally passing them through a fully connected layer to obtain the position information of the reference box.

[0092] This embodiment uses a progressive regression method to efficiently and accurately determine the location information of the region of interest in the image to be detected.

[0093] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, acquiring the image to be detected and determining the region of interest in the image to be detected includes:

[0094] Acquire the image to be detected;

[0095] A feature extraction network is used to determine the multi-scale feature map of the image to be detected;

[0096] Feature fusion is performed on the multi-scale feature maps to obtain a fused feature map;

[0097] The region of interest in the image to be detected is determined based on the fused feature map.

[0098] In this embodiment, the feature extraction network refers to a network used to extract features from the image to be detected. This embodiment does not limit the specific type of feature extraction network; for example, it may include ResNet50, Swin-T, and MobileNet v3. In this embodiment, the image to be detected is input into the feature extraction network, which then outputs a multi-scale feature map corresponding to the image to be detected.

[0099] Among them, multi-scale feature maps refer to image features at different levels, such as low-level features and high-level features. It should be noted that in the determined multi-scale feature maps, low-level features have higher resolution and contain more location and detail information, but due to fewer convolutions, their semantics are lower and there is more noise. High-level features have stronger semantic information, but their resolution is very low and their ability to perceive details is poor.

[0100] Feature fusion refers to the operation of leveraging the complementarity between low-level and high-level features, combining the advantages of different features, and thus improving model performance. In this embodiment, after determining the multi-scale feature maps, feature fusion is performed on the multi-scale feature maps to obtain a fused feature map.

[0101] Specifically, the scale and ratio of anchor boxes can be determined based on prior knowledge or by performing K-means clustering analysis on images of the corresponding scene type, resulting in different anchor boxes. In this embodiment, the side length of the anchor box can be {4, 8, 16, 32}. Then, a preset anchor box is assigned to each pixel in the fused feature map, and the feature map corresponding to each anchor box is input into the pooling layer for region of interest pooling. The size of the features is fixed, and anchor boxes with an Intersection over Union (IoU) greater than a preset threshold are determined. These anchor boxes are the reference boxes corresponding to the region of interest, thus determining the region of interest.

[0102] This embodiment determines a multi-scale feature map of the image to be detected and performs feature fusion on the multi-scale feature map to obtain a fused feature map. Based on the fused feature map, the region of interest in the image to be detected is determined. This can effectively extract and highlight the features corresponding to each scale of the image to be detected. Target detection based on this region of interest can improve the accuracy of target detection.

[0103] In one embodiment, feature fusion is performed on the multi-scale feature maps to obtain a fused feature map, including:

[0104] For each feature map in the multi-scale feature map, an intermediate feature map corresponding to the feature map is determined based on the feature map and the upsampling layer with a preset multiple.

[0105] For each intermediate feature map, a fused feature map corresponding to the intermediate feature map is determined based on the intermediate feature map and the preset downsampling layer.

[0106] Specifically, after determining the multi-scale feature maps of the image to be detected, for each feature map in the multi-scale feature maps, a preset upsampling layer corresponding to the feature map is first determined, and then an intermediate feature map corresponding to the feature map is determined based on the feature map and the preset upsampling layer; for each intermediate feature map, a preset downsampling layer corresponding to the intermediate feature map is first determined, and then a fusion feature map corresponding to the intermediate feature map is determined based on the intermediate feature map and the preset downsampling layer; in this example, the preset upsampling layer can be a double upsampling layer, and the preset downsampling layer can be a double downsampling layer.

[0107] By determining the fused feature map according to the method of this embodiment, more features in the image to be detected can be obtained, thereby improving the accuracy of target detection.

[0108] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, a feature extraction network is used to determine the multi-scale feature map of the image to be detected, including:

[0109] Determine the scene type of the image to be detected;

[0110] Determine the feature extraction network corresponding to the scene type;

[0111] The image to be detected is input into a feature extraction network, which is used to extract features and obtain a multi-scale feature map.

[0112] The scene type refers to the scene corresponding to the image to be detected. If the image to be detected is a traffic scene image, the scene type includes street scenes of different complexity, traffic road scenes of different complexity, and residential area scenes of different complexity. If the image to be detected is a PCB image, the scene type includes work scenes of different complexity.

[0113] In practice, feature extraction networks of different types are pre-determined for different scenario types. For example, for complex street scenarios, deep backbone networks such as Swin-T can be used for feature extraction; for simple residential area scenarios, lighter backbone networks such as Mobilenet v3 can be used for feature extraction; or for complex work scenarios, deep backbone networks such as Swin-T can be used for feature extraction; and for simple work scenarios, lighter backbone networks such as Mobilenet v3 can be used for feature extraction.

[0114] Specifically, after determining the feature extraction network corresponding to the scene type, the image to be detected is input into the feature extraction network, and the feature extraction network is used to extract features to obtain a multi-scale feature map.

[0115] In this embodiment, a feature extraction network corresponding to the scene type of the image to be detected is used to extract features from the image to be detected, which can improve the efficiency of determining multi-scale feature maps and improve the efficiency of edge calculation.

[0116] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be described in detail below with reference to practical application scenarios. Figure 3 The diagram shown illustrates another target detection process provided in this embodiment of the application. The specific steps of a target detection method in this embodiment are as follows:

[0117] Acquire the image to be detected in the work scenario. (RGB channels); The image to be detected is a PCB image containing complex interference factors;

[0118] The image to be detected is input into a feature extraction network (such as ResNet50), and feature maps from layers 1 to 5 of ResNet101, i.e., Layer 1, Layer 2, Layer 3, Layer 4, and Layer 5, are extracted to obtain multi-scale feature maps {x}. 1 ,x 2 ,x 3 ,x 4 ,x 5}, denoted as To filter out certain image noise, feature map x is deleted. 1 ,x2 To obtain multi-scale feature maps

[0119] Feature fusion is performed using a feature mixing module based on reverse path aggregation to obtain a fused feature map. In a specific embodiment, the process of determining the fused feature map is as follows:

[0120] Multi-scale feature maps The input is fed into a feature mixing module based on reverse path aggregation. In the top-down feature aggregation path, high-level features are used to enhance low-level features, resulting in an intermediate feature map. Specifically as follows:

[0121]

[0122] Where conv(·) represents convolution calculation, and u(·) represents a double upsampling layer.

[0123] In the bottom-up feature aggregation path, low-level features are used to enhance high-level features, resulting in a fused feature map. Specifically as follows:

[0124]

[0125] Where conv(·) represents convolution calculation, which has a larger stride than the convolution calculation when calculating intermediate feature maps, and d(·) represents a double downsampling layer.

[0126] Then, the fused feature map is input into the region candidate network to determine the region of interest, and the region of interest is cropped from the image to be detected. The cropped feature map is then input into the ensemble classification module (classification model) and the progressive regression module, respectively.

[0127] An analysis of existing PCB image sets corresponding to various work scenarios was conducted. K-means clustering was used to determine the side lengths of the anchor boxes as {4, 8, 16, 32}. In the region candidate network, a preset anchor box was assigned to each pixel in the fused feature map. The feature maps corresponding to each anchor box were then input into a pooling layer for region of interest pooling. With the feature size fixed, anchor boxes with an intersection-union ratio (IU) greater than a preset threshold were identified. These anchor boxes are the reference boxes corresponding to the region of interest, thus determining the region of interest.

[0128] The region of interest is input into the ensemble classification module. Each base learner in the ensemble classification module outputs a predicted classification score corresponding to each candidate label. Based on the predicted classification scores, the classification label corresponding to the region of interest is determined from the candidate labels. Specifically, each base learner obtains the predicted classification score output by the previous base learner for each candidate label. Based on the minimum value of the obtained predicted classification scores and the region of interest, the predicted classification score corresponding to each candidate label is output.

[0129] The region of interest (ROI) is input into the progressive regression module to determine the initial coordinates of the reference box corresponding to the ROI. Multiple regressors within the progressive regression module then perform iterative calculations based on the ROI and the initial coordinates to obtain the location information corresponding to the ROI. Current technical solutions share a fully connected layer between the classification and regression tasks, leading to decreased prediction performance and inaccurate regression. This embodiment provides a target detection method that decouples the classification and regression tasks, using a progressive regression module independent of the integrated classification module to determine the location information of the detected target, thereby improving the accuracy and efficiency of location determination.

[0130] Based on the classification labels and location information, the corresponding detection results are marked on the image to be inspected and the marked image is output to identify PCB images with faults, damage, or abnormal solder joints.

[0131] This application provides an object detection method. After determining the region of interest (ROI) of an image to be detected, the ROI is input into multiple base learners in a classification model. Each base learner outputs a predicted classification score for a candidate label corresponding to the ROI, and the classification label corresponding to the ROI is determined from the candidate labels based on the predicted classification scores. Since each base learner is trained based on sample features of different feature types, this method can accurately determine the corresponding predicted classification score based on the features of the image to be detected, and then accurately determine the classification label based on the predicted classification score. Therefore, this method can improve the accuracy of object detection.

[0132] This application also provides a method for training a classification model, wherein the classification model in this embodiment includes multiple base learners. Figure 4 The flowchart illustrates a training method for a classification model provided in this embodiment of the application. The training method for a classification model provided in this embodiment includes the following steps:

[0133] S410: Train each base learner based on training samples of different feature types to obtain the corresponding base learner; the output of the base learner is the predicted classification score of each candidate label corresponding to the region of interest of the image to be detected;

[0134] S420: Determine the classification model based on each base learner.

[0135] Specifically, the classification model in this embodiment includes multiple base learners. Each base learner in the classification model is trained using training samples of different feature types to obtain a corresponding base learner. The input of each trained base learner is the image to be detected, and the output is the predicted classification score of each candidate label corresponding to the region of interest in the image to be detected. Based on the determination of each base learner, the corresponding classification model is determined according to the trained base learners.

[0136] The classification model is trained according to the training method provided in this embodiment, and the target detection is performed based on the classification model, which can perform target detection comprehensively and accurately, thereby improving the accuracy of target detection.

[0137] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Specifically, in this embodiment, multiple base learners are connected sequentially, and each base learner is trained based on training samples of different feature types to obtain the corresponding base learner, including:

[0138] For any base learner other than the first one, the base learner is trained based on training samples of different feature types and the sample output of the previous base learner to obtain the corresponding base learner; the training samples include regions of interest and sample labels; the previous base learner is the base learner connected to the previous base learner; the sample output is the predicted classification score output by the previous base learner for each sample label.

[0139] It is understandable that the feature types of the training samples are different, and the feature types of the sample features learned by the base learner based on the training samples will correspond to different feature types. In this embodiment, for any base learner other than the first one, the sample output result is obtained. The sample output result is the predicted classification score output by the previous base learner for each sample label. The training samples with different feature types and the sample output results are input into the base learner. The base learner learns and trains based on the training samples and the sample output results, and outputs the predicted classification score corresponding to each sample label.

[0140] According to the method of this embodiment, for any base learner that is not the first base learner, the sample output results of the previous base learner are used to constrain the base learner, enhance the weight of sample features that the previous base learner did not learn well, and avoid the base learners from becoming homogeneous; therefore, using the corresponding classification model for object detection can improve the accuracy of object detection.

[0141] In a specific embodiment, for any base learner other than the first one, the base learner is trained based on training samples of different feature types and the sample output results of the previous base learner to obtain the corresponding base learner, including:

[0142] For any base learner that is not the first base learner, the classification weight of the base learner for the training samples is determined based on the sample features of the training samples and the sample output of the previous base learner.

[0143] The base learner is obtained by learning and training based on the region of interest and classification weights.

[0144] In this embodiment, for the first base learner, training samples are input into it for training. The base learner determines its classification weights for the training samples based on their features, and then trains based on the region of interest and the classification weights to obtain the corresponding base learner. For any base learner other than the first one, training samples and their output results are input into it. The base learner determines its classification weights for the training samples based on their features and the output results of the previous base learner, and then trains based on the region of interest and the classification weights to obtain the corresponding base learner.

[0145] Figure 5 This is a schematic diagram of a base learner provided in an embodiment of this application; each base learner includes at least one fully connected layer and at least one activation layer; specifically, in each base learner, the region of interest is first flattened and then fed into a hybrid network of fully connected layers and convolutional layers: the sample type of the training sample is determined, including foreground samples and background samples; the predicted classification score of the previous base learner connected to the base learner corresponding to each sample label of the training sample is obtained; the classification weight of the training sample is determined according to the sample type of the training sample and the classification score of the previous sample.

[0146] Specifically, for the j-th training sample of the i-th base learner, its classification weights are represented as follows:

[0147]

[0148] Where ω (ω≥0) represents sensitivity; c represents the sample label; F represents the predicted classification score of the previous base learner i-1 for sample label c; F represents the foreground sample set, and B represents the background sample set.

[0149] According to the method of this embodiment, the base learner for different learning features is determined by adjusting the classification weight of each base learner on the training samples. The operation is convenient and easy to implement. Moreover, each base learner is different from the other base learners, which reduces the prediction variance of multiple base learners. Furthermore, each base learner can enhance the learning of features corresponding to complex interference factors by strengthening the classification weights of the training samples that the previous base learner did not learn well. Therefore, it can improve the robustness and generalization of the detection model and improve the accuracy of target detection.

[0150] In one embodiment, a base learner is obtained by learning and training based on the region of interest and classification weights, including:

[0151] The classification error of the base learner is determined based on the region of interest, the sample labels corresponding to the predicted classification scores, and the classification weights.

[0152] The model parameters of the base learner are adjusted based on the classification error to determine the corresponding base learner.

[0153] Specifically, the classification error of the base learner is first determined based on the region of interest, the sample labels corresponding to the predicted classification scores, and the classification weights. After determining the classification error of the base learner in the current training stage, the loss function is determined based on the classification error. The model parameters of the base learner are adjusted based on the loss function. When the base learner achieves the expected effect, the base learner in the corresponding training stage is determined as the final trained base learner.

[0154] For the first base learner in the sequentially connected base learners, all its corresponding regions of interest are set to 1, and the classification error of the i-th base learner is... It can be represented as:

[0155]

[0156] Where k represents the reference box corresponding to the region of interest in the training sample; K represents the number of reference boxes in the training sample; Let represent the predicted classification score of base learner i for the sample label c corresponding to reference box k, and This represents the classification label of the base learner i for the sample label c corresponding to the reference box k; C represents the number of categories of the sample label.

[0157] After determining each base learner according to the method of this embodiment, when performing target detection on the image to be detected, each base learner is used to determine the predicted classification score corresponding to each candidate label in the image to be detected that corresponds to the region of interest. For each candidate label, the classification score corresponding to that candidate label is obtained by weighted averaging of the predicted classification scores corresponding to that candidate label. By comparing the size relationship of the classification scores of each candidate label, the classification label corresponding to the region of interest is determined from multiple candidate labels. Specifically, the calculation of the classification score of the candidate label... The calculation method is as follows:

[0158]

[0159] Where N represents the number of base learners.

[0160] The base learners are trained according to the method of this embodiment. Each base learner enhances the classification weights corresponding to training samples that the previous base learner did not learn well, thereby enhancing the learning of features corresponding to complex interference factors. Therefore, it can improve the robustness and generalization of the detection model and improve the accuracy of target detection.

[0161] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0162] Figure 6 The diagram shown is a structural schematic of a target detection device provided in an embodiment of this application. Figure 6 As shown, the target detection device in this embodiment includes an acquisition module 610, a score determination module 620, a classification module 630, a position determination module 640, and an output module 660; wherein,

[0163] The acquisition module 610 is used to acquire the image to be detected and determine the region of interest in the image to be detected;

[0164] The score determination module 620 is used to input the region of interest into multiple base learners in the classification model, and use the multiple base learners to output the predicted classification scores of each candidate label corresponding to the region of interest; each base learner is trained according to sample features of different feature types.

[0165] Classification module 630 is used to determine the classification label corresponding to the region of interest from each candidate label based on each predicted classification score;

[0166] The location determination module 640 is used to determine the location information corresponding to the region of interest;

[0167] The output module 650 is used to output the detection results of the image to be detected based on the classification label and location information.

[0168] The target detection device provided in this application embodiment has the same beneficial effects as the target detection method described above.

[0169] In one embodiment, the inputs and outputs of multiple base learners are connected sequentially; the score determination module 620 includes:

[0170] The prediction score determination submodule is used to input the region of interest and the output of the previous base learner into any base learner among multiple base learners in the classification model, and output the predicted classification score corresponding to each candidate label. The output result is the predicted classification score output by the previous base learner for each candidate label.

[0171] In one embodiment, the location determination module 640 includes:

[0172] The information determination submodule is used to determine the initial coordinates of the reference box corresponding to the region of interest;

[0173] The location information determination submodule is used to perform iterative calculations based on the region of interest and initial coordinates using multiple regressors to obtain the location information corresponding to the region of interest.

[0174] In one embodiment, the acquisition module 610 includes:

[0175] The image acquisition submodule is used to acquire the image to be detected;

[0176] The feature extraction submodule is used to determine the multi-scale feature map of the image to be detected using a feature extraction network.

[0177] The feature fusion submodule is used to fuse features from multi-scale feature maps to obtain a fused feature map.

[0178] The region determination submodule is used to determine the region of interest in the image to be detected based on the fused feature map.

[0179] In one embodiment, the feature fusion submodule includes:

[0180] The first feature fusion unit is used to determine the intermediate feature map corresponding to each feature map in the multi-scale feature map based on the feature map and the upsampling layer with a preset multiple.

[0181] The second feature fusion unit is used to determine the fused feature map corresponding to each intermediate feature map based on the intermediate feature map and the preset downsampling layer.

[0182] In one embodiment, the feature extraction submodule includes:

[0183] The scene determination unit is used to determine the scene type of the image to be detected;

[0184] The network determination unit is used to determine the feature extraction network corresponding to the scene type;

[0185] The feature extraction unit is used to input the image to be detected into the feature extraction network, and use the feature extraction network to extract features to obtain a multi-scale feature map.

[0186] Figure 7 The diagram shown is a structural schematic of a training device for a classification model provided in an embodiment of this application. Figure 7 As shown, a training apparatus for a classification model in this embodiment includes a model training module 710 and a model determination module 720; wherein,

[0187] The model training module 710 is used to train each base learner based on training samples of different feature types to obtain the corresponding base learner; the output of the base learner is the predicted classification score of each candidate label corresponding to the region of interest of the image to be detected;

[0188] The model determination module 720 is used to determine the classification model based on each base learner.

[0189] In one embodiment, multiple base learners are sequentially connected, and the model training module 710 includes:

[0190] The model training submodule is used to train any base learner (not the first one) based on training samples of different feature types and the output of the previous base learner to obtain the corresponding base learner. The training samples include regions of interest and sample labels. The previous base learner is the base learner connected to the previous base learner. The output is the predicted classification score output by the previous base learner for each sample label.

[0191] In one embodiment, the model training submodule includes:

[0192] The first determining unit is used to determine the classification weight of the base learner for the training sample based on the sample features of the training sample and the output of the previous base learner for any base learner other than the first base learner.

[0193] The second determining unit is used to learn and train based on the region of interest and classification weights to obtain the corresponding base learner.

[0194] In one embodiment, the second determining unit includes:

[0195] The first determining subunit is used to determine the classification error of the base learner based on the region of interest, the sample labels corresponding to the predicted classification scores, and the classification weights.

[0196] The second determining subunit is used to adjust the model parameters of the base learner based on the classification error and determine the corresponding base learner.

[0197] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0198] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0199] Figure 8 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 8 As shown, the terminal device 800 of this embodiment includes a memory 801, a processor 802, and a computer program 803 stored in the memory 801 and executable on the processor 802. When the processor 802 executes the computer program 803, it implements the steps in the above-described object detection method embodiments or the steps in the training method embodiments of the various classification models, or when the processor 802 executes the computer program 803, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 6 The functions of the acquisition module 610, score determination module 620, classification module 630, position determination module 640, and output module 650 shown are as follows: Figure 7 The functions of the model training module 710 and the model determination module 720 are shown.

[0200] For example, computer program 803 can be divided into one or more modules / units, one or more of which are stored in memory 801 and executed by processor 802 to implement the method of this application embodiment. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 803 in terminal device 800. For example, computer program 803 can be divided into an acquisition module, a score determination module, a classification module, a position determination module, and an output module, with the specific functions of each module as follows:

[0201] The acquisition module is used to acquire the image to be detected and determine the region of interest in the image to be detected;

[0202] The score determination module is used to input the region of interest into multiple base learners in the classification model, and use the multiple base learners to output the predicted classification score of each candidate label corresponding to the region of interest; each base learner is trained based on sample features of different feature types.

[0203] The classification module is used to determine the classification label corresponding to the region of interest from each candidate label based on each predicted classification score;

[0204] The location determination module is used to determine the location information corresponding to the region of interest;

[0205] The output module is used to output the detection results of the image to be detected based on the classification label and location information.

[0206] In applications, terminal device 800 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. Terminal device 800 may include, but is not limited to, memory 801 and processor 802. Those skilled in the art will understand that... Figure 8 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine certain components, or different components. For example, a terminal device may also include input / output devices, network access devices, buses, etc.; among which, input / output devices may include cameras, audio acquisition / playback devices, displays, etc.; network access devices may include communication modules for wireless communication with external devices.

[0207] In applications, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0208] In applications, memory can be an internal storage unit of a terminal device, such as its hard drive or RAM; it can also be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card; or it can include both internal and external storage units. Memory is used to store operating systems, applications, boot loaders, data, and other programs, such as computer program code. Memory can also be used to temporarily store data that has been output or will be output.

[0209] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0210] The computer-readable storage medium provided in this application embodiment has the same beneficial effects as the target detection method described above.

[0211] This application implements all or part of the processes in the methods of the above embodiments, which can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0212] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0213] Those skilled in the art will recognize that the device and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0214] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interface, or the device may be indirectly coupled or communicated, and may be electrical, mechanical, or other forms.

[0215] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A target detection method, characterized in that, The method includes: Acquire the image to be detected and determine the region of interest in the image to be detected; The region of interest (ROI) is input into multiple base learners in a classification model, and each base learner outputs a predicted classification score for each candidate label corresponding to the ROI. Each base learner is trained based on sample features of different feature types. The inputs and outputs of the multiple base learners are connected sequentially. The process of inputting the ROI into multiple base learners in a classification model and outputting predicted classification scores for each candidate label corresponding to the ROI includes: for any base learner other than the first one in the classification model, inputting the ROI and the output of the previous base learner into that base learner, and outputting a predicted classification score corresponding to each candidate label; the output is the predicted classification score output by the previous base learner for each candidate label. Based on the predicted classification scores, determine the classification label corresponding to the region of interest from each of the candidate labels; Determine the location information corresponding to the region of interest; The detection result of the image to be detected is output based on the classification label and the location information.

2. The method according to claim 1, characterized in that, The determination of the location information corresponding to the region of interest includes: Determine the initial coordinates of the reference box corresponding to the region of interest; Multiple regressors are used to iteratively calculate based on the region of interest and the initial coordinates to obtain the location information corresponding to the region of interest.

3. The method according to claim 1, characterized in that, The step of acquiring the image to be detected and determining the region of interest in the image to be detected includes: Acquire the image to be detected; The multi-scale feature map of the image to be detected is determined using a feature extraction network; The multi-scale feature maps are fused to obtain a fused feature map; The region of interest in the image to be detected is determined based on the fused feature map.

4. The method according to claim 3, characterized in that, The step of fusing features from the multi-scale feature maps to obtain a fused feature map includes: For each feature map in the multi-scale feature map, an intermediate feature map corresponding to the feature map is determined based on the feature map and a preset multiple upsampling layer; For each intermediate feature map, a fused feature map corresponding to the intermediate feature map is determined based on the intermediate feature map and a preset downsampling layer.

5. The method according to claim 3, characterized in that, The step of determining the multi-scale feature map of the image to be detected using a feature extraction network includes: Determine the scene type of the image to be detected; Determine the feature extraction network corresponding to the scene type; The image to be detected is input into the feature extraction network, and features are extracted using the feature extraction network to obtain a multi-scale feature map.

6. A method for training a classification model, characterized in that, The classification model includes multiple base learners, and the method includes: Each base learner is trained using training samples of different feature types to obtain a corresponding base learner. The output of each base learner is the predicted classification score of each candidate label corresponding to the region of interest in the image to be detected. Multiple base learners are sequentially connected. The process of training each base learner using training samples of different feature types to obtain a corresponding base learner includes: for any base learner other than the first one, training the base learner based on training samples of different feature types and the sample output of the previous base learner to obtain a corresponding base learner. The training samples include regions of interest and sample labels. The previous base learner is the base learner connected to the previous base learner. The sample output is the predicted classification score output by the previous base learner for each sample label. The classification model is determined based on each of the base learners.

7. The method according to claim 6, characterized in that, The step of training a base learner based on training samples of different feature types and the sample output of the previous base learner for any base learner other than the first one, to obtain the corresponding base learner, includes: For any base learner that is not the first one, the classification weight of the base learner for the training sample is determined based on the sample features of the training sample and the sample output of the previous base learner. The base learner is obtained by learning and training based on the region of interest and the classification weights.

8. The method according to claim 7, characterized in that, The step of learning and training based on the region of interest and the classification weights to obtain the corresponding base learner includes: The classification error of the base learner is determined based on the region of interest, the sample labels corresponding to the predicted classification scores, and the classification weights. The model parameters of the base learner are adjusted based on the classification error to determine the corresponding base learner.

Images