Training method, electronic device, and computer-readable storage medium

Abstract

The application discloses a training method, an electronic device and a computer readable storage medium. The training method comprises: performing enhancement processing on a target feature in a plurality of sample images in a sample image set to obtain a plurality of first target images; training an initial model by using at least part of the first target images to obtain a first model; and training the first model by using at least part of the sample images to obtain a second model. The training method provided by the application can improve the sensitivity of the model to the target feature and ensure the ability of the model to detect the target.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a training method, electronic device, and computer-readable storage medium. Background Technology

[0002] Object detection is one of the most important and challenging branches of computer vision, and it has a wide range of applications in life, such as intelligent monitoring and autonomous driving.

[0003] Currently, neural network-based object detection models are commonly used for object detection. However, during the training process, these models are not trained on the features specific to the object detection task, causing the final model to easily overlook some features that can distinguish between false positives and false negatives. Summary of the Invention

[0004] This application provides a training method, an electronic device, and a computer-readable storage medium that can improve the model's sensitivity to target features and ensure the model's ability to detect targets.

[0005] The first aspect of this application provides a training method, the training method comprising: enhancing target features in multiple sample images in a sample image set to obtain multiple first target images; training an initial model using at least a portion of the multiple first target images to obtain a first model; and training the first model using at least a portion of the sample images in the sample image set to obtain a second model.

[0006] A second aspect of this application provides a detection method, the detection method comprising: inputting an image to be processed into a second model; using the second model to perform target detection on the image to be processed to obtain a target object in the image to be processed; wherein the second model is trained by the above-described training method.

[0007] A third aspect of this application provides an electronic device, which includes a processor, a memory, and a communication circuit. The processor is coupled to the memory and the communication circuit, respectively. The memory stores program data, and the processor executes the program data in the memory to implement the steps in the above method.

[0008] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that can be executed by a processor to implement the steps in the above-described method.

[0009] The beneficial effects are as follows: The training method of this application includes: enhancing the target features in multiple sample images in a sample image set to obtain multiple first target images; training an initial model using at least a portion of the multiple first target images to obtain a first model; and training the first model using at least a portion of the sample images in the sample image set to obtain a second model. This method, on the one hand, utilizes the first target images to train the model, which can improve the model's sensitivity to target features; on the other hand, by using at least a portion of the sample images to train the model, it can ensure the model's ability to perform target detection. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0011] Figure 1 This is a flowchart illustrating one implementation method of the training method of this application;

[0012] Figure 2 yes Figure 1 A flowchart illustrating step S120;

[0013] Figure 3 It is a comparison image of the style image, the original image, and the style-transferred image in an application scenario;

[0014] Figure 4 yes Figure 1 Another part of the implementation method is illustrated in the flowchart;

[0015] Figure 5 This is a flowchart illustrating another implementation of the training method of this application;

[0016] Figure 6 This is a schematic diagram of the structure of the model in this application;

[0017] Figure 7 This is a flowchart illustrating one embodiment of the detection method of this application.

[0018] Figure 8 This is a schematic diagram of the structure of one embodiment of the electronic device of this application;

[0019] Figure 9 This is a schematic diagram of another embodiment of the electronic device of this application;

[0020] Figure 10 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0022] See Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the training method of this application, which includes:

[0023] S110: Obtain a sample image set, which includes multiple sample images.

[0024] S120: Enhance the target features in multiple sample images in the sample image set to obtain multiple first target images.

[0025] This can be achieved by enhancing the target features of a subset of sample images in the sample image set to obtain multiple first target images, or by enhancing the target features of all sample images in the sample image set to obtain multiple first target images.

[0026] The purpose of this application is to enhance the model's learning of target features and prevent the model from ignoring target features during feature extraction. Enhancing target features in sample images means making the target features in the sample images more prominent. This can be achieved through methods such as style transfer processing, filtering, or sharpening, as detailed below.

[0027] In this embodiment, the target feature is the shape feature of the sample image. However, in other embodiments, the target feature can be set to other features such as texture features or color features, depending on the specific detection task, and this is not a limitation here.

[0028] See Figure 2 In this embodiment, step S120 specifically includes:

[0029] S121: Construct a style image set, which includes multiple style images.

[0030] S122: Based on the style image set, perform style transfer processing on multiple sample images to obtain multiple first target images.

[0031] Style transfer processing can present an image of style A in a different style than style A. In other words, it presents the properties of a style image in the sample image, making the target features in the resulting first target image more prominent while weakening other features.

[0032] For example, in Figure 3 In the image, each image in the first column is a style image, each image in the second column is an original image, and each image in the third column is a style-transferred image. The first image in the third column is obtained by performing style transfer on the first image in the second column based on the first image in the first column, and the second image in the third column is obtained by performing style transfer on the second image in the second column based on the second image in the first column, and so on.

[0033] In the process of style transfer processing of sample images, a sample image can be combined with different style images, and different sample images can be combined with the same style image. In summary, this application does not limit the specific process of step S122.

[0034] When performing style transfer processing, the AdaIN (real-time arbitrary style transfer based on adaptive instance normalization) algorithm can be used. This algorithm is existing technology and will not be described in detail here.

[0035] In order to ensure that the target features are obvious in the obtained first target image, in this embodiment, step S121 will screen the style images when constructing the style image set, and select the style images whose target features meet the first preset requirements to add to the style image set.

[0036] Specifically, if the target features meet the first preset requirement, it indicates that the target features in the image are obvious. Therefore, combining the style image whose target features meet the first preset requirement with the sample image achieves the purpose of enhancing the target features in the sample image.

[0037] The first preset requirement can be either that the obviousness of the target feature in the image exceeds a certain threshold, or that the ratio of the area containing the target feature to the total area of ​​the image reaches a certain ratio threshold.

[0038] To improve the efficiency and accuracy of the filtering process, a pre-trained classifier can be used to filter the style images. This classifier divides the input images into two categories: images whose target features meet a first preset requirement, and images whose target features do not meet the first preset requirement. In this embodiment, the classifier specifically divides the input images into shape-type images (images whose shape features meet the first preset requirement) and texture-type images (images whose shape features do not meet the first preset requirement).

[0039] During the training of the classifier, the images are labeled, with lines clearly labeled as shape types and others as texture types. The classifier is then trained using the labeled data.

[0040] It should be noted that in other implementations, methods such as filtering and sharpening can also be used to enhance the target features in the sample image, and this is not a limitation.

[0041] S130: Construct a target set using multiple first target images.

[0042] Add the first target image obtained in step S120 to the target set.

[0043] Alternatively, the same style image filtering method described above can be used to filter the first target image added to the target set. For the filtering process, please refer to the relevant content above, which will not be repeated here.

[0044] S140: Using the first training set composed of multiple first target images in the target set, train the initial model to obtain the first model.

[0045] The first training set includes only the first target image from the target set. The first training set can be constructed using all images from the target set, or it can be constructed using only a portion of the images from the target set.

[0046] The basic framework of the model is established to obtain an initial model. Then, images from the first training set are used as input images to train the initial model until it converges. The trained model is defined as the first model. The model structure can be a RetinaNet object detection model, a YOLO object detection model, etc. The specific structure of the model in this implementation method can be found below.

[0047] By training the model using the first training set, the model's learning of target features can be enhanced, and the model's sensitivity to target features can be improved.

[0048] S150: Train the first model using a second training set that includes at least a number of sample images from the sample image set to obtain the second model.

[0049] After obtaining the first model, the first model is trained using the second training set until the model converges, and the resulting model is defined as the second model.

[0050] The second training set includes at least a number of sample images from the sample image set.

[0051] Since the second training set includes sample images from the sample image set, it can be guaranteed that the trained model has good object detection capabilities.

[0052] In one application scenario, the second training set only includes multiple sample images from the sample image set. In other words, the first model is trained using only the original images (sample images). This can optimize the model's target detection performance, changing the optimization direction of the model from extracting target features to target detection, making the model both sensitive to target features and possessing strong target detection capabilities.

[0053] In another application scenario, the second training set includes not only multiple sample images from the sample image set, but also multiple first target images from the target set. In other words, by using the original images and the images with enhanced target features together to train the first model, the generalization ability of the model can be enhanced, enabling the final model to perform target detection on various images and enhancing the model's target detection capability.

[0054] In this embodiment, after obtaining the first model, it is necessary to test the first model to ensure that it meets the standards. (See attached document.) Figure 4 Between step S140 and step S150, the training method further includes:

[0055] S160: Using multiple test images from the test set, test the first model and obtain the test results for each test image.

[0056] In step S130, after constructing the target set, the target set is divided into the first training set and the test set mentioned above. In step S150, after training the initial model using the first training set, step S160 uses the test set to train the first model obtained from the training.

[0057] Understandably, during the training process, the first model will output a test result for each test image it receives, meaning that there is a one-to-one correspondence between the test image and the test result.

[0058] S170: Based on the test results of each test image, determine whether the first model meets the second preset requirements.

[0059] If satisfied, proceed to step S150; otherwise, proceed to step S180.

[0060] If the first model is determined to meet the second preset requirement, it means that the first model is up to standard. If the first model is determined not to meet the second preset requirement, it means that the first model is not up to standard and needs to be trained again.

[0061] In this embodiment, the test results of the test image include: a first confidence level for evaluating the category of the target object in the test image and a second confidence level for evaluating the strength of the target features of the target object. The higher the first confidence level, the higher the accuracy of detecting the target object; the higher the second confidence level, the more obvious the target features of the target object.

[0062] Specifically, step S170 includes:

[0063] (a1) Determine the recall rate of the first model based on the first confidence level in each test result.

[0064] Recall, also known as the full detection rate, refers to the proportion of predicted positive samples among the actual positive samples. The calculation process of recall is existing technology and will not be described in detail here.

[0065] (b1) Determine whether the first model meets the second preset requirement based on the recall rate and the second confidence level in each test result.

[0066] In one application scenario, step b1 specifically includes:

[0067] (b11) The recall rate and the average of the second confidence level are weighted and summed to obtain the sum.

[0068] First, calculate the average value at the second confidence level, and then determine the sum α using the following formula:

[0069] α = β × recall + γ × shape, where β is the weight corresponding to the recall rate, shape is the average value of the second confidence level, and γ is the weight corresponding to the average value of the second confidence level. β and γ can be set according to different application scenarios and are not restricted here.

[0070] In other implementations, the recall rate can be weighted and summed with the maximum, minimum, or median value of the second confidence level to obtain a sum α, which will not be described in detail here.

[0071] (b12) In response to the sum being greater than or equal to the threshold, determine that the first model meets the second preset requirement.

[0072] The threshold is preset. If the sum α is greater than or equal to the threshold, the first model is determined to meet the second preset requirement, that is, the first model meets the standard. Otherwise, the first model is determined not to meet the second preset requirement.

[0073] It should be noted that in other embodiments, the first model may be determined to meet the second preset requirement based solely on the recall rate of the first model. For example, if the recall rate of the first model is less than the recall rate threshold, the first model is determined not to meet the second preset requirement; otherwise, the first model is determined to meet the second preset requirement.

[0074] S180: Update the first training set, train the first model using the updated first training set, and use the trained model as the first model.

[0075] After step S180, return to step S160.

[0076] When it is determined that the first model does not meet the second preset requirement, the first model is trained using the updated first training set, and the trained model is used as the first model. Then, the process returns to step S160, that is, to continue testing.

[0077] In this embodiment, the step of updating the first training set in step S180 includes:

[0078] (a2) Further enhance the target features in each sample image to obtain multiple second target images.

[0079] (b2) Replace all images in the first training set with multiple second target images.

[0080] In other words, the first model is trained using the image enhanced with new target features, so that the first model meets the target.

[0081] Specifically, when style transfer processing is used to enhance the target features in the sample image, step S180 uses a new style map to perform style transfer processing on the sample image to obtain a new style transfer map. Then, the first model is trained using the new style transfer map.

[0082] The test set can be the same each time the first model is tested.

[0083] It should be noted that steps S160 to S170 are not necessary steps. In other embodiments, step S150 can be executed directly after step S140.

[0084] See Figure 5 , Figure 5 This is a flowchart illustrating another embodiment of the training method of this application, which includes:

[0085] S210: Obtain a sample image set, which includes multiple sample images.

[0086] S220: Enhance the target features in multiple sample images in the sample image set to obtain multiple first target images;

[0087] S230: Construct a target set using multiple first target images.

[0088] S240: Train the initial model using the first training set consisting of multiple first target images in the target set to obtain the first model.

[0089] S250: The first model is trained using a second training set consisting of multiple sample images from the sample image set and multiple first target images from the target set to obtain the second model.

[0090] S260: The second model is trained using a third training set consisting of multiple sample images from the sample image set to obtain the third model.

[0091] Unlike the above implementation, the training phase of this application includes three stages: the first training stage, which trains the model using only the image after target feature enhancement; the second training stage, which trains the model using both the original image (sample image) and the image after target feature enhancement; and the third training stage, which trains the model using only the original image.

[0092] The training process consists of three stages: the first stage is used to enhance the model's learning of target features and improve its sensitivity to these features; the second stage is used to enhance the model's generalization ability, enabling it to perform target detection on various types of images; and the third stage is used to optimize the model's target detection function by shifting the optimization focus from extracting target features to optimizing target detection capabilities, so that the final model combines both target detection and feature-focused characteristics.

[0093] In this embodiment, refer to Figure 6 Each model includes a feature extraction module 110 and a prediction module 120.

[0094] The feature extraction module 110 is used to extract features from the input image to obtain a feature image.

[0095] The prediction module 120, connected to the feature extraction module 110, is used to perform prediction processing on the feature image. Specifically, it includes a first prediction unit 1201, a second prediction unit 1202, and a third prediction unit 1203. The first prediction unit 1201 performs a first prediction process on the feature image to obtain the target location of the target object. The second prediction unit 1202 performs a second prediction process on the feature image to obtain a first confidence level for evaluating the category of the target object. The third prediction unit 1203 performs a third prediction process on the feature image to obtain a third confidence level for evaluating the target feature intensity of the target object. Here, the first confidence level has the same meaning as the first confidence level described above, and the second confidence level also has the same meaning as the second confidence level described above; please refer to the above description for details, which will not be repeated here.

[0096] Compared with existing technologies, this application adds a branch to the model to perform regression prediction of target features.

[0097] In this embodiment, the model adopts the RetinaNet object detection algorithm model. The feature extraction module 110 uses a feature pyramid to extract features from the input image, resulting in three feature maps of different dimensions (the number of feature maps can also be four or five, depending on the specific situation). At this time, the prediction module 120 includes three repeating units 121 and one post-processing unit 122. Each repeating unit 121 includes a first prediction unit 1201, a second prediction unit 1202, and a third prediction unit 1203.

[0098] Each output of the feature pyramid is connected to a repeating unit 121. Each repeating unit 121 performs prediction processing on the received feature map and outputs the target location, a first confidence score, and a second confidence score. The post-processing unit 122 fuses the outputs of all repeating units 121 to obtain the final output result, which includes the target location, the first confidence score, and the second confidence score. The fusion process of the post-processing unit 122 is prior art and will not be described in detail here.

[0099] It should be noted that in other embodiments, when extracting features from the input image, only one feature map may be output. In this case, the prediction module 120 includes only one repeating unit 121.

[0100] In order to further optimize the target detection performance of the model, the learning rate of the first prediction unit 1201 and the second prediction unit 1202 is increased in the third training stage compared with the first training stage and the second training stage. For example, the learning rate of the first prediction unit 1201 and the second prediction unit 1202 is doubled.

[0101] The learning rate is a hyperparameter in the model training process that controls the learning progress of the model. Specifically, the larger the learning rate, the faster the gradient descent can be performed during the model training process, which means the training speed is faster. Therefore, in the third training stage, by increasing the learning rate of the first prediction unit 1201 and the second prediction unit 1202, the training of the model can be accelerated.

[0102] Alternatively, the learning rate of only one of the first prediction unit 1201 and the second prediction unit 1202 can be increased.

[0103] In other words, when training the second model, the learning rate of the first prediction unit 1201 and / or the second prediction unit 1202 is increased.

[0104] For further reference Figure 6 After obtaining the third model, a fusion module 130 will be added to the third model to fuse the first confidence and the second confidence to obtain the third confidence, so that the third model will finally output the target position and the third confidence.

[0105] In other words, during training, the model outputs three values: target location, first confidence level, and second confidence level. After training is complete, when the model is used for target detection, it outputs two values: target location and third confidence level.

[0106] In this embodiment, the process of the fusion module 130 fusing the first confidence level and the second confidence level includes: determining the product of the first confidence level raised to the power of N and the second confidence level raised to the power of M to obtain the third confidence level, wherein the sum of N and M is one, that is, calculating the third confidence level S according to the following formula. det :

[0107] S det =a N ×b M Where N+M=1, a is the first confidence level, and b is the second confidence level.

[0108] Of course, in other implementations, the first confidence level and the second confidence level can be fused by other methods, for example, by determining the product of the first confidence level and the second confidence level to obtain the third confidence level.

[0109] See Figure 7 , Figure 7 This is a flowchart illustrating one embodiment of the detection method of this application, which includes:

[0110] S310: Input the image to be processed into the second model.

[0111] S320: Use the second model to perform target detection on the image to be processed, and obtain the target objects in the image to be processed.

[0112] The second training model is trained using the training method described in any of the above embodiments. Detailed steps regarding the training method can be found in the above embodiments and will not be repeated here. The target object can be any type of object, such as a person, vehicle, or animal; there are no restrictions.

[0113] See Figure 8 , Figure 8 This is a schematic diagram of one embodiment of the electronic device of this application. The electronic device 200 includes a processor 210, a memory 220, and a communication circuit 230. The processor 210 is coupled to the memory 220 and the communication circuit 230 respectively. The memory 220 stores program data. The processor 210 executes the program data in the memory 220 to implement the steps in any of the above embodiments. The detailed steps can be found in the above embodiments and will not be repeated here.

[0114] The electronic device 200 can be any device with algorithm processing capabilities, such as a computer or mobile phone, and there are no restrictions on it.

[0115] See Figure 9 , Figure 9 This is a schematic diagram of one embodiment of the electronic device of this application. The electronic device 300 includes an enhancement module 310, a first training module 320, and a second training module 330.

[0116] The enhancement module 310 is used to enhance the target features in multiple sample images in the sample image set to obtain multiple first target images.

[0117] The first training module 320 is connected to the enhancement module 310 and is used to train the initial model using at least some of the first target images from a plurality of first target images to obtain the first model.

[0118] The second training module 330 is connected to the first training module 320 and is used to train the first model using at least a portion of the sample images in the sample image set to obtain the second model.

[0119] The electronic device 300 can be any device with algorithm processing capabilities, such as a computer or mobile phone, and is not limited thereto. When the electronic device 300 is in operation, it executes the method steps of any of the above embodiments. Detailed steps can be found in the above embodiments and will not be repeated here.

[0120] See Figure 10 , Figure 10This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 400 stores a computer program 410, which can be executed by a processor to implement the steps in any of the above methods.

[0121] Specifically, the computer-readable storage medium 400 can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a device that can store the computer program 410. Alternatively, it can be a server that stores the computer program 410, which can send the stored computer program 410 to other devices for execution, or it can run the stored computer program 410 itself.

[0122] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A training method characterized by, The method includes: Enhancement processing is performed on target features in multiple sample images in a sample image set to obtain multiple first target images; the target features in the first target images meet a first preset requirement, the first preset requirement includes: the obviousness of the target features in the first target images exceeds a degree threshold, and / or the ratio of the region of the target features in the first target images to the total area of ​​the first target images reaches a ratio threshold. The step of enhancing the target features in multiple sample images in the sample image set to obtain multiple first target images includes: constructing a style image set, the style image set including multiple style images that meet the first preset requirements; and performing style transfer processing on the multiple sample images according to the style image set to obtain multiple first target images. Using at least a portion of the first target images from the plurality of first target images, an initial model is trained to obtain a first model; The first model is trained using at least a portion of the sample images in the sample image set to obtain the second model.

2. The method of claim 1, wherein, Before training the first model using at least a portion of the sample images in the sample image set to obtain the second model, the method further includes: Using at least a portion of the first target images among the plurality of first target images as test images, the first model is tested to obtain test results for each of the test images, wherein the first target images used as test images are different from the first target images used to train the initial model; Based on the test results of each of the test images, determine whether the first model meets the second preset requirement; If the conditions are met, then the step of training the first model using at least a portion of the sample images in the sample image set is performed; If the conditions are not met, the target features in at least a portion of the sample images in the sample image set are enhanced again to obtain multiple second target images. The first model is trained using the multiple second target images, and the trained model is used as the first model. Then, the process returns to the step of testing the first model using at least a portion of the multiple first target images as test images.

3. The method of claim 2, wherein, The test results of the test image include: a first confidence level for evaluating the category of the target object in the test image and a second confidence level for evaluating the strength of the target feature of the target object; The step of determining whether the first model meets the second preset requirement based on the test results of each of the test images includes: The recall rate of the first model is determined based on the first confidence level in each of the test results. Based on the recall rate and the second confidence level in each of the test results, determine whether the first model meets the second preset requirement.

4. The method of claim 3, wherein, The step of determining whether the first model meets the second preset requirement based on the recall rate and the second confidence level in each of the test results includes: The recall rate and the average of the second confidence levels are weighted and summed to obtain the sum value; In response to the sum being greater than or equal to a threshold, it is determined that the first model satisfies the second preset requirement.

5. The method of claim 1, wherein, The step of training the first model using at least a portion of the sample images in the sample image set to obtain the second model includes: The first model is trained using at least a portion of the sample images in the sample image set and at least a portion of the first target images in the plurality of first target images to obtain the second model.

6. The method of claim 5, wherein, After obtaining the second model, the process further includes: The second model is trained using at least a portion of the sample images in the sample image set to obtain the third model.

7. The method of claim 6, wherein, Each model includes: The prediction module includes a first prediction unit, a second prediction unit, and a third prediction unit. The first prediction unit is used to predict the position of a target object in the input image to obtain the target position. The second prediction unit is used to predict the category of the target object to obtain a first confidence level. The third prediction unit is used to predict the intensity of the target feature of the target object to obtain a second confidence level.

8. The method according to claim 7, characterized in that, When training the second model, increase the learning rate of the first prediction unit and / or the second prediction unit.

9. The method according to claim 7, characterized in that, After obtaining the third model, the following is also included: A fusion module is added to the third model to fuse the first confidence level and the second confidence level to obtain a third confidence level, so that the third model finally outputs the target position and the third confidence level.

10. The method according to claim 9, characterized in that, The fusion module is specifically used to: determine the product of the first confidence level raised to the power of N and the second confidence level raised to the power of M to obtain the third confidence level, wherein the sum of N and M is one.

11. A detection method, characterized in that, include: Input the image to be processed into the second model; The second model is used to perform target detection on the image to be processed to obtain the target object in the image to be processed. The second model is obtained by training using the training method described in any one of claims 1-10.

12. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a communication circuit. The processor is coupled to the memory and the communication circuit. The memory stores program data. The processor executes the program data in the memory to implement the steps of the method as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the steps of the method as described in any one of claims 1-11.

Images