Methods, apparatus, computer-readable media and electronic equipment for screening difficult samples

Abstract

This application provides a method, apparatus, computer-readable medium, and electronic device for screening difficult samples. The method includes: inputting a target image into a target neural network, the target neural network including a feature extraction network, a detection head, and a quality calibration module; extracting image information feature maps of the target image through the feature extraction network, and inputting the image information feature maps into the detection head and the quality calibration module; generating a detection result feature map corresponding to the image information feature map through the detection head, and inputting the detection result feature map into the quality calibration module; and performing a fusion and mapping operation on the image information feature map and the detection result feature map through the quality calibration module to obtain a quality prediction score, and determining whether the target image is a difficult sample based on the quality prediction score. This application can save manpower and time costs in screening difficult samples. This application can be applied to various scenarios such as mapping, autonomous driving, intelligent transportation, and assisted driving.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus, computer-readable medium, and electronic device for screening difficult samples. Background Technology

[0002] Difficult samples are those that, after being detected by the algorithm model, are confirmed to contain targets that were missed during the initial detection process.

[0003] With the popularization of deep learning-based artificial intelligence technology, more and more business scenarios and applications have greater demands for intelligent algorithm models. Typically, a mature algorithm model requires training on millions of data points to reach the deployment standard. Even after the algorithm model is deployed, there will still be difficult samples that are hard to detect correctly.

[0004] However, existing methods for screening difficult samples usually involve manual sorting, which is not only inefficient but also requires a lot of manpower. Summary of the Invention

[0005] The embodiments of this application provide a method, apparatus, computer-readable medium, and electronic device for screening difficult samples, which can at least to some extent improve the screening efficiency of difficult samples and save labor costs.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0007] According to one aspect of the embodiments of this application, a method for screening difficult samples is provided. The method includes: inputting a target image to be detected into a target neural network, the target neural network including a feature extraction network, a detection head connected to the feature extraction network, and a quality calibration module connected to the feature extraction network and the detection head respectively; extracting an image information feature map of the target image through the feature extraction network, and inputting the image information feature map into the detection head and the quality calibration module respectively; generating a detection result feature map corresponding to the image information feature map through the detection head, and inputting the detection result feature map into the quality calibration module, the detection result feature map being used to characterize the position and category of the target in the target image; performing a fusion and mapping operation on the input image information feature map and the detection result feature map through the quality calibration module to obtain a quality prediction score, and determining whether the target image is a difficult sample based on the quality prediction score, wherein the difficult sample is an image containing a target not detected by the target neural network.

[0008] According to one aspect of the embodiments of this application, a hard sample screening device is provided. The device includes: an image input unit for inputting a target image to be detected into a target neural network, the target neural network including a feature extraction network, a detection head connected to the feature extraction network, and a quality calibration module connected to the feature extraction network and the detection head respectively; a feature extraction and input unit for extracting an image information feature map of the target image through the feature extraction network and inputting the image information feature map into the detection head and the quality calibration module respectively; a generation and input unit for generating a detection result feature map corresponding to the image information feature map through the detection head and inputting the detection result feature map into the quality calibration module, the detection result feature map being used to characterize the position and category of the target in the target image; and a determination unit for performing a fusion and mapping operation on the input image information feature map and the detection result feature map through the quality calibration module to obtain a quality prediction score, and determining whether the target image is a hard sample based on the quality prediction score, wherein the hard sample is an image containing a target not detected by the target neural network.

[0009] In some embodiments of this application, based on the foregoing scheme, the feature extraction network includes a backbone network and a feature pyramid network connected to the backbone network. The feature extraction and input unit is configured to: extract a basic semantic feature map of the target image through the backbone network and input the basic semantic feature map into the feature pyramid network; and generate an image information feature map containing high-level semantic feature information of the image through the feature pyramid network based on the basic semantic feature map, wherein the feature depth of the high-level semantic feature information of the image is higher than the feature depth of the basic semantic feature information of the image in the basic semantic feature map.

[0010] In some embodiments of this application, based on the aforementioned scheme, the determining unit is configured to: stitch the image information feature map and the detection result feature map together to obtain a stitched feature map; fuse the stitched feature map and reduce the dimensionality of the fusion result to the target dimension; and map the dimensionality-reduced fusion result to a quality prediction score output.

[0011] In some embodiments of this application, based on the aforementioned scheme, the determining unit is configured to: fuse the concatenated feature map sequentially through a first convolutional layer and a modified linear unit activation function layer; reduce the dimensionality of the fusion result to a predetermined dimension through a second convolutional layer to obtain a fusion result of a predetermined dimension; and reduce the dimensionality of the fusion result of the predetermined dimension to a target dimension through an average pooling layer.

[0012] In some embodiments of this application, based on the foregoing scheme, before inputting the target image to be detected into the target neural network, the image input unit is further configured to: acquire a training sample set, the training sample set including multiple training samples and ground truth labels corresponding to each training sample; input the training samples in the training sample set into the original neural network to obtain the quality prediction score output by the quality calibration module of the original neural network and the detection result feature map generated by the detection head of the original neural network; determine the loss value of the detection task loss according to the ground truth labels corresponding to the training samples and the detection result feature map; perform post-processing operations on the detection result feature map to obtain the detection result, the detection result including detection boxes for marking the detected targets; determine the true quality score according to the ground truth labels corresponding to the training samples and the detection result, and determine the loss value of the quality calibration loss according to the true quality score and the quality prediction score; determine the loss value of the training loss according to the loss value of the detection task loss and the loss value of the quality calibration loss, and train the original neural network according to the loss value of the training loss to obtain the target neural network.

[0013] In some embodiments of this application, based on the foregoing scheme, the image input unit is configured to: determine the proportion of correctly detected targets among all targets in the training sample according to the ground truth label corresponding to the training sample and the detection result, and use this as the true quality score.

[0014] In some embodiments of this application, based on the foregoing scheme, after determining whether the target image is a difficult sample according to the quality prediction score, the determining unit is further configured to: if the target image is a difficult sample, store the target image in the target database; generate new training samples according to the images in the target database, and add the new training samples back to the training sample set; and continue to optimize the target neural network according to the training sample set.

[0015] In some embodiments of this application, based on the foregoing scheme, the target neural network includes multiple detection heads and multiple quality calibration modules. The feature extraction and input unit is configured to: extract image information feature maps of the target image that are respectively paired with the multiple detection heads through the feature extraction network, and input each image information feature map into the corresponding detection head and quality calibration module respectively; the determination unit is configured to: perform fusion and mapping operations on the input image information feature maps and detection result feature maps through each quality calibration module to obtain the quality prediction value output by each quality calibration module; and determine the quality prediction score based on the quality prediction value output by each quality calibration module.

[0016] In some embodiments of this application, based on the foregoing scheme, the determining unit is configured to: determine the average value of the quality prediction values ​​output by each quality calibration module as the quality prediction score.

[0017] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the difficult sample screening method as described in the above embodiments.

[0018] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the hard sample screening method as described in the above embodiments.

[0019] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, a processor of a computer device reading the computer instructions from the computer-readable storage medium, and the processor executing the computer instructions, causing the computer device to perform the hard sample screening method as described in the above embodiments.

[0020] In some embodiments of this application, the technical solutions include a feature extraction network, a detection head connected to the feature extraction network, and a quality calibration module connected to both the feature extraction network and the detection head. When the target image to be detected is input into the target neural network, the feature extraction network extracts image information feature maps, the detection head generates detection result feature maps corresponding to these feature maps, and the quality calibration module outputs a quality prediction score based on the input image information feature maps and the detection result feature maps. This allows for the determination of whether the target image is a difficult sample based on the quality prediction score. Therefore, the technical solutions provided in this application achieve online automated screening of difficult samples, significantly improving the efficiency of difficult sample screening and saving manpower and time costs.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0023] Figure 1 A schematic diagram of the process for manually picking difficult samples in related technologies is shown;

[0024] Figure 2 A schematic diagram of the process for selecting difficult samples based on an expert model in related technologies is shown;

[0025] Figure 3 A schematic diagram of the process for picking difficult samples based on active learning in related technologies is shown;

[0026] Figure 4 The diagram illustrates the detection results and missed detections of a target detection model according to an embodiment of this application.

[0027] Figure 5 The detection results and a schematic diagram of missed detections of a target detection model according to another embodiment of this application are shown;

[0028] Figure 6 A schematic diagram of an exemplary system architecture that can be used to implement the technical solutions of the embodiments of this application is shown;

[0029] Figure 7 A flowchart of a difficult sample screening method according to an embodiment of this application is shown;

[0030] Figure 8 A network structure diagram of a target neural network according to an embodiment of this application is shown;

[0031] Figure 9 A flowchart illustrating the extraction of image information feature maps of a target image using a feature extraction network according to an embodiment of this application is shown.

[0032] Figure 10 A flowchart illustrating a process for fusing and mapping input image information feature maps and detection result feature maps via a quality calibration module according to an embodiment of this application is shown.

[0033] Figure 11 A network structure diagram of a quality calibration module according to an embodiment of this application is shown;

[0034] Figure 12 An embodiment according to this application is shown. Figure 7A flowchart detailing steps 720 and 740 in the embodiment;

[0035] Figure 13 A schematic diagram of multiple quality calibration modules corresponding to multiple detection heads according to an embodiment of this application is shown;

[0036] Figure 14 A flowchart illustrating the training of a target neural network according to an embodiment of this application is shown;

[0037] Figure 15 A flowchart illustrating the training process of a target neural network according to an embodiment of this application is shown;

[0038] Figure 16 A schematic diagram illustrating the determination of a true quality score according to an embodiment of this application is shown;

[0039] Figure 17 An embodiment according to this application is shown. Figure 7 A flowchart of the steps following step 740 in the embodiment;

[0040] Figure 18 A schematic diagram of a process for quality assessment based on a target neural network according to an embodiment of this application is shown;

[0041] Figure 19 A block diagram of a difficult sample screening apparatus according to an embodiment of this application is shown;

[0042] Figure 20 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0043] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0044] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0045] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0046] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0047] Bad cases are images containing objects that have not been detected by the object detection model.

[0048] In related technologies, there are mainly three schemes for screening difficult samples:

[0049] Option 1: Manually pick out difficult samples.

[0050] Figure 1 A flowchart illustrating the process of manually picking difficult samples in related technologies is shown. Please refer to [link / reference]. Figure 1 As shown, the process of manually picking difficult samples is as follows: First, a large number of detection results are collected in the actual deployment environment of the network model; then, the quality of the detection results is judged by manual picking offline, and the difficult samples are handed over to manual labeling; finally, the model is retrained based on the labeled difficult samples to update the model.

[0051] Option 2: Selecting difficult samples based on expert models.

[0052] Another common approach is to train an expert model to help the deployed model identify difficult samples. Expert models typically have high computing power, but real-world deployment environments (such as in-vehicle systems and mobile phones) often struggle to support such high-performance models. Therefore, the process of selecting difficult samples based on an expert model generally follows... Figure 2 As shown. (Through) Figure 2It can be determined that the method of selecting difficult samples based on expert models essentially involves comparing the detection results of expert models with those of the actual deployed network models. Its advantage is the introduction of automation, avoiding manual intervention in the selection process and improving efficiency. However, the normal operation of the entire process requires that the detection accuracy of the expert model is significantly higher than that of the network model in the actual deployment environment. This is not always guaranteed. For example, in tasks with limited data, expert models, due to their larger number of parameters, are often more prone to overfitting, resulting in lower accuracy compared to the network model in the actual deployment environment. Furthermore, if the computing power of the actual deployment environment is comparable to that of the offline deployment environment, the expert model may not have an advantage in accuracy compared to the actual model. Under these two conditions, the quality of the difficult samples selected in this way cannot be guaranteed.

[0053] Option 3: Picking difficult samples based on active learning.

[0054] Figure 3 A schematic diagram of the process for picking difficult samples based on active learning in related technologies is shown. Please refer to [link / reference]. Figure 3 As shown, the selection of difficult samples based on active learning is implemented in the network model of the actual deployment environment. Specifically, deep learning-based tasks typically have a prediction confidence level. For example, in object detection, different predicted objects usually carry a confidence value between 0 and 1 to represent the certainty of the prediction. During the learning process, the confidence value of some simple targets usually approaches 1, while that of some more difficult-to-predict targets is unlikely to approach 1. Therefore, in general active learning schemes, researchers usually use whether a sample contains low-confidence predictions to define whether the sample is a difficult sample. This method can achieve difficult sample selection in the actual deployment environment without transmitting data offline or requiring expert model access, thus saving some computational resources. However, in practice, the confidence score of this method is not equivalent to the prediction quality. Therefore, the accuracy of hard samples selected using this scheme needs to be verified. In addition, not all deep learning-based tasks have this confidence prediction. Therefore, if the task does not have this confidence, the scheme of selecting hard samples based on active learning will fail. For example, in face recognition, the network usually only outputs the feature vector (embedding) of a face and does not predict the confidence.

[0055] Therefore, the relevant technology has the following drawbacks:

[0056] 1. Manually picking out difficult samples consumes a lot of manpower and time, has low screening efficiency, and cannot achieve automated screening.

[0057] 2. In the method of selecting difficult samples based on expert models, the expert model needs to backfeed the collected data in the deployment environment and then complete the screening offline, which cannot be done online; at the same time, since expert models usually have large computing power and large computing resource requirements, the applicable conditions are limited.

[0058] 3. The method of picking difficult samples based on active learning only uses the prior of the network prediction confidence to determine difficult samples. This confidence may not accurately reflect the prediction quality of the sample, and it cannot work in tasks without prediction confidence.

[0059] To address this, this application first provides a method for hard sample screening. The hard sample screening method provided by the embodiments of this application can overcome the aforementioned shortcomings. On the one hand, it can solve the problem of manpower consumption in traditional bad case screening schemes, improve screening efficiency, and reduce screening costs. On the other hand, based on deep learning, it uses a quality calibration calculation branch to determine the prediction quality of the network, achieving online hard sample screening with only a small increase in computation. This overcomes the limitation of expert models being unable to screen online, and is applicable to all deep learning tasks. It also overcomes the limitation of active learning being unsuitable for tasks without prediction confidence.

[0060] Figure 4 The diagram illustrates the detection results and missed detections of a target detection model according to an embodiment of this application. Figure 5 The diagram illustrates the detection results and missed detections of a target detection model according to another embodiment of this application. Please refer to... Figure 4 and Figure 5 As shown, Figure 4 and Figure 5 Each road scene image is shown as a hard sample for the object detection model. In the road object detection scenario, the solid bounding boxes of the road scene images represent the prediction results of the object detection model, namely vehicles and people, respectively, while the entities within the dashed bounding boxes are the objects that the object detection model did not detect.

[0061] Figure 6 A schematic diagram of an exemplary system architecture that can be used to implement the technical solutions of the embodiments of this application is shown. Figure 6 As shown, the system architecture 600 includes a vehicle 610 and a cloud 620, which can communicate with each other. Specifically, the vehicle 610 includes a camera 630 for capturing images of the environment surrounding the vehicle. The vehicle 610 also houses a target neural network for object detection in the environmental images. The target neural network includes a feature extraction network, a detection head connected to the feature extraction network, and a quality calibration module connected to both the feature extraction network and the detection head. When the difficult sample screening method provided in this embodiment is applied... Figure 6In the system architecture shown, a process can be as follows: First, the vehicle 610 starts, and the camera 630 of the vehicle 610 captures environmental images of the environment around the vehicle 610. The environmental images are then input into the target neural network, where a feature extraction network and a detection head connected to the feature extraction network identify targets in the surrounding environment to achieve assisted driving. After the environmental images are input into the target neural network, the target neural network also obtains a corresponding quality prediction score through the feature extraction network, the detection head, and the quality calibration module. Finally, the vehicle 610 determines whether the captured environmental image is a difficult sample based on the quality prediction score. If the environmental image is a difficult sample, it sends the environmental image to the cloud 620 to achieve the screening of difficult samples.

[0062] In some embodiments of this application, vehicle 610 periodically sends newly screened difficult samples to cloud 620.

[0063] In some embodiments of this application, after receiving an environmental image, the cloud 620 sends the environmental image to the annotator for annotation. The cloud 620 collects the annotated environmental image and sends it to the vehicle 610. The vehicle 610 uses the annotated environmental image as a training sample to continue to optimize and train the target neural network.

[0064] It should be understood that Figure 6 The number of vehicles and cameras shown in the image is merely illustrative. Depending on the implementation requirements, there can be any number of vehicles, and any number of cameras can be installed at various locations on the vehicles.

[0065] It should be noted that, Figure 6 The example shown is only one embodiment of this application. Although Figure 6 The solution in this embodiment is for the field of assisted driving, with the target neural network deployed on a vehicle. However, in other embodiments of this application, the solution can also be applied to various other fields, such as automated sorting scenarios in intelligent logistics, where the target neural network can also be deployed on other types of terminal devices. Although Figure 6 In the embodiments of this application, the target neural network is used for target detection tasks. However, in other embodiments of this application, the target neural network can also be used for other types of tasks such as semantic segmentation. Figure 6 In the embodiment of the scheme, the target neural network is deployed on the vehicle. However, in other embodiments of this application, the target neural network can also be deployed in the cloud. That is, the vehicle can send the captured environmental images to the cloud, where hard samples are screened and the labeled hard samples are used to optimize the target neural network. Although in Figure 6In one embodiment, the labeled environmental image is sent from the cloud to the vehicle, where the target neural network is optimized. However, in other embodiments of this application, the target neural network can be deployed in both the cloud and the vehicle. The cloud can collect environmental images sent by multiple vehicles, optimize the target neural network based on the labeled environmental images, and then send the optimized target neural network to the vehicle. This application does not limit the scope of protection of this application in any way.

[0066] As is readily understood, the difficult sample screening method provided in this application embodiment is generally executed by a user terminal, and correspondingly, the difficult sample screening device is generally located in the user terminal. However, in other embodiments of this application, the server may also have similar functions to the user terminal, thereby executing the difficult sample screening scheme provided in this application embodiment.

[0067] Therefore, the embodiments of this application can be applied to terminals or servers. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any restrictions.

[0068] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0069] Figure 7 A flowchart of a hard sample screening method according to an embodiment of this application is shown. This hard sample screening method can be executed by various computing and processing devices, such as user terminals or cloud servers. User terminals include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. Embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving. Please refer to... Figure 7 As shown, this difficult sample screening method includes at least the following steps:

[0070] In step 710, the target image to be detected is input into the target neural network, which includes a feature extraction network, a detection head connected to the feature extraction network, and a quality calibration module connected to the feature extraction network and the detection head respectively.

[0071] The target image to be detected can be an image or photograph in various formats such as .jpeg and .bmp, or it can be a video frame contained in a video. Typically, the target image can be obtained by capturing a photograph, but in some cases, it can also be automatically generated by computer equipment.

[0072] Target images typically contain various categories of targets. A target is an entity or object recorded in the target image, located at a specific position within the image. A target image can contain one or more targets. For example, if the target image is a road scene image, the targets could be people, vehicles, etc., within the road scene image; or, if the target image is a logistics sorting scene image, the targets could be various objects within the logistics sorting scene image. The feature extraction network and detection head in the target neural network are modules used in conventional network detection processes. The quality calibration module is an additional computational branch added in this application, used to evaluate the quality prediction score q∈[0,1] of the detection results.

[0073] Figure 8 A network structure diagram of a target neural network according to an embodiment of this application is shown. Please refer to... Figure 8 As shown, the feature extraction network may include a backbone network and feature pyramid networks (FPNs) connected to the backbone network. The target neural network also includes a head and a quality calibration module.

[0074] In step 720, the image information feature map of the target image is extracted by the feature extraction network, and the image information feature map is input into the detection head and the quality calibration module respectively.

[0075] Figure 9 A flowchart illustrating the extraction of image information feature maps of a target image using a feature extraction network according to an embodiment of this application is shown. Figure 9 As shown, extracting image information feature maps from a target image using a feature extraction network can specifically include the following steps:

[0076] In step 721, the basic semantic feature map of the target image is extracted through the backbone network and then input into the feature pyramid network.

[0077] Please see Figure 8 As shown, after the image to be detected is input into the backbone network, it undergoes multiple convolution operations to obtain the basic semantic feature map.

[0078] In step 722, an image information feature map containing high-level semantic feature information of the image is generated by the feature pyramid network based on the basic semantic feature map. The feature depth of the high-level semantic feature information of the image is higher than the feature depth of the basic semantic feature information of the image in the basic semantic feature map.

[0079] Please continue reading Figure 8 As shown, after receiving the input basic semantic feature map, the feature pyramid network processes it and outputs an image information feature map. The image information feature map contains more semantic information than the basic semantic feature map.

[0080] In step 730, a detection result feature map corresponding to the image information feature map is generated by the detection head, and the detection result feature map is input into the quality calibration module. The detection result feature map is used to characterize the position and category of the target in the target image.

[0081] The detection head is used for user-preset detection tasks, such as object detection and semantic segmentation. The detection head outputs the detection result features. Figure 1 Generally, it is a tensor with height, width, and channel dimensions. Taking the detection head used for target detection as an example, the detection result feature map output by the detection head can be a feature map of size H×W×C, where H and W are the height and width, respectively, and C is the channel. The values ​​on the channel represent different categories of the target and the pixel position of the target.

[0082] In step 740, the quality calibration module performs fusion and mapping operations on the input image information feature map and the detection result feature map to obtain a quality prediction score, and determines whether the target image is a hard sample based on the quality prediction score. A hard sample is an image that contains a target that has not been detected by the target neural network.

[0083] The quality prediction score can be used to obtain the hard sample screening results, which indicate whether the target image is a hard sample.

[0084] In one embodiment of this application, determining whether a target image is a difficult sample based on a quality prediction score includes: if the quality prediction score is less than a predetermined quality prediction score threshold, then determining that the target image is a difficult sample; if the quality prediction score is above the predetermined quality prediction score threshold, then determining that the target image is not a difficult sample.

[0085] The predetermined quality prediction score threshold can be set based on experience, and can be set to a decimal between 0 and 1, for example, 0.3.

[0086] Since quality prediction scores are not always perfectly accurate, setting a predetermined quality prediction score threshold can increase the likelihood that the identified hard samples are indeed the correct hard samples.

[0087] Please continue reading Figure 8 As shown, the detection quality score output by the quality calibration module is the quality prediction score γ.

[0088] This application embodiment enables the network to predict the quality of detection results by setting a quality calibration module in the target neural network. The input to the quality calibration module is the image information feature map and the detection result feature map, and the output is a quality prediction score q used to measure the quality of the detection results. Theoretically, after obtaining the image information and detection result information, the quality calibration module possesses sufficient conditions to predict the detection quality. The quality calibration module can be represented as:

[0089] q = Qualify(img context ,det context |θ)

[0090] Where Qualify represents the quality calibration function, which consists of a quality calibration module that performs quality prediction mapping of detection results combined with image semantics. θ represents the weights of the quality calibration module. context For image information feature maps, det context Here is the feature map of the detection result, and q is the quality prediction score.

[0091] Figure 10 A flowchart illustrating the fusion and mapping operation of an input image information feature map and a detection result feature map via a quality calibration module according to an embodiment of this application is shown. Please refer to... Figure 10 As shown, the fusion and mapping operation of the input image information feature map and the detection result feature map through the quality calibration module may specifically include the following steps:

[0092] In step 1010, the image information feature map and the detection result feature map are stitched together to obtain the stitched feature map.

[0093] Figure 11 A network structure diagram of a quality calibration module according to an embodiment of this application is shown. Please refer to... Figure 11 As shown, the input to the quality calibration module is the image information feature map output by the FPN and the detection information feature map output by the detection head. Here, the detection information feature map is the detection result feature map. The size of the image information feature map is H×W×C1, where H, W, and C1 represent the length, width, and number of channels, respectively. The size of the detection information feature map is H×W×C2, where H, W, and C2 represent the length, width, and number of channels, respectively. The quality calibration module first concatenates the image information feature map and the detection information feature map together through a concat operation to obtain a concatenated feature map with a size of H×W×(C1+C2).

[0094] In step 1020, the spliced ​​feature maps are fused, and the fusion result is reduced to the target dimension.

[0095] In one embodiment of this application, fusing the spliced ​​feature maps includes: fusing the spliced ​​feature maps sequentially through a first convolutional layer and a modified linear unit activation function layer;

[0096] The dimensionality reduction of the fusion result to the target dimension includes: reducing the dimensionality of the fusion result to a predetermined dimension through a second convolutional layer to obtain a fusion result of the predetermined dimension; and reducing the dimensionality of the fusion result of the predetermined dimension to the target dimension through an average pooling layer.

[0097] Specifically, the first convolutional layer can be a k×k convolution, and the second convolutional layer can be an m×m convolution, where k and m can be positive integers, and k and m can be the same or different.

[0098] Please continue reading Figure 11 As shown, after obtaining the concatenated feature map, a 1×1 convolution is used to convert the concatenated feature map into a feature map of size H×W×C3. Then, a ReLU (Rectified Linear Unit) activation function layer is used to further process the feature map of size H×W×C3. Then, a 1×1 convolution is used to reduce the dimensionality of the output of the ReLU activation function layer to size H×W×1. Finally, an average pooling operation of an AvgPool layer is used to reduce the dimensionality of the feature map of size H×W×1 to size 1×1×1.

[0099] In step 1030, the dimensionality-reduced fusion result is mapped to a quality prediction score output.

[0100] Please continue reading Figure 11 As shown, the quality calibration module finally inputs the output of the average pooling layer to the sigmoid function layer, and outputs the quality prediction score through the sigmoid function layer.

[0101] Although the solution in this application embodiment is to first perform splicing and then fuse using 1×1 convolution and ReLU, in other embodiments of this application, 1×1 convolution and ReLU can be fused first, and then splicing can be performed.

[0102] Typically, the detection results of a detection task consist of the outputs of multiple detection heads. Therefore, multiple detection heads and multiple quality calibration modules can be set in the target neural network, with each quality calibration module corresponding to one detection head, and the number of detection heads being the same as the number of quality calibration modules. In target detection tasks, different detection heads can be used to output target detection results at different scales.

[0103] Figure 12An embodiment according to this application is shown. Figure 7 A flowchart detailing steps 720 and 740 in the embodiment is provided. Please refer to [link / reference]. Figure 12 As shown, step 720 may specifically include the following steps:

[0104] In step 720', the feature extraction network extracts the image information feature maps of the target image that are paired with multiple detection heads respectively, and inputs each image information feature map into the corresponding detection head and quality calibration module respectively.

[0105] As mentioned earlier, the feature extraction network includes a feature pyramid network, which can output multi-layered feature maps, each of which is paired with a detection head.

[0106] After each image information feature map is input into its corresponding detection head, the detection head generates a detection result feature map corresponding to the image information feature map, thus obtaining the detection result feature map corresponding to each detection head; each detection head also inputs its corresponding detection result feature map into the corresponding quality calibration module.

[0107] Figure 13 A schematic diagram of multiple quality calibration modules corresponding to multiple detection heads according to an embodiment of this application is shown. Please refer to... Figure 13 As shown, assuming there are M detection heads (M>=1) and M quality calibration modules, The height, width, and number of channels of the feature map representing the image information paired with the i-th detection head. This represents the height, width, and number of channels of the feature map corresponding to the i-th detection head. Therefore, the size of the image information feature map paired with different detection heads can be different, and similarly, the size of the feature map corresponding to different detection heads can also be different. Pairing can be performed based on the similarity of the feature map's height and width. The feature maps and image information feature maps corresponding to different detection heads are then paired and sent to their respective quality calibration modules.

[0108] Please continue reading Figure 12 As shown, step 740 may specifically include the following steps:

[0109] In step 741, the input image information feature map and the detection result feature map are fused and mapped by each quality calibration module to obtain the quality prediction value output by each quality calibration module.

[0110] Please continue reading Figure 13 As shown, after the paired detection result feature maps and image information feature maps are sent to the corresponding quality calibration modules, the M quality calibration modules output the quality prediction values ​​corresponding to different detection heads.

[0111] In step 742, the quality prediction score is determined based on the quality prediction values ​​output by each quality calibration module.

[0112] Please continue reading Figure 13 As shown, the final quality prediction score can be calculated based on the quality prediction values ​​output by each quality calibration module.

[0113] In one embodiment of this application, determining a quality prediction score based on the quality prediction values ​​output by each quality calibration module includes: determining the average value of the quality prediction values ​​output by each quality calibration module as the quality prediction score.

[0114] Specifically, the quality prediction score can be calculated based on the following formula:

[0115]

[0116] Where M is the number of detection heads, q i This is the predicted quality value output by the quality calibration module corresponding to the i-th detection head. For quality prediction scores.

[0117] Of course, in other embodiments of this application, the quality prediction score can also be determined based on each quality prediction value using other calculation methods.

[0118] Figure 14 A flowchart illustrating the training of a target neural network according to an embodiment of this application is shown. Figure 14 As shown, before inputting the target image to be detected into the target neural network, the target neural network can be trained using the following process:

[0119] In step 1410, a training sample set is obtained, which includes multiple training samples and ground truth labels corresponding to each training sample.

[0120] Taking object detection as an example, the ground truth labels corresponding to the training samples can include the category, location, and size of the object in the training samples.

[0121] In step 1420, the training samples in the training sample set are input into the original neural network to obtain the quality prediction score output by the quality calibration module of the original neural network and the detection result feature map generated by the detection head of the original neural network.

[0122] A raw neural network is a neural network model that has not yet been fully trained; it can be an initialized neural network model.

[0123] Figure 15 A schematic flowchart illustrating the training process of a target neural network according to an embodiment of this application is shown. Please refer to... Figure 15As shown, the training of the quality calibration module and the user-preset detection task are performed simultaneously. After the image is input into the network, the network will output the detection semantics through the detection head, that is, the detection result feature map. The quality calibration module will output the quality prediction value, that is, the quality prediction score.

[0124] In step 1430, the loss value of the detection task loss is determined based on the ground truth labels corresponding to the training samples and the feature map of the detection results.

[0125] Please see Figure 15 As shown, the predicted value can be obtained from the feature map of the detection result output by the detection head. The detection head loss, i.e. the detection task loss, can be calculated based on the ground truth label and the predicted value.

[0126] In step 1440, post-processing operations are performed on the feature map of the detection result to obtain the detection result, which includes detection boxes used to mark the detected targets.

[0127] Post-processing refers to the processing steps that obtain the detection results based on the output of the detection head. Taking target detection as an example, post-processing operations typically include steps such as non-maximum suppression and coordinate decoding, and finally output the detection box.

[0128] In step 1450, the true quality score is determined based on the ground truth label corresponding to the training sample and the detection result, and the loss value of the quality calibration loss is determined based on the true quality score and the predicted quality score.

[0129] The quality of the network prediction results for this sample can be evaluated using metrics such as accuracy and recall.

[0130] In one embodiment of this application, determining the true quality score based on the ground truth label corresponding to the training sample and the detection result includes: determining the proportion of correctly detected targets in the training sample based on the ground truth label corresponding to the training sample and the detection result, and using this as the true quality score.

[0131] Figure 16 A schematic diagram illustrating the determination of a true quality score according to one embodiment of this application is shown. See also... Figure 16 As shown, it illustrates an input sample in a road target detection scenario. For a road target detection task, the 6 targets marked with numbers in the figure should be detected. However, if only the 3 targets marked with solid lines are detected, and the remaining 3 targets marked with dashed lines are missed, then the true quality score = number of correctly detected targets / total number of targets = 0.5.

[0132] Please continue reading Figure 15As shown, quality assessment is performed based on the detection box and true value label obtained from post-processing to obtain the true quality value. The quality calibration loss is then calculated based on the predicted quality value and the true quality value.

[0133] In one embodiment of this application, determining the loss value of quality calibration loss based on the true quality score and the predicted quality score includes: determining the distance between the true quality score and the predicted quality score as the loss value of quality calibration loss.

[0134] The quality calibration loss can be calculated using equal distance losses such as L1 and L2, and its expression is as follows:

[0135] loss q =distance(q pred ,q gt )

[0136] Where, loss q For quality calibration loss, distance is the distance calculation function, which can be L1 or L2 equal distance loss, q pred For the quality prediction score, q gt This is the true score for quality.

[0137] In step 1460, the loss value of the training loss is determined based on the loss value of the detection task loss and the loss value of the quality calibration loss, and the original neural network is trained based on the loss value of the training loss to obtain the target neural network.

[0138] In one embodiment of this application, determining the training loss value based on the loss value of the detection task loss and the loss value of the quality calibration loss includes: determining a weighted sum of the loss value of the detection task loss and the loss value of the quality calibration loss as the training loss value.

[0139] Specifically, the training loss can be constructed based on the following expression:

[0140] loss = a·loss q +(1-a)·loss det

[0141] Where loss represents the training loss, loss q This represents the quality calibration loss. det This represents the original detection task loss, where 'a' is a preset value used to balance the loss weights, and 'a' ∈ (0, 1).

[0142] The weights of the original neural network can be adjusted based on the training loss, thereby training the target neural network.

[0143] Please continue reading Figure 15As shown, after obtaining the detection task loss and the quality calibration loss, the overall loss is obtained by combining the two, and the network is updated based on the overall loss.

[0144] Figure 17 An embodiment according to this application is shown. Figure 7 A flowchart of the steps following step 740 in the embodiment. (See attached flowchart.) Figure 17 As shown, step 740 may be followed by the following steps:

[0145] In step 750, if the target image is a difficult sample, the target image is stored in the target database.

[0146] The target database can be specifically used to store difficult samples.

[0147] In step 760, new training samples are generated based on the images in the target database, and the new training samples are added back to the training sample set.

[0148] Images from the target database can be sent to annotators, and the annotated images returned by the annotators can be used as new training samples.

[0149] In step 770, the target neural network is further optimized based on the training sample set.

[0150] The parameters of the target neural network are further optimized using the updated training sample set.

[0151] Figure 18 A schematic diagram illustrating a process for quality assessment based on a target neural network according to an embodiment of this application is shown. Please refer to... Figure 18 As shown, for a trained network with a quality calibration module, we can directly deploy it to the production environment. After inputting a sample into the network, we obtain the corresponding quality calibration prediction score. Then, based on the quality calibration prediction score, we determine whether the sample is a hard sample according to the preset Badcase judgment rule. If it is, the sample is fed back into the database. The database continuously collects hard samples to form a hard sample dataset. Finally, we manually label the hard samples in the hard sample dataset and then add them back to the training set to help the network iteratively optimize.

[0152] In summary, the difficult sample screening method provided by the embodiments of this application can achieve at least the following technical effects: it provides an automated screening scheme for difficult samples, which can save manpower and time costs; this scheme is an online difficult sample screening method, which can determine whether a sample is a difficult sample in the actual deployment environment of the model, saving a lot of sample collection and reflow processes.

[0153] The following describes an apparatus embodiment of this application, which can be used to execute the difficult sample screening method in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the difficult sample screening method described above.

[0154] Figure 19 A block diagram of a difficult sample screening apparatus according to an embodiment of this application is shown.

[0155] Reference Figure 19 As shown, a difficult sample screening apparatus 1900 according to an embodiment of this application includes: an image input unit 1910, a feature extraction and input unit 1920, a generation and input unit 1930, and a determination unit 1940. The image input unit 1910 is used to input the target image to be detected into the target neural network. The target neural network includes a feature extraction network, a detection head connected to the feature extraction network, and a quality calibration module connected to the feature extraction network and the detection head respectively. The feature extraction and input unit 1920 is used to extract the image information feature map of the target image through the feature extraction network and input the image information feature map into the detection head and the quality calibration module respectively. The generation and input unit 1930 is used to generate a detection result feature map corresponding to the image information feature map through the detection head and input the detection result feature map into the quality calibration module. The detection result feature map is used to characterize the position and category of the target in the target image. The determination unit 1940 is used to perform fusion and mapping operations on the input image information feature map and the detection result feature map through the quality calibration module to obtain a quality prediction score, and determine whether the target image is a hard sample based on the quality prediction score. The hard sample is an image containing a target that has not been detected by the target neural network.

[0156] In some embodiments of this application, based on the foregoing scheme, the feature extraction network includes a backbone network and a feature pyramid network connected to the backbone network. The feature extraction and input unit 1920 is configured to: extract the basic semantic feature map of the target image through the backbone network and input the basic semantic feature map into the feature pyramid network; generate an image information feature map containing high-level semantic feature information of the image through the feature pyramid network based on the basic semantic feature map, wherein the feature depth of the high-level semantic feature information of the image is higher than the feature depth of the basic semantic feature information of the image in the basic semantic feature map.

[0157] In some embodiments of this application, based on the aforementioned scheme, the determining unit 1940 is configured to: stitch the image information feature map and the detection result feature map together to obtain a stitched feature map; fuse the stitched feature map and reduce the dimensionality of the fusion result to the target dimension; and map the dimensionality-reduced fusion result to a quality prediction score output.

[0158] In some embodiments of this application, based on the aforementioned scheme, the determining unit 1940 is configured to: fuse the spliced ​​feature map sequentially through a first convolutional layer and a modified linear unit activation function layer; reduce the dimensionality of the fusion result to a predetermined dimension through a second convolutional layer to obtain a fusion result of a predetermined dimension; and reduce the dimensionality of the fusion result of the predetermined dimension to a target dimension through an average pooling layer.

[0159] In some embodiments of this application, based on the foregoing scheme, before inputting the target image to be detected into the target neural network, the image input unit 1910 is further configured to: acquire a training sample set, the training sample set including multiple training samples and ground truth labels corresponding to each training sample; input the training samples in the training sample set into the original neural network to obtain the quality prediction score output by the quality calibration module of the original neural network and the detection result feature map generated by the detection head of the original neural network; determine the loss value of the detection task loss according to the ground truth labels corresponding to the training samples and the detection result feature map; perform post-processing operations on the detection result feature map to obtain a detection result, the detection result including a detection box for marking the detected target; determine the true quality score according to the ground truth labels corresponding to the training samples and the detection result, and determine the loss value of the quality calibration loss according to the true quality score and the quality prediction score; determine the loss value of the training loss according to the loss value of the detection task loss and the loss value of the quality calibration loss, and train the original neural network according to the loss value of the training loss to obtain the target neural network.

[0160] In some embodiments of this application, based on the foregoing scheme, the image input unit 1910 is configured to: determine the proportion of correctly detected targets among all targets in the training sample according to the ground truth label corresponding to the training sample and the detection result, and use it as the true quality score.

[0161] In some embodiments of this application, based on the foregoing scheme, after determining whether the target image is a difficult sample according to the quality prediction score, the determining unit 1940 is further configured to: if the target image is a difficult sample, store the target image in the target database; generate new training samples according to the images in the target database, and add the new training samples back to the training sample set; and continue to optimize the target neural network according to the training sample set.

[0162] In some embodiments of this application, based on the foregoing scheme, the target neural network includes multiple detection heads and multiple quality calibration modules. The feature extraction and input unit 1920 is configured to: extract image information feature maps of the target image that are respectively paired with the multiple detection heads through the feature extraction network, and input each image information feature map into the corresponding detection head and quality calibration module respectively; the determination unit 1940 is configured to: perform fusion and mapping operations on the input image information feature maps and detection result feature maps through each quality calibration module to obtain the quality prediction value output by each quality calibration module; and determine the quality prediction score based on the quality prediction value output by each quality calibration module.

[0163] In some embodiments of this application, based on the aforementioned scheme, the determining unit 1940 is configured to: determine the average value of the quality prediction values ​​output by each quality calibration module as the quality prediction score.

[0164] Figure 20 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0165] It should be noted that, Figure 20 The computer system 2000 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0166] like Figure 20 As shown, the computer system 2000 includes a Central Processing Unit (CPU) 2001, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 2002 or programs loaded from Storage Unit 2008 into Random Access Memory (RAM) 2003, such as performing the methods described in the above embodiments. Various programs and data required for system operation are also stored in RAM 2003. The CPU 2001, ROM 2002, and RAM 2003 are interconnected via bus 2004. An Input / Output (I / O) interface 2005 is also connected to bus 2004.

[0167] The following components are connected to I / O interface 2005: an input section 2006 including a keyboard, mouse, etc.; an output section 2007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 2008 including a hard disk, etc.; and a communication section 2009 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 2009 performs communication processing via a network such as the Internet. Drive 2010 is also connected to I / O interface 2005 as needed. Removable media 2011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 2010 as needed so that computer programs read from them can be installed into storage section 2008 as needed.

[0168] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 2009, and / or installed from removable medium 2011. When the computer program is executed by central processing unit (CPU) 2001, it performs various functions defined in the system of this application.

[0169] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0170] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0171] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0172] In one aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0173] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0174] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0175] It is understood that in the specific embodiments of this application, data related to image detection is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0176] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0177] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A difficult sample screening method, characterized in that, The method comprises: obtaining a training sample set comprising a plurality of training samples and a true value label corresponding to each training sample; inputting the training samples in the training sample set into an original neural network to obtain a quality prediction score output by a quality calibration module of the original neural network and a detection result feature map generated by a detection head of the original neural network; determining a loss value of a detection task loss according to the true value label corresponding to the training sample and the detection result feature map; performing a post-processing operation on the detection result feature map to obtain a detection result, the detection result comprising a detection box for marking a detected target; determining a quality true score according to the true value label corresponding to the training sample and the detection result, and determining a loss value of a quality calibration loss according to the quality true score and the quality prediction score; determining a loss value of a training loss according to the loss value of the detection task loss and the loss value of the quality calibration loss, and training the original neural network according to the loss value of the training loss to obtain a target neural network; inputting a target image to be detected into the target neural network, the target neural network comprising a feature extraction network, a detection head connected to the feature extraction network, and a quality calibration module connected to the feature extraction network and the detection head respectively; extracting an image information feature map of the target image through the feature extraction network, and inputting the image information feature map into the detection head and the quality calibration module respectively; generating a detection result feature map corresponding to the image information feature map through the detection head, and inputting the detection result feature map into the quality calibration module, the detection result feature map being used to represent the position and category of a target in the target image; performing fusion and mapping operations on the input image information feature map and detection result feature map through the quality calibration module to obtain a quality prediction score, and determining whether the target image is a difficult sample according to the quality prediction score, wherein the difficult sample is an image containing a target that is not detected by the target neural network.

2. The difficult sample screening method of claim 1, wherein, The feature extraction network comprises a backbone network and a feature pyramid network connected to the backbone network, and the extraction of the image information feature map of the target image through the feature extraction network comprises: extracting a basic semantic feature map of the target image through the backbone network, and inputting the basic semantic feature map into the feature pyramid network; generating an image information feature map containing image high-level semantic feature information through the feature pyramid network according to the basic semantic feature map, the feature depth of the image high-level semantic feature information being higher than that of image basic semantic feature information in the basic semantic feature map.

3. The difficult sample screening method of claim 1, wherein, The fusion and mapping operations on the input image information feature map and detection result feature map through the quality calibration module to obtain a quality prediction score comprise: splicing the image information feature map and the detection result feature map together to obtain a spliced feature map; Fusing the spliced feature maps and dimensionally reducing the fusion result to a target dimension; Mapping the dimensionally reduced fusion result as a quality prediction score output.

4. The difficult sample screening method of claim 3, wherein, The fusing the spliced feature maps comprises: sequentially fusing the spliced feature maps through a first convolutional layer and a rectified linear unit activation function layer; The dimensionally reducing the fusion result to the target dimension comprises: dimensionally reducing the fusion result to a predetermined dimension through a second convolutional layer to obtain a fusion result of the predetermined dimension; dimensionally reducing the fusion result of the predetermined dimension to the target dimension through an average pooling layer.

5. The difficult sample screening method of claim 1, wherein, The determining a quality real score according to the ground truth label corresponding to the training sample and the detection result comprises: determining a proportion of the number of targets correctly detected in all targets of the training sample as the quality real score according to the ground truth label corresponding to the training sample and the detection result.

6. The difficult sample screening method of claim 1, wherein, After determining whether the target image is a difficult sample according to the quality prediction score, the method further comprises: if the target image is a difficult sample, storing the target image into a target database; generating a new training sample according to the images in the target database and re-adding the new training sample to the training sample set; continuing to optimize the target neural network according to the training sample set.

7. The difficult sample screening method of any one of claims 1-6, wherein, The target neural network comprises a plurality of detection heads and a plurality of quality calibration modules, and the extracting an image information feature map of the target image through the feature extraction network and inputting the image information feature map into the detection head and the quality calibration module respectively comprises: extracting image information feature maps respectively matched with the plurality of detection heads through the feature extraction network and inputting each image information feature map into the corresponding detection head and quality calibration module; The fusing and mapping the input image information feature map and the detection result feature map through the quality calibration module to obtain a quality prediction score comprises: fusing and mapping the input image information feature map and the detection result feature map through each quality calibration module to obtain a quality prediction value output by each quality calibration module; determining a quality prediction score according to the quality prediction value output by each quality calibration module.

8. The difficult sample screening method of claim 7, wherein, The determining a quality prediction score according to the quality prediction value output by each quality calibration module comprises: determining an average value of the quality prediction value output by each quality calibration module as the quality prediction score.

9. A difficult sample screening device, comprising: The apparatus comprises: an image input unit configured to input a target image to be detected into a target neural network, the target neural network comprising a feature extraction network, a detection head connected to the feature extraction network, and a quality calibration module connected to the feature extraction network and the detection head respectively; a feature extraction and input unit configured to extract an image information feature map of the target image through the feature extraction network and input the image information feature map into the detection head and the quality calibration module respectively; The generation and input unit is configured to generate a detection result feature map corresponding to the image information feature map by the detection head, and input the detection result feature map into the quality calibration module, where the detection result feature map is used to represent the position and category of the target in the target image. The determination unit is configured to fuse and map the input image information feature map and the detection result feature map by the quality calibration module to obtain a quality prediction score, and determine whether the target image is a difficult sample according to the quality prediction score, where the difficult sample is an image containing a target that is not detected by the target neural network. Before the target image to be detected is input into the target neural network, the image input unit is further configured to: obtain a training sample set, where the training sample set includes a plurality of training samples and a true value label corresponding to each training sample; input the training sample in the training sample set into the original neural network to obtain a quality prediction score output by a quality calibration module of the original neural network and a detection result feature map generated by a detection head of the original neural network; determine a loss value of a detection task loss according to the true value label corresponding to the training sample and the detection result feature map; perform a post-processing operation on the detection result feature map to obtain a detection result, where the detection result includes a detection box used to mark a detected target; determine a quality real score according to the true value label corresponding to the training sample and the detection result, and determine a loss value of a quality calibration loss according to the quality real score and the quality prediction score; determine a loss value of a training loss according to the loss value of the detection task loss and the loss value of the quality calibration loss, and train the original neural network according to the loss value of the training loss to obtain a target neural network.

10. A computer readable medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the difficult sample screening method in any one of claims 1 to 8.

11. An electronic device, comprising: comprise: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the difficult sample screening method in any one of claims 1 to 8.

12. A computer program product, characterised in that, The computer program product comprises computer instructions stored in a computer readable storage medium, and a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the difficult sample screening method in any one of claims 1 to 8.

Images