Target detection method, target detection model, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a target detection method, a target detection model, electronic equipment and a storage medium, and belongs to the technical field of target detection. According to the method, a target detection model is used for carrying out the following processes on a target detection image: step-by-step feature extraction is carried out on the target detection image through a backbone network to obtain a plurality of first feature maps with different scales; performing forward feature fusion on the plurality of first feature maps through a top-down module in the head network to obtain a plurality of transition feature maps; performing reverse feature fusion on the plurality of transition feature maps through a bottom-up module containing a coordinate attention module in the head network to obtain a plurality of second feature maps; and determining a prediction frame corresponding to each second feature map through the detection network, and determining a target frame in the plurality of determined prediction frames, so as to determine a target detection result based on the target frame and the target detection image. According to the embodiment of the invention, the detection effect of target detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a target detection method and a target detection model, an electronic device and a storage medium. Background Art

[0002] Object detection methods can be applied to a variety of scenarios, such as road sign detection, vehicle type detection, etc. However, outdoor targets to be detected (such as road signs) are easily affected by environmental factors, resulting in slow detection speed and low detection accuracy. In particular, for some smaller targets or densely distributed targets, false detection and missed detection are prone to occur, resulting in poor detection results. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a target detection method and a target detection model, an electronic device and a storage medium, aiming to improve the detection effect of target detection.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a target detection method, the target detection method comprising:

[0005] The target detection image is input into the target detection model to perform the following process through the target detection model:

[0006] Performing step-by-step feature extraction on the target detection image through a backbone network to obtain a plurality of first feature maps with different scales;

[0007] Performing forward feature fusion on a plurality of the first feature maps with different scales through a top-down module in the head network to obtain a plurality of transition feature maps;

[0008] Performing reverse feature fusion on the plurality of transition feature maps through a bottom-up module including a coordinate attention module in the head network to obtain a plurality of second feature maps, wherein the coordinate attention module is used to perform feature focusing on at least one of the transition feature maps;

[0009] A prediction box corresponding to each of the second feature maps is determined through a detection network, and a target box is determined among the determined multiple prediction boxes, so as to determine a target detection result based on the target box and the target detection image.

[0010] In some embodiments, the plurality of transition feature maps include a transition shallow feature map, a transition middle feature map, and a transition deep feature map;

[0011] The reverse feature fusion is performed on the plurality of transition feature maps through the bottom-up module including the coordinate attention module in the head network to obtain a plurality of second feature maps, including:

[0012] Based on the coordinate attention module, feature focusing is performed on the transition shallow feature map to obtain a first focused feature map;

[0013] Fusing the first focused feature map and the transitional middle layer feature map to obtain a first fusion result;

[0014] Performing feature focusing on the first fusion result based on the coordinate attention module to obtain a second focused feature map;

[0015] Fusing the second focused feature map and the transition deep feature map to obtain a second fusion result;

[0016] Based on the transition shallow feature map, the first fusion result and the second fusion result, a plurality of second feature maps are determined.

[0017] In some embodiments, the plurality of first feature maps with different scales include a first shallow feature map, a first middle feature map, and a first deep feature map, and the plurality of first feature maps with different scales are forwardly fused by a top-down module in the head network to obtain a plurality of transition feature maps, including:

[0018] Processing the first deep feature map based on the dilated space pyramid pooling module to obtain a transition deep feature map;

[0019] Fusing the transition deep layer feature map and the first middle layer feature map to obtain a transition middle layer feature map;

[0020] The transitional middle layer feature map and the first shallow layer feature map are fused to obtain a transitional shallow layer feature map.

[0021] In some embodiments, the target detection model is trained in the following manner:

[0022] Acquire a training data set, the training data set comprising a plurality of training images and annotated images corresponding to the training images, the annotated images comprising a plurality of real boxes;

[0023] Clustering the multiple real boxes of the annotated image to obtain multiple candidate anchor boxes;

[0024] Processing the training image and the plurality of candidate anchor frames by the target detection model to obtain a plurality of predicted training frames;

[0025] Based on the predicted training frame and the real frame, determining a relative distance and an intersection-over-union ratio between the predicted training frame and the real frame;

[0026] Based on the relative distance and the intersection-over-union ratio, a bounding box loss value of the target detection model is determined, and the target detection model is trained by the bounding box loss value until the target detection model meets a preset condition.

[0027] In some embodiments, clustering the multiple real boxes of the annotated image to obtain multiple candidate anchor boxes includes:

[0028] Selecting a cluster center box from multiple real boxes in the labeled image;

[0029] Adding the cluster center box to the clustering data set;

[0030] Based on the distance between the cluster center box and the other real boxes, assigning a selection probability to the other real boxes;

[0031] updating the cluster center box based on the selection probabilities of the other real boxes, and adding the updated cluster center box to the cluster data set to repeatedly assign the selection probabilities until the number of the cluster center boxes in the cluster data set is equal to a preset number;

[0032] Mean clustering is performed on the multiple cluster center boxes in the cluster data set to obtain multiple candidate anchor boxes.

[0033] In some embodiments, determining a target frame from the determined plurality of prediction frames comprises:

[0034] sorting the plurality of prediction boxes in descending order based on the confidence scores of the plurality of prediction boxes to obtain a first list;

[0035] Determine an index value of each prediction box based on a position of the prediction box in the first list;

[0036] Traversing the plurality of prediction frames in the first list, and taking the currently traversed prediction frame as a reference frame, so as to update the confidence score of the prediction frame whose index value is greater than the current reference frame;

[0037] Based on the updated confidence scores of the multiple prediction boxes, a target box is determined from the multiple prediction boxes.

[0038] In some embodiments, traversing the plurality of prediction boxes in the first list and taking the currently traversed prediction box as a reference box to update the confidence score of the prediction box having an index value greater than the current reference box includes:

[0039] Taking the predicted frame at the first position in the first list as a reference frame, and taking the index value of the reference frame as a reference value;

[0040] Determine the overlap between the prediction frame whose index value is greater than the reference value and the reference frame;

[0041] Lowering the confidence score of the predicted frame whose overlap with the reference frame is greater than a preset threshold;

[0042] The next predicted frame of the reference frame in the first list is determined as the updated reference frame, and the confidence score of the predicted frame whose index value is greater than the current reference frame is updated based on the updated reference frame until multiple predicted frames in the first list are traversed.

[0043] To achieve the above purpose, a second aspect of an embodiment of the present application proposes a target detection model, wherein the target detection model includes:

[0044] A backbone network, wherein the backbone network is used to extract features of the target detection image step by step to obtain a plurality of first feature maps with different scales;

[0045] A head network, the head network includes a top-down module and a bottom-up module, the top-down module is used to perform forward feature fusion on multiple first feature maps with different scales to obtain multiple transition feature maps; the bottom-up module is used to perform reverse feature fusion on multiple transition feature maps to obtain multiple second feature maps, and the top-down module includes a coordinate attention module, the coordinate attention module is used to perform feature focusing on at least one of the transition feature maps;

[0046] A detection network is used to determine a prediction box corresponding to each of the second feature maps, and to determine a target box among the determined multiple prediction boxes, so as to determine a target detection result based on the target box and the target detection image.

[0047] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0048] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0049] The target detection method and device, electronic device and storage medium proposed in the present application process the target detection image through the target detection model. Specifically, the target detection image is firstly subjected to step-by-step feature extraction through the backbone network to obtain multiple first feature maps with different scales. Afterwards, the multiple first feature maps are forwardly fused through the top-down module in the head network to obtain multiple transition feature maps, and the multiple transition feature maps are reversely fused through the bottom-up module including the coordinate attention module in the head network to obtain multiple second feature maps. The coordinate attention module in the head network is used to perform feature focus on at least one transition feature map, which can obtain the feature information of the target to be detected in the horizontal and vertical directions, ensure the long-distance correlation, thereby strengthening the feature extraction of the location of the target to be detected, and improving the detection ability of the target detection model for small targets. Then, the target detection result finally obtained retains the detection result of the small target, reduces the occurrence of missed detection, and improves the detection effect of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is an overall flow chart of the target detection method provided in the embodiment of the present application;

[0051] Figure 2 is a schematic diagram of a target detection model provided in an embodiment of the present application;

[0052] Figure 3 is a schematic diagram of a coordinate attention module provided in an embodiment of the present application;

[0053] Figure 4 yes Figure 1 A flowchart of step S104 in FIG.

[0054] Figure 5 yes Figure 1 A flowchart of step S103 in FIG.

[0055] Figure 6 is a schematic diagram of a hole space pyramid pooling module provided in an embodiment of the present application;

[0056] Figure 7 Schematic diagram of a principle of a hole space pyramid pooling module provided in an embodiment of the present application;

[0057] Figure 8 It is a flow chart of the training method of the target detection model provided in the embodiment of the present application;

[0058] Fig. 9 is a schematic diagram of a predicted training frame and a real frame provided in an embodiment of the present application;

[0059] Fig.10 yes Figure 8 A flowchart of step S802 in FIG.

[0060] Fig.11 yes Figure 1 A flowchart of step S105 in FIG.

[0061] Fig.12 yes Fig.11 A flowchart of step S1103 in FIG.

[0062] Fig.13 is a schematic diagram of the structure of a target detection device provided in an embodiment of the present application;

[0063] Fig.14 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0067] First, some nouns involved in this application are analyzed:

[0068] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] Object Detection: The task of object detection is to find all the objects of interest in the image and determine their categories and locations. It is one of the core issues in the field of computer vision. Since various objects have different appearances, shapes and postures, and are interfered by factors such as illumination and occlusion during imaging, object detection has always been the most challenging problem in the field of computer vision. However, the targets to be detected outdoors (such as road signs) are usually located outdoors and are easily affected by environmental factors, resulting in slow detection speed and low detection accuracy. In particular, for some smaller targets or densely distributed targets, it is easy to have problems such as false detection and missed detection, resulting in poor detection results.

[0070] Based on this, the embodiments of the present application provide a target detection method and a target detection model, an electronic device and a storage medium, aiming to improve the detection effect of target detection. The target detection method and target detection model provided in the embodiments of the present application can be applied to a variety of scenarios, such as but not limited to being applied to target detection of road signs (such as zebra crossings, speed limit signs, turn signs or lane lines, etc.), road facilities (such as traffic lights, gates or entrances and exits, etc.), pedestrians, vehicles or vehicle types in traffic scenarios, and can also be applied to target detection in hospital scenes, stations, airports, exhibition halls, conferences and other scenarios; technical personnel in this field can flexibly apply the target detection method provided in the embodiments of the present application to the required scenarios according to actual business needs, and perform detection on one or more targets in the required scenarios.

[0071] The target detection method and target detection model, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the target detection method in the embodiments of the present application is described.

[0072] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0073] The target detection method provided in the embodiment of the present application relates to the field of target detection technology and artificial intelligence technology. The target detection method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. Cloud servers; software can be applications that implement target detection methods, etc., but are not limited to the above forms.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] It should be noted that all data and information involved in the embodiments of the present application comply with the requirements of relevant laws and regulations.

[0076] Figure 1 is an optional flowchart of the target detection method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0077] Step S101, input the target detection image into the target detection model to execute the following steps S102 to S105 through the target detection model.

[0078] Step S102: extract features of the target detection image step by step through the backbone network to obtain a plurality of first feature maps with different scales.

[0079] Step S103, performing forward feature fusion on multiple first feature maps with different scales through a top-down module in the head network to obtain multiple transition feature maps.

[0080] Step S104, performing reverse feature fusion on multiple transition feature maps through a bottom-up module including a coordinate attention module in the head network to obtain multiple second feature maps, and the coordinate attention module is used to perform feature focusing on at least one transition feature map.

[0081] Step S105, determining a prediction box corresponding to each second feature map through a detection network, and determining a target box among the determined multiple prediction boxes, so as to determine a target detection result based on the target box and the target detection image.

[0082] The following describes the target and target detection image for target detection in the embodiments of the present application:

[0083] The target detected by the target detection image in the embodiment of the present application can be a conventional target or a small target; wherein the standards for defining a small target in different scenarios are different, such as being able to define a small target based on, but not limited to, the relative size and / or absolute size of the target, and several examples are given below:

[0084] When defining small targets based on relative size: that is, defining small targets from the perspective of the relative ratio of the target to the image, such as, but not limited to, defining targets whose median of the relative area (i.e., the ratio of the target's bounding box area to the image area) of all target instances in the same category is within a preset range (such as, but not limited to, 0.08% to 0.58% or 0.10% to 0.60%) as small targets; targets whose ratio of the width and height of the target's bounding box to the width and height of the image is less than a certain value (such as 0.1, 0.15 or 0.2, etc.) can be defined as small targets; targets whose square root of the ratio of the target's bounding box area to the image area is less than a certain value can be defined as small targets; small targets can also be defined based on the ratio between the target's actual coverage pixels and the total pixels of the image, etc.;

[0085] When defining small targets based on absolute size: that is, defining small targets from the perspective of the absolute pixel size of the target; the most common definition currently comes from the general dataset in the field of target detection, the MS COCO dataset, which defines small targets as targets with a resolution less than 32 pixels × 32 pixels; in addition to MS COCO, there are other definitions based on absolute scale, such as in the aerial image dataset DOTA and the face detection dataset WIDER FACE, targets with pixel values ​​in the range of [10, 50] are defined as small targets; in the pedestrian recognition dataset CityPersons, small targets are defined as targets with a height less than 75 pixels. The small pedestrian dataset TinyPerson based on aerial images defines small targets as targets with pixel values ​​in the range of [20, 32], etc.

[0086] The target detection image in the embodiment of the present application is an image that needs to be detected, such as an image collected for a scene that may contain the aforementioned conventional target and / or small target. For ease of understanding, several examples of targets for target detection and corresponding target detection images in the embodiment of the present application are given as follows:

[0087] Example 1: When the target to be detected is a traffic sign in a traffic scene (which can be understood as a small target in a traffic scene), the target detection image may be an image collected of the traffic environment and / or road environment around the vehicle during the driving process of the vehicle, wherein the traffic sign may be a sign set up on the road or around the road to provide driving, walking or service guidance for vehicles and pedestrians; the aforementioned traffic signs may include but are not limited to at least one of warning signs, instruction signs and service signs; warning signs are used to alert drivers of dangerous conditions under special circumstances, including but not limited to indicating at least one of bends, slopes, road conditions, etc.; instruction signs are used to indicate the direction of the road ahead for the driver or pedestrian, including but not limited to indicating at least one of intersections, road sections, crossroads, etc.; service signs are used to indicate the location of service facilities, including but not limited to indicating at least one service facility such as gas stations, restaurants, hotels, etc.

[0088] Example 2: When the target to be detected is a cow / horse / sheep grazing in a pasture, the target detection image may be an image taken by a drone of the pasture.

[0089] Example 3: When the target to be detected is a person in a high-speed railway station, the target detection image may be an image taken by a camera in the high-speed railway station within the high-speed railway station.

[0090] It should be understood that the target detection model in the aforementioned step S101 may be a trained model, which is used to perform target detection on the target detection image. In road sign detection, the target detection model is used to perform road sign detection on the target detection image.

[0091] In step S102 of some embodiments, the backbone network is part of the target detection model, and the input of the backbone network is the input of the target detection model. Based on the backbone network, the target detection image can be subjected to step-by-step feature extraction to obtain multiple first feature maps, and the scales of the multiple first feature maps are not the same, thereby obtaining information of different levels in the target detection image.

[0092] In step S103 of one embodiment, the head network is also part of the target model, and the output of the backbone network is connected to the input of the head network. The head network includes a top-down module, which is used to perform forward feature fusion on multiple first feature maps with different scales to obtain multiple transition feature maps. The scales of the multiple transition feature maps are also different from each other. Assuming that the multiple first feature maps are respectively a first shallow feature map, a first deep feature map, and multiple first middle feature maps, then the multiple transition feature maps are respectively a first transition feature map, a first transition feature map, and multiple first transition feature maps.

[0093] In step S104 of one embodiment, the head network further includes a bottom-up module, and the bottom-up module includes a coordinate attention module. The bottom-up module is used to perform reverse feature fusion on multiple transition feature maps to obtain multiple second feature maps. The scales of the multiple second feature maps are also different from each other. In addition, Figure 2 As shown, the top-down modules and the bottom-up modules are interconnected to form a pyramid structure.

[0094] The coordinate attention module is used to focus on at least one transition feature map, thereby capturing the feature information of the target to be detected, such as road signs in the horizontal and vertical directions, to ensure long-distance correlation.

[0095] The coordinate attention module is a coordinate attention mechanism (CA), which can embed the horizontal and vertical coordinate information of targets, such as road signs, into feature channels, so that the generated features contain the coordinate information of the targets in the image.

[0096] Reference Figure 3 For the input features, the coordinate attention module uses average pooling layers of size (1, W) and (H, 1) to encode each channel in the vertical direction Y and horizontal direction X, generating feature outputs with coordinate information while ensuring long-distance correlation, where the vertical direction Y is the height direction and the horizontal direction X is the width direction. The process can be expressed as:

[0097]

[0098] Among them, x c is the input feature with c channels, W is the width of the input feature, H is the height of the input feature, is the output of the average pooling of the input features along the horizontal direction, and h is the height variable corresponding to the average pooling of the input features along the horizontal direction. It is the output of the average pooling of the input features along the vertical direction, w is the width variable corresponding to the average pooling of the input features along the vertical direction, and i and j are used to determine the position of the feature points.

[0099] Afterwards, the features and Features By concatenating and convolving, we can obtain the intermediate mapping f of the encoded spatial information in the horizontal and vertical directions:

[0100] f=δ(F1([Z h ,Z w ])),

[0101] Among them, δ represents the activation function, F1 represents 1×1 convolution, and Z h That is Z w That is

[0102] Finally, the intermediate map f is split into two separate tensors f along different spatial dimensions h and f w , and use two 1×1 convolution operators to convert f h and f w Convert to g with the same number of channels h and g w :

[0103] g h =σ(F h (f h )),

[0104] g w =σ(F w (f w )),

[0105] And use this as the weight value of attention, the weight value can be expressed as:

[0106]

[0107] Among them, σ represents the Sigmoid activation function, F h and F wrepresents 1×1 convolution, f h is the tensor feature of the intermediate mapping along the horizontal dimension, f w is the tensor feature of the intermediate mapping along the vertical dimension, g h f h The output obtained by sequentially passing through the 1×1 convolution transformation function and the activation function, g w f w The output obtained by sequentially passing through the 1×1 convolution transformation function and the activation function.

[0108] It can be seen that the coordinate attention module decomposes the channel attention into two independent 1D feature codes, and then aligns them in the horizontal and vertical directions for feature aggregation, so that the features contain the position information of the target in the other direction while containing the long-distance correlation in the corresponding direction. Finally, these two types of features are fused to generate attention features that are more sensitive to the target position information, thereby strengthening the feature extraction of the target area and improving the accuracy of feature expression.

[0109] In step S105 of an embodiment, on each second feature map, the detection network adjusts the prior frame according to the adjustment vector corresponding to each anchor point to obtain a prediction frame corresponding to the feature map, and then obtains multiple prediction frames corresponding to the target detection image. The adjustment vector corresponding to the anchor point is preset, which is specifically determined according to the training process of the target detection model. The prior frame is determined by the target detection model processing the target detection image.

[0110] After determining the prediction boxes corresponding to the multiple second feature maps, redundant boxes in the multiple prediction boxes are removed to determine the target box in the multiple prediction boxes. The target box is projected on the target detection image to obtain the target detection result. The target detection result includes the target box, and the target box contains the target to be detected.

[0111] The target box can be determined in a variety of ways, such as the non-maximum suppression algorithm (NMS).

[0112] The detection network includes a plurality of re-parameterized refocusing convolution modules (REP) and a second convolution module (CBM), and the plurality of re-parameterized refocusing convolution modules (REP) and the second convolution module (CBM) are connected to the second feature Figure 1One-to-one setting. Assuming that the second feature map includes a second deep feature map, a second shallow feature map, and a second middle feature map, the detection network includes three refocusing convolution modules (REPs) and a second convolution module (CBM). The second deep feature map, the second shallow feature map, and the second middle feature map are respectively input into the corresponding refocusing convolution module (REP) and the second convolution module (CBM) to obtain the prediction box corresponding to each second feature map, and then to multiple prediction boxes corresponding to the target detection image. Afterwards, the target box is determined in the multiple prediction boxes to determine the target detection result based on the target box and the target detection image.

[0113] In the training phase, the REP module is responsible for adjusting the parameters of the model according to the training data so that the target detection model can better fit the data. In the inference phase, the REP module is responsible for generating the final detection results based on the input data. CBM consists of a convolution layer, a batch normalization layer (Batch normalization, BN) and a Sigmoid activation function. The convolution layer is responsible for performing convolution operations and extracting the features of the input data. The batch normalization layer normalizes the convolved data to improve the stability and convergence speed of the model. The Sigmoid activation function is responsible for introducing nonlinear factors so that the target detection model can learn more complex mapping relationships. In addition, CBM is a contextual attention module, which is used to introduce global information to improve detection efficiency and accuracy.

[0114] Steps S101 to S105 shown in the embodiment of the present application process the target detection image through the target detection model. Specifically, first, the target detection image is subjected to step-by-step feature extraction through the backbone network to obtain multiple first feature maps with different scales. Afterwards, the multiple first feature maps are forwardly fused through the top-down module in the head network to obtain multiple transition feature maps, and the multiple transition feature maps are reversely fused through the bottom-up module including the coordinate attention module in the head network to obtain multiple second feature maps. The coordinate attention module in the head network is used to perform feature focusing on at least one transition feature map, and can obtain feature information of the target to be detected in the horizontal and vertical directions, ensuring long-distance correlation, thereby strengthening the feature extraction of the location of the target to be detected, and improving the detection ability of the target detection model for small targets. Then, the target detection result finally obtained retains the detection result of the small target, reduces the occurrence of missed detection, and improves the detection effect of target detection.

[0115] See also Figure 4 In some embodiments, the plurality of transition feature maps include a transition shallow feature map, a transition middle feature map, and a transition deep feature map. Step S104 may include, but is not limited to, steps S401 to S405:

[0116] Step S401, based on the coordinate attention module, feature focusing is performed on the transition shallow feature map to obtain a first focused feature map.

[0117] Step S402, fusing the first focused feature map and the transition middle layer feature map to obtain a first fusion result.

[0118] Step S403: Feature focusing is performed on the first fusion result based on the coordinate attention module to obtain a second focused feature map.

[0119] Step S404, fusing the second focused feature map and the transition deep feature map to obtain a second fusion result.

[0120] Step S405, determining a plurality of second feature maps based on the transition shallow feature map, the first fusion result and the second fusion result.

[0121] It should be noted that multiple transition feature maps can be divided into three categories according to their scale, namely, transition shallow feature maps, transition middle feature maps and transition deep feature maps. Among them, the scale of the transition shallow feature map is larger, the scale of the transition deep feature map is smaller, and the scale of the transition middle feature map is between the transition shallow feature map and the transition deep feature map.

[0122] In step S401 of some embodiments, feature focusing is performed on the transition shallow feature map based on the coordinate attention module to obtain a first focused feature map.

[0123] Reference Figure 2 First, the transition shallow feature map is input into the Efficient Layer Aggregation Networks (ELAN), and then the output of ELAN is feature focused based on the coordinate attention module to obtain the first focused feature map. ELAN is an efficient network structure that controls the shortest and longest gradient paths, allowing the network to learn more features and have stronger robustness.

[0124] In step S402 of some embodiments, the first focus feature map and the transition middle layer feature map are fused and spliced ​​to obtain a first fusion result.

[0125] Reference Figure 2 First, the first focused feature map is input into the pooling module (Max-Pooling, MP) to downsample the first focused feature map so that the width and height of the first focused feature map and the transition middle-layer feature map are consistent, so that the first focused feature map and the transition middle-layer feature map can be fused and spliced ​​to obtain a first fusion result.

[0126] In step S403 of some embodiments, feature focusing is performed on the first fusion result based on the coordinate attention module to obtain a second focused feature map.

[0127] Reference Figure 2 ,First, the first fusion result is input into the efficient layer aggregation network (ELAN), and then the output result of ELAN is feature focused based on the coordinate attention module, and the second focused feature map is obtained.

[0128] In step S404 of some embodiments, the second focus feature map and the transition deep feature map are fused to obtain a second fusion result.

[0129] Reference Figure 2 In the embodiment of the present application, the second focus feature map is first input into the pooling module (MP) to downsample the second focus feature map so that the width and height of the first focus feature map and the transition middle layer feature map are consistent. After that, the second focus feature map and the transition deep layer feature map are fused to obtain a second fusion result.

[0130] In step S405 of some embodiments, a plurality of second feature maps are determined based on the transition shallow feature map, the first fusion result, and the second fusion result.

[0131] Reference Figure 2 , the second feature map can be divided into the second deep feature map, the second shallow feature map and the second middle feature map according to the scale. First, the transition shallow feature map is directly used as the second shallow feature map, and the first fusion result is directly used as a second middle feature map. The transition deep feature map is input into the efficient layer aggregation network (ELAN) to obtain a second deep feature map.

[0132] In the process of reverse feature fusion through the above-mentioned embodiments of steps S401 to S407, the transition feature maps of the shallow and middle layers are focused by the coordinate attention module and then transmitted back to the small-scale detection head of the deep features. The purpose of this is to obtain the features of the small target in the shallow and middle layers and capture the features of its position and direction by the coordinate attention module to ensure long-distance correlation. As a result, the features of the small target in different dimensions will be retained in the second deep feature map to ensure the detection accuracy of the small target. In the second shallow feature map and the second middle feature map, the features of the small target in different dimensions are also integrated to reduce the occurrence of missed detection of small target features at other scales and improve the detection capability of small targets.

[0133] See also Figure 5 In some embodiments, the plurality of first feature maps of different scales include a first shallow feature map, a first middle feature map, and a first deep feature map. Step S103 may include, but is not limited to, steps S501 to S503:

[0134] Step S501, processing the first deep feature map based on the dilated spatial pyramid pooling module to obtain a transition deep feature map.

[0135] Step S502, fusing the transition deep layer feature map and the first middle layer feature map to obtain a transition middle layer feature map.

[0136] Step S503, fusing the transition middle layer feature map and the first shallow layer feature map to obtain a transition shallow layer feature map.

[0137] It should be noted that the multiple first feature maps can be divided into three categories according to their scale, namely, the first shallow feature map, the first middle feature map and the first deep feature map. Among them, the scale of the first shallow feature map is larger, the scale of the first deep feature map is smaller, and the scale of the first middle feature map is between the first shallow feature map and the first deep feature map.

[0138] In step S501 of some embodiments, the embodiments of the present application process the first deep feature map through an atrous spatial pyramid pooling module (ASPP). The atrous spatial pyramid pooling module expands the receptive field of the convolution operator when performing a pooling operation on the feature map, but does not reduce the resolution of the target object, ensures that the feature information of small targets is not lost, and can further reduce the situation of small targets.

[0139] Specifically, refer to Figure 6 ,The dilated spatial pyramid pooling module consists of a 1×1 convolution, three dilated convolutional layers with different dilation rates, and a spatial pyramid pooling layer. Figure 7 The schematic diagram of the principle of the dilated space pyramid pooling module provided in the embodiment of the present application shows that for the input features, the dilated space pyramid pooling module uses dilated convolution layers with different sampling rates, i.e., dilated rates, which can produce receptive fields of different scales, thereby obtaining multi-scale object information. Subsequently, the features extracted in parallel at each sampling rate are fused and spliced, and the number of channels is reduced to the expected value through a 1×1 convolution, which is equivalent to capturing the contextual features of the target detection image at multiple scales. Therefore, the dilated space pyramid pooling module can capture the contextual information of the target at different scales by receptive fields of different scales while ensuring the resolution of small targets, thereby reducing the loss of effective features and improving the detection effect.

[0140] Reference Figure 6, if the size of the first deep feature map is 20×20×1024, after the first deep feature map is input into the dilated spatial pyramid pooling module, the dilated spatial pyramid pooling module first calculates the first deep feature map through a parallel 1×1 convolution, three 3×3 dilated convolution layers with dilation rates of 6, 12 and 18, and a spatial pyramid pooling layer including a pooling layer, a 1×1 convolution and an upsampling. The 1×1 convolution will perform linear calculations on the first deep feature map and output a feature map of size 20×20×1024. The purpose is to increase the model depth without changing the receptive field to learn more complex feature information. A single dilated convolution layer outputs feature maps of 12×12×1024, 20×20×1024, and 48×48×1024, respectively. Different receptive fields are generated by different dilation rates, so that the model can have a better understanding of features of different scales on three different branches, while improving the model's perception of objects of different sizes. The serial spatial pyramid pooling layer will output a 40×40×1024 feature map, which aims to reduce the dimension of the captured feature information and integrate it into fixed-dimensional features, which will help the model network better understand the feature identification of small targets at different scales. Finally, the parallel output features are fused and a 1D convolution linear calculation is performed to output a 140×140×1024 feature to obtain a transition deep feature map.

[0141] In step S502 of some embodiments, the transition deep layer feature map and the first middle layer feature map are fused to obtain a transition middle layer feature map.

[0142] Reference Figure 2 In the embodiment of the present application, the transition deep feature map is first input into the first convolution module (ChannelBlock Squeeze, CBS) and the upsampling layer (UPSample) in sequence, and the first middle-layer feature map is input into the first convolution module (CBS), and the feature map output by the upsampling layer and the feature degree corresponding to the first middle-layer feature map output by the first convolution module are fused and spliced ​​to obtain the transition middle-layer feature map. Among them, the structure of CBS is basically the same as that of CBM, both of which include a convolution layer, a batch normalization layer and a Sigmoid activation function. The processing of the upsampling layer makes the width and height of the transition deep feature map and the first middle-layer feature map consistent, which is convenient for fusion and splicing. In addition, CBS can be considered as a channel compression module, which processes the transition deep feature map through a series of convolution operations to reduce the amount of calculation and the number of parameters.

[0143] In step S503 of some embodiments, the transitional middle layer feature map and the first shallow layer feature map are fused to obtain a transitional shallow layer feature map.

[0144] Reference Figure 2, which is the same as the generation process of the transitional middle layer feature map, the embodiment of the present application first inputs the transitional middle layer feature map into the first convolution module (CBS) and the upsampling layer (UPSample) in sequence, and inputs the first shallow layer feature map into the first convolution module (CBS), and fuses and splices the processed transitional middle layer feature map and the first shallow layer feature map. After that, the fused and spliced ​​result is input into the efficient layer aggregation network (ELAN) to obtain a transitional shallow layer feature map.

[0145] In the process of forward feature fusion through the above-mentioned embodiments of step S501 to step S503, the first deep feature output by the backbone network is subjected to the atrous spatial pyramid pooling module, and the contextual information of the small target at different scales of the deep features is captured through receptive fields of different sizes, thereby reducing the possibility of the deep features of the small target being lost in the traditional pooling operation, thereby further reducing the possibility of missed detection of small target objects and improving the detection effect of target detection.

[0146] See also Figure 8 In some embodiments, the training method of the target detection model may include but is not limited to steps S801 to S805.

[0147] Step S801: Acquire a training data set, where the training data set includes a plurality of training images and annotated images corresponding to the training images, where the annotated images include a plurality of real boxes.

[0148] Step S802: cluster multiple real boxes of the annotated image to obtain multiple candidate anchor boxes.

[0149] Step S803: Process the training image and multiple candidate anchor frames through the object detection model to obtain multiple predicted training frames.

[0150] Step S804: Based on the predicted training frame and the real frame, determine the relative distance and intersection-over-union ratio between the predicted training frame and the real frame.

[0151] Step S805, based on the relative distance and the intersection-over-union ratio, determine the bounding box loss value of the target detection model, and train the target detection model through the bounding box loss value until the target detection model meets the preset conditions.

[0152] In step S801 of some embodiments, a training data set is obtained. The training data set may be composed of a self-collected data set containing images of the target to be detected and a public image data set of the target to be detected.

[0153] To improve the detection effect, the images in the training data set include images containing the target to be detected in different weather conditions such as day, night, sunny, cloudy, and rainy days. In addition, if the target to be detected is a road sign, the public data sets can include the TT-100K data set, CCTSDB data set, CTSDB data set, etc.

[0154] The training data set includes a training image and an annotated image corresponding to the training image, wherein the annotated image is an annotated result of the training image. The annotated image includes multiple real frames, and in the annotated image, the target to be detected has been framed by the real frame.

[0155] It should be noted that the embodiment of the present application can use the image graphic annotation tool label Img to annotate the training image, and the format of the annotation label is the YOLO label format. After annotation, each training image will have a label file with the same name, with the extension ".txt". Each line of the label file represents the annotation information of a target to be detected in the training image, and its format is class_id x_center y_center width height, where class_id is the category label of the target to be detected, x_center and y_center are the center point coordinate values ​​of the real box relative to the width and height of the training image, and width and height are the width and height of the bounding box of the target to be detected relative to the width and height of the image.

[0156] In step S802 of some embodiments, the candidate anchor boxes are obtained by clustering multiple real boxes of the annotated image. The clustering of the real boxes can adopt a variety of clustering analysis algorithms, such as K-means clustering (K-means), mean shift algorithm (MeanShift), etc.

[0157] In step S803 of some embodiments, the training image is detected by the backbone network and the head network of the target detection model to obtain multiple prediction training feature maps. Figure 2 The predicted training feature map corresponds to the second feature map in step S104. Then, in the detection network, multiple candidate anchor frames are evenly distributed to multiple predicted training feature maps, so as to adjust the corresponding candidate anchor frames based on the predicted training feature maps to obtain multiple predicted training frames.

[0158] The embodiment of the present application first sorts multiple candidate anchor boxes according to their scales and evenly distributes them to multiple generated prediction training feature maps, so that each prediction training feature map is divided into an H×W grid, and an anchor point is set at the center of each grid unit. H×W is the size of the prediction training feature map.

[0159] Assuming that the number of candidate anchor boxes is 9 and the number of predicted training feature maps is 3, then each predicted training feature map corresponds to 3 candidate anchor boxes, and each anchor point covers the 3 candidate anchor boxes of the corresponding predicted training feature map. Each anchor point on each predicted training feature map corresponds to a vector, and by splitting the vector into dimensions, a one-dimensional adjustment vector for each candidate anchor box is obtained. Based on the adjustment vector, the adjustment information of the candidate anchor box can be determined, and then the position and size of the candidate anchor box can be adjusted to obtain multiple predicted training boxes.

[0160] In step S804 of some embodiments, the predicted training frame and the real frame are usually rectangular. For ease of representation, the relative distance between the predicted training frame and the real frame is embodied as the first distance and the second distance. The predicted training frame is represented by the first coordinate and the second coordinate, and the real frame is represented by the third coordinate and the fourth coordinate. The first coordinate and the second coordinate can be the upper left corner coordinate and the lower right corner coordinate of the predicted training frame, and the corresponding third coordinate and the fourth coordinate are the upper left corner coordinate and the lower right corner coordinate of the real frame. In addition, the first coordinate and the second coordinate can also be the upper right corner coordinate and the lower left corner coordinate of the predicted training frame, and the corresponding third coordinate and the fourth coordinate are the upper right corner coordinate and the lower left corner coordinate of the real frame. The first distance is the distance between the first coordinate and the third coordinate, and the second distance is the second distance between the second coordinate and the fourth coordinate.

[0161] Reference Fig.11 , assuming that the first coordinate of the predicted training box is The third coordinate of the ground-truth box is expressed as The first distance can be expressed as:

[0162] Assume that the second coordinate of the predicted training box is The fourth coordinate of the ground-truth box is expressed as The first distance can be expressed as:

[0163] The intersection-over-union ratio between the predicted training box and the true box is the ratio of the intersection and union between the predicted training box and the true box.

[0164] Assuming that the predicted training box is represented as prd and the real box is represented as gt, the intersection-over-union (IOU) ratio between the predicted training box and the real box is expressed as:

[0165]

[0166] In step S805 of some embodiments, a bounding box loss value of the object detection model is determined based on the relative distance and the intersection-over-union ratio.

[0167] In the calculation of the bounding box loss value, the first distance and the square of the second distance are first determined:

[0168]

[0169] Afterwards, determine the bounding box loss value for the object detection model:

[0170]

[0171] Among them, L MPDIoU is the bounding box loss value, d1 is the first distance, d2 is the second distance, w is the width of the training image, and h is the height of the training image.

[0172] The bounding box loss value can represent the difference between the predicted training box and the real box corresponding to the predicted training box, and thus represent the model detection accuracy of the target detection model. Therefore, the target detection model is trained based on the bounding box loss value to make the size and position of the predicted training box as close to the real box as possible, or even consistent with the real box, so as to better ensure the accuracy of the bounding box regression, make the predicted bounding box have less redundancy, and locate the predicted position of the target to be detected more accurately.

[0173] The preset condition for target detection model training is model convergence, which can be determined by the loss function and the number of model training times.

[0174] In addition, the embodiments of the present application may also use a warm-up strategy (warm up) and an early stop strategy (early stop) to reduce the occurrence of overfitting in training, so that the model converges faster and the model effect is better.

[0175] Through the embodiments of the above steps S801 to S805, multiple real boxes in the annotated image corresponding to the training image are clustered to obtain multiple candidate anchor boxes. Afterwards, the training image and the multiple candidate anchor boxes are processed by the target detection model to obtain multiple predicted training boxes. Based on the predicted training box and the real box, the relative distance and intersection-over-union ratio between the predicted training box and the real box are determined, and then the bounding box loss value of the target detection model is determined. The target detection model is trained based on the bounding box loss value of the target detection model until the target detection model meets the preset conditions, and a trained target detection model is obtained, thereby improving the detection accuracy of the target detection model. In addition, the bounding box loss value takes into account the overlapping area and the distance between the predicted training box and the real box during the calculation process, which can more accurately locate the location where the target to be detected is densely or overlapping, thereby improving the detection accuracy.

[0176] After step S801 in some embodiments, the training data set is subjected to image enhancement preprocessing. Image enhancement preprocessing may include geometric transformation, color space transformation, and hybrid transformation. In geometric transformation, the image is subjected to operations such as horizontal flipping, vertical flipping, random cropping, rotation, scaling, translation, and perspective transformation to randomly change the position, direction, and size of the target to be detected in the image. In color transformation, operations such as brightness adjustment, contrast change, saturation adjustment, and hue shift may be used to simulate natural lighting conditions and white balance differences of camera equipment during acquisition. In addition, Gaussian noise and salt and pepper noise may be added to the image to simulate changes in image quality. In hybrid transformation, image mixing and sample splicing operations are used to combine parts of multiple images. For other objects that are prone to interfere with the correct detection of the target to be detected, an erasing operation may be used to reduce their impact.

[0177] In addition, for some images where the target to be detected appears less frequently, a certain number of samples can be generated by copying and pasting.

[0178] It should be noted that the image enhancement preprocessing operations used for the training image and the annotated image corresponding to the training image are the same. The training image may also be subjected to image enhancement preprocessing first, and then the preprocessed training image may be annotated.

[0179] Image enhancement preprocessing can increase the diversity of training data, alleviate the sample imbalance problem to a certain extent, optimize the adaptability of the target detection model to scale changes, and improve the generalization ability and robustness of the target detection model.

[0180] See also Fig.10 In some embodiments, step S802 may include but is not limited to steps S1001 to S1005.

[0181] Step S1001, selecting a cluster center box from multiple real boxes of the annotated image.

[0182] Step S1002: adding the cluster center frame to the cluster data set.

[0183] Step S1003: assigning selection probabilities to other real frames based on the distances between the cluster center frame and other real frames.

[0184] Step S1004, updating the cluster center box based on the selection probabilities of other real boxes, and adding the updated cluster center box to the cluster data set to repeatedly assign the selection probabilities until the number of cluster center boxes in the cluster data set is equal to a preset number.

[0185] Step S1005 , performing mean clustering on the multiple cluster center boxes in the cluster data set to obtain multiple candidate anchor boxes.

[0186] In step S1001 of some embodiments, the annotated image generally includes multiple real boxes. The cluster center box selected for the first time can be randomly determined from the multiple real boxes of the annotated image.

[0187] In step S1002 of some embodiments, the cluster center box is added to the cluster data set. It can be said that the cluster data set stores multiple cluster center boxes selected in the process of determining the candidate anchor boxes.

[0188] In step S1003 of some embodiments, after the cluster center box is determined, selection probabilities are assigned to other real boxes.

[0189] Specifically, the distance from each true box to the cluster center box is calculated, and the square of the distance is used as the weight to assign a selection probability to each point, and the sum of the selection probabilities of each true box is 1. Therefore, the selection probability of the true box farther away from the cluster center box is greater.

[0190] In step S1004 of some embodiments, based on the selection probabilities of each real frame except the cluster center frame, a real frame is randomly selected to update the cluster center frame. In this way, the real frame far away from the cluster center frame will be more likely to be selected.

[0191] The updated cluster center box is added to the cluster data set to repeatedly assign the selection probability, that is, repeat steps S1002 to S1004 until the number of cluster center boxes in the cluster data set is equal to a preset number. The preset number can be set as needed.

[0192] In step S1005 of some embodiments, mean clustering is performed on multiple cluster center boxes in the clustering data set to obtain multiple candidate anchor boxes.

[0193] Specifically, the embodiment of the present application uses the K-means clustering algorithm (K-means) to cluster multiple cluster center boxes in the clustering data set to obtain multiple candidate anchor boxes.

[0194] The embodiments of the above steps S1001 to S1005 will only select one cluster center box at a time. After determining the selection probabilities of other real boxes except the cluster center box, the cluster center box will be updated based on the selection probability. Therefore, the method for determining the candidate anchor box can not only solve the problem that the standard K-means algorithm is prone to fall into the local optimal solution, but also is more conducive to obtaining the global optimal solution. It is more suitable for scenes where the targets to be detected are small and similar in shape, and can calculate candidate anchor boxes that are more in line with the targets to be detected.

[0195] After step S805 in some embodiments, the validation set can also be used to evaluate the performance of the target detection model. Evaluation indicators include precision, recall, average precision (AP) and mean average precision (mAP). Precision refers to the proportion of samples correctly predicted as positive to all samples predicted as positive, and the precision can be expressed as:

[0196]

[0197] Among them, TP is the number of samples predicted as positive samples, and FP is the number of samples predicted as positive samples.

[0198] The recall rate refers to the ratio of the number of correctly predicted positive samples to the number of actual positive samples. The recall rate can be expressed as:

[0199]

[0200] Among them, FN is the number of samples predicted as negative samples.

[0201] The average precision refers to the area under the curve with recall as the horizontal axis and precision as the vertical axis. The larger the area, the higher the average precision and the better the performance of the target detection model. The mean average precision is the average of the average precisions of images of each category.

[0202] If the performance indicators of the target detection model do not meet expectations, the target detection model can be iteratively trained again by optimizing operations such as increasing the data set, and finally a target detection model that meets expectations can be obtained.

[0203] See also Fig.11 In one embodiment, step S105 may include but is not limited to steps S1101 to S1105.

[0204] Step S1101: sort the multiple prediction boxes in descending order based on their confidence scores to obtain a first list.

[0205] Step S1102: Determine the index value of each predicted box based on the position of the predicted box in the first list.

[0206] Step S1103: traverse multiple prediction boxes in the first list, and use the currently traversed prediction box as a reference box to update the confidence score of the prediction box whose index value is greater than the current reference box.

[0207] Step S1104: determine a target box from the multiple prediction boxes based on the updated confidence scores of the multiple prediction boxes.

[0208] In step S1101 of some embodiments, the multiple prediction boxes are sorted in descending order based on the confidence scores of the multiple prediction boxes to obtain a first list. In the first list, the multiple prediction boxes are sorted from large to small according to the confidence scores.

[0209] The confidence score is the product of the probability that the target to be detected exists in the predicted box and the maximum overlap between the predicted box and all real boxes, that is, the intersection and union ratio.

[0210] In step S1102 of some embodiments, the index value of the prediction box is determined based on the position of each prediction box in the first list, and the prediction box with a higher confidence score has a smaller index value. Assuming that the confidence scores corresponding to the multiple prediction boxes in the first list are 0.82, 0.67, and 0.45, respectively, the index value of the prediction box with a confidence score of 0.82 is 1, the index value of the prediction box with a confidence score of 0.67 is 2, and the index value of the prediction box with a confidence score of 0.45 is 3.

[0211] In step S1103 of some embodiments, the prediction boxes are sequentially taken out from the first list as reference boxes in the order of the index values ​​from small to large, and the confidence scores of the prediction boxes with index values ​​greater than the current reference box are updated according to the current reference box. For example, if the index value of the current reference box is 1, the confidence scores of the prediction boxes with index values ​​of 2 and 3 are updated. If the index value of the current reference box is 2, the confidence score of the prediction box with index value of 3 is updated.

[0212] In step S1104 of some embodiments, after the confidence scores of the multiple prediction boxes are updated, a target box is determined among the multiple prediction boxes based on the updated confidence scores of the multiple prediction boxes.

[0213] If the number of target frames is a preset first number, the first number of reference frames can be selected as target frames. In addition, a score threshold can be set to select reference frames with confidence scores greater than the score threshold as target frames.

[0214] The embodiments of steps 1101 to 1104 above update the confidence scores of the prediction boxes and determine the target box according to the confidence scores of the updated prediction boxes. Compared with the related art in which the prediction boxes that do not meet the conditions are directly discarded, the embodiments of the present application can reduce the occurrence of discarding the correct prediction boxes by updating the confidence scores, thereby improving the performance of target detection.

[0215] See also Fig.12 In one embodiment, step S1103 may include but is not limited to steps S1201 to S1204.

[0216] Step S1201: The predicted box at the first position in the first list is used as a reference box, and the index value of the reference box is used as a reference value.

[0217] Step S1202, determining the overlap between the prediction box whose index value is greater than the reference value and the reference box.

[0218] Step S1203 , lowering the confidence score of the prediction box whose overlap with the reference box is greater than a preset threshold.

[0219] Step S1204, determine the next predicted frame of the reference frame in the first list as the updated reference frame, and update the confidence score of the predicted frame whose index value is greater than the current reference frame based on the updated reference frame, until the multiple predicted frames in the first list are traversed.

[0220] In step S1201 of one embodiment, when traversing multiple prediction boxes in the first list, the prediction box at the first position in the first list is first selected, and the prediction box at the first position in the first list, that is, the prediction box with the highest confidence score in the first list is used as the reference box.

[0221] In step S1202 of an embodiment, the overlap between the prediction box whose index value is greater than the reference value and the reference box is determined, and the overlap can be determined by the ratio of the intersection and the union between the prediction box and the reference box in the first list.

[0222] In step S1203 of one embodiment, the confidence score of the prediction box whose overlap with the reference box is greater than a preset threshold is reduced. For the prediction box whose index value is greater than the reference value, if the overlap between the prediction box and the reference box is greater than the preset threshold, it means that the overlap between the prediction box and the reference box is high and there may be redundancy. Therefore, it is necessary to reduce the confidence score of the prediction box.

[0223] In the embodiment of the present application, based on the overlap between the predicted frame whose index value is greater than the reference value and the reference frame in the first list, the confidence score of the predicted frame whose overlap with the reference frame is greater than a preset threshold can be reduced.

[0224] Specifically, the embodiment of the present application can reduce the confidence score of the prediction box whose overlap with the reference box is greater than a preset threshold by linear weight descent or Gaussian weight descent.

[0225] In step S1204 of one embodiment, the next predicted frame of the reference frame in the first list is determined as the updated reference frame, and the confidence score of the predicted frame whose index value is greater than the current reference frame is updated based on the updated reference frame until multiple predicted frames in the first list are traversed.

[0226] After the confidence scores of each prediction box in the first list are updated based on the currently traversed prediction box, the next prediction box of the reference box in the first list is determined as the updated reference box, and steps 1203 and 1204 are repeated to update the confidence scores of the prediction boxes with index values ​​greater than the current reference box based on the updated reference box until multiple prediction boxes in the first list are traversed.

[0227] The above-mentioned embodiment of step S1201 to step S1204 can select a prediction frame with better effect from multiple overlapping prediction frames as the target frame, and better handle small objects such as the target to be detected that appear densely or overlap. When the redundant frame is small or mostly overlapped, it can also be accurately removed, retaining the correct prediction frame, thereby improving the performance of target detection.

[0228] The target detection method provided in the embodiment of the present application processes the target detection image through the target detection model. Specifically, first, the target detection image is subjected to step-by-step feature extraction through the backbone network to obtain multiple first feature maps with different scales. Afterwards, the multiple first feature maps are forwardly fused through the top-down module in the head network to obtain multiple transition feature maps, and the multiple transition feature maps are reversely fused through the bottom-up module including the coordinate attention module in the head network to obtain multiple second feature maps. The coordinate attention module in the head network is used to perform feature focusing on at least one transition feature map, and can obtain feature information of the target to be detected in the horizontal and vertical directions, ensuring long-distance correlation, thereby strengthening the feature extraction of the location of the target to be detected, and improving the detection ability of the target detection model for small targets. Then, the target detection result finally obtained retains the detection result of the small target, reduces the occurrence of missed detection, and improves the detection effect of target detection.

[0229] In addition, refer to Figure 2 , the embodiment of the present application further provides a target detection model, the target detection model includes:

[0230] A backbone network, which is used to extract features of the target detection image step by step to obtain a plurality of first feature maps with different scales;

[0231] The head network includes a top-down module and a bottom-up module, the top-down module is used to perform forward feature fusion on multiple first feature maps with different scales to obtain multiple transition feature maps; the bottom-up module is used to perform reverse feature fusion on multiple transition feature maps to obtain multiple second feature maps, and the top-down module includes a coordinate attention module, the coordinate attention module is used to perform feature focusing on at least one transition feature map;

[0232] The detection network is used to determine the prediction box corresponding to each second feature map, and determine the target box among the determined multiple prediction boxes, so as to determine the target detection result based on the target box and the target detection image.

[0233] The specific implementation of the target detection model is basically the same as the specific implementation of the above-mentioned target detection method, and will not be repeated here.

[0234] See also Fig.13 The present application also provides a target detection device that can implement the above target detection method. The device includes:

[0235] The image input module is used to input the target detection image into the target detection model.

[0236] The feature extraction module is used to extract features of the target detection image step by step through the backbone network to obtain multiple first feature maps with different scales.

[0237] The first fusion module is used to perform forward feature fusion on multiple first feature maps with different scales through a top-down module in the head network to obtain multiple transition feature maps.

[0238] The second fusion module is used to perform reverse feature fusion on multiple transition feature maps through a bottom-up module including a coordinate attention module in the head network to obtain multiple second feature maps. The coordinate attention module is used to perform feature focusing on at least one transition feature map.

[0239] The result determination module is used to determine the prediction box corresponding to each second feature map through the detection network, and determine the target box among the determined multiple prediction boxes, so as to determine the target detection result based on the target box and the target detection image.

[0240] The specific implementation of the target detection device is basically the same as the specific embodiment of the above-mentioned target detection method, and will not be repeated here.

[0241] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above target detection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0242] See also Fig.14 , Fig.14 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0243] The processor 1401 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0244] The memory 1402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1402 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1402, and the processor 1401 calls and executes the target detection method of the embodiment of this application;

[0245] Input / output interface 1403, used to implement information input and output;

[0246] The communication interface 1404 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0247] A bus 1405 that transmits information between the various components of the device (e.g., the processor 1401, the memory 1402, the input / output interface 1403, and the communication interface 1404);

[0248] The processor 1401 , the memory 1402 , the input / output interface 1403 and the communication interface 1404 are connected to each other in communication within the device via a bus 1405 .

[0249] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned target detection method is implemented.

[0250] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0251] In addition, the embodiment of the present application also provides a program product, which can execute the above-mentioned target detection method, and the program product is determined based on the specific application field of the target detection method. If the target detection method provided in the embodiment of the present application is applied to road sign detection, the program product is installed in a vehicle for road sign detection to detect road signs. In addition, the program product can also be set in the cloud, and through communication with the autonomous driving vehicle, the detection of road signs is realized, and the autonomous driving vehicle is notified.

[0252] The target detection method, target detection model, electronic device and storage medium provided in the embodiment of the present application process the target detection image through the target detection model. Specifically, the target detection image is first subjected to step-by-step feature extraction through the backbone network to obtain multiple first feature maps with different scales. Afterwards, the multiple first feature maps are forwardly fused through the top-down module in the head network to obtain multiple transition feature maps, and the multiple transition feature maps are reversely fused through the bottom-up module including the coordinate attention module in the head network to obtain multiple second feature maps. The coordinate attention module in the head network is used to perform feature focusing on at least one transition feature map, and can obtain feature information of the target to be detected in the horizontal and vertical directions, ensuring long-distance correlation, thereby strengthening the feature extraction of the location of the target to be detected, and improving the detection ability of the target detection model for small targets. Then, the target detection result finally obtained retains the detection result of the small target, reduces the occurrence of missed detection, and improves the detection effect of target detection.

[0253] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0254] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0255] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0256] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0257] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0258] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0259] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0260] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0261] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0262] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0263] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A target detection method, characterized in that: The target detection method comprises: The target detection image is input into the target detection model to perform the following process through the target detection model: Performing step-by-step feature extraction on the target detection image through a backbone network to obtain a plurality of first feature maps with different scales; Performing forward feature fusion on a plurality of the first feature maps with different scales through a top-down module in the head network to obtain a plurality of transition feature maps; Performing reverse feature fusion on the plurality of transition feature maps through a bottom-up module including a coordinate attention module in the head network to obtain a plurality of second feature maps, wherein the coordinate attention module is used to perform feature focusing on at least one of the transition feature maps; A prediction box corresponding to each of the second feature maps is determined through a detection network, and a target box is determined among the determined multiple prediction boxes, so as to determine a target detection result based on the target box and the target detection image.

2. The target detection method according to claim 1, characterized in that: The multiple transition feature maps include a transition shallow feature map, a transition middle feature map and a transition deep feature map; The reverse feature fusion is performed on the plurality of transition feature maps through the bottom-up module including the coordinate attention module in the head network to obtain a plurality of second feature maps, including: Based on the coordinate attention module, feature focusing is performed on the transition shallow feature map to obtain a first focused feature map; Fusing the first focused feature map and the transitional middle layer feature map to obtain a first fusion result; Performing feature focusing on the first fusion result based on the coordinate attention module to obtain a second focused feature map; Fusing the second focused feature map and the transition deep feature map to obtain a second fusion result; Based on the transition shallow feature map, the first fusion result and the second fusion result, a plurality of second feature maps are determined.

3. The target detection method according to claim 2, characterized in that: The plurality of first feature maps with different scales include a first shallow feature map, a first middle feature map, and a first deep feature map. The plurality of first feature maps with different scales are forwardly fused by a top-down module in the head network to obtain a plurality of transition feature maps, including: Processing the first deep feature map based on the dilated space pyramid pooling module to obtain a transition deep feature map; Fusing the transition deep layer feature map and the first middle layer feature map to obtain a transition middle layer feature map; The transitional middle layer feature map and the first shallow layer feature map are fused to obtain a transitional shallow layer feature map.

4. The target detection method according to claim 1, characterized in that: The object detection model is trained in the following way: Acquire a training data set, the training data set comprising a plurality of training images and annotated images corresponding to the training images, the annotated images comprising a plurality of real boxes; Clustering the multiple real boxes of the annotated image to obtain multiple candidate anchor boxes; Processing the training image and the plurality of candidate anchor frames by the target detection model to obtain a plurality of predicted training frames; Based on the predicted training frame and the real frame, determining a relative distance and an intersection-over-union ratio between the predicted training frame and the real frame; Based on the relative distance and the intersection-over-union ratio, a bounding box loss value of the target detection model is determined, and the target detection model is trained by the bounding box loss value until the target detection model meets a preset condition.

5. The target detection method according to claim 4, characterized in that: The clustering of the multiple real boxes of the annotated image to obtain multiple candidate anchor boxes includes: Selecting a cluster center box from multiple real boxes in the labeled image; Adding the cluster center box to the clustering data set; Based on the distance between the cluster center box and the other real boxes, assigning a selection probability to the other real boxes; updating the cluster center box based on the selection probabilities of the other real boxes, and adding the updated cluster center box to the cluster data set to repeatedly assign the selection probabilities until the number of the cluster center boxes in the cluster data set is equal to a preset number; Mean clustering is performed on the multiple cluster center boxes in the cluster data set to obtain multiple candidate anchor boxes.

6. The target detection method according to claim 1, characterized in that: The determining of the target frame from the determined multiple prediction frames includes: sorting the plurality of prediction boxes in descending order based on the confidence scores of the plurality of prediction boxes to obtain a first list; Determine an index value of each prediction box based on a position of the prediction box in the first list; Traversing the plurality of prediction frames in the first list, and taking the currently traversed prediction frame as a reference frame, so as to update the confidence score of the prediction frame whose index value is greater than the current reference frame; Based on the updated confidence scores of the multiple prediction boxes, a target box is determined from the multiple prediction boxes.

7. The target detection method according to claim 6, characterized in that: The traversing the plurality of prediction frames in the first list and taking the currently traversed prediction frame as a reference frame to update the confidence score of the prediction frame having an index value greater than the current reference frame includes: Taking the predicted frame at the first position in the first list as a reference frame, and taking the index value of the reference frame as a reference value; Determine the overlap between the prediction frame whose index value is greater than the reference value and the reference frame; Lowering the confidence score of the predicted frame whose overlap with the reference frame is greater than a preset threshold; The next predicted frame of the reference frame in the first list is determined as the updated reference frame, and the confidence score of the predicted frame whose index value is greater than the current reference frame is updated based on the updated reference frame until multiple predicted frames in the first list are traversed.

8. A target detection model, characterized in that: The target detection model includes: A backbone network, wherein the backbone network is used to extract features of the target detection image step by step to obtain a plurality of first feature maps with different scales; A head network, the head network includes a top-down module and a bottom-up module, the top-down module is used to perform forward feature fusion on multiple first feature maps with different scales to obtain multiple transition feature maps; the bottom-up module is used to perform reverse feature fusion on multiple transition feature maps to obtain multiple second feature maps, and the top-down module includes a coordinate attention module, the coordinate attention module is used to perform feature focusing on at least one of the transition feature maps; A detection network is used to determine a prediction box corresponding to each of the second feature maps, and to determine a target box among the determined multiple prediction boxes, so as to determine a target detection result based on the target box and the target detection image.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the target detection method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the target detection method according to any one of claims 1 to 7 is implemented.

Images