Target detection method, device, electronic device and computer readable medium
By constructing a multi-scale feature extraction network and a three-dimensional maximum value filtering network, the problem that existing target detection models cannot achieve end-to-end detection is solved, the detection performance and feature representation ability are improved, and the model structure is simplified.
Patent Information
- Application Number
- CN202011489172.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-16
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-12-16
AI Technical Summary
Existing target detection models cannot achieve end-to-end detection, resulting in less than ideal detection performance and long inference time.
By constructing a multi-scale feature extraction network and combining it with a localization network, a category detection network, and a three-dimensional maximum filtering network, end-to-end target detection is achieved. The three-dimensional maximum filtering network is used to preserve local features and enhance feature representation capabilities.
It improves the detection performance of the object detection model, simplifies the model structure, reduces the training difficulty, and enhances the discriminative power of local features during the inference process.
Smart Images

Figure CN112613544B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and specifically to a target detection method, device, electronic device, and computer-readable medium. Background Art
[0002] Object detection is a technology that finds objects in a given image and determines the category, size, and location of the object. It is one of the basic technologies for computer vision-related businesses. Object detection models usually need to be combined with non-maximum suppression algorithms to obtain the final detection results. Since end-to-end object detection cannot be achieved, the detection performance of the network is not ideal.
[0003] In the existing technology, modules such as recursive neural networks and attention mechanisms can be combined to enable the model to achieve end-to-end target detection, thereby improving the model performance to a certain extent. However, this method will increase the time consumption of model reasoning, and the detection performance of the model is still not ideal. Summary of the invention
[0004] The embodiments of the present application propose a target detection method, device, electronic device and computer-readable medium to solve the technical problem of low detection performance of target detection models in the prior art.
[0005] In a first aspect, an embodiment of the present application provides a target detection method, the method comprising: performing feature extraction on an image to be tested to obtain a multi-scale first feature map; inputting the multi-scale first feature map into a positioning network and a category detection network in a pre-trained target detection model, respectively, to obtain a positioning result and a first category detection result; inputting the feature output by one of the network layers of the positioning network into a three-dimensional maximum filtering network in the target detection model to obtain a second category detection result; fusing the first category detection result and the second category detection result, and obtaining a target detection result based on the positioning result and the fused category detection result.
[0006] In a second aspect, an embodiment of the present application provides a target detection device, comprising: a first detection unit, configured to input the multi-scale first feature map into a positioning network and a category detection network in a pre-trained target detection model, respectively, to obtain a positioning result and a first category detection result; a second detection unit, configured to input the feature output by one of the network layers of the positioning network into a three-dimensional maximum filtering network in the target detection model, to obtain a second category detection result; a fusion unit, configured to fuse the first category detection result and the second category detection result, and obtain a target detection result based on the positioning result and the fused category detection result.
[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method in the first aspect.
[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method in the first aspect.
[0009] The target detection method, device, electronic device and computer-readable medium provided in the embodiments of the present application extract features from the image to be tested to obtain a multi-scale first feature map; then the multi-scale first feature map is respectively input into the positioning network and the category detection network in the pre-trained target detection model to obtain the positioning result and the first category detection result; then the feature output by one of the network layers of the positioning network is input into the three-dimensional maximum filter network in the target detection model to obtain the second category detection result; finally, the first category detection result and the second category detection result are fused, and the target detection result is obtained based on the positioning result and the fused category detection result. Since a three-dimensional maximum filter network is constructed in the target detection model, the model can retain better local features during the reasoning process, so that the model's discrimination of local features is enhanced. Fusion of the output of the three-dimensional maximum filter network with the output of the category detection network can further enhance the expressiveness of the features, thereby improving the detection performance of the target detection model. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:
[0011] Figure 1 is a flow chart of an embodiment of a target detection method according to the present application;
[0012] Figure 2 is a schematic diagram of the structure of a target detection model according to the target detection method of the present application;
[0013] Figure 3 is a schematic diagram of the structure of a three-dimensional maximum filtering network according to the target detection method of the present application;
[0014] Figure 4 is a schematic structural diagram of an embodiment of a target detection device according to the present application;
[0015] Figure 5 It is a schematic diagram of the structure of a computer system for implementing an electronic device in an embodiment of the present application. DETAILED DESCRIPTION
[0016] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0017] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0018] Please refer to Figure 1 , which shows a process 100 of an embodiment of a target detection method according to the present application. The execution subject of the target detection method can be various electronic devices or processors running in electronic devices. For example, the above electronic devices can include but are not limited to servers, smart phones, tablet computers, laptop computers, desktop computers, and wearable devices. The target detection method includes the following steps:
[0019] Step 101: extract features from the image to be tested to obtain a multi-scale first feature map.
[0020] In this embodiment, the execution subject of the target detection method can use the image to be detected as the image to be tested, perform feature extraction on the image to be tested, and obtain a multi-scale first feature map. In practice, a pre-trained feature extraction network can be deployed in the above-mentioned execution subject. The feature extraction network is used to extract features from the image to generate a multi-scale feature map. The above-mentioned execution subject can input the image to be tested into the pre-trained feature extraction network to obtain a multi-scale first feature map.
[0021] In some examples, the feature extraction network may be a feature extraction layer in a pre-trained object detection model for extracting multi-scale features. The network structure of the feature extraction network may be pre-set as needed, for example, a network structure such as a Feature Pyramid Network (FPN) may be used but is not limited thereto.
[0022] Taking the feature pyramid network as an example, the feature pyramid network includes two network structures: bottom-up and top-down. Among them, the bottom-up network structure is the traditional convolutional neural network, which is used to extract image features. As the convolution goes deeper, the scale and resolution of the feature map gradually decrease, the detail features gradually decrease, and the semantic features gradually increase, and finally a multi-scale feature map with scales from high to low can be obtained. The top-down network structure is used to fuse the obtained feature maps in order from small to large scales to obtain the final multi-scale feature map. In the process of generating each feature map, the top-down network structure fuses features of different resolutions and different semantic strengths, thereby ensuring that the feature map of each scale has a suitable resolution and strong semantic features.
[0023] Step 102: Input the multi-scale first feature map into the positioning network and the category detection network in the pre-trained target detection model respectively to obtain the positioning result and the first category detection result.
[0024] In this embodiment, the pre-trained target detection model may include a positioning network and a category detection network. The positioning network may be used to locate the detection target in the image (such as each object in the foreground). The category detection network may be used to detect the category of the detection target in the image.
[0025] As an example, Figure 2 Figure 2 shows a schematic diagram of the structure of the target detection model. Figure 2 As shown, the target detection model includes a feature extraction network, a positioning network, a three-dimensional maximum filtering network and a category detection network. Among them, the feature extraction network can be a feature pyramid network. The feature extraction network is connected to the positioning network and the category detection network respectively, so that the extracted multi-scale first feature map is used as the input of the positioning network and the category detection network. The positioning network and the category detection network can respectively include multiple network layers, and the three-dimensional maximum filtering network can be connected to one of the network layers of the positioning network to use the features output by the network layer as input.
[0026] In this embodiment, the above-mentioned execution subject can input the multi-scale first feature map into the positioning network to obtain the positioning result of the detection target in the image to be tested. The above-mentioned positioning result can characterize the position of the detection target in the image to be tested, and specifically can characterize the position of the detection frame of the target. At the same time, the above-mentioned execution subject can input the multi-scale first feature map into the category detection network to obtain the first category detection result of the detection target in the image to be tested. The above-mentioned first category detection result can characterize the category of the detection target in the image to be tested.
[0027] In some optional implementations of this embodiment, the positioning network and the category detection network may include multiple convolutional layers respectively, and the three-dimensional maximum filtering network may be connected to one of the convolutional layers of the positioning network. Since the first feature map input to the positioning network has multiple scales, after the convolutional layer in the positioning network performs convolution processing on each first feature map, the output second feature map also includes multiple scales.
[0028] Continue to see Figure 2 , the localization network can contain 4 convolutional layers. In addition to the 4 convolutional layers, the category detection network can also contain another network layer (such as Figure 2 The activation function is used to introduce nonlinear factors into the neural network and make the neural network nonlinear. The activation function is continuous and differentiable. Here, the Sigmoid function can be used as the activation function. The Sigmoid function is an S-type function, also known as the S-type growth curve. It can map the input finger to the (0,1) interval.
[0029] In some optional implementations of this embodiment, the target detection model can be pre-trained through the following sub-steps S11 to S13:
[0030] Sub-step S11, obtaining a sample set.
[0031] The sample set may include a large number of image samples. Each image sample may be annotated with the location of the detection target, such as the coordinates of the annotation box (Ground Truth, GT). At the same time, each image sample may be annotated with the category of the detection target, such as the category identifier of the annotation box.
[0032] Sub-step S12, inputting the image sample into the basic model to obtain the candidate detection result output by the basic model.
[0033] The basic model is the model before training, which may include untrained feature extraction network, positioning network, three-dimensional maximum filtering network and category detection network. Its structure can be found in Figure 2 Here, the image sample can be input into the basic model to obtain multiple candidate detection results output by the basic model. Each candidate detection result may include a positioning result (such as the coordinates of the detection box) and a category detection result (such as the category of the detection box).
[0034] Sub-step S13, using the Hungarian algorithm to determine the target detection result among the candidate detection results.
[0035] Here, if the number of candidate detection results is recorded as N, and the number of annotation boxes is recorded as G, then N is usually much larger than G. Here, the Hungarian algorithm can be used to select the best set of candidate detection results from the N candidate detection results, so that the selected best candidate detection results correspond to the annotation boxes one by one. That is, G best candidate detection results are selected from the N candidate detection results.
[0036] As an example, an objective function can be pre-set to evaluate the matching quality of the selected G candidate detection results and the G annotation boxes. The matching quality can be used to characterize the degree of matching. The higher the matching quality, the closer the candidate detection result is to the annotation box, and the better the candidate detection result is. The objective function can be pre-set as needed and is not specifically limited here. In practice, the above objective function can be maximized by the Hungarian algorithm to obtain the optimal G candidate detection results.
[0037] By selecting target detection results through the Hungarian algorithm, the dependence on the non-maximum suppression algorithm can be eliminated, allowing the model to achieve end-to-end training.
[0038] Sub-step S14, training a basic model based on the target detection results and the annotations of the image samples.
[0039] Here, the loss value can be first determined based on the target detection results and the annotations of the image samples. The loss value is the value of the loss function, which is a non-negative real-valued function that can be used to characterize the difference between the detection result and the true result. In general, the smaller the loss value, the better the robustness of the model. The loss function can be set according to actual needs. Afterwards, the loss value can be used to update the parameters of the basic model. In practice, the back-propagation algorithm can be used to obtain the gradient of the loss value relative to the model parameters, and then the gradient descent algorithm can be used to update the model parameters based on the gradient.
[0040] Optionally, two loss functions may be pre-set, namely a first loss function and a second loss function. Based on the target detection result, the annotation of the image sample and the preset first loss function, the first loss value may be determined. Based on the category detection result output by the basic model, the annotation of the image sample (specifically the category annotation) and the preset second loss function, the second loss value may be determined; based on the first loss value and the second loss value, the total loss value may be obtained, so that the basic model may be trained based on the loss value. Among them, the first loss value may act on each network structure in the basic model, and the second loss value may only act on the category detection model in the basic model. By combining the second loss function, more supervision information may be provided to the model, thereby improving the detection performance of the model.
[0041] Sub-step S15, when the basic model training is completed, the trained basic model is determined as the target detection model.
[0042] Here, each time an image sample is input, the parameters of the basic model can be updated once. The above sub-steps S11 to S14 can be iteratively performed multiple times until the model training is completed. In practice, whether the model training is completed can be detected in a variety of ways. As an example, when the accuracy of the detection result output by the basic model reaches a preset value (for example, 99%), it can be determined that the training is completed. As another example, if the number of training times of the basic model is equal to the preset number of times, it can be determined that the training is completed.
[0043] Since there is no need to use the non-maximum suppression algorithm during the training process, end-to-end target detection is achieved, thereby improving the detection performance of the model.
[0044] Step 103: Input the features output by one of the network layers of the positioning network into the three-dimensional maximum filtering network in the target detection model to obtain a second category detection result.
[0045] In this embodiment, the execution subject can input the features output by one of the network layers of the positioning network into the three-dimensional maximum filtering network in the target detection model to obtain the second category detection result. Here, the three-dimensional maximum filtering network can scale the multi-scale second feature map to obtain a three-dimensional feature map, and can perform three-dimensional maximum pooling processing on the three-dimensional feature map to obtain a filtered feature map.
[0046] The three-dimensional maximum filtering network can efficiently filter out the better features and retain the most discriminative parts, thereby increasing the discriminative power of the target detection model for local features. At the same time, the three-dimensional maximum filtering network can fuse multi-scale features to increase the feature expression ability. As a result, the detection performance of the target detection model can be improved.
[0047] In some optional implementations of this embodiment, such as Figure 3 As shown, the three-dimensional maximum filtering network may include a first convolution layer, a three-dimensional maximum filtering layer, a normalization layer and a second convolution layer.
[0048] The above-mentioned normalization layer can be normalized by group normalization (GN) and nonlinear processing can be performed using an activation function. Here, ReLU (Rectified Linear Unit) can be used as an activation function. ReLU, also known as the rectified linear unit, is a maximum value function. It does not have the problem of gradient disappearance, and the calculation is simple, so the operation speed is faster. The output of the above-mentioned second convolutional layer can also be processed using an activation function, such as using a Sigmoid function as an activation function.
[0049] It should be noted that the activation functions used by the normalization layer and the second convolutional layer may also be set in other ways as needed and are not limited to the above examples.
[0050] In some optional implementations of this embodiment, based on the above network structure of the three-dimensional maximum filtering network, the second category detection result can be obtained through the following sub-steps S21 to S23:
[0051] In sub-step S21, the features output by one of the network layers of the positioning network are input into the first convolutional layer to obtain a multi-scale second feature map.
[0052] The features output by one of the network layers of the positioning network may also be a multi-scale feature map. After the features output by the network layer are input to the first convolutional layer, the first convolutional layer may perform convolution processing on the features and output a multi-scale feature map after convolution processing. Here, the feature map after convolution processing output by the first convolutional layer may be referred to as a second feature map.
[0053] Sub-step S22, inputting the multi-scale second feature map into a three-dimensional maximum filtering layer to obtain a multi-scale third feature map.
[0054] After the multi-scale second feature map is input to the three-dimensional maximum filtering layer, the three-dimensional maximum filtering layer can perform interpolation, three-dimensional maximum filtering and other processing on the second feature map, and output a multi-scale filtered feature map. Here, the filtered feature map output by the three-dimensional maximum filtering layer can be called a third feature map.
[0055] In sub-step S23, the multi-scale first feature map and the multi-scale third feature map are fused and input into a normalization layer, and the features output by the normalization layer are input into a second convolutional layer to obtain a second category detection result.
[0056] The execution subject may fuse the multi-scale first feature map and the multi-scale third feature map by element-wise addition or the like. Then, the fused feature map may be input to a normalization layer for normalization. Thereafter, the features output by the normalization layer may be input to a second convolution layer, thereby obtaining a second category detection result.
[0057] In some optional implementations of the present embodiment, in the above sub-step S22, the three-dimensional maximum filtering layer may take each scale of the second feature map as the target scale in turn, and perform bilinear interpolation on the second feature map of at least one scale other than the target scale to convert the at least one scale into the target scale; then, the second feature map of the target scale may be subjected to three-dimensional maximum pooling processing to obtain a third feature map of the target scale.
[0058] In the above implementation, the selection method of at least one scale other than the target scale can be preset as needed. As an example, there are a total of 5 scales of second feature maps, namely A, B, C, D, and E. If the second feature map of scale A is used as the target feature map, one or more scales can be selected from the remaining 4 scales of B, C, D, and E (such as selecting scale B), and the second feature map of the selected scale (such as the second feature map of scale B) is bilinearly interpolated to convert its scale to scale A. As another example, if the second feature map of scale B is used as the target feature map, one or more scales can be selected from the remaining 4 scales of A, C, D, and E (such as selecting scale A and scale C), and the second feature map of the selected scale (such as the second feature map of scale A and scale C) is bilinearly interpolated to convert its scale to scale B.
[0059] In some optional implementations, the following formula may be used for bilinear interpolation:
[0060]
[0061] Wherein, s represents the scale order of the second feature map. For example, if there are five scales of the second feature map, the value range of s is an integer in [1,5]. τ is a preset value, such as 2 or 4, which is not limited here. k is an integer in the value range to which it belongs. is a universal quantification symbol, a mathematical symbol used to represent universal quantifiers. k Represents the second feature map of the k-th scale. Indicates that x k The feature map obtained after interpolation to the sth scale. Bilinear(x k ) represents the k Bilinear interpolation operation. Represents the result of bilinear interpolation of the second feature map of each scale.
[0062] In some optional implementations, an adjacent scale of the target scale may be determined (ie, τ is 2), and bilinear interpolation may be performed on the three-dimensional feature map of the adjacent scale to convert the scale of the three-dimensional feature map of the adjacent scale to the target scale.
[0063] like Figure 3 As shown, for the second feature map x of the sth scale s , if τ is 2, then the second feature map of the s-1th scale and the second feature map of the s+1th scale can be bilinearly interpolated respectively. The result after bilinear interpolation is The interpolated scale is equal to the sth scale.
[0064] In some optional implementations of this embodiment, the above-mentioned execution body may use the following formula to perform three-dimensional maximum pooling processing:
[0065]
[0066] Where i is the coordinate of the feature point in the feature map. The meanings of s, k and τ refer to the previous formula and will not be repeated here. represents the set of feature points in the φ×φ neighborhood of i. j is The feature points within. φ can be set manually, such as 3×3 or 5×5. Representation feature map The maximum eigenvalue of the feature points in the φ×φ neighborhood of i in . It represents the result of the three-dimensional maximum pooling processing at position i when the sth scale is used as the target scale. After the three-dimensional maximum pooling processing is performed on each position i, the pooled feature map is obtained, such as Figure 3 The y in s .
[0067] Through bilinear interpolation, two-dimensional features can be converted into three-dimensional features, so as to support subsequent three-dimensional maximum pooling processing. Through three-dimensional maximum pooling processing, it is possible to efficiently filter out the better features and retain the most discriminative parts to increase the discriminative power of the target detection model for local features. At the same time, it is possible to fuse multi-scale features to increase the feature expression capability. As a result, the detection performance of the target detection model is improved.
[0068] Step 104: Fusing the first category detection result and the second category detection result, and obtaining a target detection result based on the positioning result and the fused category detection result.
[0069] In this embodiment, the execution subject may adopt various fusion methods to fuse the first category detection result and the second category detection result, and obtain the target detection result based on the positioning result and the fused category detection result. By fusing category detection results with different characteristics and sources, the advantages of different network structures can be combined to improve the accuracy of category detection results.
[0070] As an example, the first category detection result and the second category detection result are both multi-channel feature maps, each channel in the multi-channel corresponds to a category. The above execution entity may multiply the feature maps corresponding to the same category element by element to achieve the fusion of the first category detection result and the second category detection result.
[0071] The method provided by the above-mentioned embodiment of the present application obtains a multi-scale first feature map by extracting features from the image to be tested; then the multi-scale first feature map is respectively input into the positioning network and the category detection network in the pre-trained target detection model to obtain the positioning result and the first category detection result; then the feature output by one of the network layers of the positioning network is input into the three-dimensional maximum filter network in the target detection model to obtain the second category detection result; finally, the first category detection result and the second category detection result are fused, and the target detection result is obtained based on the positioning result and the fused category detection result. Since a three-dimensional maximum filter network is constructed in the target detection model, the model can retain better local features during the reasoning process, so that the model's discrimination of local features is enhanced. Fusion of the output of the three-dimensional maximum filter network with the output of the category detection network can further enhance the expressiveness of the features, thereby improving the detection performance of the target detection model.
[0072] Furthermore, the target detection model in the above embodiment does not need to use a non-maximum suppression algorithm for post-processing, which greatly simplifies the structure of the target detection model and reduces the training difficulty. At the same time, it enables the target detection model to achieve end-to-end target detection, thereby improving the detection performance of the target detection model.
[0073] Further references Figure 4 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a target detection device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0074] like Figure 4As shown, the target detection device 400 of this embodiment includes: a feature extraction unit 401, which is configured to perform feature extraction on the image to be tested to obtain a multi-scale first feature map; a first detection unit 402, which is configured to input the above-mentioned multi-scale first feature map into the positioning network and the category detection network in the pre-trained target detection model respectively, to obtain a positioning result and a first category detection result; a second detection unit 403, which is configured to input the features output by one of the network layers of the above-mentioned positioning network into the three-dimensional maximum value filtering network in the above-mentioned target detection model to obtain a second category detection result; a fusion unit 404, which is configured to fuse the above-mentioned first category detection result and the above-mentioned second category detection result, and obtain a target detection result based on the above-mentioned positioning result and the fused category detection result.
[0075] In some optional implementations of this embodiment, the above-mentioned positioning network and the above-mentioned category detection network respectively include multiple convolutional layers, and the above-mentioned three-dimensional maximum filtering network is connected to one of the convolutional layers of the above-mentioned positioning network.
[0076] In some optional implementations of this embodiment, the three-dimensional maximum filtering network includes a first convolution layer, a three-dimensional maximum filtering layer, a normalization layer and a second convolution layer.
[0077] In some optional implementations of this embodiment, the second detection unit 403 is further configured to: input the features output by one of the network layers of the positioning network into the first convolutional layer to obtain a multi-scale second feature map; input the multi-scale second feature map into the three-dimensional maximum filtering layer to obtain a multi-scale third feature map; fuse the multi-scale first feature map and the multi-scale third feature map and input them into the normalization layer, and input the features output by the normalization layer into the second convolutional layer to obtain a second category detection result.
[0078] In some optional implementations of this embodiment, the three-dimensional maximum filtering layer generates the multi-scale third feature map through the following steps: taking each scale of the second feature map as the target scale in turn, and executing the following steps: performing bilinear interpolation on the second feature map of at least one scale other than the target scale to convert the at least one scale into the target scale; performing three-dimensional maximum pooling processing on the second feature map of the target scale to obtain the third feature map of the target scale.
[0079] In some optional implementations of this embodiment, the bilinear interpolation of the second feature map of at least one scale other than the target scale to convert the at least one scale into the target scale includes: determining an adjacent scale of the target scale; and bilinear interpolation of the three-dimensional feature map of the adjacent scale to convert the scale of the three-dimensional feature map of the adjacent scale into the target scale.
[0080] In some optional implementations of this embodiment, the above-mentioned first category detection results and the above-mentioned second category detection results are both multi-channel feature maps, each channel in the above-mentioned multi-channels corresponds to a category; and the further fusion unit 404 is configured to: multiply the feature maps corresponding to the same category element by element.
[0081] In some optional implementations of the present embodiment, the target detection model is trained through the following steps: obtaining a sample set, wherein the sample set includes labeled image samples; inputting the image samples into the basic model to obtain multiple candidate detection results output by the basic model, wherein each candidate detection result includes a category detection result and a positioning result; selecting a target detection result from the multiple candidate detection results using the Hungarian algorithm; training the basic model based on the target detection results and the annotations of the image samples; and when the training of the basic model is completed, determining the trained basic model as the target detection model.
[0082] In some optional implementations of the present embodiment, the above-mentioned basic model is trained based on the above-mentioned detection target and the annotation of the above-mentioned image sample, including: determining a first loss value based on the above-mentioned target detection result, the annotation of the above-mentioned image sample and a preset first loss function; determining a second loss value based on the category detection result output by the above-mentioned basic model, the annotation of the above-mentioned image sample and a preset second loss function; training the above-mentioned basic model based on the above-mentioned first loss value and the above-mentioned second loss value.
[0083] The device provided by the above-mentioned embodiment of the present application obtains a multi-scale first feature map by extracting features from the image to be tested; then the multi-scale first feature map is respectively input into the positioning network and the category detection network in the pre-trained target detection model to obtain the positioning result and the first category detection result; then the feature output by one of the network layers of the positioning network is input into the three-dimensional maximum filter network in the target detection model to obtain the second category detection result; finally, the first category detection result and the second category detection result are fused, and the target detection result is obtained based on the positioning result and the fused category detection result. Since a three-dimensional maximum filter network is constructed in the target detection model, the model can retain better local features during the reasoning process, so that the model's discrimination of local features is enhanced. Fusion of the output of the three-dimensional maximum filter network with the output of the category detection network can further enhance the expressiveness of the features, thereby improving the detection performance of the target detection model.
[0084] Reference below Figure 5 , which shows a schematic structural diagram of an electronic device for implementing some embodiments of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0085] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0086] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a disk, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 5Each block shown in the figure may represent one device, or may represent multiple devices as required.
[0087] In particular, according to some embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of some embodiments of the present application are executed.
[0088] It should be noted that the computer-readable medium described in some embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0089] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0090] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: extracts features from the image to be tested to obtain a multi-scale first feature map; inputs the multi-scale first feature map into the positioning network and the category detection network in the pre-trained target detection model, respectively, to obtain a positioning result and a first category detection result; inputs the features output by one of the network layers of the positioning network into the three-dimensional maximum filtering network in the target detection model to obtain a second category detection result; fuses the first category detection result and the second category detection result, and obtains a target detection result based on the positioning result and the fused category detection result.
[0091] Computer program code for performing the operations of some embodiments of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++; and also conventional procedural programming languages such as "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, or may be connected to an external computer (e.g., through the Internet using an Internet service provider), including a local area network (LAN) or a wide area network (WAN).
[0092] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0093] The units described in some embodiments of the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a first determination unit, a second determination unit, a selection unit, and a third determination unit. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0094] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0095] The above description is only some preferred embodiments of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present application is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with (but not limited to) the technical features with similar functions disclosed in the embodiments of the present application to form a technical solution.
Claims
1. A target detection method, characterized in that: The method comprises: Perform feature extraction on the image to be tested to obtain a multi-scale first feature map; Inputting the multi-scale first feature map into a positioning network and a category detection network in a pre-trained target detection model respectively to obtain a positioning result and a first category detection result, wherein the positioning network is used to locate the detection target in the image, and the category detection network is used to detect the category of the detection target in the image; Inputting the features output by one of the network layers of the positioning network into the three-dimensional maximum filtering network in the target detection model to obtain a second category detection result, wherein the three-dimensional maximum filtering network is used to scale the multi-scale second feature map output by the positioning network to obtain a three-dimensional feature map, and perform three-dimensional maximum pooling processing on the three-dimensional feature map to obtain a filtered feature map, and based on the multi-scale first feature map and the filtered feature map, obtain the second category detection result; The first category detection result and the second category detection result are fused, and a target detection result is obtained based on the positioning result and the fused category detection result.
2. The method according to claim 1, characterized in that The positioning network and the category detection network respectively include a plurality of convolutional layers, and the three-dimensional maximum filtering network is connected to one of the convolutional layers of the positioning network.
3. The method according to claim 1 or 2, characterized in that: The three-dimensional maximum filtering network includes a first convolutional layer, a three-dimensional maximum filtering layer, a normalization layer and a second convolutional layer.
4. The method according to claim 3, characterized in that The step of inputting the features output by one of the network layers of the positioning network into the three-dimensional maximum filtering network in the target detection model to obtain the second category detection result includes: Inputting the features output by one of the network layers of the positioning network into the first convolutional layer to obtain a multi-scale second feature map; Inputting the multi-scale second feature map into the three-dimensional maximum filtering layer to obtain a multi-scale third feature map; The multi-scale first feature map and the multi-scale third feature map are fused and input into the normalization layer, and the features output by the normalization layer are input into the second convolutional layer to obtain a second category detection result.
5. The method according to claim 4, characterized in that The three-dimensional maximum filtering layer generates the multi-scale third feature map through the following steps: Take each scale of the second feature map as the target scale in turn and perform the following steps: Performing bilinear interpolation on the second feature map of at least one remaining scale except the target scale to convert the at least one remaining scale into the target scale; Perform three-dimensional maximum pooling processing on the second feature map of the target scale to obtain a third feature map of the target scale.
6. The method according to claim 5, characterized in that The performing bilinear interpolation on the second feature map of at least one remaining scale except the target scale to convert the at least one remaining scale into the target scale includes: Determining adjacent scales of the target scale; Bilinear interpolation is performed on the three-dimensional feature map of the adjacent scale to convert the scale of the three-dimensional feature map of the adjacent scale to the target scale.
7. The method according to any one of claims 1-2 or 4-6, characterized in that: The first category detection result and the second category detection result are both multi-channel feature maps, and each channel in the multi-channel corresponds to a category; And, fusing the first category detection result and the second category detection result includes: Multiply the feature maps corresponding to the same category element-wise.
8. The method according to claim 3, characterized in that The first category detection result and the second category detection result are both multi-channel feature maps, and each channel in the multi-channel corresponds to a category; And, fusing the first category detection result and the second category detection result includes: Multiply the feature maps corresponding to the same category element-wise.
9. The method according to any one of claims 1-2 or 4-6 or 8, characterized in that: The target detection model is trained through the following steps: Acquire a sample set, wherein the sample set includes labeled image samples; Inputting the image sample into a basic model to obtain a plurality of candidate detection results output by the basic model, each candidate detection result including a category detection result and a positioning result; Selecting a target detection result from the multiple candidate detection results using the Hungarian algorithm; Training the basic model based on the target detection results and the annotations of the image samples; When the basic model training is completed, the trained basic model is determined as the target detection model.
10. The method according to claim 3, characterized in that The target detection model is trained through the following steps: Acquire a sample set, wherein the sample set includes labeled image samples; Inputting the image sample into a basic model to obtain a plurality of candidate detection results output by the basic model, each candidate detection result including a category detection result and a positioning result; Selecting a target detection result from the multiple candidate detection results using the Hungarian algorithm; Training the basic model based on the target detection results and the annotations of the image samples; When the basic model training is completed, the trained basic model is determined as the target detection model.
11. The method according to claim 7, characterized in that The target detection model is trained through the following steps: Acquire a sample set, wherein the sample set includes labeled image samples; Inputting the image sample into a basic model to obtain a plurality of candidate detection results output by the basic model, each candidate detection result including a category detection result and a positioning result; Selecting a target detection result from the multiple candidate detection results using the Hungarian algorithm; Training the basic model based on the target detection results and the annotations of the image samples; When the basic model training is completed, the trained basic model is determined as the target detection model.
12. The method according to claim 9, characterized in that The training of the basic model based on the target detection result and the annotation of the image sample includes: Determining a first loss value based on the target detection result, the annotation of the image sample and a preset first loss function; Determine a second loss value based on the category detection result output by the basic model, the annotation of the image sample and a preset second loss function; The basic model is trained based on the first loss value and the second loss value.
13. The method according to claim 10 or 11, characterized in that: The training of the basic model based on the target detection result and the annotation of the image sample includes: Determining a first loss value based on the target detection result, the annotation of the image sample and a preset first loss function; Determine a second loss value based on the category detection result output by the basic model, the annotation of the image sample and a preset second loss function; The basic model is trained based on the first loss value and the second loss value.
14. A target detection device, characterized in that: The device comprises: A feature extraction unit is configured to perform feature extraction on the image to be tested to obtain a multi-scale first feature map; A first detection unit is configured to input the multi-scale first feature map into a positioning network and a category detection network in a pre-trained target detection model, respectively, to obtain a positioning result and a first category detection result, wherein the positioning network is used to locate the detection target in the image, and the category detection network is used to detect the category of the detection target in the image; A second detection unit is configured to input the features output by one of the network layers of the positioning network into a three-dimensional maximum filtering network in the target detection model to obtain a second category detection result, wherein the three-dimensional maximum filtering network is used to scale the multi-scale second feature map output by the positioning network to obtain a three-dimensional feature map, and perform a three-dimensional maximum pooling process on the three-dimensional feature map to obtain a filtered feature map, and obtain the second category detection result based on the multi-scale first feature map and the filtered feature map; The fusion unit is configured to fuse the first category detection result and the second category detection result, and obtain a target detection result based on the positioning result and the fused category detection result.
15. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 13.
16. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.
