Target detection method and device, storage medium and electronic device

Abstract

The present application discloses a target detection method and device, a storage medium and an electronic device, wherein the method comprises: inputting an image to be detected into a first detection network, obtaining a first detection result output by the first detection network, wherein the first detection result is used to indicate a first target frame containing a detection target in the image to be detected; based on the first target frame, feature fusion is performed on a first multi-scale feature corresponding to the first detection network, obtaining a first fused feature map, wherein the first multi-scale feature is a feature map extracted from the image to be detected by multiple feature extraction networks of the first detection network; inputting the first fused feature map into a second detection network, obtaining a second detection result, wherein the second detection result is used to indicate a second target frame containing a detection target in the image to be detected. Through the present application, the problem of low accuracy in predicting the position of the detection target in the image in the target detection method in the related art is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to a target detection method and device, a storage medium, and an electronic device. Background Art

[0002] At present, deep learning-based target detection methods can be used to automatically detect target objects. When detecting targets, the entire image can be gridded, with different cells responsible for predicting the categories and locations of different targets. However, due to the gridding of the entire image, the prediction accuracy of the target location is low.

[0003] It can be seen that the target detection method in the related art has the problem of low accuracy in predicting the position of the detected target in the image. Summary of the invention

[0004] The embodiments of the present application provide a target detection method and device, a storage medium and an electronic device to at least solve the problem of low accuracy in predicting the position of a detected target in an image in the target detection method in the related art.

[0005] According to one aspect of an embodiment of the present application, a target detection method is provided, including: inputting an image to be detected into a first detection network to obtain first position information of a detection target contained in the image to be detected; performing feature fusion on multiple first sub-features of the image to be detected based on the first position information to obtain a first fusion feature, wherein the multiple first sub-features are obtained by extracting features of the image to be detected based on multiple sub-feature extraction networks connected in series in the first detection network; performing position detection on the detection target based on the first fusion feature to obtain second position information of the detection target.

[0006] In an exemplary embodiment, the step of inputting the image to be detected into a first detection network to obtain first position information of a detection target contained in the image to be detected includes: inputting the image to be detected into the plurality of sub-feature extraction networks connected in series to obtain a target sub-feature output by a last sub-feature extraction network of the plurality of sub-feature extraction networks; and performing position detection on the detection target based on the target sub-feature to obtain the first position information.

[0007] In an exemplary embodiment, the method of fusing the multiple first sub-features of the image to be detected based on the first position information to obtain a first fused feature includes: processing the first position information based on the mapping relationship corresponding to each sub-feature extraction network in the multiple sub-feature extraction networks to obtain intermediate position information corresponding to each sub-feature extraction network; using the intermediate position information corresponding to each sub-feature extraction network, performing feature extraction on the first sub-features obtained by performing feature extraction on the image to be detected by each sub-feature extraction network to obtain the intermediate features corresponding to each sub-feature extraction network; and fusing the intermediate features corresponding to each sub-feature extraction network according to the channel direction to obtain the first fused feature.

[0008] In an exemplary embodiment, the use of the intermediate position information corresponding to each of the sub-feature extraction networks to perform feature extraction on the first sub-features obtained by the sub-feature extraction networks for feature extraction of the image to be detected includes: by retaining floating-point operations and adopting bilinear interpolation, according to the intermediate position information corresponding to each of the sub-feature extraction networks, performing feature extraction on the area of ​​interest in the first sub-features obtained by the sub-feature extraction networks for feature extraction of the image to be detected.

[0009] In an exemplary embodiment, performing position detection on the detection target based on the first fusion feature to obtain second position information of the detection target includes: inputting the first fusion feature into a feature extraction network of a second detection network to obtain a target feature vector corresponding to the first fusion feature; inputting the target feature vector into a regression layer of the second detection network to obtain the second position information output by the regression layer of the second detection network.

[0010] In an exemplary embodiment, before inputting the image to be detected into the first detection network, the method also includes: inputting the sample image into the first detection network to obtain third position information of the sample target contained in the sample image; performing feature fusion on multiple second sub-features of the sample image based on the third position information to obtain a second fused feature, wherein the multiple second sub-features are obtained by extracting features from the sample image based on the multiple sub-feature extraction networks connected in series; repeating the following steps until a stop condition is met: inputting the second fused feature into the initial detection network to obtain fourth position information of the sample target, wherein the initial detection network corresponds to the second detection network; calculating the regression loss between the fourth position information and the labeled position information of the sample target; adjusting the model parameters of the initial detection network according to the regression loss to obtain the updated initial detection network; wherein the stop condition is at least one of the following: the regression loss is less than or equal to the target loss threshold, and the number of repeated executions reaches the target number of times.

[0011] In an exemplary embodiment, after performing position detection on the detection target based on the first fusion feature to obtain second position information of the detection target, the method further includes: performing weighted fusion on the first position information and the second position information to obtain target position information of the detection target.

[0012] According to another aspect of an embodiment of the present application, a target detection device is also provided, including: a first input unit, used to input an image to be detected into a first detection network to obtain first position information of a detection target contained in the image to be detected; a first fusion unit, used to perform feature fusion on multiple first sub-features of the image to be detected based on the first position information to obtain a first fusion feature, wherein the multiple first sub-features are obtained by extracting features from the image to be detected based on multiple sub-feature extraction networks connected in series in the first detection network; a detection unit, used to perform position detection on the detection target based on the first fusion feature to obtain second position information of the detection target.

[0013] In an exemplary embodiment, the first input unit includes: a first input module, used to input the image to be detected into the multiple sub-feature extraction networks connected in series to obtain the target sub-feature output by the last sub-feature extraction network of the multiple sub-feature extraction networks; a detection module, used to perform position detection on the detection target based on the target sub-feature to obtain the first position information.

[0014] In an exemplary embodiment, the first fusion unit includes: a processing module, which is used to process the first position information based on the mapping relationship corresponding to each sub-feature extraction network in the multiple sub-feature extraction networks, and obtain the intermediate position information corresponding to each sub-feature extraction network; an extraction module, which is used to use the intermediate position information corresponding to each sub-feature extraction network to extract the first sub-feature obtained by each sub-feature extraction network performing feature extraction on the image to be detected, and obtain the intermediate feature corresponding to each sub-feature extraction network; a fusion module, which is used to fuse the intermediate features corresponding to each sub-feature extraction network according to the channel direction to obtain the first fused feature.

[0015] In an exemplary embodiment, the extraction module includes: an extraction submodule, which is used to extract features of an area of ​​interest in the first sub-feature obtained by each sub-feature extraction network performing feature extraction on the image to be detected according to the intermediate position information corresponding to each sub-feature extraction network by retaining floating-point operations and adopting bilinear interpolation.

[0016] In an exemplary embodiment, the detection unit includes: a second input module, used to input the first fusion feature into a feature extraction network of a second detection network to obtain a target feature vector corresponding to the first fusion feature; and a third input module, used to input the target feature vector into a regression layer of the second detection network to obtain the second position information output by the regression layer of the second detection network.

[0017] In an exemplary embodiment, the device also includes: a second input unit, which is used to input the sample image into the first detection network before inputting the image to be detected into the first detection network to obtain third position information of the sample target contained in the sample image; a second fusion unit, which is used to perform feature fusion on multiple second sub-features of the sample image based on the third position information to obtain a second fused feature, wherein the multiple second sub-features are obtained by extracting features from the sample image based on the multiple sub-feature extraction networks connected in series; an execution unit, which is used to repeatedly perform the following steps until a stop condition is met: input the second fused feature into the initial detection network to obtain fourth position information of the sample target, wherein the initial detection network corresponds to the second detection network; calculate the regression loss between the fourth position information and the labeled position information of the sample target; adjust the model parameters of the initial detection network according to the regression loss to obtain the updated initial detection network; wherein the stop condition is at least one of the following: the regression loss is less than or equal to the target loss threshold, and the number of repeated executions reaches the target number of times.

[0018] In an exemplary embodiment, the device also includes: a third fusion unit, which is used to perform weighted fusion on the first position information and the second position information to obtain the target position information of the detection target after performing position detection on the detection target based on the first fusion feature to obtain the second position information of the detection target.

[0019] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned target detection method when running.

[0020] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the target detection method through the computer program.

[0021] In an embodiment of the present application, a method of further correcting and fusing the position output of the network is adopted, and the first position information of the detection target contained in the image to be detected is obtained by inputting the image to be detected into the first detection network; based on the first position information, multiple first sub-features of the image to be detected are feature fused to obtain a first fused feature, wherein the multiple first sub-features are obtained by extracting features from the image to be detected based on multiple sub-feature extraction networks connected in series in the first detection network; based on the first fused feature, the position of the detection target is detected to obtain the second position information of the detection target. After obtaining the coarse positioning information of the detection target (i.e., the first position information), the position of the detection target is accurately predicted by combining the coarse positioning information of the detection target and the multi-scale features of the image (i.e., the multiple first sub-features), thereby achieving the technical effect of improving the accuracy of target detection, thereby solving the problem of low accuracy in predicting the position of the detection target in the image in the target detection method in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 is a schematic diagram of a hardware environment of an optional target detection method according to an embodiment of the present application;

[0025] Figure 2 is a flowchart of an optional target detection method according to an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of an optional target detection method according to an embodiment of the present application;

[0027] Figure 4 is a flowchart of another optional target detection method according to an embodiment of the present application;

[0028] Figure 5 is a structural block diagram of an optional target detection device according to an embodiment of the present application;

[0029] Figure 6 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] First, the main technical terms involved in the embodiments of the present application are explained:

[0033] YOLO (You Only Look Once) is a real-time target detection model that can process the entire image end-to-end through a convolutional neural network and decode the detection layer results to directly obtain the location and category of the target object in the image.

[0034] ROI (Region Of Interest) pooling: The ROI pooling operation maps the features of ROIs of different sizes to a fixed size through average or maximum pooling.

[0035] ROI-ALIGN (Region of Interest Matching): A more accurate regional feature extraction operation than ROI pooling. It extracts features from the region of interest by retaining floating-point operations and using bilinear interpolation to avoid losing the original feature map information during the extraction process, thereby more accurately mapping regions of interest of different sizes to a fixed size.

[0036] Anchor box: A preset box of different sizes obtained by statistical or clustering calculations based on the real boxes in the training set. It serves as a reference box for predicting the target position and size, which helps the model converge quickly.

[0037] IOU (Intersection Over Union): It is used to measure the overlap between the predicted area and the true area. The higher the IOU value, the higher the overlap between the two.

[0038] According to one aspect of an embodiment of the present application, a target detection method is provided. Optionally, in this embodiment, the target detection method can be applied to Figure 1 In the hardware environment composed of the terminal 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the terminal 102 via a network, and can be used to provide services (such as target detection services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104.

[0039] The network may include but is not limited to at least one of the following: a wired network, a wireless network. The wired network may include but is not limited to at least one of the following: a wide area network, a metropolitan area network, a local area network, and the wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal 102 may not be limited to a PC, a mobile phone, a tablet computer, etc.

[0040] The target detection method of the embodiment of the present application may be executed by the server 104, or by the terminal 102, or by both the server 104 and the terminal 102. The target detection method of the embodiment of the present application may be executed by the terminal 102 or by a client installed thereon.

[0041] Taking the target detection method in this embodiment executed by the server 104 as an example, Figure 2is a flow chart of an optional target detection method according to an embodiment of the present application, such as Figure 2 As shown, the process of the method may include the following steps:

[0042] Step S202: input the image to be detected into a first detection network to obtain first position information of the detection target contained in the image to be detected.

[0043] The target detection method in this embodiment can be applied to the scenario of predicting the category and position of multiple target objects in a video image. For example, it can be applied to the following scenarios: a traffic checkpoint obtains a target image through a video surveillance device, and then uses target detection technology to identify the target of interest in the image, outputs the classification category and regresses its position coordinates, and gives its bounding box according to the regression result, thereby realizing automatic detection of the target and providing a basis for subsequent analysis and prediction of target attributes, behaviors, etc.

[0044] Optionally, the target detection method in the embodiments of the present application is applicable to real-time detection scenarios of common objects such as pedestrians, vehicles, and non-motor vehicles, for example, motor vehicle detection at highway intersections in infrared scenes at night, pedestrian detection at traffic checkpoints in thermal imaging scenes at night, small target pedestrian detection with panoramic cameras, etc., but is not limited to such targets.

[0045] The server can obtain the image to be detected (which can be a real-time video screen), and input the image to be detected into the first detection network to obtain the first position information of the detection target contained in the image to be detected. The first detection network can be a coarse detection network, and the first position information can be coarse positioning information of the detection target (i.e., the target of interest), which can be a first target frame used to indicate that the detection target is contained in the image to be detected.

[0046] The first detection network can be a detection network based on deep learning, for example, R-FCN (a region-based fully convolutional neural network), YOLO. R-FCN has a good detection effect on the target, but the amount of calculation is large and time-consuming, and it cannot be well applied to real-time target detection scenarios. YOLO directly performs target regression based on the entire input image, which greatly reduces the time consumption and makes full use of the image target context information. Compared with R-FCN, it reduces the possibility of background false alarms. Therefore, for non-real-time target detection scenarios, the first detection network can use R-FCN, and for real-time target detection scenarios, YOLO can be used. In this embodiment, YOLO is used as an example for explanation.

[0047] Step S204, performing feature fusion on multiple first sub-features of the image to be detected based on the first position information to obtain a first fused feature, wherein the multiple first sub-features are obtained by performing feature extraction on the image to be detected based on multiple sub-feature extraction networks connected in series in the first detection network.

[0048] The first detection network may include multiple sub-feature extraction networks, for example, two sub-feature extraction networks, and the connection mode of the multiple sub-feature extraction networks may be a serial connection (i.e., series connection). The image to be detected input into the first detection network will pass through multiple sub-feature extraction networks, and the feature extracted by each sub-feature extraction network is a first sub-feature, and the multiple sub-feature extraction networks output multiple first sub-features. The first sub-feature extracted by each sub-feature extraction network may be an image feature of a scale of the image to be detected, and the multiple first sub-features are multiple scale features corresponding to the image to be detected, that is, multi-scale features corresponding to the image to be detected.

[0049] It should be noted that the sub-features extracted by each sub-feature extraction network may be feature maps (or feature matrices), and the first sub-feature may be a first sub-feature map, which may include at least one feature map. Correspondingly, the multiple first sub-features may be multiple first sub-feature maps. The sizes of the feature maps included in different first sub-features are different, and the number of feature maps included may be the same or different.

[0050] Based on the first position information, the server may perform feature fusion on the multiple first sub-features to obtain a first fused feature. The feature fusion may be performed by splicing or superposition of features. For example, the first target frame and the multiple first sub-feature maps are input into a feature fusion network to obtain a first fused feature map output by the feature fusion network.

[0051] Step S206: Perform position detection on the detection target based on the first fusion feature to obtain second position information of the detection target.

[0052] After obtaining the first fusion feature, the server may perform position detection on the detection target based on the first fusion feature to obtain second position information of the detection target. For example, the server may input the first fusion feature into the second detection network. The second detection network may be a precise detection network, and the second position information may be precise positioning information of the detection target, which may be used to indicate a second target frame containing the detection target in the image to be detected.

[0053] In this embodiment, the first detection network and the second detection network can be considered to be constructed secondary convolutional neural networks. Since the feature fusion uses the sub-features output by the sub-feature extraction network of the first detection network, it is equivalent to the first detection network and the second detection network sharing the convolution layer, which can greatly reduce the computational complexity of the precise detection network model, thereby greatly reducing the model time consumption.

[0054] For example, we can add feature fusion network and precise detection network to the original YOLO detection network, further correct and fuse the position regression output of the network, and achieve precise detection of the target. In addition, we use feature extraction and fusion technology to extract and fuse the intermediate feature layer of the YOLO network, realize feature reuse and reorganization, greatly reduce the calculation amount of the precise detection network model, and shorten the model time.

[0055] Through the above steps, the image to be detected is input into the first detection network to obtain the first position information of the detection target contained in the image to be detected; based on the first position information, multiple first sub-features of the image to be detected are feature fused to obtain a first fused feature, wherein the multiple first sub-features are obtained by extracting features from the image to be detected based on multiple sub-feature extraction networks connected in series in the first detection network; based on the first fused feature, the position of the detection target is detected to obtain the second position information of the detection target, which solves the problem of low accuracy in predicting the position of the detection target in the image in the target detection method in the related art and improves the accuracy of target detection.

[0056] In an exemplary embodiment, inputting an image to be detected into a first detection network to obtain first position information of a detection target contained in the image to be detected includes:

[0057] S11, inputting the image to be detected into a plurality of sub-feature extraction networks connected in series, and obtaining a target sub-feature output by the last sub-feature extraction network of the plurality of sub-feature extraction networks;

[0058] S12: Perform position detection on the detection target based on the target sub-feature to obtain first position information.

[0059] The multiple sub-feature extraction networks can all be lightweight convolutional neural networks, including several convolutional layers and downsampling layers. The multiple sub-feature extraction networks can be connected in sequence (i.e., serially connected), and for adjacent sub-feature extraction networks, the sub-features output by the previous sub-feature extraction network are input to the next sub-feature extraction network after passing through the pooling layer corresponding to the previous sub-feature extraction network.

[0060] When performing rough detection of the detection target (i.e., position detection), the image to be detected can be input into multiple sub-feature extraction networks. Multiple sub-feature extraction networks are connected in series, and different sub-feature extraction networks can extract different sub-features. The output of the last sub-feature extraction network of the multiple sub-feature extraction networks, i.e., the target sub-feature, can be used for position detection.

[0061] Optionally, before the rough detection of the detection target is performed, the image to be detected can be preprocessed, for example, the images of different channels of the image to be detected can be normalized. The above-mentioned preprocessing operation can be implemented by the input layer of the first detection network. The images of multiple channels of the image to be detected can be normalized by the input layer of the first detection network, and uniformly scaled to the target size to obtain a preprocessed image. The above-mentioned multiple channels can include three channels of R (Red), G (Green), and B (Blue). Correspondingly, the server can input the preprocessed image into multiple sub-feature extraction networks in series to obtain the target sub-features output by the last sub-feature extraction network of the multiple sub-feature extraction networks.

[0062] Based on the target sub-features, the position of the detection target can be detected to obtain the first position information. The above position detection can be performed by the detection layer and the regression layer (i.e., the first regression layer) of the first detection network, that is, the target sub-features can be input into the detection layer of the first detection network, and the output of the detection layer is input into the first regression layer of the first detection network to obtain the first position information output by the first regression layer.

[0063] For example, Figure 3 As shown in FIG. 1 , the coarse detection network (for example, the YOLO coarse detection network) mainly includes an input layer, a feature extraction network 1, a feature extraction network 2, a detection layer, and a regression layer 1. The feature extraction network 1 and the feature extraction network 2 are both lightweight convolutional neural networks, including several convolutional layers and downsampling layers. The image to be detected can be input into the input layer of the coarse detection network, and the three channels of R, G, and B are normalized and uniformly scaled to 604*448 size (an example of the target size) to obtain the preprocessed image I 0 .

[0064] Feature extraction network 1 for preprocessed image I 0 Perform multiple convolution and downsampling processes to obtain feature map I 1 , size is 88*56*192; feature extraction network 2 pairs of feature maps I 1 Perform multiple convolution and downsampling processes to obtain feature map I 2 (An example of a target sub-feature), I 2 The size is 44*28*256. Feature map I 2 It can be input into the detection layer, and then into the regression layer 1 (the first regression layer) to perform rough position detection of the target, and obtain the target box A containing the detected target 0 .

[0065] Through this embodiment, the sub-feature output by the last sub-feature extraction network of multiple sub-feature extraction networks is used to perform coarse position detection of the detection target, which can improve the convenience of position detection.

[0066] In an exemplary embodiment, performing feature fusion on a plurality of first sub-features of an image to be detected based on the first position information to obtain a first fused feature includes:

[0067] S21, based on the mapping relationship corresponding to each sub-feature extraction network in the plurality of sub-feature extraction networks, processing the first position information to obtain the intermediate position information corresponding to each sub-feature extraction network;

[0068] S22, using the intermediate position information corresponding to each sub-feature extraction network, extracting features from the first sub-features obtained by each sub-feature extraction network through feature extraction of the image to be detected, to obtain intermediate features corresponding to each sub-feature extraction network;

[0069] S23, fusing the intermediate features corresponding to each sub-feature extraction network according to the channel direction to obtain a first fused feature.

[0070] The sizes of the first sub-features extracted by different sub-feature extraction networks are different, and correspondingly, the mapping relationships used to map the first position information to the position information corresponding to each sub-feature are different, that is, each sub-feature extraction network has its own corresponding mapping relationship. Based on the mapping relationships corresponding to each sub-feature extraction network, the first position information can be processed (for example, position mapping) to obtain the intermediate position information corresponding to each sub-feature extraction network. Optionally, the first position information can be a first target frame, and correspondingly, the intermediate position information corresponding to each sub-feature extraction network can be an intermediate frame corresponding to each sub-feature extraction network.

[0071] For example, a feature fusion network can contain multiple mapping layers, such as Figure 3 As shown, it may include mapping layer 1 (corresponding to mapping relationship 1) and mapping layer 2 (corresponding to mapping relationship 2). The output feature map I obtained by processing the image to be tested by the YOLO network feature extraction layer 1 1 The output feature map I obtained by processing the feature extraction layer 2 2 Input to the feature fusion network. And the target box output by the YOLO network regression layer is input to mapping layer 1 and mapping layer 2 respectively, and the corresponding feature maps I and 1 and I 2 The target frame A 1 and A 2 .

[0072] For the intermediate position information corresponding to each sub-feature extraction network, the server can use the intermediate position information corresponding to each sub-feature extraction network to perform feature extraction on the first sub-feature obtained by each sub-feature extraction network for feature extraction of the image to be detected, and obtain the intermediate feature (first intermediate feature) corresponding to each sub-feature extraction network. The intermediate feature corresponding to each sub-feature extraction network can be an intermediate feature map. The number of intermediate feature maps corresponding to each sub-feature extraction network is the same as the number of first sub-feature maps extracted by each sub-feature extraction network.

[0073] For example, Figure 3 As shown, using target frame A 1 and A 2 For feature map I 1 (88*56*192) and I 2 (44*28*256) is used for feature extraction to obtain feature maps of s*s*192 dimensions and s*s*256 dimensions. The size of s can be configured as needed. For example, s can be 5.

[0074] After obtaining the intermediate features corresponding to each sub-feature extraction network, the server can fuse (e.g., splice) the intermediate features corresponding to each sub-feature extraction network in the channel direction to obtain the first fused feature. For example, the result after feature extraction can be input into the splicing layer of the feature fusion network, and the feature map splicing can be performed along the channel direction to obtain the feature map I with s*s*448 dimensions. 3 .

[0075] Through this embodiment, the feature fusion network is used to extract features through the rough detection results of the target and the multi-scale features, so that the accuracy of feature extraction can be improved.

[0076] In an exemplary embodiment, using the intermediate position information corresponding to each sub-feature extraction network, feature extraction is performed on the first sub-feature obtained by each sub-feature extraction network for feature extraction of the image to be detected, including:

[0077] S31, by retaining floating-point operations and using bilinear interpolation, according to the intermediate position information corresponding to each sub-feature extraction network, feature extraction is performed on the region of interest in the first sub-feature obtained by each sub-feature extraction network for feature extraction of the image to be detected.

[0078] When performing regional feature extraction, feature extraction and fusion can be performed through ROI pooling. Optionally, in this embodiment, ROI-ALIGN can be used instead of ROI pooling for feature extraction and fusion: by retaining floating-point operations and using bilinear interpolation, the intermediate position information corresponding to each sub-feature extraction network can be used (the first intermediate position information, for example, the middle box corresponding to each sub-feature extraction network), and the region of interest in the first sub-feature extracted by each sub-feature extraction network is: the region identified by the intermediate position information corresponding to each sub-feature extraction network in the first sub-feature extracted by each sub-feature extraction network. Feature extraction of the region of interest can avoid the loss of the original feature map information during the extraction process, thereby improving the overall position detection accuracy of the model.

[0079] For example, you can 1 and A 2 The images are input to ROI-ALIGN layer 1 and ROI-ALIGN layer 2 respectively, and feature extraction is performed on the target frame to obtain feature maps of 5*5*192 and 5*5*256 dimensions respectively.

[0080] Through this embodiment, ROI-ALIGN is used for feature extraction and fusion, which can avoid the loss of original feature map information during the extraction process and improve the overall position detection accuracy of the model.

[0081] In an exemplary embodiment, performing position detection on a detection target based on the first fusion feature to obtain second position information of the detection target includes:

[0082] S41, inputting the first fused feature into a feature extraction network of a second detection network to obtain a target feature vector corresponding to the first fused feature;

[0083] S42, input the target feature vector into the regression layer of the second detection network to obtain the second position information output by the regression layer of the second detection network.

[0084] The second detection network (for example, the precise position detection network or the precise detection network) may include a feature extraction network and a regression layer, that is, a second regression layer. The feature extraction network may be a lightweight convolutional neural network, and the second regression layer may be a fully connected layer. The server may first input the first fused feature into the feature extraction network of the second detection network to obtain a target feature vector corresponding to the first fused feature; then, the target feature vector may be input into the regression layer of the second detection network, and the output result of the second regression layer is the second position information mentioned above.

[0085] For example, the precise location detection network may include a feature extraction network 3 and a regression layer 2, wherein the feature extraction network 3 is a lightweight convolutional neural network and the regression layer 2 is a fully connected layer. The precise location detection network may receive the feature map I output by the feature fusion network. 3 After the feature extraction network 3 to I 3 Perform convolution and downsampling to obtain the target feature vector I 4 , and then pass through regression layer 2 to obtain the accurate position detection result.

[0086] Through this embodiment, the feature vector of the fused feature is extracted through the feature extraction network, and the accurate position detection result is output through the regression layer, which can improve the convenience of determining the detection result.

[0087] In an exemplary embodiment, before inputting the image to be detected into the first detection network, the method further includes:

[0088] S51, inputting the sample image into the first detection network to obtain third position information of the sample target contained in the sample image;

[0089] S52, performing feature fusion on a plurality of second sub-features of the sample image based on the third position information to obtain a second fused feature, wherein the plurality of second sub-features are obtained by extracting features from the sample image based on a plurality of sub-feature extraction networks connected in series;

[0090] S53, repeatedly performing the following steps until a stop condition is met: inputting the second fused feature into the initial detection network to obtain fourth position information of the sample target, wherein the initial detection network corresponds to the second detection network; calculating the regression loss between the fourth position information and the position information of the labeled sample target; adjusting the model parameters of the initial detection network according to the regression loss to obtain an updated initial detection network;

[0091] Among them, the stopping condition is at least one of the following: the regression loss is less than or equal to the target loss threshold, and the number of repeated executions reaches the target number of times.

[0092] In order to obtain the second detection network, the second detection network can be trained first. The second detection network can be trained with sample images. The training samples (i.e., sample images) can include common traffic scene images of sample targets such as pedestrians, vehicles, and non-motor vehicles, and the positions of the sample targets are marked in the form of annotation boxes, and the categories of the sample targets can also be marked. Optionally, the first detection network can be trained first, and the second detection network can be trained based on the trained first detection network. Here, the images used for training the first detection network can also be sample images. The first detection network can include multiple sub-feature extraction networks, and can also include an input layer, a detection layer, and a first regression layer.

[0093] For example, a YOLO detection network (i.e., a YOLO rough detection network) can be constructed first, mainly including an input layer, a feature extraction network 1, a feature extraction network 2, a detection layer, and a regression layer 1. The sample image can first be input into the input layer of the initial YOLO detection network, the R, G, and B channels are normalized, and random brightness transformation and random contrast transformation are performed, and then uniformly scaled to 604*448 size (target size, or other sizes, this is only an example), to obtain a preprocessed image F 0 .

[0094] Feature extraction network 1 for preprocessed image F 0 Perform multiple convolution and downsampling processes to obtain the feature map F 1 , F 1 The size can be 88*56*192; the feature extraction network has 2 pairs of feature maps F 1 Perform multiple convolution and downsampling processes to obtain the feature map F 2 , F 2 The size can be 44*28*256.

[0095] When training the YOLO detection network, the feature map F 2 Input to the detection layer, the detection layer detects the feature map F 2 Perform two convolutions to obtain the feature map F 3 , F 3 The size can be 44*28*24, and then the feature map F 3 The input is converted into a rough detection result of the target position by regression layer 1 through the transformation as shown in formula (1). The rough detection result can be an output box corresponding to the target:

[0096]

[0097] Among them, (b x ,b y ,b w ,b h ) is the center point coordinates and length and width of the rough detection result, b x and b y are the horizontal and vertical coordinates of the center point of the rough detection result, b w and b h is the length and width of the rough detection result, (t x ,t y ,t w ,t h ) is the feature map F 3 The prediction result of any pixel in (i.e., the position of the target predicted based on the pixel), t x and ty are the horizontal and vertical coordinates of the center point of the prediction result, t w and t h is the length and width of the prediction result, p w and p h is the length and width of the pixel's anchor box (reference box), c x and c y is the position of the pixel where the center point of the anchor box is located, and σ is the sigmoid function.

[0098] Here, each pixel point will predict at least one result (target position) and the probability value corresponding to each prediction result. Only the prediction result whose corresponding probability value reaches the set threshold will be retained (considered to be a real target box). If the prediction result whose corresponding probability value reaches the set threshold, multiple prediction results can be fused, for example, based on the intersection-over-union ratio, to obtain the above-mentioned rough detection result.

[0099] When training the second detection network, the sample image can be first input into the trained first detection network. After the sample image is input into the first detection network, the third position information (for example, the third target box) output by the regression layer of the first feature network can be obtained, and the third position information can be used to indicate the position of the sample target contained in the sample image in the sample image. The features obtained by extracting features from the sample image by multiple sub-feature extraction networks connected in series in the first detection network are: multiple second sub-features (for example, the multi-scale features in the YOLO coarse detection network can be multiple second sub-feature maps), and the multiple second sub-features correspond one-to-one to the multiple sub-feature extraction networks.

[0100] In this embodiment, the feature fusion network and the second detection network (i.e., the precise detection network) are mainly trained for the prediction results of the first detection network (i.e., the coarse detection network, such as the YOLO coarse detection network) that are close to the real target, so that the prediction results are closer to the real target to achieve precise detection. The prediction results close to the real target can be: the distance between the corresponding third position information and the marked position information is less than or equal to the target distance threshold, and the distance can be the distance of the center point or the intersection and union ratio of the area.

[0101] The feature fusion network can extract and fuse multiple second sub-features to obtain a second fused feature (for example, a second fused feature map). For example, the second position information can be processed based on the mapping relationship corresponding to each sub-feature extraction network to obtain the second intermediate position information corresponding to each sub-feature extraction network; the second intermediate position information corresponding to each sub-feature extraction network can be used to extract the second sub-features obtained by each sub-feature extraction network performing feature extraction on the sample image to obtain the second intermediate features corresponding to each sub-feature extraction network; the second intermediate features corresponding to each sub-feature extraction network can be fused according to the channel direction to obtain the second fused feature.

[0102] The feature fusion network has no trainable parameters and only maps and fuses the features of the first detection network (e.g., the YOLO coarse detection network). Therefore, the feature fusion network does not need to be trained. Figure 3 As shown, the feature fusion network mainly includes feature layer 1, feature layer 2, mapping layer 1, mapping layer 2, ROI-ALIGN layer 1, ROI-ALIGN layer 2 and splicing layer. The detection results of the YOLO coarse detection network are respectively mapped to the feature maps extracted by feature extraction network 1 and feature extraction network 2, the region of interest on the feature map is determined, and the region of interest is processed by ROI-ALIGN to extract the features of the region of interest. The extracted features can be spliced ​​to obtain a fused feature map (second fused feature map).

[0103] During the training phase, the weight parameters of the first detection network can be fixed. Assuming that the similarity between a certain prediction result of the first detection network (third position information, the model parameters corresponding to the prediction result are the model parameters of the first detection network) and the actual target position reaches a set similarity threshold, the prediction result can be used to train the second detection network.

[0104] For example, a prediction result (predicted output box B 0 ) and the IOU value of the real target area is I, and the IOU threshold of the target screening is set to I th , when I>I th When the output box B of the regression layer of the YOLO coarse detection network is 0 Input to mapping layer 1 and mapping layer 2 respectively, and obtain the target box B corresponding to feature layer 1 1 and the target box B corresponding to feature layer 2 2 . 1 and B 2Perform ROI-ALIGN processing on feature layer 1 and feature layer 2 respectively, and obtain feature maps with 5*5*192 dimensions and 5*5*256 dimensions respectively. Then input the above two feature maps into the concatenation layer, concatenate the feature maps along the channel direction, and obtain a feature map F with 5*5*448 dimensions. 4 (i.e., the second fused feature map).

[0105] The second detection network is obtained by training the initial detection network. After obtaining the second fused features, the second fused features can be used to perform multiple rounds of training on the initial detection network to obtain the second detection network. In each round of training, the second fused features can be input into the initial detection network to obtain the fourth position information of the sample target; the regression loss between the fourth position information and the position information of the labeled sample target is calculated; the model parameters of the initial detection network are adjusted according to the regression loss to obtain an updated initial detection network.

[0106] For example, Figure 3 As shown, the feature map F output by the feature fusion network 4 (i.e., the region of interest feature) is input into the precise detection network, and the feature extraction network 3 is used to extract F 4 Perform multiple convolution and downsampling processes to obtain the target feature vector F 5 , and then input into regression layer 2 for accurate position prediction to obtain accurate position detection results.

[0107] When training the precise detection network, the regression loss l between the prediction result of the precise location detection network and the true target of the sample image can be calculated. reg , then according to l reg Back propagation is performed to train and optimize the model parameters of the precise location detection network. When the number of iterations reaches the specified value, or the regression loss l reg When it is less than the specified value, the training is stopped, thereby completing the model training of the precise location detection network.

[0108] Here, the regression loss l reg Smooth L1 loss is used, which is defined as shown in formula (2).

[0109]

[0110] Among them, x is the input data of the loss function.

[0111] Through this embodiment, the characteristic parameters of the network model are obtained through multiple iterative trainings, and the characteristic parameters are gradually optimized during the training process until a better parameter model is obtained, which can improve the rationality of the adjustment of the model characteristic parameters and thus improve the accuracy of target detection.

[0112] In an exemplary embodiment, after performing position detection on the detection target based on the first fusion feature to obtain second position information of the detection target, the method further includes:

[0113] S61, performing weighted fusion on the first position information and the second position information to obtain target position information of the detection target.

[0114] After obtaining the second position information, the second position information can be directly output as the position detection result of the detection target. Optionally, in this embodiment, the first position information and the second position information can be weighted fused to obtain the target position information of the detection target, which is the position detection result of the detection target.

[0115] For example, the coarse detection result output by the YOLO network and the precise detection result output by the precise detection network are weighted and fused to obtain the final position detection result of each target object in the image. The above weighting coefficient can be 1, but is not limited to 1.

[0116] In this embodiment, in addition to the first position information, the first detection result output by the first detection network may also include first category information, and the first category information is used to indicate the target type of the detection target. The target category indicated by the first category information may be determined as the category of the detection target. Optionally, in addition to the second position information, the second detection result output by the second detection network may also include second category information, and the second category information is used to indicate the target type of the detection target. The server may also perform weighted fusion on the first category information and the second category information to obtain the target category information of the detection target. If there are multiple detection targets, the server may obtain the final position information and category information of each detection target in the image to be detected.

[0117] Through this embodiment, the coarse detection result and the precise detection result are weighted and fused, so that the accuracy of target position detection can be improved.

[0118] The target detection method in the embodiment of the present application is explained below in conjunction with an optional example. This optional example provides a target object detection solution based on feature fusion, using the following example: Figure 3 The secondary convolutional neural network based on feature fusion is shown. In this optional example, the first detection network is a YOLO coarse detection network, and the second detection network is a precise detection network (ie, a precise position detection network).

[0119] Combination Figure 4 As shown, the process of the target detection method in this optional example may include the following steps:

[0120] Step S402, using the YOLO coarse detection network to obtain a coarse positioning result and classification result of the target of interest in the input image.

[0121] The detection network can be used to process the input image (an example of the image to be detected) to obtain the coarse positioning result and classification result of the target of interest.

[0122] Step S404, using a feature fusion network to extract and fuse regional features of the multi-scale features and coarse positioning results of the YOLO coarse detection network to obtain fused features.

[0123] The feature fusion network can be constructed by using multiple intermediate feature layers of the YOLO coarse detection network, and the multi-scale features of the image can be extracted and fused in combination with the ROI-ALIGN operation.

[0124] Step S406: Use the precise detection network to combine the coarse positioning information of the target of interest and the multi-scale features of the image to accurately predict the position of the target.

[0125] The constructed precise detection network can be used to accurately predict the position of the target by combining the coarse positioning information of the target of interest and the multi-scale features of the image, thereby improving the detection accuracy of the target of interest.

[0126] Through this optional example, a feature fusion network and a precise detection network are added to the original YOLO detection network, and the network's position regression output is further corrected and fused to achieve precise detection of the target; feature extraction and fusion technology is used to extract and fuse the intermediate feature layer of the YOLO network, realizing the reuse and reorganization of features, greatly reducing the amount of calculation of the precise detection network model and reducing the model time consumption; ROI-ALIGN is used to extract and fuse local features, further improving the detection accuracy of the model.

[0127] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0129] According to another aspect of the embodiments of the present application, a target detection device for implementing the above target detection method is also provided. Figure 5 is a structural block diagram of an optional target detection device according to an embodiment of the present application, such as Figure 5 As shown, the device may include:

[0130] A first input unit 502 is used to input the image to be detected into the first detection network to obtain first position information of the detection target contained in the image to be detected;

[0131] A first fusion unit 504 is connected to the first input unit 502 and is used to perform feature fusion on a plurality of first sub-features of the image to be detected based on the first position information to obtain a first fused feature, wherein the plurality of first sub-features are obtained by extracting features from the image to be detected based on a plurality of sub-feature extraction networks connected in series in the first detection network;

[0132] The detection unit 506 is connected to the first fusion unit 504, and is used to perform position detection on the detection target based on the first fusion feature to obtain second position information of the detection target.

[0133] It should be noted that the first input unit 502 in this embodiment can be used to execute the above step S202, the first fusion unit 504 in this embodiment can be used to execute the above step S204, and the detection unit 506 in this embodiment can be used to execute the above step S206.

[0134] Through the above module, the image to be detected is input into the first detection network to obtain the first position information of the detection target contained in the image to be detected; based on the first position information, multiple first sub-features of the image to be detected are feature fused to obtain a first fused feature, wherein the multiple first sub-features are obtained by extracting features from the image to be detected based on multiple sub-feature extraction networks connected in series in the first detection network; based on the first fused feature, the position of the detection target is detected to obtain the second position information of the detection target, which solves the problem of low prediction accuracy of the target position of interest in the image in the target detection method in the related art and improves the detection accuracy of the target of interest.

[0135] In an exemplary embodiment, the first input unit 502 includes:

[0136] A first input module is used to input the image to be detected into a plurality of sub-feature extraction networks connected in series to obtain a target sub-feature output by a last sub-feature extraction network of the plurality of sub-feature extraction networks;

[0137] The detection module is used to perform position detection on the detection target based on the target sub-feature to obtain first position information.

[0138] In an exemplary embodiment, the first fusion unit 504 includes:

[0139] A processing module, configured to process the first position information based on the mapping relationship corresponding to each sub-feature extraction network in the plurality of sub-feature extraction networks, to obtain the intermediate position information corresponding to each sub-feature extraction network;

[0140] An extraction module is used to use the intermediate position information corresponding to each sub-feature extraction network to extract the first sub-feature obtained by each sub-feature extraction network through feature extraction of the image to be detected, so as to obtain the intermediate features corresponding to each sub-feature extraction network;

[0141] The fusion module is used to fuse the intermediate features corresponding to each sub-feature extraction network according to the channel direction to obtain the first fused feature.

[0142] In an exemplary embodiment, the extraction module includes:

[0143] The extraction submodule is used to extract features of the region of interest in the first sub-features obtained by each sub-feature extraction network for feature extraction of the image to be detected, by retaining floating-point operations and adopting bilinear interpolation, according to the intermediate position information corresponding to each sub-feature extraction network.

[0144] In an exemplary embodiment, the detection unit 506 includes:

[0145] A second input module is used to input the first fused feature into a feature extraction network of a second detection network to obtain a target feature vector corresponding to the first fused feature;

[0146] The third input module is used to input the target feature vector into the regression layer of the second detection network to obtain the second position information output by the regression layer of the second detection network.

[0147] In an exemplary embodiment, the above device further comprises:

[0148] A second input unit is used to input the sample image into the first detection network before inputting the image to be detected into the first detection network to obtain third position information of the sample target contained in the sample image;

[0149] A second fusion unit is used to perform feature fusion on a plurality of second sub-features of the sample image based on the third position information to obtain a second fusion feature, wherein the plurality of second sub-features are obtained by extracting features from the sample image based on a plurality of sub-feature extraction networks connected in series;

[0150] The execution unit is used to repeatedly execute the following steps until a stop condition is met: inputting the second fused feature into the initial detection network to obtain fourth position information of the sample target, wherein the initial detection network corresponds to the second detection network; calculating the regression loss between the fourth position information and the position information of the labeled sample target; adjusting the model parameters of the initial detection network according to the regression loss to obtain an updated initial detection network;

[0151] Among them, the stopping condition is at least one of the following: the regression loss is less than or equal to the target loss threshold, and the number of repeated executions reaches the target number of times.

[0152] In an exemplary embodiment, the above device further comprises:

[0153] The third fusion unit is used to perform position detection on the detection target based on the first fusion feature to obtain second position information of the detection target, and then perform weighted fusion on the first position information and the second position information to obtain target position information of the detection target.

[0154] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware, wherein the hardware environment includes a network environment.

[0155] According to another aspect of the embodiments of the present application, a storage medium is further provided. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above target detection methods in the embodiments of the present application.

[0156] Optionally, in this embodiment, the storage medium may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0157] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0158] S1, inputting the image to be detected into the first detection network to obtain the first position information of the detection target contained in the image to be detected;

[0159] S2, based on the first position information, performing feature fusion on a plurality of first sub-features of the image to be detected to obtain a first fused feature, wherein the plurality of first sub-features are obtained by extracting features of the image to be detected by a plurality of sub-feature extraction networks connected in series in the first detection network;

[0160] S3, performing position detection on the detection target based on the first fusion feature to obtain second position information of the detection target.

[0161] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0162] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0163] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned target detection method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0164] Figure 6 is a structural block diagram of an optional electronic device according to an embodiment of the present application, such as Figure 6 As shown, it includes a processor 602, a communication interface 604, a memory 606 and a communication bus 608, wherein the processor 602, the communication interface 604 and the memory 606 communicate with each other through the communication bus 608, wherein,

[0165] Memory 606, used to store computer programs;

[0166] The processor 602 is used to implement the following steps when executing the computer program stored in the memory 606:

[0167] S1, inputting the image to be detected into the first detection network to obtain the first position information of the detection target contained in the image to be detected;

[0168] S2, based on the first position information, performing feature fusion on a plurality of first sub-features of the image to be detected to obtain a first fused feature, wherein the plurality of first sub-features are obtained by extracting features of the image to be detected by a plurality of sub-feature extraction networks connected in series in the first detection network;

[0169] S3, performing position detection on the detection target based on the first fusion feature to obtain second position information of the detection target.

[0170] Optionally, in this embodiment, the communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The communication interface is used for communication between the electronic device and other devices.

[0171] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0172] As an example, the memory 606 may include, but is not limited to, the first input unit 502, the first fusion unit 504, and the detection unit 506 in the target detection device. In addition, it may also include, but is not limited to, other module units in the target detection device, which will not be repeated in this example.

[0173] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0174] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0175] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only. The device for implementing the above target detection method may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 6 The structure of the electronic device is not limited. Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 6 Different configurations shown.

[0176] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0177] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0178] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0179] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0181] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A target detection method, characterized in that: include: Inputting the image to be detected into the first detection network to obtain first position information of the detection target contained in the image to be detected; Based on the first position information, a plurality of first sub-features of the image to be detected are subjected to feature fusion to obtain a first fusion feature, wherein the plurality of first sub-features are obtained by extracting features of the image to be detected based on a plurality of sub-feature extraction networks connected in series in the first detection network; Performing position detection on the detection target based on the first fusion feature to obtain second position information of the detection target; The step of fusing the plurality of first sub-features of the image to be detected based on the first position information to obtain a first fused feature includes: Based on the mapping relationship corresponding to each sub-feature extraction network in the multiple sub-feature extraction networks, the first position information is processed to obtain the intermediate position information corresponding to each sub-feature extraction network; Using the intermediate position information corresponding to each of the sub-feature extraction networks, feature extraction is performed on the first sub-features obtained by each of the sub-feature extraction networks performing feature extraction on the image to be detected, so as to obtain the intermediate features corresponding to each of the sub-feature extraction networks; The intermediate features corresponding to each of the sub-feature extraction networks are fused according to the channel direction to obtain the first fused feature.

2. The method according to claim 1, characterized in that The step of inputting the image to be detected into the first detection network to obtain first position information of the detection target contained in the image to be detected includes: Inputting the image to be detected into the plurality of sub-feature extraction networks connected in series to obtain a target sub-feature output by a last sub-feature extraction network of the plurality of sub-feature extraction networks; Perform position detection on the detection target based on the target sub-feature to obtain the first position information.

3. The method according to claim 1, characterized in that The step of using the intermediate position information corresponding to each of the sub-feature extraction networks to extract features from the first sub-features obtained by each of the sub-feature extraction networks on the image to be detected includes: By retaining floating-point operations and adopting bilinear interpolation, feature extraction is performed on the region of interest in the first sub-feature obtained by each sub-feature extraction network through feature extraction of the image to be detected according to the intermediate position information corresponding to each sub-feature extraction network.

4. The method according to claim 1, characterized in that: The performing position detection on the detection target based on the first fusion feature to obtain second position information of the detection target includes: Inputting the first fused feature into a feature extraction network of a second detection network to obtain a target feature vector corresponding to the first fused feature; The target feature vector is input into the regression layer of the second detection network to obtain the second position information output by the regression layer of the second detection network.

5. The method according to claim 4, characterized in that Before inputting the image to be detected into the first detection network, the method further includes: Inputting the sample image into the first detection network to obtain third position information of the sample target contained in the sample image; Based on the third position information, a plurality of second sub-features of the sample image are subjected to feature fusion to obtain a second fused feature, wherein the plurality of second sub-features are obtained by extracting features from the sample image based on the plurality of sub-feature extraction networks connected in series; Repeat the following steps until a stop condition is met: input the second fused feature into an initial detection network to obtain fourth position information of the sample target, wherein the initial detection network corresponds to the second detection network; calculate the regression loss between the fourth position information and the labeled position information of the sample target; adjust the model parameters of the initial detection network according to the regression loss to obtain an updated initial detection network; Among them, the stopping condition is at least one of the following: the regression loss is less than or equal to the target loss threshold, and the number of repeated executions reaches the target number of times.

6. The method according to any one of claims 1 to 5, characterized in that After performing position detection on the detection target based on the first fusion feature to obtain second position information of the detection target, the method further includes: The first position information and the second position information are weightedly fused to obtain target position information of the detection target.

7. A target detection device, characterized in that: include: A first input unit, used to input the image to be detected into the first detection network to obtain first position information of the detection target contained in the image to be detected; A first fusion unit is used to perform feature fusion on a plurality of first sub-features of the image to be detected based on the first position information to obtain a first fusion feature, wherein the plurality of first sub-features are obtained by extracting features of the image to be detected based on a plurality of sub-feature extraction networks connected in series in the first detection network; a detection unit, configured to perform position detection on the detection target based on the first fusion feature to obtain second position information of the detection target; The device is also used to process the first position information based on the mapping relationship corresponding to each sub-feature extraction network in the multiple sub-feature extraction networks to obtain the intermediate position information corresponding to each sub-feature extraction network; use the intermediate position information corresponding to each sub-feature extraction network to extract the first sub-feature obtained by the sub-feature extraction network for feature extraction of the image to be detected to obtain the intermediate feature corresponding to each sub-feature extraction network; and fuse the intermediate features corresponding to each sub-feature extraction network according to the channel direction to obtain the first fused feature.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

Images