Target detection method, device, electronic device and storage medium

Abstract

The embodiments of the present application disclose a target detection method, device, electronic device and storage medium, which relate to the field of smart home technology. The method includes: obtaining an image to be detected; inputting the image to be detected into a pre-trained network model, the network model is used to fuse the multi-scale traditional features and multi-scale depth features of the image to be detected according to the corresponding scale, and output the detection result of the target to be detected in the image to be detected based on the fused multi-scale traditional features and multi-scale depth features; and obtaining the detection result of the target to be detected output by the network model. The embodiments of the present application introduce multi-scale traditional features in the network model to guide attention, which can improve the target detection accuracy of the network model in the image to be detected.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a target detection method, device, electronic device and storage medium. Background Art

[0002] Object detection is a very important research direction in the field of computer vision. It can accurately identify and locate objects in images and videos, enable computers to understand the surrounding environment, and achieve good human-computer interaction. Object detection has extremely high application value and broad application prospects. It has been widely used in the fields of autonomous driving, environmental monitoring, traffic security, etc. However, existing object detection methods still have certain defects and need to be optimized. Summary of the invention

[0003] The embodiments of the present application provide a target detection method, device, electronic device and storage medium, which can improve the accuracy of target detection.

[0004] In a first aspect, an embodiment of the present application provides a target detection method, the method comprising: acquiring an image to be detected; inputting the image to be detected into a pre-trained network model, the network model being used to fuse multi-scale traditional features and multi-scale depth features of the image to be detected according to corresponding scales, and outputting a detection result of the target to be detected in the image to be detected based on the fused multi-scale traditional features and multi-scale depth features; and obtaining the detection result of the target to be detected output by the network model.

[0005] In the second aspect, an embodiment of the present application provides a target detection device, which includes: an image acquisition module, used to acquire an image to be detected; an image input module, used to input the image to be detected into a pre-trained network model, the network model is used to fuse the multi-scale traditional features and multi-scale depth features of the image to be detected according to the corresponding scales, and output the detection result of the target to be detected in the image to be detected based on the fused multi-scale traditional features and multi-scale depth features; an image detection module, used to obtain the detection result of the target to be detected output by the network model.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the target detection method as described in the first aspect above.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the target detection method described in the first aspect above are implemented.

[0008] In an embodiment of the present application, an image to be detected is obtained and input into a pre-trained network model to obtain a detection result of the target to be detected output by the network model. The network model is used to fuse the multi-scale traditional features and multi-scale depth features of the image to be detected according to the corresponding scale, and output the detection result of the target to be detected in the image to be detected based on the fused multi-scale traditional features and multi-scale depth features. As a result, the present application introduces more interpretable traditional image features into the network model to guide feature fusion, realize the attention mechanism, optimize the feature extraction of the network model, increase the interpretability of the network model, and improve the target detection accuracy of the network model for the image to be detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic diagram of the structure of a target detection network suitable for an embodiment of the present application is shown.

[0011] Figure 2 A flow chart of a target detection method provided by an embodiment of the present application is shown.

[0012] Figure 3 A flow chart of a target detection method provided by another embodiment of the present application is shown.

[0013] Figure 4 The embodiment of the present application is shown Figure 3 Flowchart of step S220 in FIG.

[0014] Figure 5 A schematic block diagram of a process for performing feature fusion using a first fusion module provided in an embodiment of the present application is shown.

[0015] Figure 6 Another schematic block diagram of the process of performing feature fusion using the first fusion module provided in an embodiment of the present application is shown.

[0016] Figure 7 A schematic block diagram of another process for performing feature fusion using the first fusion module provided in an embodiment of the present application is shown.

[0017] Figure 8 A schematic block diagram of a process for performing feature fusion using a second fusion module provided in an embodiment of the present application is shown.

[0018] Fig. 9 The embodiment of the present application is shown Figure 4 Flowchart of step S225 in FIG.

[0019] Fig.10 A schematic block diagram of a process for performing feature fusion using a third fusion module provided in an embodiment of the present application is shown.

[0020] Fig.11 Another schematic block diagram of the process of performing feature fusion using the third fusion module provided in an embodiment of the present application is shown.

[0021] Fig.12 Another schematic block diagram of the process of performing feature fusion using the third fusion module provided in an embodiment of the present application is shown.

[0022] Fig.13 A schematic diagram of the overall structure of a network model applicable to a target detection method according to an embodiment of the present application is shown.

[0023] Fig.14 A flow chart of a network model training method provided in an embodiment of the present application is shown.

[0024] Fig.15 A block diagram of a target detection device provided by an embodiment of the present application is shown.

[0025] Fig.16 It is a hardware structure block diagram of an electronic device according to an embodiment of the present application.

[0026] Fig.17 It is a computer-readable storage medium in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0028] Although the accuracy of object detection has been greatly improved with the development of deep learning, the existing object detection network SSD (single shot multibox detector) usually uses the backbone network (backbone network, the network part used to extract features from images) of VGG (Visual Geometry Group). Figure 1As shown in the figure, the original image of 300*300*3 (width*height*depth) is feature extracted through 6 convolutional layers, thus obtaining feature maps of 38*38, 19*19, 10*10, 5*5, 3*3, and 1*1 scales, and predictions are made on 38*38, 19*19, 10*10, 5*5, 3*3, and 1*1 branches, respectively. Each branch is independent and not fused, and the feature extraction of the backbone is also not selective and explanatory. Therefore, the existing target detection network still has certain defects and needs to be optimized.

[0029] The inventors found in their research that although the features extracted using convolutional neural networks are not very interpretable, if traditional image features are used to guide the convolutional neural network to extract features, the detection accuracy can be greatly improved for specific tasks, and traditional image features (i.e., manual image features) are also very interpretable. And, as we all know, high-level semantic information is suitable for classification, and low-level semantic information is suitable for positioning. Therefore, after long-term research, the inventors proposed a target detection method, device, electronic device, and storage medium, which can be based on the SSD framework, integrate low-level and high-level semantic features, and introduce traditional image features to guide attention on the detection branch, which can optimize the convolutional neural network feature extraction on the detection branch, increase the interpretability of the system, and improve the target detection accuracy of the network.

[0030] The various embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0031] See also Figure 2 , Figure 2 The following is a flow chart of a target detection method provided by an embodiment of the present application. In a specific embodiment, the target detection method can be applied to electronic devices. The electronic device can be a terminal device such as a mobile phone, a laptop computer, a smart wearable device, a smart TV, a car terminal, or a server such as a local server or a cloud server. Figure 2 The process shown is described in detail, and the target detection method shown can specifically include the following steps:

[0032] Step S110: Acquire the image to be detected.

[0033] In an embodiment of the present application, when target detection is required, the electronic device may first obtain an image to be detected. Among them, the image to be detected may be an image that requires a target detection task, and the image may be a video frame image in a video, which is not limited here. In some embodiments, the image to be detected may include images or photos of various objects and various creatures, etc., which are not limited here. For example, it may be an image containing a pet or a person, or it may be an image of a living room containing various household appliances.

[0034] In some embodiments, the electronic device can collect environmental images from the surrounding environment in real time through its own image acquisition device (such as a camera). Optionally, the collected environmental image can be used as the acquired image to be detected; or the processed image obtained by performing a series of processing on the collected environmental image can be used as the acquired image to be detected, which is not limited here.

[0035] In other embodiments, the electronic device may also obtain the image to be detected from other devices. For example, as a method, after the image acquisition device of the external device acquires the environment image, the environment image is sent to the electronic device, so that the electronic device acquires the image to be detected.

[0036] Of course, it is to be understood that the above-mentioned image to be detected and the acquisition method of the image to be detected are only examples, and do not represent the limitation of the image to be detected and the acquisition method of the image to be detected in the embodiment of the present application. For example, the electronic device may also download the image to be detected from the server. The electronic device may also save the downloaded image to be detected to the storage space of the electronic device, so that the electronic device may also obtain the image to be detected from the local storage.

[0037] Step S120: input the image to be detected into a pre-trained network model, and the network model is used to fuse the multi-scale traditional features and multi-scale depth features of the image to be detected according to the corresponding scales, and output the detection result of the target to be detected in the image to be detected based on the fused multi-scale traditional features and multi-scale depth features.

[0038] The traditional target detection network SSD usually only predicts on feature maps of 38*38, 19*19, 10*10, 5*5, 3*3, and 1*1 scales, and each branch is independent, the extracted features are not selective and interpretative, and the network's target detection accuracy is not high. Therefore, in an embodiment of the present application, a network model that integrates multi-scale traditional features and multi-scale deep features is used to detect targets on the image to be detected. Specifically, the electronic device can input the acquired image to be detected into a pre-trained network model, in which the multi-scale traditional features and multi-scale deep features of the image to be detected can be integrated according to the corresponding scale, and the detection result of the target to be detected in the image to be detected is output based on the fused multi-scale traditional features and multi-scale deep features. Among them, the traditional features can be understood as features that can be extracted by simple operations based on the image to be detected itself. For example, it can be the underlying features of the image such as color, texture, shape, gradient, etc., or it can be a feature obtained by multi-feature fusion processing based on the underlying features, such as the diversity fusion of color and texture. Deep features can be understood as mining and extracting deeper and more abstract features of images through deep neural network models. The features of each scale in the network model (including traditional features and deep features) can be understood as a feature map composed of a set of vectors.

[0039] In an embodiment of the present application, the above-mentioned pre-trained network model can be obtained by training the neural network model in advance according to a large number of training samples. In some embodiments, the training sample may include an image sample and a target annotation sample corresponding to the image sample, wherein the target annotation sample may be a data sample that has been annotated for an object, a creature, etc. in the image sample. Thus, the pre-trained preset model can be used to output the detection result of the target to be detected in the image to be detected according to the acquired image to be detected. Among them, the detection result of the target to be detected may be the annotation of the target to be detected in the image to be detected. The annotation may be annotating the target to be detected in the form of a bounding box (a quadrilateral box with a boundary) in the image to be detected, and the position of the target to be detected in the image to be detected can be determined. In some embodiments, the category attributes of the target to be detected that are identified can also be displayed at the same time. For example, the target to be detected "person" and "car" are framed in red in the image to be detected, as well as the attributes corresponding to the target to be detected, such as "person" and "car".

[0040] In an embodiment of the present application, the network model can be fused according to the corresponding scales based on the multi-scale traditional features and multi-scale deep features of the image to be detected. Among them, fusion according to the corresponding scale can be understood as fusing the traditional features and deep features of the same scale. For example, the traditional features of the 38*38 scale and the deep features of the 38*38 scale are fused; the traditional features of the 19*19 scale and the deep features of the 19*19 scale are fused. By fusing according to the corresponding scale, the correctness and rationality of the feature fusion operation can be guaranteed, and the feature maps at each scale can also have the interpretability of the corresponding scale.

[0041] In some embodiments, the multi-scale traditional features of the image to be detected can be extracted by a traditional feature extraction module outside the network model, and input into the network model, and fused with the multi-scale depth features extracted by the convolutional neural network in the network model. In some embodiments, the traditional feature extraction module can also be placed in the network model to extract the multi-scale traditional features of the image to be detected through the traditional feature extraction module in the network model, extract the multi-scale depth features of the image to be detected through the convolutional neural network in the network model, and fuse the multi-scale traditional features and the multi-scale depth features. In this way, for the network model, only the image to be detected needs to be input, which reduces the complexity of the input parameters of the network model.

[0042] In some embodiments, the multi-scale depth features of the image to be detected can be extracted by a convolutional neural network in a network model, and the convolutional neural network includes multiple convolutional layers, and the feature map with a certain scale output by each convolutional layer is a depth feature of a certain scale. Among them, the convolutional neural network can be a convolutional neural network in an existing SSD network, or it can be a convolutional neural network reasonably set according to a specific target detection scenario. Similarly, the multi-scale depth features of the image to be detected can be depth features of multiple scales that can be extracted using an existing SSD network, or it can be depth features of more or fewer scales that can be extracted after changing the existing SSD network. That is, the embodiments of the present application do not limit the number and size of scales of traditional features and depth features.

[0043] Compared with the existing target detection network that only predicts targets based on the deep feature information on a single independent scale branch, the network model of the present application can integrate traditional features and deep features of multiple scales, and can be combined according to the corresponding scales to associate the traditional feature information and deep feature information of each scale for target prediction, thereby optimizing feature extraction, increasing the interpretability of the network model, and enabling the network model to have better target detection capabilities.

[0044] In some embodiments, the electronic device may preprocess the acquired image to be detected to obtain an image that meets the input standard of the pre-trained network model. Preprocessing may include operations such as deduplication, cropping, splicing, edge fusion, noise reduction, rotation and / or flipping. For example, taking the size of the input parameters of the pre-set network model as "300*300*3 (width*height*depth)", the image to be detected in the original data set can be cropped to a size of "300*300*3". Of course, other preprocessing operations can be further performed on these images to be detected.

[0045] Step S130: Obtain the detection result of the target to be detected output by the network model.

[0046] In an embodiment of the present application, after the image to be detected is input into a pre-trained network model, the detection result of the target to be detected output by the network model can be obtained.

[0047] In some embodiments, the targets to be detected may also be all targets that the network model can identify and predict in the image to be detected, such as outputting the labeling results of all objects in the image to be detected. It is understandable that the targets that the network model can identify can be determined based on the training samples. When the number and types of training samples involved are greater, the network model automatically learns more detection target data and types based on the training samples.

[0048] In other embodiments, the target to be detected may also be a pre-set detection target task. For example, if it is necessary to detect whether there is a face image in a certain picture or a batch of pictures, the network model may only output the labeling results of human images in the images to be detected, where the face image is the target to be detected.

[0049] It is understandable that the above-mentioned various steps in this embodiment can be performed locally by the above-mentioned terminal device, or can be performed in the above-mentioned server, or can be performed by the terminal device and the server in a division of labor. According to different actual application scenarios, tasks can be allocated as needed to achieve the most optimized target detection method, which is not limited here. For example, due to the large amount of computation of the network model, the network model can be deployed on the server. After the terminal device obtains the image to be detected, it uploads the image to be detected to the server. When the server obtains the detection result of the target to be detected in the image to be detected through the network model, the detection result can be returned to the terminal device. It is convenient for the terminal device to be used later.

[0050] An object detection method provided by an embodiment of the present application obtains an image to be detected, and inputs the image to be detected into a pre-trained network model to obtain the detection result of the target to be detected output by the network model. The network model is used to fuse the multi-scale traditional features and multi-scale deep features of the image to be detected according to the corresponding scale, and outputs the detection result of the target to be detected in the image to be detected based on the fused multi-scale traditional features and multi-scale deep features, wherein the network model is obtained by training the neural network model based on the training samples. The present application introduces more interpretable traditional image features in the network model to guide feature fusion, realize the attention mechanism, optimize the feature extraction of the network model, increase the interpretability of the network model, and improve the target detection accuracy of the network model for the image to be detected.

[0051] See also Figure 3 , Figure 3 FIG. 1 is a flow chart of a target detection method provided by another embodiment of the present application, which can be applied to the above electronic device. Figure 3 The process shown is described in detail, and the target detection method shown can specifically include the following steps:

[0052] Step S210: Acquire the image to be detected.

[0053] Step S220: input the image to be detected into a pre-trained network model, the network model is used to fuse the multi-scale traditional features and multi-scale depth features of the image to be detected according to the corresponding scales, and output the detection result of the target to be detected in the image to be detected based on the fused multi-scale traditional features and multi-scale depth features, wherein the network model is obtained by training the neural network model according to the training samples.

[0054] In the embodiment of the present application, step S210 and step S220 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0055] In some embodiments, the traditional features involved in the present application may include but are not limited to: any one or more of Harris corner features, SIFT scale-invariant features, HOG directional gradient histogram features, and LBP local binary features. In some embodiments, traditional features that have been studied for a long time can be directly selected for use according to specific detection tasks. For example, the HOG directional gradient histogram features are used in pedestrian detection, and the Harr features are used in face detection. In some embodiments, the image to be detected can also be filtered for noise reduction, sharpening, edge detection, grayscale mapping, color conversion, etc., and even different interpolation methods can be used to scale the image to be detected to obtain an image to be detected that meets the network model standard.

[0056] Since the extracted high-level semantic features (usually the corresponding small-scale features) are suitable for classification, and the underlying detail features (usually the corresponding large-scale features) are suitable for positioning, the high-level features can be passed down to the bottom layer to be combined with the underlying features, so that the low-order feature maps can also be suitable for classification, thereby improving the prediction accuracy of the network model. Specifically, in some embodiments, the network model can first fuse the multi-scale traditional features to increase the feature information volume of the traditional features of each scale, and avoid inaccurate classification of the underlying features due to the single information. Then, the fused multi-scale traditional features are fused with the multi-scale depth features of the image to be detected according to the corresponding scale fusion. In this way, by first fusing the multi-scale traditional features and then fusing them with the multi-scale depth features of the image to be detected, the accuracy of traditional feature extraction is improved, and the network model can have better feature recognition capabilities.

[0057] The network model may include a first fusion module, a second fusion module and a prediction module. The first fusion module may be used to fuse high-level traditional features with low-level traditional features to obtain a first fused feature after fusion, that is, a fused multi-scale traditional feature; the second fusion module is used to fuse the fused multi-scale traditional features with the multi-scale depth features according to the corresponding scale to obtain a second fused feature after fusion, that is, a fused multi-scale depth feature; the prediction module is used to perform prediction processing on the fused multi-scale depth features to output the detection result of the target to be detected in the image to be detected.

[0058] Specifically, see Figure 4 , step S220 may include:

[0059] Step S221: obtaining the nth layer of traditional features of the image to be detected, where the nth layer is any layer scale of the multi-scale traditional features, and n is a positive integer.

[0060] Step S222: obtaining a high-level traditional feature corresponding to the n-th level traditional feature, wherein the high-level traditional feature is any traditional feature having a scale higher than the n-th level scale.

[0061] Step S223: inputting the n-th layer traditional features and the high-level traditional features into the first fusion module for feature fusion, and obtaining the first fusion features corresponding to the fused n-th layer output by the first fusion module.

[0062] In some embodiments, the network model may include a convolutional neural network, which may include a multi-level convolutional layer for extracting multi-scale depth features. For example, when the convolutional neural network in the network model includes 6 levels of convolutional layers, the multi-scale depth features can be extracted through the 6 levels of convolutional layers. Specifically, the image to be detected is input into the first level of the convolutional layer for feature extraction, and a depth feature map of the first level scale can be obtained; then the depth feature map of the first level scale is input into the second level of the convolutional layer for feature extraction, and a depth feature map of the second level scale can be obtained; then the depth feature map of the second level scale is input into the third level of the convolutional layer for feature extraction, and a depth feature map of the third level scale can be obtained; then the depth feature map of the third level scale is input into the fourth level of the convolutional layer for feature extraction, and a depth feature map of the fourth level scale can be obtained, and the depth feature map of the fourth level scale is continuously input into the fifth level of the convolutional layer for feature extraction, and a depth feature map of the fifth level scale can be obtained; and the depth feature map of the fifth level scale is continuously input into the sixth level of the convolutional layer for feature extraction, and the features of the sixth level scale can be obtained. Therefore, through the feature extraction of multiple levels of convolutional layers, multi-scale deep feature maps can be obtained. It can be understood that the more convolutional layers are experienced, the more abstract the extracted features are, and the richer the semantic information of the features is.

[0063] It should be noted that the level of the convolution layer for feature extraction can be set according to the actual application and is not limited here. The parameters of each level of convolution layer can also be set according to the actual application, which can be 3×3, 5×5, or 7×7, and are not limited here. It can be understood that since the level of the convolution layer is not limited, the parameters of the convolution layer are not obvious, and therefore, the scale of the deep features extracted by the convolutional neural network is not limited.

[0064] In some embodiments, the scale parameters of the multi-scale traditional features of the image to be detected may correspond to the scale parameters of the multi-scale deep features. The scale parameter correspondence may be that the size of the traditional feature map is consistent with the size of the deep feature map. For example, in the deep feature image 5*5*256 processed by the convolutional neural network and the traditional feature image 5*5*1 obtained by the traditional feature processing, the scale size is 5*5.

[0065] As an implementation method, a traditional feature image of the same scale can be generated according to the scale of the extracted deep features. For example, if the deep feature images extracted by the convolutional neural network are of 38*38, 19*19, and 10*10 scales, correspondingly, traditional feature images of 38*38, 19*19, and 10*10 scales can be generated according to the image to be detected.

[0066] In some embodiments, since it is basically difficult to extract traditional features with too small a scale, the number of scale layers of multi-scale traditional features can be smaller than the number of scale layers of multi-scale deep features. For example, if the deep feature maps extracted by the convolutional neural network are of 38*38, 19*19, 10*10, 5*5, 3*3, and 1*1 scales, then correspondingly, traditional feature maps of 38*38, 19*19, 10*10, and 5*5 scales can be generated according to the image to be detected, respectively. For 3*3 and 1*1 scales that are too small, the generation process of the traditional feature map is not performed, because it is basically difficult to extract the traditional features if it is processed.

[0067] In an embodiment of the present application, when multi-scale traditional features and multi-scale depth features of an image to be detected are acquired, in order to improve the accuracy and interpretability of target detection, the multi-scale traditional features and the multi-scale depth features may be fused.

[0068] In some embodiments, before fusing the multi-scale traditional features with the multi-scale deep features, the multi-scale traditional features may be fused to further improve the accuracy of the traditional feature extraction at each scale, thereby improving the accuracy of the features after the fusion of the multi-scale traditional features and the multi-scale deep features. In some embodiments, the fusion of the multi-scale traditional features may be to transmit the high-level traditional features to the low-level traditional features, so that the high-level traditional features are integrated into the low-level traditional features, so that the low-level traditional features also have the semantic information of the high-level traditional features, enriching the feature information of the low-level traditional features, so that after the low-level traditional features are fused with the low-level deep features, the fused low-level deep features may have stronger interpretability.

[0069] Specifically, the n-th layer of traditional features of the image to be detected can be obtained, and the high-level traditional features corresponding to the n-th layer of traditional features can be obtained, and the n-th layer of traditional features and the high-level scale traditional features can be input into the first fusion module for feature fusion, so that the first fusion features corresponding to the fused n-th layer output by the first fusion module can be obtained, thereby realizing the combination of the above-mentioned high-level traditional features and low-level traditional features. Among them, the first fusion module can be to add and fuse the n-th layer of traditional features and their corresponding high-level scale traditional features, that is, to add and sum the corresponding pixel mapping values ​​in the two feature maps.

[0070] The nth layer can be understood as any layer of scale in the above-mentioned multi-scale traditional features, n is a positive integer, and the high-level traditional features can be understood as any traditional features in the above-mentioned multi-scale traditional features whose scale is higher than the nth layer. Since the more convolution layers are passed, the higher and more abstract the scale features extracted are, so the scale higher than the nth layer can also be understood as the number of convolution layers corresponding to the scale is greater than the number of convolution layers corresponding to the nth layer. For example, if the nth layer traditional features are the traditional features of the first layer scale of 38*38, the corresponding high-level traditional features can be the traditional features of the second layer scale of 19*19, the traditional features of the third layer scale of 10*10, or the traditional features of the fourth layer scale of 5*5. Feature fusion can be the fusion of 38*38 traditional features with 19*19 traditional features, or the fusion of 38*38 traditional features with 10*10 traditional features.

[0071] In some embodiments, since the scales of the traditional features to be fused are different, the scales of the traditional features corresponding to different scales need to be adjusted. As a method, a target traditional feature can be determined as the nth layer traditional feature in the multi-scale traditional features, the scale of the target traditional feature can be obtained, and the scale of the corresponding high-level traditional feature can be adjusted according to the scale of the target traditional feature to adjust the scale to be consistent with the scale of the target traditional feature. For example, if the target traditional feature is a traditional feature of 38*38 scale, then its corresponding high-level traditional feature, a traditional feature of 19*19 scale, can be adjusted to 38*38 scale.

[0072] In some embodiments, the scale of the high-level traditional features may be adjusted by an upsampling method, that is, the first fusion module may include an upsampling layer, which may be used to adjust the scale of the high-level traditional features to the scale of the nth layer of traditional features.

[0073] Among them, the upsampling method may refer to inserting new elements between pixels based on the original image pixels of the small-scale feature map (such as high-level traditional features) using a suitable interpolation algorithm to expand the scale of the small-scale feature map. For example, as a method, the scale of the original small-scale feature map is 2×2, and there are 4 pixel mapping values ​​S1, S2, S3 and S4 in the small-scale feature map. 0 can be filled around the four values ​​of S1, S2, S3 and S4, so that an enlarged small-scale feature map with a scale of 4×4 can be obtained. It should be noted that the above-mentioned upsampling method is only an example, and the specific upsampling method is not limited in the embodiments of the present application.

[0074] For example, see Figure 5 , Figure 5A schematic flow chart of feature fusion using the first fusion module is shown, wherein the scale of the n-th layer traditional features is N*N*1, and the scale of the high-level traditional features corresponding to the n-th layer traditional features is N / 2*N / 2*1. Figure 5 It can be seen that after the n-th layer traditional features and the high-level traditional features corresponding to the n-th layer traditional features are input into the first fusion module, the high-level traditional features with a scale of N / 2*N / 2*1 can be expanded to high-level traditional features with a scale of N*N*1 after the upsampling layer in the first fusion module, and then added and fused with the n-th layer traditional features with a scale of N*N*1, so as to output the first fused features corresponding to the n-th layer with a fused scale of N*N*1. Addition and fusion can be understood as the addition of corresponding pixel values. For example, there are 4 pixel mapping values ​​S1, S2, S3 and S4 in the scale feature map A, and there are also 4 pixel mapping values ​​Y1, Y2, Y3 and Y4 in the corresponding positions in the scale feature map B. After the scale feature map A and the scale feature map B are added and fused, the pixel mapping values ​​of the corresponding positions are S1+Y1, S2+Y2, S3+Y3 and S4+Y4.

[0075] Further, in some embodiments, the fusion of multi-scale traditional features may be a step-by-step fusion from high level to low level. Specifically, the high-level traditional features obtained in the above step S222 may be the first fused features corresponding to the n+1th layer after fusion, wherein the first fused features corresponding to the n+1th layer are the fused features output after the n+1th layer traditional features and the high-level traditional features corresponding to the n+1th layer traditional features are input into the first fusion module. In other words, the high-level traditional features corresponding to the nth layer traditional features may be the n+1th layer traditional features that have been fused with higher-level traditional features, so that after the nth layer traditional features are fused with the n+1th layer traditional features that have been fused with the higher-level traditional features, in addition to obtaining the original semantic information of the n+1th layer traditional features, the semantic information of the higher-level traditional features can also be obtained.

[0076] Exemplarily, when traditional features of multiple scales such as 38*38, 19*19, 10*10, and 5*5 are generated respectively according to the image to be detected, the fusion of the multi-scale traditional features can be to fuse the highest-level traditional features (i.e., the fourth-level traditional features) with a scale of 5*5 with the lower-level traditional features (i.e., the third-level traditional features) with a scale of 10*10, thereby obtaining the third-level traditional features fused with the high-level traditional features, i.e., the first fused features with a scale of 10*10, i.e., the first fused features corresponding to the third layer; and then fuse the third-level traditional features fused with the high-level traditional features. The features are fused with the traditional features of the next lower layer with a scale of 19*19 (i.e., the second-layer traditional features) to obtain the second-layer traditional features integrated with the high-level traditional features, i.e., the first fused features with a scale of 19*19, which are also the first fused features corresponding to the second layer; and then the second-layer traditional features integrated with the high-level traditional features are fused with the bottom-layer traditional features with a scale of 38*38 (i.e., the first-layer traditional features) to obtain the first-layer traditional features integrated with the high-level traditional features, i.e., the first fused features with a scale of 38*38, which are also the first fused features corresponding to the first layer.

[0077] For example, see Figure 6 , Figure 6 Another schematic flow chart of feature fusion using the first fusion module is shown, wherein the scale of the traditional features of the nth layer is N*N*1, the scale of the traditional features of the n+1th layer is N / 2*N / 2*1, and the scale of the high-level traditional features corresponding to the traditional features of the n+1th layer is N / 4*N / 4*1. Figure 6 It can be seen that after the high-level traditional features corresponding to the n+1th layer traditional features and the n+1th layer traditional features are input into the first fusion module, the high-level traditional features with a scale of N / 4*N / 4*1 can be expanded to high-level traditional features with a scale of N / 2*N / 2*1 after the upsampling layer in the first fusion module, and then added and fused with the n+1th layer traditional features with a scale of N / 2*N / 2*1, so as to output the first fused features corresponding to the n+1th layer with a fused scale of N / 2*N / 2*1. The first fused features corresponding to the n+1th layer can also be understood as the traditional features of the n+1th layer that integrate the semantic information of the high-level traditional features.

[0078] Then, the traditional features of the n+1th layer fused by the first fusion module can be further fused with the nth traditional features again, so that the nth traditional features can also fuse the semantic information of the high-level traditional features. Specifically, the first fused features corresponding to the n+1th layer and the nth layer traditional features can be input into the first fusion module, and then the first fused features corresponding to the n+1th layer with a scale of N / 2*N / 2*1 can be expanded to the first fused features corresponding to the n+1th layer with a scale of N*N*1 after passing through the upsampling layer in the first fusion module, and then added and fused with the nth layer traditional features with a scale of N*N*1, so as to output the first fused features corresponding to the nth layer with a fused scale of N*N*1, that is, the nth layer traditional features with the semantic information of the high-level traditional features are obtained, wherein the semantic information of the high-level traditional features includes the semantic information of the n+1th layer traditional features and the semantic information of the higher-level traditional features corresponding to the n+1th layer traditional features.

[0079] In other embodiments, the fusion of multi-scale traditional features may also be to transfer all high-level features to low-level features for fusion. Specifically, the high-level traditional features obtained in the above step S222 may also be to obtain all high-level traditional features corresponding to the n-th layer traditional features. In this way, by directly transferring all high-level traditional features to low-level traditional features, since they are directly connected to the lower layer, the semantic information in the high-level traditional features is not changed, so that the first fused features after fusion at each scale can contain the original high-level traditional features, thereby ensuring the semantic information quality of the first fused features at each scale and effectively improving the detection accuracy of the network model.

[0080] For example, see Figure 7 , Figure 7 Another schematic block diagram of a process of feature fusion using the first fusion module is shown, wherein the scale of the traditional features of the nth layer is N*N*1, the scale of the traditional features of the n+1th layer is N / 2*N / 2*1, and the scale of the traditional features of the n+2th layer is N / 4*N / 4*1. Figure 7It can be seen that the nth layer traditional features and the n+1th layer traditional features, the n+2th layer traditional features... can be input into the first fusion module, wherein after being input into the first fusion module, the n+1th layer traditional features with a scale of N / 2*N / 2*1 can be expanded to the n+1th layer traditional features with a scale of N*N*1 after passing through the upsampling layer in the first fusion module, and the n+2th layer traditional features with a scale of N / 4*N / 4*1 can also be expanded to the n+2th layer traditional features with a scale of N*N*1 after passing through the upsampling layer in the first fusion module..., and then expanded to the n+1th layer traditional features with a scale of N*N*1, the n+2th layer traditional features... are added and fused with the nth layer traditional features with a scale of N*N*1, so as to output the first fused features corresponding to the nth layer with a fused scale of N*N*1, and the first fused features corresponding to the nth layer can also be understood as the traditional features of the nth layer that fuse the semantic information of all high-level traditional features.

[0081] In some embodiments, all high-level traditional features may be fused with low-level traditional features, or only some high-level traditional features may be fused with low-level traditional features, which is not limited here. For example, which high-level traditional features to be fused may be reasonably selected based on the amount of computation and the detection accuracy.

[0082] Step S224: input the first fusion feature and the n-th layer depth feature into the second fusion module for feature fusion, and obtain the second fusion feature of the n-th layer after fusion output by the second fusion module, wherein the n-th layer depth feature is a depth feature in the multi-scale depth feature corresponding to the scale of the n-th layer traditional feature.

[0083] In some embodiments, after the fusion of multi-scale traditional features, the fused multi-scale traditional features and multi-scale depth features can be fused so that the fused multi-scale depth features can have stronger explanatory power. Specifically, after obtaining the first fused feature corresponding to the nth layer after the fusion of the nth layer traditional features and the corresponding high-level traditional features, the fused first fused features and the nth layer depth features can be input into the second fusion module for feature fusion. Among them, the nth layer depth features are the depth features corresponding to the scale of the nth layer traditional features in the multi-scale depth features, that is, the scale of the nth layer depth features is consistent with the scale of the nth layer traditional features. Thereby realizing the aforementioned fusion of multi-scale depth features and multi-scale traditional features according to the corresponding scale. Among them, the second fusion module can be to multiply and fuse the fused first fused features with the nth layer depth features.

[0084] In some embodiments, since the number of channels required for the input and output of each convolutional layer in the convolutional neural network is likely to be different, the number of channels can be understood as the number of feature maps, that is, the depth value in the scale parameter. For example, the depth value 3 in 300*300*3 (width*height*depth). Therefore, the scales of the first fused feature (the fused multi-scale traditional feature) and the depth feature to be fused are likely to be different, so it is necessary to adjust the scale of the first fused feature so that the scale of the first fused feature is the same as the scale of the depth feature. Specifically, the scale of the first fused feature corresponding to the nth layer can be adjusted according to the scale of the nth layer depth feature so that the scale of the first fused feature corresponding to the nth layer is consistent with the scale of the nth layer depth feature. For example, when the scale of the nth layer depth feature is 38*38*512, the first fused feature corresponding to the nth layer with a scale of 38*38*1 can be adjusted to 38*38*512.

[0085] In some embodiments, the scale of the first fused feature can be adjusted by convolution. That is, the second fusion module may include a convolution layer, which may include multiple convolution kernels for adjusting the scale of the first fused feature to the scale of the nth layer of depth features. Among them, the number of convolution kernels can be determined according to the number of feature maps of the nth layer of depth features (depth values ​​in the scale parameters), and the parameters of the convolution kernel can be 3*3. Of course, the specific number of convolution kernels and the parameters of the convolution kernel are not limited in the embodiments of the present application. For example, when the scale of the nth layer of depth features is 38*38*512, the first fused feature corresponding to the nth layer with a scale of 38*38*1 can be convolved with 512 convolution kernels to obtain a first fused feature of a scale of 38*38*512.

[0086] In some embodiments, after adjusting the scale of the first fused feature through convolution, the second fusion module can also be used to multiply the first fused feature after scale adjustment with the n-th layer deep feature, and then pass it through the ReLU (Rectified Linear Unit, linear rectification function) layer (the ReLU layer can be used to enhance the nonlinear feature extraction capability of the network model, that is, to express the extracted features nonlinearly) and a 3*3 convolution kernel to obtain the second fused feature corresponding to the fused n-th layer. Among them, the scale of the second fused feature corresponding to the fused n-th layer is the same as the scale of the n-th layer deep feature. Thus, the final feature map on the n-th layer deep feature branch after adding multi-scale traditional features to guide attention can be obtained, and then the target prediction can be performed based on the final feature map, thereby obtaining a more accurate prediction result.

[0087] For example, see Figure 8 , Figure 8A schematic flow chart of feature fusion using the second fusion module is shown, wherein the scale of the first fusion feature corresponding to the nth layer is N*N*1, and the scale of the depth feature of the nth layer is N*N*M. Figure 5 It can be seen that after the first fusion feature corresponding to the nth layer and the nth layer depth feature are input into the second fusion module, the first fusion feature with a scale of N*N*1 is convolved with M 3*3 convolution kernels to obtain the first fusion feature with a scale of N*N*M, and then multiplied with the nth layer depth feature with a scale of N*N*M to obtain the fusion feature with a scale of N*N*M. After that, the multiplied feature map is convolved with the ReLU layer and the 3*3 convolution kernel to obtain the second fusion feature with the same size as the original N*N*M scale of the nth layer.

[0088] Step S225: inputting the second fusion feature into the prediction module, and the prediction module is used to output the detection result of the target to be detected in the image to be detected according to the second fusion feature.

[0089] In some embodiments, after obtaining the second fused feature corresponding to the n-th level scale after the first fused feature and the n-th level depth feature are fused, the second fused feature of the n-th level scale can be predicted. Specifically, the second fused feature can be input into a prediction module, and the prediction module is used to output the detection result of the target to be detected in the image to be detected according to the second fused feature.

[0090] It can be understood that by repeating the aforementioned feature fusion process for each scale of the multi-scale deep features and the multi-scale traditional features, a multi-scale second fusion feature can be obtained, so that the multi-scale second fusion feature can be input into the prediction module, so that the prediction module can output the prediction results at each scale according to the second fusion features at each scale, and comprehensively consider the prediction results at each scale to output the final detection result of the target to be detected in the image to be detected.

[0091] In some embodiments, the prediction module may pre-assign weights of the second fusion feature branches of each scale according to the detection task.

[0092] Since the more convolution layers the image to be detected passes through, the smaller the scale of the generated feature map will be. For smaller targets, the higher the layer of the feature map, the easier it is to lose the feature information of the small target. If the high-level feature map is used for prediction, the accuracy of the prediction result is not high. Therefore, a relatively large feature map (low-level feature) is usually used to detect relatively small targets, and a smaller feature map is used to detect relatively large targets. Therefore, as a way, the weight of the feature map corresponding to the second fusion feature corresponding to each scale can be set according to the size of the target to be detected. For example, when detecting a smaller target, the weight of the low-level second fusion feature branch (usually a larger scale, such as 38*38) can be increased so that the prediction result is mainly affected by this branch, thereby improving the accuracy of the prediction.

[0093] In some embodiments, the prediction module may also pre-allocate the weights of classification and detection in the prediction results based on the importance of the target classification task and the target position detection task. For example, when detection is prioritized, the detection weight may be set to 0.6 and the classification branch may be set to 0.4, so that the weight of the detection branch is greater than that of the classification branch.

[0094] In some embodiments, after obtaining the multi-scale second fusion feature after the fusion of the multi-scale traditional feature and the multi-scale deep feature, the multi-scale second fusion feature can be fused again on the prediction path, and the target inspection can be performed according to the fused prediction path. Therefore, in some embodiments, the network model can also include a third fusion module. The third fusion module can be used to fuse the high-level second fusion feature with the low-level second fusion feature to obtain the fused multi-scale second fusion feature.

[0095] Specifically, see Fig. 9 , the above step S225 may also include:

[0096] Step S2251: Obtain the second fusion features of the m-th layer, where the m-th layer is any layer scale of the multi-scale second fusion features, and m is a positive integer.

[0097] Step S2252: obtaining a high-level fusion feature corresponding to the second fusion feature of the m-th layer, wherein the high-level fusion feature is any second fusion feature having a scale higher than that of the m-th layer.

[0098] Step S2253: input the second fusion feature of the m-th layer and the high-level fusion feature into the third fusion module for feature fusion, and obtain the fused second fusion feature of the m-th layer output by the third fusion module.

[0099] In some embodiments, before performing target prediction based on the second fused features that fuse the multi-scale traditional features and the multi-scale deep features, the multi-scale second fused features may be fused to further improve the accuracy of feature extraction at each scale, thereby improving the prediction accuracy on each scale branch. In some embodiments, the fusion between the multi-scale second fused features may be to transmit the high-level second fused features to the low-level second fused features, so that the high-level second fused features are integrated into the low-level second fused features, so that the low-level second fused features also have the semantic information and interpretability of the high-level second fused features, enriching the feature information and interpretability of the low-level second fused features, and also improving the prediction accuracy on each scale branch.

[0100] Specifically, the second fusion feature of the mth layer of the image to be detected can be obtained by obtaining the second fusion feature of the mth layer and the high-level fusion feature corresponding to the second fusion feature of the mth layer, and the second fusion feature of the mth layer and the high-level fusion feature can be input into the third fusion module for feature fusion, so that the fused second fusion feature of the mth layer output by the third fusion module can be obtained, thereby realizing the combination of the above-mentioned high-level second fusion feature and the low-level second fusion feature. Among them, the third fusion module can be to add and fuse the second fusion feature corresponding to the mth layer and its corresponding high-level fusion feature, that is, to add and sum the corresponding pixel mapping values ​​in the two feature maps.

[0101] The mth layer can be understood as any layer scale of the multi-scale second fusion features, m is a positive integer, and the high-level fusion features can be understood as any second fusion features with a scale higher than the mth layer scale in the multi-scale second fusion features. For example, if the mth layer traditional features are the second fusion features corresponding to the first layer scale of 38*38, the corresponding high-level fusion features can be the second fusion features corresponding to the second layer scale of 19*19, or the second fusion features corresponding to the third layer scale of 10*10, or the second fusion features corresponding to the fourth layer scale of 5*5. Feature fusion can be the fusion of the second fusion features corresponding to 38*38 and the second fusion features corresponding to 19*19, or the fusion of the second fusion features corresponding to 38*38 and the second fusion features corresponding to 10*10.

[0102] In some embodiments, since the scales of the second fusion features to be fused are different, it is necessary to adjust the scales of the second fusion features corresponding to different scales. As a method, a target second fusion feature can be determined as the second fusion feature corresponding to the mth layer in the multi-scale traditional features, the scale of the target second fusion feature can be obtained, and the scale of the corresponding high-level fusion feature can be adjusted according to the scale of the target second fusion feature to adjust the scale to be consistent with the scale of the target second fusion feature. For example, if the target second fusion feature is the second fusion feature corresponding to the scale of 38*38*512, then its corresponding high-level fusion feature, the second fusion feature corresponding to the scale of 19*19*1024, can be adjusted to a scale of 38*38*512.

[0103] In some embodiments, the scale of the high-level fusion feature can be adjusted by convolution and upsampling methods. That is, the first fusion module may include an upsampling layer and a convolution layer. The upsampling layer can be used to adjust the width and height values ​​in the scale of the high-level fusion feature to the width and height values ​​in the scale of the second fusion feature corresponding to the mth layer, and the convolution layer can be used to adjust the depth value (i.e., the number of feature maps) in the scale of the high-level fusion feature to the depth value in the scale of the second fusion feature corresponding to the mth layer.

[0104] For example, see Fig.10 , Fig.10 A schematic flow chart of feature fusion using the third fusion module is shown, wherein the scale of the second fusion feature corresponding to the mth layer is N*N*M, and the scale of the high-level fusion feature corresponding to the second fusion feature corresponding to the mth layer is N / 2*N / 2*P. Fig.10 It can be seen that after the second fusion feature corresponding to the m-th layer and its corresponding high-level fusion feature are input into the third fusion module, the high-level fusion feature with a scale of N / 2*N / 2*P can be expanded to a high-level fusion feature with a scale of N*N*M after passing through multiple 3*3 convolution kernels and upsampling layers (the upsampling layer is not shown in the figure) in the third fusion module, so as to have the same scale as the second fusion feature branch corresponding to the m-th layer, and then expanded to a high-level fusion feature with a scale of N*N*M and added and fused with the second fusion feature corresponding to the m-th layer with a scale of N*N*M, so as to output the second fusion feature corresponding to the m-th layer with a fused scale of N*N*M, and the fused second fusion feature of the m-th layer after the fusion integrates the semantic information and interpretability of the high-level second fusion feature.

[0105] Similar to the fusion of the aforementioned multi-scale traditional features, in some embodiments, the fusion of the multi-scale second fusion features can also be to transfer all high-level fusion features to the second fusion features of the lower layer for fusion. Specifically, the acquisition of high-level fusion features in the above step S2252 can be to obtain all high-level fusion features corresponding to the second fusion features of the mth layer. In this way, by directly transferring all high-level fusion features to the low-level fusion features, since they are directly connected to the lower layer, the semantic information in the high-level fusion features is not changed, so that the second fusion features after fusion at each scale can contain the original high-level fusion features, thereby ensuring the semantic information quality of the second fusion features after fusion at each scale, and effectively improving the detection accuracy of the network model.

[0106] Exemplarily, when second fusion features of multi-scale sizes such as 38*38*512, 19*19*1024, 10*10*512, and 5*5*256 are obtained after fusion of multi-scale traditional features and multi-scale deep features according to the image to be detected, the fusion of the multi-scale second fusion features can be to fuse the second fusion features corresponding to the scales of 5*5*256, 10*10*512, and 19*19*1024 with the second fusion features corresponding to the bottom layer with a scale of 38*38*512 (that is, the second fusion features corresponding to the first layer), so that the second fusion features corresponding to the first layer fuse all high-level fusion features to obtain the second fusion features of the first layer after fusion, that is, the second fusion features with a scale of 38*38*512 after fusion are obtained; and then the second fusion features with a scale of 5*5* The second fusion features corresponding to the scale size of 256 and 10*10*512 are fused with the second fusion features corresponding to the scale size of 19*19*1024 (that is, the second fusion features corresponding to the second layer), so that the second fusion features corresponding to the second layer fuse all its high-level fusion features to obtain the second fusion features of the second layer after fusion, that is, the second fusion features with a fused scale size of 19*19*1024 are obtained; and then the second fusion features corresponding to the scale size of 5*5*256 are fused with the second fusion features corresponding to the scale size of 10*10*512 (that is, the second fusion features corresponding to the third layer), so that the second fusion features corresponding to the third layer fuse all its high-level fusion features to obtain the second fusion features of the third layer after fusion, that is, the second fusion features with a fused scale size of 10*10*512 are obtained.

[0107] For example, see Fig.11 , Fig.11A schematic block diagram of a process of feature fusion using the third fusion module is shown, wherein the scale of the second fusion feature corresponding to the m+1th layer is N / 2*N / 2*P, the scale of the second fusion feature corresponding to the m+2th layer is N / 4*N / 4*Q, and the scale of the second fusion feature corresponding to the mth layer is N*N*M. Fig.11 It can be seen that the second fusion features corresponding to the mth layer, the second fusion features corresponding to the m+1th layer, the second fusion features corresponding to the m+2th layer...are input into the third fusion module, wherein after being input into the third fusion module, the second fusion features corresponding to the m+1th layer with a scale of N / 2*N / 2*P are expanded to the second fusion features corresponding to the m+1th layer with a scale of N*N*M after the 3*3 convolution kernel and upsampling layer (the upsampling layer is not drawn in the figure) in the third fusion module, and the second fusion features corresponding to the m+2th layer with a scale of N / 4*N / 4*Q are expanded to the second fusion features corresponding to the m+1th layer with a scale of N*N*M after the 3*3 convolution kernel and upsampling layer (the upsampling layer is not drawn in the figure) in the third fusion module. After the accumulation kernel and the upsampling layer (the upsampling layer is not drawn in the figure), it can also be expanded to the second fused features corresponding to the m+2th layer with a scale of N*N*M..., and then the second fused features corresponding to the m+1th layer expanded to a scale of N*N*M, the second fused features corresponding to the m+2th layer... are added and fused with the second fused features corresponding to the mth layer with a scale of N*N*M, so as to output the second fused features corresponding to the mth layer with a fused scale of N*N*M. The second fused features corresponding to the mth layer can also be understood as the second fused features of the mth layer that integrate the semantic information and explanatory information of all high-level fused features.

[0108] Similar to the fusion of the aforementioned multi-scale traditional features, in other embodiments, the fusion of the multi-scale traditional features may also be a step-by-step fusion from high level to low level. Specifically, the acquisition of the high-level fusion feature in the above step S2252 may also be the acquisition of the second fusion feature after the fusion of the m+1th layer. The second fusion feature after the fusion of the m+1th layer is the fusion feature output after the second fusion feature of the m+1th layer and the high-level fusion feature corresponding to the second fusion feature of the m+1th layer are input into the third fusion module. In other words, the high-level fusion feature corresponding to the second fusion feature of the mth layer may be the second fusion feature of the m+1th layer that has been fused with the fusion feature of a higher level, so that the second fusion feature of the mth layer, after the feature fusion of the second fusion feature of the m+1th layer with the second fusion feature of the m+1th layer that has been fused with the fusion feature of the higher level, can obtain the semantic information and explanatory properties of the second fusion feature of the higher level in addition to the original semantic information and explanatory properties of the second fusion feature of the m+1th layer.

[0109] Exemplarily, when second fusion features of multi-scale sizes such as 38*38*512, 19*19*1024, 10*10*512, and 5*5*256 are obtained after fusion of multi-scale traditional features and multi-scale deep features according to the image to be detected, the fusion of the multi-scale second fusion features can be to fuse the second fusion features of the scale size of 5*5*256 (i.e., the second fusion features of the fourth layer) with the second fusion features of the lower layer of the scale size of 10*10*512 (i.e., the second fusion features of the third layer), thereby obtaining the second fusion features of the third layer fused with the high-level fusion features, that is, obtaining the second fusion features of the fused scale size of 10*10*512; and then The second fusion feature of the third layer that is fused with the high-level fusion feature is fused with the second fusion feature of the next lower layer with a scale of 19*19*1024 (i.e., the second fusion feature of the second layer) to obtain the second fusion feature of the second layer fused with the high-level fusion feature, that is, the second fusion feature with a fusion scale of 19*19*1024; and then the second fusion feature of the second layer that is fused with the high-level fusion feature is fused with the second fusion feature of the bottom layer with a scale of 38*38*512 (i.e., the second fusion feature of the first layer) to obtain the second fusion feature of the first layer fused with the high-level fusion feature, that is, the second fusion feature with a fusion scale of 38*38*512.

[0110] For example, see Fig.12 , Fig.12 A schematic block diagram of another process for feature fusion using the third fusion module is shown, wherein the scale of the second fusion feature corresponding to the m+1th layer is N / 2*N / 2*P, the scale of the high-level fusion feature corresponding to the second fusion feature corresponding to the m+1th layer is N / 4*N / 4*Q, and the scale of the second fusion feature corresponding to the mth layer is N*N*M.

[0111] from Fig.12 It can be seen that the second fusion feature corresponding to the m+1th layer and the high-level fusion feature corresponding to the m+1th layer can be fused first, and then the second fusion feature after fusion of the m+1th layer can be fused with the second fusion feature corresponding to the mth layer to obtain the second fusion feature after fusion of the mth layer. The step-by-step fusion of the second fusion feature from high level to low level is realized.

[0112] Specifically, after the second fusion features corresponding to the m+1th layer and the high-level fusion features corresponding to the second fusion features corresponding to the m+1th layer are input into the third fusion module, the high-level fusion features with a scale of N / 4*N / 4*Q can be expanded to high-level fusion features with a scale of N / 2*N / 2*P (achieving the same scale as the second fusion features of the m+1th layer) after passing through the 3*3 convolution kernel and the upsampling layer (the upsampling layer is not shown in the figure) in the third fusion module, and then added and fused with the second fusion features corresponding to the m+1th layer with a scale of N / 2*N / 2*P, so as to output the second fusion features of the m+1th layer with a fused scale of N / 2*N / 2*P. The fused second fusion features of the m+1th layer can also be understood as the second fusion features of the m+1th layer integrating the semantic information of the high-level fusion features.

[0113] Then, the second fusion features of the m+1th layer fused by the third fusion module can be further fused with the second fusion features of the mth layer, so that the second fusion features of the mth layer can also fuse the semantic information of the high-level fusion features. Specifically, the obtained second fused features of the m+1th layer and the second fused features of the mth layer can be input into the third fusion module, and then the second fused features of the m+1th layer with a scale of N / 2*N / 2*P can be expanded to the second fused features of the m+1th layer with a scale of N*N*M (achieving the same scale as the second fused features of the mth layer) after passing through the 3*3 convolution kernel and the upsampling layer (the upsampling layer is not shown in the figure) in the third fusion module, and then added and fused with the second fused features of the mth layer with a scale of N*N*M, so as to output the second fused features of the mth layer with a fused scale of N*N*M, that is, the second fused features of the mth layer with the semantic information of the high-level fused features are obtained, wherein the semantic information of the high-level fused features includes the semantic information of the second fused features of the m+1th layer and the semantic information of the higher-level fused features corresponding to the second fused features of the m+1th layer.

[0114] In some embodiments, all high-level second fusion features may be fused with low-level second fusion features, or only some high-level second fusion features may be fused with low-level second fusion features, which is not limited here. For example, which high-level second fusion features to fuse may be reasonably selected based on the amount of computation and the detection accuracy.

[0115] Step S2254: inputting the fused second fusion features of the mth layer into the prediction module, and the prediction module is used to output the detection result of the target to be detected in the image to be detected according to the fused second fusion features of the mth layer.

[0116] In some embodiments, after obtaining the second fusion feature corresponding to the fused mth layer, the second fusion feature of the fused mth layer scale can be predicted. Specifically, the fused second fusion feature of the mth layer can be input into a prediction module, and the prediction module is used to output the detection result of the target to be detected in the image to be detected according to the fused second fusion feature of the mth layer.

[0117] It can be understood that by repeating the above-mentioned feature fusion process for the second fusion features of each scale, the second fusion features after multi-scale fusion can be obtained, so that the second fusion features after multi-scale fusion can be input into the prediction module, so that the prediction module can output the prediction results at each scale according to the second fusion features fused at each scale, and comprehensively consider the prediction results at each scale to output the detection result of the target to be detected in the final image to be detected.

[0118] Step S230: Obtain the detection result of the target to be detected output by the network model.

[0119] In the embodiment of the present application, step S230 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0120] Therefore, by introducing high-order features into low-order features in both the traditional feature extraction part and the prediction part of the second fusion feature after the fusion of traditional features and deep features, the existing target detection network can be optimized. And because high-order features are suitable for classification and low-order features are suitable for positioning, after the high-order features are introduced into the low-order features, the low-order images can also be suitable for classification. Thereby improving the accuracy of the network model.

[0121] For example, see Fig.13 , Fig.13 The overall structure diagram of the network model applicable to the target detection method of the embodiment of the present application is shown. Among them, the C module in the figure is the first fusion module mentioned above, the A module in the figure is the second fusion module mentioned above, and the F module in the figure is the third fusion module mentioned above. Fig.13It can be seen that, based on the original image of the image to be detected of size 300*300*3, the convolutional neural network can extract features to obtain depth feature maps of scales 38*38*512, 19*19*1024, 10*10*512, 5*5*256, 3*3*256, and 1*1*256, and then generate traditional feature maps of scales 38*38*1, 19*19*1, 10*10*1, and 5*5*1 respectively based on the original image of size 300*300*3. Then, the multi-scale traditional feature maps are fused step by step from high to low layers through the C module, and then the fused traditional feature maps of each scale are fused with the depth feature maps of the corresponding scales through the A module to obtain the fused depth feature maps of each scale. Then, on the prediction path, the F module is used to introduce the fused depth feature maps of all high-level scales into the fused depth feature maps of the low-level scales for fusion, so as to obtain the final depth feature maps on the prediction paths of each scale. Finally, the prediction module is used to predict the final depth feature maps on the prediction paths of each scale respectively, and finally the detection results of the image to be detected are output.

[0122] Specifically, the 5*5*1 traditional image map and the 5*5*256 deep feature map extracted by the convolutional neural network can be first fused through the A module, the 5*5*1 traditional image map and the 10*10*1 traditional image map can be fused through the C module and then fused with the 10*10*512 deep feature map extracted by the convolutional neural network through the A module, the 10*10*1 traditional image map fused through the C module and the 19*19*1 traditional image features can be fused through the C module and then fused with the 19*19*1024 deep feature map extracted by the convolutional neural network through the A module, and the 19*19*1 traditional image map fused through the C module and the 38*38*1 traditional image map features can be fused through the C module and then fused with the 38*38*512 deep feature map extracted by the convolutional neural network through the A module. In this way, multi-scale traditional features can be added to the branches of the deep feature maps of each scale to guide attention. Then, the feature maps fused by the A module can be fused again at multiple scales. Specifically, the feature map branches of 19*19*1024, 10*10*512, and 5*5*256 are fused with the feature map branches of 38*38*512 through the F module, the feature map branches of 10*10*512 and 5*5*256 are fused with the feature map branches of 19*19*1024 through the F module, and the feature map branches of 5*5*256 are fused with the feature map branches of 10*10*512 through the F module, thereby obtaining the final deep feature maps on the prediction paths of each scale. Finally, the fused 38*38*512, 19*19*1024, 10*10*512, 5*5*256 branches and the original 3*3*256 and 1*1*256 branches are predicted respectively, and the final prediction results are output.

[0123] In some embodiments, see Fig.14 , the above pre-trained network model can be trained by the following method:

[0124] Step S310: Obtain an initial detection model, where the initial detection model includes a convolutional neural network.

[0125] Among them, Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. The network may include multiple levels of convolutional layers. In some embodiments, the initial detection model can be directly reconstructed for training, or the existing network model can be directly improved and then trained. This is not limited here. In some embodiments, the initial detection model obtained may include a convolutional neural network, which includes at least 6 convolutional layers to achieve the extraction of multi-scale deep features.

[0126] It is understandable that when designing the initial detection model, various convolutional neural networks can be used for design and training to obtain a model with the best learning effect. It is understandable that while increasing the number of convolutional layer structures in the convolutional neural network, the learning effect of the model will also be improved, but the amount of calculation will also increase accordingly.

[0127] Step S320: Acquire a training sample set, where the training sample set includes image samples and target annotation samples corresponding to the image samples.

[0128] In some embodiments, the original data set can be first obtained from a database or server, and then the images of the original data set are preprocessed to obtain image samples that meet the input standards of the initial detection model. Then, the preprocessed image samples are annotated with the detection targets, such as annotating the coordinate information, category information and confidence level of the object, to obtain multiple target annotated samples of the annotated detection results. Among them, an image sample and the target annotated sample corresponding to the image sample are a set of training samples, and multiple sets of training samples can construct a training sample set. Among them, the preprocessing can include operations such as deduplication, cropping, rotation and / or flipping, and other preprocessing operations can be further performed on these images.

[0129] Step S330: using the image sample as the input of the initial detection model, using the target annotation sample as the expected output of the initial detection model, training the initial detection model, and obtaining the pre-trained network model.

[0130] In an embodiment of the present application, the above-mentioned image sample is used as the input of the initial detection model, and the target annotation sample corresponding to the image sample is used as the expected output (i.e., the learning target) of the initial detection model, so that the initial detection model can be trained through the actual output and expected output of the model to obtain the aforementioned pre-trained network model.

[0131] Specifically, after obtaining the image sample, the image sample can be input into the initial detection model for forward calculation to obtain the actual output of the initial detection model, which is the target detection result predicted by the initial detection model. Since the target annotation sample corresponding to the image sample is used as the expected output of the initial detection model, the model parameters can be updated according to the error between the predicted target detection result and the expected output, and the aforementioned pre-trained network model can be obtained through a large number of iterative training.

[0132] It can be understood that the above-mentioned steps in this embodiment can be performed locally by the above-mentioned terminal device, or can be performed in the above-mentioned server, or can be performed by the terminal device and the server in a division of labor. Depending on the actual application scenario, tasks can be allocated as needed to achieve optimized voice control of home appliances, which is not limited here.

[0133] An object detection method provided by an embodiment of the present application obtains the nth layer of traditional features of an image to be detected, and the high-level traditional features corresponding to the nth layer of traditional features, and inputs the nth layer of traditional features and the high-level traditional features into the first fusion module for feature fusion, thereby obtaining the first fusion features corresponding to the fused nth layer output by the first fusion module. In this way, by introducing high-order traditional image features into low-order traditional image features, the semantic information of low-order traditional image features can be enriched, the fusion of multi-scale traditional features is achieved, and the accuracy of traditional feature extraction is improved. By inputting the first fusion features and the nth layer of deep features into the second fusion module for feature fusion, the second fusion features of the nth layer after fusion output by the second fusion module are obtained. In this way, by introducing multi-scale traditional image features to guide attention on multi-scale deep features, the feature extraction of the convolutional neural network on the detection branch is optimized, the interpretability of the system is increased, and the target detection accuracy of the network is improved. By obtaining the second fusion features of the mth layer and the high-level fusion features corresponding to the second fusion features of the mth layer, the second fusion features of the mth layer and the high-level fusion features are input into the third fusion module for feature fusion, the fused second fusion features of the mth layer output by the third fusion module are obtained, and the fused second fusion features of the mth layer are input into the prediction module, and the prediction module is used to output the detection result of the target to be detected in the image to be detected according to the fused second fusion features of the mth layer. In this way, by introducing high-order fusion features into low-order fusion features, since high-order features are suitable for classification and low-order features are suitable for positioning, after processing, the low-order fusion features can also be suitable for classification, thereby improving the accuracy of the target detection network.

[0134] See also Fig.15 , Fig.15The structural block diagram of a target detection device 500 provided by the present application is shown, and the device may include: an image acquisition module 510, an image input module 520, and an image detection module 530. Among them, the image acquisition module 510 is used to acquire the image to be detected; the image input module 520 is used to input the image to be detected into a pre-trained network model, and the network model is used to fuse the multi-scale traditional features and multi-scale depth features of the image to be detected according to the corresponding scale, and output the detection result of the target to be detected in the image to be detected according to the fused multi-scale traditional features and multi-scale depth features; the image detection module 530 is used to obtain the detection result of the target to be detected output by the network model.

[0135] In some embodiments, the above network model includes a first fusion module, a second fusion module and a prediction module, and the image input module 520 may include: a feature acquisition unit, a high-level feature acquisition unit, a first fusion unit, a second fusion unit and a prediction unit. Among them, the feature acquisition unit is used to acquire the n-th layer traditional features of the image to be detected, the n-th layer is any layer scale in the multi-scale traditional features, and n is a positive integer; the high-level feature acquisition unit is used to acquire the high-level traditional features corresponding to the n-th layer traditional features, and the high-level traditional features are any traditional features with a scale higher than the n-th layer scale; the first fusion unit is used to input the n-th layer traditional features and the high-level traditional features into the first fusion module for feature fusion, and obtain the first fusion features corresponding to the n-th layer after fusion output by the first fusion module; the second fusion unit is used to input the first fusion features and the n-th layer depth features into the second fusion module for feature fusion, and obtain the second fusion features of the n-th layer after fusion output by the second fusion module, and the n-th layer depth features are the depth features in the multi-scale depth features corresponding to the scale of the n-th layer traditional features; the prediction unit is used to input the second fusion features into the prediction module, and the prediction module is used to output the detection result of the target to be detected in the image to be detected according to the second fusion features.

[0136] In some embodiments, the high-level traditional features in the above-mentioned high-level feature acquisition unit can be the first fused features corresponding to the n+1th layer after fusion, wherein the first fused features corresponding to the n+1th layer are the fused features output after the n+1th layer traditional features and the high-level traditional features corresponding to the n+1th layer traditional features are input into the first fusion module.

[0137] In some embodiments, the network model also includes a third fusion module, and the above-mentioned prediction unit may include: a fusion feature acquisition unit, a high-level fusion feature acquisition unit, a third fusion unit, and a fusion prediction unit. Among them, the fusion feature acquisition unit is used to obtain the second fusion feature of the mth layer, the mth layer is any layer scale of the multi-scale second fusion feature, and m is a positive integer; the high-level fusion feature acquisition unit is used to obtain the high-level fusion feature corresponding to the second fusion feature of the mth layer, and the high-level fusion feature is any second fusion feature with a scale higher than the scale of the mth layer; the third fusion unit is used to input the second fusion feature of the mth layer and the high-level fusion feature into the third fusion module for feature fusion, and obtain the fused second fusion feature of the mth layer output by the third fusion module; the fusion prediction unit is used to input the fused second fusion feature of the mth layer into the prediction module, and the prediction module is used to output the detection result of the target to be detected in the image to be detected according to the fused second fusion feature of the mth layer.

[0138] Furthermore, in some embodiments, the high-level fusion feature acquisition unit may be specifically used to: acquire all high-level fusion features corresponding to the second fusion features of the m-th layer.

[0139] In some embodiments, the above-mentioned pre-trained network model can be trained through the following steps: obtaining an initial detection model, the initial detection model includes a convolutional neural network; obtaining a training sample set, the training sample set includes image samples and target annotation samples corresponding to the image samples; using the image samples as the input of the initial detection model, and the target annotation samples as the expected output of the initial detection model, training the initial detection model, and obtaining the pre-trained network model.

[0140] In some embodiments, the above-mentioned traditional features may include any one or more of Harris corner features, SIFT scale-invariant features, HOG directional gradient histogram features, and LBP local binary features.

[0141] Those skilled in the art can clearly understand that the target detection device provided in the embodiment of the present application can achieve Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 , Fig.11 , Fig.12 , Fig.13 and Fig.14For the convenience and brevity of description, the specific working processes of the above-described devices, modules, and units in any method embodiment can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0142] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0143] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0144] In summary, an embodiment of the present application provides a target detection device, which includes: an image acquisition module 510, an image input module 520 and an image detection module 530. Among them, the image acquisition module 510 is used to acquire the image to be detected; the image input module 520 is used to input the image to be detected into a pre-trained network model, and the network model is used to fuse the multi-scale traditional features and multi-scale depth features of the image to be detected according to the corresponding scale, and output the detection result of the target to be detected in the image to be detected according to the fused multi-scale traditional features and multi-scale depth features; the image detection module 530 is used to obtain the detection result of the target to be detected output by the network model. In this way, by introducing more interpretable traditional image features into the network model to guide feature fusion and realize the attention mechanism, the feature extraction of the network model is optimized, the interpretability of the network model is increased, and the target detection accuracy of the network model for the image to be detected is improved.

[0145] The present application embodiment provides an electronic device, the electronic device includes a processor and a memory, the memory stores at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method provided in the above method embodiment. Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 , Fig.11 , Fig.12 , Fig.13 and Fig.14Any target detection method. In this embodiment, the electronic device can be a mobile phone, a laptop computer, a smart wearable device, a smart TV, a smart speaker, a vehicle terminal, a gateway, and other terminal devices that can run applications. It can also be a local server, a cloud server, and other servers.

[0146] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0147] Fig.16 1 is a hardware structure block diagram of an electronic device 1200 provided in one embodiment of the present application. Specifically, when the electronic device is used as an electronic device in a wireless network system, it can execute and implement the method provided in the above method embodiment. Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 , Fig.11 , Fig.12 , Fig.13 and Fig.14 Any target detection method, wherein the electronic device can be an image acquisition device, a server, or other electronic device capable of running an application.

[0148] like Fig.16As shown, the electronic device 1200 may have relatively large differences due to different configurations or performances, and may include one or more processors 1210 (the processor 1210 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1230 for storing data, and one or more storage media 1220 (such as one or more mass storage devices) for storing application programs 1223 or data 1222. Among them, the memory 1230 and the storage medium 1220 can be short-term storage or permanent storage. The program stored in the storage medium 1220 may include one or more modules, each of which may include a series of instruction operations in the electronic device 1200. Furthermore, the processor 1210 may be configured to communicate with the storage medium 1220 to execute a series of instruction operations in the storage medium 1220 on the electronic device 1200. The electronic device 1200 may also include one or more power supplies 1260, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1240, and / or one or more operating systems 1221, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, FreeBSD TM, etc.

[0149] It should be noted that the power supply 1260 may include an independent power supply module to supply power to the electronic device 1200 , or may be used to connect to an external power supply to supply power to the electronic device 1200 .

[0150] The input / output interface 1240 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 1200. In one example, the input / output interface 1240 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 1240 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0151] It can be understood by those skilled in the art that Fig.16 The structure shown is for illustration only and does not limit the structure of the above electronic device. Fig.16 More or fewer components as shown, or with Fig.16 Different configurations are shown.

[0152] In summary, an electronic device provided by an embodiment of the present application performs the following steps: obtaining an image to be detected; inputting the image to be detected into a pre-trained network model, the network model is used to fuse the multi-scale traditional features and multi-scale deep features of the image to be detected according to the corresponding scale, and output the detection result of the target to be detected in the image to be detected based on the fused multi-scale traditional features and multi-scale deep features, wherein the network model is obtained by training a neural network model based on training samples; and the detection result of the target to be detected output by the network model is obtained. In this way, by introducing more interpretable traditional image features into the network model to guide feature fusion and realize the attention mechanism, the feature extraction of the network model is optimized, the interpretability of the network model is increased, and the target detection accuracy of the network model for the image to be detected is improved.

[0153] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements each process of the above-mentioned target detection method embodiment and can achieve the same technical effect. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0154] Fig.17 800 is a block diagram of a computer-readable storage medium 800 provided in an embodiment of the present application. The computer-readable storage medium 800 stores program code, which can be called by a processor to execute the method provided in the above embodiment. Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 , Fig. 9 , Fig.10 , Fig.11 , Fig.12 , Fig.13 and Fig.14Any target detection method. The computer-readable storage medium 800 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that executes any method step in the above method. These program codes can be read from one or more computer program products or written to one or more computer program products. The program code 810 can be compressed, for example, in an appropriate form.

[0155] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0157] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A target detection method, characterized in that: The method comprises: Acquire the image to be detected; Inputting the image to be detected into a pre-trained network model, the network model is used to fuse the multi-scale traditional features of the image to be detected, and fuse the fused multi-scale traditional features with the multi-scale depth features of the image to be detected according to the corresponding scales, and outputting the detection result of the target to be detected in the image to be detected according to the fused multi-scale traditional features and the multi-scale depth features; Obtaining a detection result of the target to be detected output by the network model.

2. The method according to claim 1, characterized in that The network model includes a first fusion module, a second fusion module and a prediction module. After the image to be detected is input into the pre-trained network model, the method further includes: Acquire the nth layer of traditional features of the image to be detected, where the nth layer is any layer scale of the multi-scale traditional features, and n is a positive integer; Acquire a high-level traditional feature corresponding to the n-th level traditional feature, wherein the high-level traditional feature is any traditional feature having a scale higher than the n-th level scale; Inputting the n-th layer traditional features and the high-level traditional features into the first fusion module for feature fusion, and obtaining the first fusion features corresponding to the n-th layer after fusion output by the first fusion module; Inputting the first fusion feature and the n-th layer depth feature into the second fusion module for feature fusion, obtaining the fused n-th layer second fusion feature output by the second fusion module, wherein the n-th layer depth feature is a depth feature in the multi-scale depth feature corresponding to the scale of the n-th layer traditional feature; The second fusion feature is input into the prediction module, and the prediction module is used to output a detection result of the target to be detected according to the second fusion feature.

3. The method according to claim 2, characterized in that The high-level traditional features are the first fused features corresponding to the n+1th layer after fusion, wherein the first fused features corresponding to the n+1th layer are the fused features output after the n+1th layer traditional features and the high-level traditional features corresponding to the n+1th layer traditional features are input into the first fusion module.

4. The method according to claim 2, characterized in that: The network model further includes a third fusion module. Before the second fusion feature is input into the prediction module, and the prediction module is used to output a detection result of the target to be detected according to the second fusion feature, the method further includes: Obtaining the second fusion features of the mth layer, where the mth layer is any layer scale of the multi-scale second fusion features, and m is a positive integer; Obtaining a high-level fusion feature corresponding to the second fusion feature of the m-th layer, wherein the high-level fusion feature is any second fusion feature having a scale higher than that of the m-th layer; Inputting the second fusion feature of the mth layer and the high-level fusion feature into the third fusion module for feature fusion, and obtaining the fused second fusion feature of the mth layer output by the third fusion module; The step of inputting the second fusion feature into the prediction module, wherein the prediction module is configured to output a detection result of the target to be detected according to the second fusion feature, comprises: The fused second fusion features of the mth layer are input into the prediction module, and the prediction module is used to output the detection result of the target to be detected according to the fused second fusion features of the mth layer.

5. The method according to claim 4, characterized in that The obtaining of the high-level fusion feature corresponding to the second fusion feature of the m-th layer includes: Obtain all high-level fusion features corresponding to the second fusion features of the m-th layer.

6. The method according to any one of claims 1 to 5, characterized in that: The pre-trained network model is trained by the following method: Acquire an initial detection model, wherein the initial detection model includes a convolutional neural network; Acquire a training sample set, wherein the training sample set includes image samples and target annotation samples corresponding to the image samples; The image sample is used as the input of the initial detection model, the target annotation sample is used as the expected output of the initial detection model, the initial detection model is trained, and the pre-trained network model is obtained.

7. The method according to any one of claims 1 to 5, characterized in that: The traditional features include any one or more of Harris corner features, SIFT scale-invariant features, HOG directional gradient histogram features, and LBP local binary features.

8. A target detection device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be detected; An image input module, used for inputting the image to be detected into a pre-trained network model, wherein the network model is used for fusing the multi-scale traditional features of the image to be detected, and fusing the fused multi-scale traditional features with the multi-scale depth features of the image to be detected according to the corresponding scales, and outputting the detection result of the target to be detected in the image to be detected according to the fused multi-scale traditional features and the multi-scale depth features; The image detection module is used to obtain the detection result of the target to be detected output by the network model.

9. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the target detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the target detection method according to any one of claims 1 to 7 are implemented.

Images