Target detection and model training method and device, edge computing equipment and medium

Abstract

The invention discloses a target detection and model training method and device, edge computing equipment and a medium. The target detection method comprises the following steps: performing feature processing on an original image to obtain feature data; performing position prediction on the feature data to obtain a position prediction result of the target object; performing category prediction on the feature data according to the position prediction result to obtain a category prediction result of the target object; and determining the position prediction result and the category prediction result as a detection result of the target object.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target detection and model training method, apparatus, edge computing device and medium. Background Art

[0002] Object detection is an important technology in the field of computer vision, which refers to the detection and identification of objects of interest in images or videos. Object detection has a wide range of applications in autonomous driving, surveillance and security, smart retail, medical image analysis, etc.

[0003] The target detection methods in related technologies usually perform position detection and category detection simultaneously, which has the disadvantage of poor detection effect. Summary of the invention

[0004] The embodiments of the present application aim to solve at least one of the technical problems in the related art to a certain extent. To this end, the purpose of the embodiments of the present application is to propose a target detection and model training method, apparatus, edge computing device, medium and program product.

[0005] An embodiment of the present application provides a target detection method, which includes: performing feature processing on an original image to obtain feature data; performing position prediction on the feature data to obtain a position prediction result of a target object; performing category prediction on the feature data based on the position prediction result to obtain a category prediction result of the target object; and determining the position prediction result and the category prediction result as a detection result of the target object.

[0006] Exemplarily, category prediction is performed on feature data according to the position prediction result to obtain a category prediction result of the target object, including: extracting local features located at a corresponding position from the feature data according to the position prediction result; performing category prediction based on the local features at the corresponding position to obtain a category prediction result of the target object; and / or, feature processing is performed on the original image to obtain feature data, including: performing feature extraction on the original image to obtain multi-scale features; performing feature extraction on the multi-scale features to obtain multiple feature maps; and performing feature fusion on the multiple feature maps to obtain the feature data.

[0007] Exemplarily, feature extraction is performed on multi-scale features to obtain multiple feature maps, including: feature extraction is performed on multi-scale features to obtain a first differential fusion feature; feature extraction is performed on the first differential fusion feature to obtain a first spatial depth decoupling feature; feature extraction is performed on the n-1th spatial depth decoupling feature to obtain an nth differential fusion feature, wherein n is an integer greater than or equal to 2 and less than or equal to N, and N is an integer greater than or equal to 2; spatial depth decoupling feature extraction is performed on the nth differential fusion feature to obtain an nth spatial depth decoupling feature; at least part of the N spatial depth decoupling features are determined as multiple feature maps.

[0008] Exemplarily, feature extraction is performed on the original image to obtain multi-scale features, including: feature extraction is performed on the original image to obtain the first convolution feature; feature extraction is performed on the m-1th convolution feature to obtain the mth convolution feature, where m is an integer greater than or equal to 2 and less than or equal to M, and M is an integer greater than or equal to 2; channel merging is performed on the M convolution features to obtain multi-scale features.

[0009] Exemplarily, each of the N differential fusion features is obtained in the following manner: performing spatial deep decoupling feature extraction on the input feature to obtain the first feature; performing residual calculation on the first feature to obtain the second feature; performing spatial deep decoupling feature extraction on the input feature to obtain the third feature; performing channel merging on the second feature and the third feature to obtain the fourth feature; performing convolution processing on the fourth feature to obtain the differential fusion feature.

[0010] Exemplarily, position prediction is performed on feature data to obtain a position prediction result of a target object, including: performing residual calculation on the feature data to obtain a fifth feature; and performing spatial depth decoupling feature extraction on the fifth feature to obtain a position prediction result of the target object.

[0011] Exemplarily, the residual calculation includes: performing spatial depth decoupling feature extraction on the input feature to obtain the sixth feature; performing spatial depth decoupling feature extraction on the sixth feature to obtain the seventh feature; performing channel merging on the input feature and the seventh feature to obtain the eighth feature; performing spatial depth decoupling feature extraction on the eighth feature to obtain the result of the residual calculation.

[0012] Exemplarily, feature fusion is performed on multiple feature maps to obtain feature data, including: based on multiple feature maps, at least one of pooling aggregation, feature extraction fusion, feature connection and spatial depth decoupling feature extraction is performed to obtain feature data; and / or the local features at the corresponding positions include local features at multiple scales; category prediction is performed based on the local features at the corresponding positions to obtain a category prediction result of the target object, including: average pooling the local features of multiple scales and aggregating them into a feature vector; based on the feature vector, a category prediction result is obtained.

[0013] Exemplarily, spatial deep decoupling feature extraction includes: performing a 1×1 convolution operation on the input features to obtain a first multi-channel feature; performing a depth-separable convolution operation on each channel of the first multi-channel feature to obtain a second multi-channel feature; merging channels of the second multi-channel feature to obtain a merged feature; performing batch normalization and activation function processing on the merged feature to obtain the result of spatial deep decoupling feature extraction.

[0014] Another embodiment of the present application provides a training method for a target detection model, wherein the target detection model includes a feature acquisition layer, a target position prediction layer and a target category prediction layer, and the method includes: performing feature processing on a training image based on the feature acquisition layer to be trained to obtain first feature data; performing position prediction on the first feature data based on the target position prediction layer to be trained to obtain a first position prediction result of the target object, and adjusting parameters of the feature acquisition layer to be trained and parameters of the target position prediction layer to be trained based on the first position prediction result and the position label of the training image to obtain a trained feature acquisition layer and a trained target position prediction layer; performing feature processing on the training image based on the trained feature acquisition layer to obtain second feature data; performing category prediction on the second feature data based on the target category prediction layer to be trained according to the target position information of the target object in the training image to obtain a category prediction result of the target object, and adjusting parameters of the target category prediction layer to be trained based on the category prediction result and the category label of the training image to obtain a trained target category prediction layer.

[0015] Another embodiment of the present application provides a target detection device, which includes: a feature acquisition layer, which is used to perform feature processing on an original image to obtain feature data; a target position prediction layer, which is used to perform position prediction on the feature data to obtain a position prediction result of the target object; a target category prediction layer, which is used to perform category prediction on the feature data based on the position prediction result to obtain a category prediction result of the target object; wherein the position prediction result and the category prediction result are used as the detection result of the target object.

[0016] Exemplarily, the feature acquisition layer includes: a target extraction backbone layer, which is used to extract features from the original image and obtain multiple feature maps; a sequential fusion convolution layer, which is used to fuse multiple feature maps to obtain feature data; the target extraction backbone layer includes: a multi-scale perception layer, which is used to extract features from the original image and obtain multi-scale features; a differential fusion convolution layer and a spatial depth decoupling convolution layer, which are used to extract multi-scale features and obtain multiple feature maps.

[0017] Exemplarily, there are N differential fusion convolution layers and N spatial depth decoupling convolution layers, where N is an integer greater than or equal to 2: the first differential fusion convolution layer is used to extract features from multi-scale features to obtain a first differential fusion feature; the first spatial depth decoupling convolution layer is used to extract features from the first differential fusion feature to obtain a first spatial depth decoupled feature; the nth differential fusion convolution layer is used to extract features from the n-1th spatial depth decoupled feature to obtain the nth differential fusion feature, where n is an integer greater than or equal to 2 and less than or equal to N; the nth spatial depth decoupling convolution layer is used to extract features from the nth differential fusion feature to obtain the nth spatial depth decoupled feature; wherein the multiple feature maps are at least part of the N spatial depth decoupled features.

[0018] Exemplarily, the multi-scale perception layer includes M convolution kernels, where M is an integer greater than or equal to 2: the first convolution kernel is used to extract features of the original image to obtain the first convolution feature; the m-th convolution kernel is used to extract features of the m-1th convolution feature to obtain the m-th convolution feature, where m is an integer greater than or equal to 2 and less than or equal to M; wherein the M convolution features are channel-merged to obtain multi-scale features.

[0019] Another embodiment of the present application provides a training device for a target detection model, the device comprising: a feature acquisition layer to be trained, used to perform feature processing on a training image to obtain first feature data; a target position prediction layer to be trained, used to perform position prediction on the first feature data to obtain a first position prediction result of the target object; a first parameter adjustment layer, used to adjust the parameters of the feature acquisition layer to be trained and the parameters of the target position prediction layer to be trained based on the first position prediction result and the position label of the training image, to obtain a trained feature acquisition layer and a trained target position prediction layer; a target category prediction layer to be trained, used to perform category prediction on the second feature data according to the target position information of the target object in the training image, to obtain a category prediction result of the target object, wherein the second feature data is obtained by performing feature processing on the training image based on the trained feature acquisition layer; and a second parameter adjustment layer, used to adjust the parameters of the target category prediction layer to be trained based on the category prediction result and the category label of the training image, to obtain a trained target category prediction layer.

[0020] Another embodiment of the present application provides an edge computing device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of any of the above embodiments when executing the computer program.

[0021] Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method of any of the above embodiments are implemented.

[0022] Another embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device is enabled to perform the steps of the method of any of the above embodiments.

[0023] In the above implementation, feature data is obtained by performing feature processing on the original image; position prediction is performed on the feature data to obtain a position prediction result of the target object; category prediction is performed on the feature data based on the position prediction result to obtain a category prediction result of the target object; the position prediction result and the category prediction result are determined as the detection result of the target object. The target detection method based on a convolutional neural network of the present application can first predict the position of the target object, and then use the position prediction result as prior knowledge to further predict the category of the target object, thereby improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a process of a target detection method based on a convolutional neural network provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the process of an end-to-end object detection method based on a convolutional neural network provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the workflow of the target extraction backbone layer provided for the implementation mode of the present application;

[0027] Figure 4 A schematic diagram of the structure of the sequential fusion convolutional layer provided in the implementation manner of the present application;

[0028] Figure 5 A schematic diagram of a flow chart of a training method for a target detection model based on a convolutional neural network provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of a target detection device provided in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of a training device for a target detection model provided in an embodiment of the present application;

[0031] Figure 8 A block diagram of an edge computing device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0032] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0033] Object detection is an important technology in the field of computer vision, which refers to the detection and identification of objects of interest in images or videos. Object detection has a wide range of applications in autonomous driving, surveillance and security, smart retail, medical image analysis, etc.

[0034] Related target detection methods include classical machine learning methods, deep learning methods, template matching methods, etc.

[0035] Methods based on classical machine learning, such as the Viola–Jones object detection framework (Viola-Jones) and support vector machine (SVM), require manual feature design and classifier training.

[0036] Deep learning-based methods, such as R-CNN (Region with CNN feature), YOLO (You Only Look Once), and SSD (Single Shot MultiBox Detector), can automatically extract features and perform end-to-end target detection. The detection accuracy and speed of deep learning methods have been significantly improved.

[0037] The template matching method detects targets by matching the input image with a predefined template, which is suitable for target detection tasks in specific scenarios.

[0038] However, classical machine learning methods require a large number of manually designed features and have poor generalization. Deep learning methods have high computational complexity and require a large amount of training data. Template matching methods have poor flexibility and can only detect specific types of target objects. In addition, target detection models based on related technologies still have some problems to be solved, and there is room for improvement as follows:

[0039] (1) Improve target detection accuracy;

[0040] (2) Improve the real-time performance of detection, optimize the algorithm computational complexity and memory usage, increase the processing frame rate, and achieve real-time detection of large-scale target scenes;

[0041] (3) Enhanced robustness: Improve the algorithm's ability to handle complex situations such as occlusion, rapid movement, and target appearance and disappearance, thereby improving the robustness of the model algorithm in various environments.

[0042] (4) Existing object detection models can only recognize a limited number of objects;

[0043] (5) The target detection of existing target detection algorithms predicts the target position and category at the same time. However, when the human eye actually determines the type of target in an image, the first step is to locate the target, and then determine the type of the located target based on the local information after location. Compared with these existing algorithms, the human eye's location and category determination method has the following characteristics:

[0044] (5.1) These existing object detection algorithms use datasets containing certain types of objects during the training phase. When the types of objects in the dataset to be identified change, the entire network needs to be retrained. Even if the new dataset has only minor changes compared to the previous dataset (for example, only one category is added or modified), online training is not possible.

[0045] (5.2) Although these existing object detection algorithms use parallel computing to predict the categories and locations of all objects in the image at one time, in practical applications, not all object detections are meaningful, which wastes a lot of computing resources.

[0046] In view of this, the embodiments of the present application provide an optimized target detection method based on a convolutional neural network and a training method for a target detection model, which can first predict the position of the target object, and then use the position prediction result as prior knowledge to further predict the category of the target object, thereby improving the accuracy of target detection and at least partially solving the problems existing in the related technology.

[0047] The object positioning and category detection methods of the human eye are the result of billions of years of biological evolution, and better balance real-time, energy consumption and detection performance. Referring to the step-by-step mechanism of the biological vision system, the target detection method of this application decouples positioning and recognition, while ensuring real-time, it can also improve recognition accuracy and computing efficiency.

[0048] The target detection method of this application can be applied to the following scenarios:

[0049] Autonomous driving: Unmanned driving scenarios have high requirements for the accuracy and real-time performance of target detection, and the advantages of target detection algorithms are very suitable; they can more intelligently detect and identify various obstacles on the road and improve driving safety.

[0050] Intelligent monitoring: Monitoring scenes usually have complex backgrounds, occlusions and other problems, and target detection algorithms have stronger generalization capabilities.

[0051] Robot Vision: Robots need to accurately perceive their surroundings to perform complex interactive tasks. Object detection algorithms can provide robots with more accurate object detection and recognition capabilities, enhancing the robot's intelligent perception.

[0052] Enhanced human-computer interaction: In human-computer interaction scenarios such as virtual reality and augmented reality, accurately detecting and identifying users' movements and gestures is very important for achieving natural interaction; target detection algorithms can more accurately detect the position and movements of users' hands, body and other parts in space, thereby providing a more reliable perception basis for these interactions.

[0053] Behavior analysis and intent recognition: In scenarios such as intelligent monitoring and robot collaboration, analysis of human behavior and intent recognition are very important; target detection algorithms can more accurately detect and track key parts of the human body, providing richer input data for subsequent behavior analysis and intent inference.

[0054] Optimization of human-machine collaboration: In the scenario of human-machine collaboration, machines need to accurately perceive human actions and intentions in order to better coordinate the behaviors of both parties; target detection algorithms can help robots detect human actions more accurately, thereby better predicting human intentions and optimizing the human-machine collaboration process.

[0055] Human safety monitoring: In some dangerous working environments, real-time monitoring of the status and behavior of workers is very important. The target detection algorithm can accurately capture the movement characteristics of workers and combine other physiological sensor data to conduct real-time analysis and early warning of the safety status of workers.

[0056] Human motion analysis: In the fields of sports training, rehabilitation therapy, etc., accurately capturing and analyzing human motion data is important for optimizing training plans and evaluating treatment effects. Target detection algorithms can more accurately detect and track the motion trajectory of key joints, providing richer data support for the above applications.

[0057] Figure 1 A schematic diagram of the flow of a convolutional neural network-based target detection method provided in an embodiment of the present application.

[0058] like Figure 1 As shown, the convolutional neural network-based target detection method 100 provided in an embodiment of the present application includes, for example, steps S110-S140.

[0059] Step S110, performing feature processing on the original image to obtain feature data.

[0060] Step S120, performing position prediction on the feature data to obtain a position prediction result of the target object.

[0061] Step S130, performing category prediction on the feature data according to the position prediction result to obtain a category prediction result of the target object.

[0062] Step S140: determine the position prediction result and the category prediction result as the detection result of the target object.

[0063] Exemplarily, the embodiment of the present application first performs feature processing on the original image to obtain feature data, then performs position prediction on the feature data to obtain a position prediction result, and then further performs category prediction on the feature data with reference to the position prediction result to obtain a category prediction result. The position prediction result and the category prediction result can be used as the final detection result.

[0064] When biological vision systems perform target recognition, they usually pre-process the position information of the target object first, and then combine this position information to perform more accurate category recognition. The target detection method of the present application also adopts a similar idea. First, the position information of the target object is predicted, and then this position information is used to perform more accurate category prediction. This can better utilize the important prior knowledge of the target position for category prediction and improve the overall detection accuracy.

[0065] In addition, when learning target recognition, the biological vision system not only learns the appearance characteristics of the target, but also learns the distribution patterns and relationships of the target in space. The target detection method of this application explicitly models the position information of the target object, so it can better learn the spatial relationship and distribution characteristics of the target object. Among them, the knowledge of position information, spatial information and distribution information helps the target detection algorithm maintain strong generalization performance in scenes such as complex backgrounds and occlusions.

[0066] In another example, the target detection method of the present application can be performed by a target detection model, which includes a feature acquisition layer, a target position prediction layer, and a target category prediction layer.

[0067] For example, feature processing can be performed on the original image based on the feature acquisition layer to obtain feature data. Position prediction can be performed on the feature data based on the target position prediction layer to obtain a position prediction result of the target object. Category prediction can be performed on the feature data based on the position prediction result to obtain a category prediction result of the target object. The position prediction result and the category prediction result can then be determined as the detection result of the target object.

[0068] Figure 2 A schematic diagram of the process of an end-to-end object detection method based on a convolutional neural network provided in an embodiment of the present application.

[0069] like Figure 2 As shown, the target detection model includes a feature acquisition layer, a target position prediction layer and a target category prediction layer, wherein the feature acquisition layer includes a target extraction backbone layer and a sequential fusion convolution layer.

[0070] Exemplarily, feature processing is performed on the original image to obtain feature data, including first inputting the original image into a target extraction backbone layer, extracting features from the original image based on the target extraction backbone layer to obtain multiple feature maps, and then inputting the multiple feature maps into a sequential fusion convolution layer, and performing feature fusion on the multiple feature maps based on the sequential fusion convolution layer to obtain feature data. After obtaining the feature data, the feature data is input into a target position prediction layer for position prediction to obtain a position prediction result. Finally, the position prediction result and the feature data are input into a target category prediction layer for category prediction to obtain a category prediction result.

[0071] The target extraction backbone layer may include a multi-scale perception layer, at least one differential fusion convolution layer, and at least one first spatial depth decoupling convolution layer. The first spatial depth decoupling convolution layer, the multi-scale perception layer, and the differential fusion convolution layer may exist in parallel. The calculation process of the target extraction backbone layer is as follows: Figure 3 shown.

[0072] Each differential fusion convolution layer may include a second spatial depth decoupled convolution layer, a third spatial depth decoupled convolution layer, and a first residual layer.

[0073] The object position prediction layer may include a second residual layer and a fourth spatial depth decoupled convolution layer.

[0074] The first residual layer and the second residual layer may each include a fifth spatial depth decoupled convolution layer, a sixth spatial depth decoupled convolution layer, and a seventh spatial depth decoupled convolution layer.

[0075] The sequential fusion convolutional layers may include a pooling aggregation layer, a feature extraction fusion layer, a connection layer, and an eighth spatial depth decoupled convolutional layer.

[0076] The residual calculation includes: performing spatial depth decoupling feature extraction on the input feature to obtain the sixth feature; performing spatial depth decoupling feature extraction on the sixth feature to obtain the seventh feature; merging the input feature and the seventh feature through channels to obtain the eighth feature; performing spatial depth decoupling feature extraction on the eighth feature to obtain the result of the residual calculation.

[0077] The spatial deep decoupling feature extraction includes: performing a 1×1 convolution operation on the input features to obtain a first multi-channel feature; performing a depth-separable convolution operation on each channel of the first multi-channel feature to obtain a second multi-channel feature; merging the channels of the second multi-channel feature to obtain a merged feature; performing batch normalization and activation function processing on the merged feature to obtain the result of spatial deep decoupling feature extraction.

[0078] Residual computation can be performed by a residual layer, and spatial-depth decoupled feature extraction can be performed by a spatial-depth decoupled convolutional layer.

[0079] First, the structure and function of the residual layer and the spatial depth decoupled convolutional layer are explained.

[0080] Specifically, each of the first residual layer and the second residual layer can be used to perform:

[0081] For the feature R1 input to the residual layer, feature extraction is performed on the input feature R1 based on the fifth spatial depth decoupled convolution layer to obtain the sixth feature R2;

[0082] Based on the sixth spatial depth decoupling convolution layer, feature extraction is performed on the sixth feature R2 to obtain the seventh feature R3;

[0083] Merge the input feature R1 and the seventh feature R3 to obtain the eighth feature R4;

[0084] The eighth feature R4 is extracted based on the seventh spatial depth decoupled convolutional layer to obtain the output feature R5 of the residual layer.

[0085] Among them, the input feature R1 and the output feature R5 of the residual layer can have the same size.

[0086] For the first spatial depth decoupled convolution layer, the second spatial depth decoupled convolution layer, ..., the eighth spatial depth decoupled convolution layer, each of the eight spatial depth decoupled convolution layers can perform a convolution operation based on a 1×1 convolution kernel and a depthwise separable convolution operation, and each spatial depth decoupled convolution layer can be used to perform:

[0087] For the features input into the spatial depth decoupled convolution layer, the size of the input feature is, for example, H1×W1×C1, and the input feature is denoted as I1. A convolution operation is performed on the input feature I1 based on a 1×1 convolution kernel to obtain a first multi-channel feature I2; the size of the 1×1 convolution kernel is 1×1×C1, the number of 1×1 convolution kernels is C2, C2 is the number of output channels, and the size of the first multi-channel feature I2 can be H1×W1×C2.

[0088] A depthwise separable convolution operation is performed on each channel of the first multi-channel feature I2 to obtain a second multi-channel feature I3, wherein the depthwise separable convolution can perform a convolution operation on each channel of the first multi-channel feature I2 to obtain the second multi-channel feature I3.

[0089] Channels of the second multi-channel feature I3 are merged to obtain a merged feature I3. For example, calculation results of all channels of the second multi-channel feature I3 are merged to obtain the merged feature I3. The size of the merged feature I3 is, for example, H2×W2×C2.

[0090] The merged feature I3 is batch normalized and processed by an activation function to obtain the output feature I4 of the spatial depth decoupled convolutional layer.

[0091] Among them, the spatial depth decoupled convolution layer can specify any number of output channels and the size of output features.

[0092] Exemplarily, performing feature extraction on the original image to obtain multiple feature maps includes: performing feature extraction on the original image to obtain multi-scale features; performing feature extraction on the multi-scale features to obtain multiple feature maps.

[0093] Among them, feature extraction is performed on the multi-scale features to obtain multiple feature maps, including: feature extraction is performed on the multi-scale features to obtain the first differential fusion feature; feature extraction is performed on the first differential fusion feature to obtain the first spatial depth decoupling feature; feature extraction is performed on the n-1th spatial depth decoupling feature to obtain the nth differential fusion feature, wherein n is an integer greater than or equal to 2 and less than or equal to N, and N is an integer greater than or equal to 2; spatial depth decoupling feature extraction is performed on the nth differential fusion feature to obtain the nth spatial depth decoupling feature; at least part of the N spatial depth decoupling features are determined as multiple feature maps.

[0094] Among them, feature extraction is performed on the original image to obtain multi-scale features, including: feature extraction is performed on the original image to obtain the first convolution feature; feature extraction is performed on the m-1th convolution feature to obtain the mth convolution feature, wherein m is an integer greater than or equal to 2 and less than or equal to M, and M is an integer greater than or equal to 2; channel merging processing is performed on the M convolution features to obtain multi-scale features.

[0095] Specifically, the process of extracting features from the original image to obtain multiple feature maps can be performed by the target extraction backbone layer.

[0096] The workflow of the target extraction backbone layer is further described below.

[0097] Exemplarily, the target extraction backbone layer includes a multi-scale perception layer, a differential fusion convolution layer, and a first spatial depth decoupling convolution layer.

[0098] First, feature extraction can be performed on the original image based on the multi-scale perception layer to obtain multi-scale features.

[0099] Then, based on the differential fusion convolution layer and the first spatial depth decoupling convolution layer, multi-scale features are extracted to obtain multiple feature maps.

[0100] In one example, there are N differential fusion convolution layers and N first spatial depth decoupling convolution layers, where N is an integer greater than or equal to 2. When extracting multi-scale features based on the differential fusion convolution layer and the first spatial depth decoupling convolution layer to obtain multiple feature maps, the following process may be included:

[0101] Based on the first differential fusion convolution layer, feature extraction is performed on the multi-scale features to obtain the first differential fusion feature. Then, based on the first first spatial depth decoupling convolution layer, feature extraction is performed on the first differential fusion feature to obtain the first spatial depth decoupling feature.

[0102] Based on the nth differential fusion convolution layer, feature extraction is performed on the n-1th spatial depth decoupling feature to obtain the nth differential fusion feature, where n is an integer greater than or equal to 2 and less than or equal to N; then, based on the nth first spatial depth decoupling convolution layer, feature extraction is performed on the nth differential fusion feature to obtain the nth spatial depth decoupling feature.

[0103] Finally, at least part of the N spatial depth decoupled features are determined as a plurality of feature maps.

[0104] In order to facilitate the understanding of the workflow of the target extraction backbone layer, the following Figure 3 Here is a specific example. It can be understood that Figure 3 The example takes N=4 as an example, that is, taking 4 differential fusion convolutional layers and 4 first spatial depth decoupling convolutional layers as an example, and N can also be 2, 3, 5, 6, etc.

[0105] Figure 3 A schematic diagram of the workflow of the target extraction backbone layer provided for an implementation manner of the present application.

[0106] like Figure 3 As shown, the workflow of the target extraction backbone layer may include steps D0-D9.

[0107] D0: Get the original image. The original image may be a three-channel RGB color image, and its size may be H×W×C. The original image is denoted as T0.

[0108] D1: The original image T0 is input into the multi-scale perception layer. After being processed by the multi-scale perception layer, the number of channels and the size of the original image T0 are changed to (H / 2)×(W / 2)×(8×C1). After being processed by the multi-scale perception layer, the multi-scale feature is obtained, and the multi-scale feature is recorded as T1.

[0109] D2: Input T1 into the first differential fusion convolution layer, perform differential fusion convolution on T1 to obtain a feature map of size (H / 2)×(W / 2)×(8×C1), denoted as T2.

[0110] D3: Input T2 into the first spatial depth decoupled convolution layer, perform spatial depth decoupled convolution on T2 to obtain a feature map of size (H / 4)×(W / 4)×(16×C1), denoted as T3.

[0111] D4: Input T3 into the second differential fusion convolution layer, perform differential fusion convolution on T3 to obtain a feature map with a size of (H / 4)×(W / 4)×(16×C1), denoted as T4.

[0112] D5: Input T4 into the second first spatial depth decoupled convolution layer, perform spatial depth decoupled convolution calculation on T4 to obtain a feature map of size (H / 8)×(W / 8)×(32×C1), denoted as T5.

[0113] D6: Input T5 into the third differential fusion convolution layer, perform differential fusion convolution on T5 to obtain a feature map with a size of (H / 8)×(W / 8)×(32×C1), denoted as T6.

[0114] D7: Input T6 into the third first spatial depth decoupled convolution layer, perform spatial depth decoupled convolution calculation on T6 to obtain a feature map of size (H / 16)×(W / 16)×(64×C1), denoted as T7.

[0115] D8: Input T7 into the fourth differential fusion convolution layer, perform differential fusion convolution calculation on T7 to obtain a feature map of size (H / 16)×(W / 16)×(64×C1), denoted as T8.

[0116] D9: Input T8 into the fourth first spatial depth decoupled convolution layer, perform spatial depth decoupled convolution calculation on T8 to obtain a feature map of size (H / 32)×(W / 32)×(128×C1), denoted as T9.

[0117] Finally, at least part of the N spatial depth decoupled features is determined as a plurality of feature maps. For example, T5, T7, and T9 are used as the target to extract the plurality of feature maps output by the backbone layer.

[0118] The workflow of the multi-scale perception layer is further described below.

[0119] Exemplarily, the multi-scale perception layer includes M convolution kernels, where M is an integer greater than or equal to 2; and extracting features from the original image based on the multi-scale perception layer to obtain multi-scale features includes the following:

[0120] First, the first convolution kernel can be used to extract features from the original image to obtain the first convolution feature. Then, the m-1th convolution feature can be extracted based on the mth convolution kernel to obtain the mth convolution feature, where m is an integer greater than or equal to 2 and less than or equal to M. Finally, the M convolution features are channel-merged to obtain multi-scale features.

[0121] For ease of understanding, take M=8 as an example, that is, the multi-scale perception layer includes 8 convolution kernels, and M may also be 2, 3, 4, 5, 6, 10, etc. The workflow of the multi-scale perception layer includes the following steps C1-C9.

[0122] C1: Assume that the size of a feature map T1 is H×W×C, where H, W, and C represent the height, width, and number of channels of the feature map, respectively. The feature map T1 can be the original image or a feature map obtained by preprocessing the original image.

[0123] C2: After the feature map T1 is convolved, the size of the feature map U1 is (H / 2)×(W / 2)×C1. Here, the value of C1 is an even power of 2. Here, the convolution kernel size is 1×1 and the convolution step size is 2.

[0124] C3: After the feature map U1 is convolved, the size of the feature map U2 is (H / 2)×(W / 2)×C1. Here, the convolution kernel size is 3×3 and the convolution step size is 2.

[0125] C4: After the convolution of feature map U2, the size of feature map U3 is (H / 2)×(W / 2)×C1. Here, the convolution kernel size is 5×5 and the convolution step size is 2.

[0126] C5: After convolution processing, feature map U3 is convolved to obtain feature map U4 of size (H / 2)×(W / 2)×C1. Here, the convolution kernel size is 7×7 and the convolution step size is 2.

[0127] C6: After convolution processing, feature map U4 is convolved to obtain feature map U5 of size (H / 2)×(W / 2)×C1; here the convolution kernel size is 1×1 and the convolution step size is 2.

[0128] C7: After convolution processing, feature map U5 is convolved to obtain feature map U6 of size (H / 2)×(W / 2)×C1. Here, the convolution kernel size is 3×3 and the convolution step size is 2.

[0129] C8: After convolution processing, feature map U6 obtains feature map U7 with a size of (H / 2)×(W / 2)×C1. Here, the convolution kernel size is 5×5 and the convolution step size is 2.

[0130] C8: After convolution processing, feature map U7 is processed to obtain feature map U8 of size (H / 2)×(W / 2)×C1; the convolution kernel size is 7×7 and the convolution step size is 2.

[0131] C9: The eight feature maps U1, U2, U3, U4, U5, U6, U7 and U8 are processed by the channel merging layer to obtain the multi-scale feature U9, whose size is (H / 2)×(W / 2)×(8×C1).

[0132] It can be understood that the multi-scale perception layer of other YOLO target detection algorithms is usually deployed at the back end of the algorithm, resulting in the loss of some information when the features reach the multi-scale perception layer, and the features extracted by the multi-scale perception layer are missing, which affects the detection accuracy. Unlike other YOLO target detection algorithms, the multi-scale perception layer of this application is deployed at the front end of the algorithm, directly processing the original image, and using different receptive fields to extract features from the original image at the very beginning of target detection, with less information loss, which is conducive to improving the performance of the model. Finally, the features of different scales are merged through channels, and this information is integrated to enhance the perception ability of the model.

[0133] Exemplarily, each of the N differential fusion features is obtained by: performing spatial depth decoupling feature extraction on the input feature to obtain a first feature; performing residual calculation on the first feature to obtain a second feature; performing spatial depth decoupling feature extraction on the input feature to obtain a third feature; merging the second feature and the third feature through channels to obtain a fourth feature; and performing convolution processing on the fourth feature to obtain the differential fusion feature. Specifically, the differential fusion feature can be obtained by performing feature extraction on the differential fusion convolution layer.

[0134] The following is a further description of the workflow of the differential fusion convolutional layer.

[0135] Exemplarily, each differential fusion convolution layer includes a second spatial depth decoupled convolution layer, a third spatial depth decoupled convolution layer and a first residual layer; each differential fusion convolution layer is used to perform the following steps B1-B6.

[0136] B1: For the features input into the differential fusion convolutional layer, the input features are recorded as P1.

[0137] B2: Extract the input feature P1 based on the second spatial depth decoupled convolutional layer to obtain the first feature P2.

[0138] B3: Calculate the first feature P2 based on the first residual layer to obtain the second feature P3. For example, the first feature P2 is calculated in sequence through at least one residual layer to obtain the second feature P3.

[0139] B4: Based on the third spatial depth decoupled convolutional layer, feature extraction is performed on the input feature P1 to obtain the third feature P4.

[0140] B5: Merge the second feature P3 and the third feature P4 to obtain the fourth feature P5;

[0141] B6: Perform convolution processing on the fourth feature P5 to obtain the output feature P6 of the differential fusion convolution layer. For example, P5 is calculated by the convolution layer to obtain P6.

[0142] Exemplarily, the size of the input feature P1 of the differential fusion convolution layer has the same dimension as the size of the output feature P6. The difference from the C3 structure in YOLO is that the differential fusion convolution layer of the present application performs feature extraction based on the spatial depth decoupled convolution layer. The spatial depth decoupled convolution layer can perform 1×1 convolution operations and depth-separable convolution operations. While reducing the number of parameters, it can also increase the learning ability between model channels, better fuse deep and shallow feature information, effectively combine shallow detail features and deep semantic features, improve the expressiveness and generalization performance of the model, and alleviate the gradient vanishing problem.

[0143] The target extraction backbone layer has been described in detail above. The sequential fusion convolution layer is described below. The sequential fusion convolution layer can be used to perform at least one of pooling aggregation, feature extraction fusion, feature connection and spatial depth decoupled feature extraction based on multiple feature maps to obtain feature data.

[0144] Exemplarily, the sequential fusion convolutional layers include a pooling aggregation layer, a feature extraction fusion layer, a connection layer, and an eighth spatial depth decoupled convolutional layer.

[0145] The multiple feature maps T5, T7, and T9 output by the feature extraction backbone layer are input into the sequential fusion convolution layer for feature fusion to obtain feature data.

[0146] The calculation of the sequential fusion convolution layer in the embodiment of the present application is consistent with the neck structure of YOLOV9, in which the parts involving upsampling and downsampling are all replaced by spatial depth decoupled convolution layers to reduce the number of parameters. After the sequential fusion convolution layer fuses the input multiple feature maps T5, T7, and T9, the output feature data is recorded as T10, T11, and T12. Among them, the sizes of T10, T11, and T12 can be: (H / 8)×(W / 8)×(32×C1), (H / 16)×(W / 16)×(64×C1), and (H / 32)×(W / 32)×(128×C1), respectively.

[0147] For ease of understanding, the following Figure 4 A schematic diagram of the structure of sequential fusion convolutional layers is shown.

[0148] Figure 4 A schematic diagram of the structure of the sequential fusion convolutional layers provided in an embodiment of the present application.

[0149] like Figure 4 As shown, the sequential fusion convolution layer includes a pooling aggregation layer, a feature extraction fusion layer, a connection layer and an eighth spatial depth decoupled convolution layer.

[0150] The pooling aggregation layer includes SPPELAN, the feature extraction fusion layer includes RepNCSPELAN4, the connection layer includes Concat layer, and the eighth spatial depth decoupled convolution layer includes 4 spatial depth decoupled convolution layers, which are represented by A1-A4. A1 and A2 correspond to the upsampling module of the neck structure of YOLOV9, and A3 and A4 correspond to the downsampling module of the neck structure of YOLOV9.

[0151] After the sequential fusion convolutional layer outputs the feature data T10, T11, and T12, the feature data T10, T11, and T12 are input into the target position prediction layer for position prediction.

[0152] Exemplarily, performing position prediction on the feature data to obtain a position prediction result of the target object includes: performing residual calculation on the feature data to obtain a fifth feature; performing spatial depth decoupling feature extraction on the fifth feature to obtain a position prediction result of the target object. Specifically, the position prediction can be performed by a target position prediction layer.

[0153] Exemplarily, the target position prediction layer includes a second residual layer and a fourth spatial depth decoupled convolution layer. The feature data is calculated based on the second residual layer to obtain a fifth feature, and the fifth feature is calculated based on the fourth spatial depth decoupled convolution layer to obtain a position prediction result of the target object.

[0154] Specifically, the calculation process of the target position prediction layer may include steps E1-E3.

[0155] E1: The feature data T10 is processed by the second residual layer and the fourth spatial depth decoupled convolution layer in sequence to obtain a first prediction head. The size of the first prediction head is, for example, (H / 8)×(W / 8)×4.

[0156] E2: The feature data T11 is processed by the second residual layer and the fourth spatial depth decoupled convolution layer in sequence to obtain a second prediction head. The size of the second prediction head is, for example, (H / 16)×(W / 16)×4.

[0157] E3: The feature data T12 is processed by the second residual layer and the fourth spatial depth decoupled convolution layer in sequence to obtain a third prediction head. The size of the third prediction head is, for example, (H / 32)×(W / 32)×4.

[0158] The first prediction head, the second prediction head, and the third prediction head can all be feature maps. The first prediction head, the second prediction head, and the third prediction head are the position prediction results of the target object. It can be seen that the target position prediction layer performs position prediction at three scales to obtain three prediction heads respectively. The number of channels of each prediction head is 4. The first two-dimensional channels can represent the center point position of the target box, and the latter two-dimensional channels can represent the width and height of the target box. The target box is the area where the predicted target object is located.

[0159] After obtaining the position prediction result, the position prediction result is input into the target category prediction layer, and the target category prediction layer predicts the category of the target object based on the position prediction result and the feature data T10, T11, and T12.

[0160] Specifically, the target category prediction layer extracts local features at corresponding positions from feature data T10, T11, and T12 according to the position prediction results, and then performs category prediction based on the local features at the corresponding positions to obtain the category prediction results of the target object.

[0161] For example, the local features of the corresponding position include local features of multiple scales, such as local features of three scales, the local features of the first scale are extracted from feature data T10, the local features of the second scale are extracted from feature data T11, and the local features of the third scale are extracted from feature data T12. Next, the local features of the three scales are average-pooled and aggregated into a feature vector, and a category prediction result is obtained based on the feature vector.

[0162] Specifically, the calculation process of the target category prediction layer is as follows:

[0163] Feature aggregation: Use the position prediction result as an index to extract the local features at the corresponding positions from the feature data T10, T11, and T12, and then perform average pooling on the extracted local features to aggregate them into a feature vector of fixed length.

[0164] Category prediction: The aggregated feature vector is input into a fully connected layer. The output dimension of the fully connected layer is equal to the number of target categories. Each output element represents the prediction score of the corresponding category. A Softmax activation function is added after the fully connected layer to convert the prediction score into a probability distribution. Both the prediction score and probability distribution of the category can be used as the category prediction results.

[0165] The above describes in detail the method of target detection based on the target detection model. Next, the training method of the target detection model is described.

[0166] Figure 5 A flowchart of a method for training a convolutional neural network-based target detection model provided in an embodiment of the present application.

[0167] like Figure 5 As shown, the training method 500 of the target detection model based on the convolutional neural network provided in the embodiment of the present application includes steps S510-S540, wherein the target detection model includes a feature acquisition layer, a target position prediction layer and a target category prediction layer.

[0168] Step S510, performing feature processing on the training image based on the feature acquisition layer to be trained to obtain first feature data;

[0169] Step S520, based on the target position prediction layer to be trained, the position prediction of the first feature data is performed to obtain a first position prediction result of the target object, and based on the first position prediction result and the position label of the training image, the parameters of the feature acquisition layer to be trained and the parameters of the target position prediction layer to be trained are adjusted to obtain a trained feature acquisition layer and a trained target position prediction layer.

[0170] Step S530: Perform feature processing on the training image based on the trained feature acquisition layer to obtain second feature data.

[0171] Step S540, based on the target category prediction layer to be trained, according to the target position information of the target object in the training image, perform category prediction on the second feature data to obtain the category prediction result of the target object, and based on the category prediction result and the category label of the training image, adjust the parameters of the target category prediction layer to be trained to obtain the trained target category prediction layer.

[0172] For example, the target position information of the target object in the training image may be information pre-marked in the training image. Alternatively, the position prediction of the second feature data may be performed based on the trained target position prediction layer to obtain the position prediction result of the target object, and the position prediction result may be used as the target position information.

[0173] Specifically, when training the target detection model, the target position prediction layer can be trained first to obtain a relatively stable target positioning capability. Then, the parameters of the target position prediction layer and the feature extraction layer are fixed, and the target category prediction layer is further trained.

[0174] Exemplarily, the true label of the position is compared with the position result predicted by the model, and the true label of the category is compared with the category probability distribution result, and the gradient is calculated by back propagation and the model parameters are updated. The loss function used in model training includes a cross entropy loss function, which is used to optimize the accuracy of the prediction.

[0175] It can be understood that training the target position prediction and target category prediction separately allows the two modules to be optimized relatively independently. This decoupled design enhances the versatility of the algorithm and makes it easier to apply to different target detection tasks. When the detection task changes, there is no need to completely retrain the entire target detection model. In addition, since the position prediction and category prediction are relatively independent, they are very pluggable. In addition, according to actual needs, the position prediction and category prediction modules can be replaced with other better models respectively, thereby further improving the overall performance. Its flexibility is difficult to achieve with a traditional single network.

[0176] Figure 6 A schematic diagram of a target detection device provided in an embodiment of the present application.

[0177] like Figure 6 As shown, the target detection device 600 provided in the embodiment of the present application includes: a feature acquisition layer 610, a target position prediction layer 620 and a target category prediction layer 630.

[0178] The feature acquisition layer 610 is used to perform feature processing on the original image to obtain feature data.

[0179] The target position prediction layer 620 is used to perform position prediction on the feature data to obtain a position prediction result of the target object.

[0180] The target category prediction layer 630 is used to perform category prediction on the feature data according to the position prediction result to obtain the category prediction result of the target object.

[0181] Exemplarily, the position prediction result and the category prediction result are used as the detection result of the target object.

[0182] Exemplarily, the target category prediction layer 630 is also used to: extract local features at corresponding positions from feature data according to the position prediction results; perform category prediction based on the local features at the corresponding positions to obtain category prediction results of the target object.

[0183] Exemplarily, the feature acquisition layer includes: a target extraction backbone layer, which is used to extract features from the original image to obtain multiple feature maps; and a sequential fusion convolution layer, which is used to perform feature fusion on multiple feature maps to obtain feature data.

[0184] Exemplarily, the target extraction backbone layer includes: a multi-scale perception layer, which is used to extract features from the original image to obtain multi-scale features; a differential fusion convolution layer and a spatial depth decoupling convolution layer, which are used to extract features from the multi-scale features to obtain multiple feature maps.

[0185] Exemplarily, there are N differential fusion convolution layers and N spatial depth decoupling convolution layers, where N is an integer greater than or equal to 2: the first differential fusion convolution layer is used to extract features from multi-scale features to obtain a first differential fusion feature; the first spatial depth decoupling convolution layer is used to extract features from the first differential fusion feature to obtain a first spatial depth decoupled feature; the nth differential fusion convolution layer is used to extract features from the n-1th spatial depth decoupled feature to obtain the nth differential fusion feature, where n is an integer greater than or equal to 2 and less than or equal to N; the nth spatial depth decoupling convolution layer is used to extract features from the nth differential fusion feature to obtain the nth spatial depth decoupled feature; wherein the multiple feature maps are at least part of the N spatial depth decoupled features.

[0186] Exemplarily, the multi-scale perception layer includes M convolution kernels, where M is an integer greater than or equal to 2: the first convolution kernel is used to extract features of the original image to obtain the first convolution feature; the m-th convolution kernel is used to extract features of the m-1th convolution feature to obtain the m-th convolution feature, where m is an integer greater than or equal to 2 and less than or equal to M; wherein the M convolution features are channel-merged to obtain multi-scale features.

[0187] Exemplarily, each differential fusion convolution layer includes a second spatial depth decoupled convolution layer, a third spatial depth decoupled convolution layer and a first residual layer; each differential fusion convolution layer is used to perform: for the features input into the differential fusion convolution layer, feature extraction is performed on the input features based on the second spatial depth decoupled convolution layer to obtain a first feature; the first feature is calculated based on the first residual layer to obtain a second feature; the input feature is extracted based on the third spatial depth decoupled convolution layer to obtain a third feature; the second feature and the third feature are channel-merged to obtain a fourth feature; the fourth feature is convolved to obtain the output feature of the differential fusion convolution layer.

[0188] Exemplarily, the target position prediction layer includes a second residual layer and a fourth spatial depth decoupled convolution layer; the second residual layer is used to calculate the feature data to obtain the fifth feature; the fourth spatial depth decoupled convolution layer is used to calculate the fifth feature to obtain the position prediction result of the target object.

[0189] Exemplarily, the residual layer includes a fifth spatial depth decoupled convolution layer, a sixth spatial depth decoupled convolution layer and a seventh spatial depth decoupled convolution layer; the residual layer is used to perform: for the features input into the residual layer, feature extraction is performed on the input features based on the fifth spatial depth decoupled convolution layer to obtain the sixth feature; feature extraction is performed on the sixth feature based on the sixth spatial depth decoupled convolution layer to obtain the seventh feature; channel merging is performed on the input features and the seventh feature to obtain the eighth feature; feature extraction is performed on the eighth feature based on the seventh spatial depth decoupled convolution layer to obtain the output features of the residual layer.

[0190] Exemplarily, the sequential fusion convolution layer includes at least one of a pooling aggregation layer, a feature extraction fusion layer, a connection layer, and an eighth spatial depth decoupled convolution layer.

[0191] Exemplarily, the spatial depth decoupled convolution layer is used to perform the following operations: for the features input into the spatial depth decoupled convolution layer, convolution operation is performed on the input features based on a 1×1 convolution kernel to obtain a first multi-channel feature; depth-separable convolution operation is performed on each channel of the first multi-channel feature to obtain a second multi-channel feature; channels of the second multi-channel feature are merged to obtain merged features; batch normalization and activation function processing are performed on the merged features to obtain output features of the spatial depth decoupled convolution layer.

[0192] Exemplarily, the local features at the corresponding position include local features at multiple scales; performing category prediction based on the local features at the corresponding position to obtain a category prediction result of the target object includes: averaging and pooling the local features at multiple scales and aggregating them into a feature vector; and obtaining a category prediction result based on the feature vector.

[0193] It can be understood that the specific implementation process of the target detection device 600 can refer to the implementation process of the target detection method based on the convolutional neural network mentioned above, and will not be repeated here.

[0194] Figure 7 A schematic diagram of a training device for a target detection model provided in an embodiment of the present application.

[0195] like Figure 7 As shown, the training device 700 of the target detection model provided in the embodiment of the present application includes: a feature acquisition layer 710 to be trained, a target position prediction layer 720 to be trained, a first parameter adjustment layer 730, a target category prediction layer 740 to be trained and a second parameter adjustment layer 750.

[0196] The feature acquisition layer 710 to be trained is used to perform feature processing on the training image to obtain first feature data.

[0197] The target position prediction layer 720 to be trained is used to perform position prediction on the first feature data to obtain a first position prediction result of the target object.

[0198] The first parameter adjustment layer 730 is used to adjust the parameters of the feature acquisition layer to be trained and the parameters of the target position prediction layer to be trained based on the first position prediction result and the position label of the training image to obtain the trained feature acquisition layer and the trained target position prediction layer.

[0199] The target category prediction layer 740 to be trained is used to perform category prediction on the second feature data according to the target position information of the target object in the training image to obtain the category prediction result of the target object, wherein the second feature data is obtained by feature processing the training image based on the trained feature acquisition layer.

[0200] The second parameter adjustment layer 750 adjusts the parameters of the target category prediction layer to be trained based on the category prediction result and the category label of the training image to obtain a trained target category prediction layer.

[0201] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0202] One embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.

[0203] Figure 8 A block diagram of an edge computing device provided for an embodiment of the present application.

[0204] An embodiment of the present application provides an edge computing device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0205] like Figure 8 As shown, for ease of understanding, an embodiment of the present application shows a specific edge computing device 800.

[0206] The edge computing device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The edge computing device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0207] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In RAM 803, various programs and data required for the operation of the edge computing device 800 can also be stored. The computing unit 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0208] Multiple components in the edge computing device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the edge computing device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0209] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods described above. For example, in some embodiments, any one or more of the above methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the edge computing device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of any one or more of the various methods described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform any one or more of the above methods in any other appropriate manner (e.g., by means of firmware).

[0210] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this application, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0211] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0212] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0213] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0214] In addition, the terms "first", "second", etc. used in the embodiments of the present application are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present application can explicitly or implicitly indicate that at least one of the features is included in the embodiment. In the description of the present application, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0215] In this application, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "connected" and "fixed" etc. appearing in the embodiments should be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or an integrated connection. It can be understood that it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements, or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific implementation situation.

[0216] In the present application, unless otherwise clearly specified and limited, a first feature being “above” or “below” a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being “above”, “above”, and “above” a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being “below”, “below”, and “below” a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0217] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Perform feature processing on the original image to obtain feature data; Performing position prediction on the feature data to obtain a position prediction result of the target object; Performing category prediction on the feature data according to the position prediction result to obtain a category prediction result of the target object; The position prediction result and the category prediction result are determined as the detection result of the target object.

2. The method according to claim 1, characterized in that The performing category prediction on the feature data according to the position prediction result to obtain the category prediction result of the target object includes: extracting local features at corresponding positions from the feature data according to the position prediction result; performing category prediction based on the local features at corresponding positions to obtain the category prediction result of the target object; and / or, The feature processing of the original image to obtain feature data includes: extracting features from the original image to obtain multi-scale features; extracting features from the multi-scale features to obtain multiple feature maps; and fusing features from the multiple feature maps to obtain the feature data.

3. The method according to claim 2, characterized in that The extracting the multi-scale features to obtain a plurality of feature maps includes: Extracting features from the multi-scale features to obtain a first differential fusion feature; Perform feature extraction on the first differential fusion feature to obtain the first spatial depth decoupling feature; Extract the n-1th spatial depth decoupling feature to obtain the nth differential fusion feature, where n is an integer greater than or equal to 2 and less than or equal to N, and N is an integer greater than or equal to 2; Perform spatial depth decoupling feature extraction on the nth differential fusion feature to obtain the nth spatial depth decoupling feature; At least part of the N spatial depth decoupling features are determined as the multiple feature maps.

4. The method according to claim 3, characterized in that Each of the N differential fusion features is obtained in the following way: Perform spatial depth decoupling feature extraction on the input features to obtain the first feature; Performing residual calculation on the first feature to obtain a second feature; Perform spatial depth decoupling feature extraction on the input features to obtain the third feature; Merge the second feature and the third feature through channels to obtain a fourth feature; The fourth feature is convolved to obtain the differential fusion feature.

5. The method according to claim 1, characterized in that: The performing position prediction on the feature data to obtain a position prediction result of the target object includes: Performing residual calculation on the feature data to obtain a fifth feature; Perform spatial depth decoupling feature extraction on the fifth feature to obtain a position prediction result of the target object.

6. The method according to claim 4 or 5, characterized in that: The residual calculation includes: Perform spatial depth decoupling feature extraction on the input features to obtain the sixth feature; Performing spatial depth decoupling feature extraction on the sixth feature to obtain a seventh feature; Merge the input feature and the seventh feature through channels to obtain an eighth feature; Perform spatial depth decoupling feature extraction on the eighth feature to obtain a residual calculation result.

7. The method according to claim 2, characterized in that: The performing feature fusion on the multiple feature maps to obtain the feature data includes: performing at least one of pooling aggregation, feature extraction fusion, feature connection and spatial depth decoupling feature extraction based on the multiple feature maps to obtain the feature data; and / or, The local features of the corresponding positions include local features of multiple scales; performing category prediction based on the local features of the corresponding positions to obtain a category prediction result of the target object includes: averaging and pooling the local features of multiple scales and aggregating them into a feature vector; and obtaining a category prediction result based on the feature vector.

8. The method according to claim 3, 4, 5 or 7, characterized in that The spatial depth decoupling feature extraction comprises: Perform a 1×1 convolution operation on the input features to obtain the first multi-channel features; Performing a depth-wise separable convolution operation on each channel of the first multi-channel feature to obtain a second multi-channel feature; Merging the second multi-channel features to obtain merged features; The merged features are batch normalized and processed with activation functions to obtain the results of spatial deep decoupled feature extraction.

9. A method for training a target detection model, characterized in that: The target detection model comprises the target detection model according to any one of claims 1 to 8, the target detection model comprises a feature acquisition layer, a target position prediction layer and a target category prediction layer, and the method comprises: Performing feature processing on the training image based on the feature acquisition layer to be trained to obtain first feature data; Performing position prediction on the first feature data based on the target position prediction layer to be trained to obtain a first position prediction result of the target object, and adjusting parameters of the feature acquisition layer to be trained and parameters of the target position prediction layer to be trained based on the first position prediction result and the position label of the training image to obtain a trained feature acquisition layer and a trained target position prediction layer; Perform feature processing on the training image based on the trained feature acquisition layer to obtain second feature data; Based on the target category prediction layer to be trained, category prediction is performed on the second feature data according to the target position information of the target object in the training image to obtain a category prediction result of the target object, and based on the category prediction result and the category label of the training image, the parameters of the target category prediction layer to be trained are adjusted to obtain a trained target category prediction layer.

10. A target detection device, characterized in that: The device comprises: The feature acquisition layer is used to perform feature processing on the original image to obtain feature data; The target position prediction layer is used to perform position prediction on the feature data to obtain a position prediction result of the target object; A target category prediction layer, used to perform category prediction on the feature data according to the position prediction result to obtain a category prediction result of the target object; The position prediction result and the category prediction result are used as the detection results of the target object.

11. The device according to claim 10, characterized in that The feature acquisition layer includes: The target extraction backbone layer is used to extract features from the original image and obtain multiple feature maps; A sequential fusion convolution layer is used to perform feature fusion on the multiple feature maps to obtain the feature data; The target extraction backbone layer includes: The multi-scale perception layer is used to extract features from the original image and obtain multi-scale features; The differential fusion convolution layer and the spatial depth decoupling convolution layer are used to extract the multi-scale features to obtain multiple feature maps.

12. The device according to claim 11, characterized in that The number of the differential fusion convolutional layers is N, and the number of the spatial depth decoupling convolutional layers is N, where N is an integer greater than or equal to 2: The first differential fusion convolution layer is used to extract the multi-scale features to obtain the first differential fusion features; The first spatial depth decoupling convolution layer is used to extract the first differential fusion feature to obtain the first spatial depth decoupling feature; The nth differential fusion convolution layer is used to extract the n-1th spatial depth decoupling feature to obtain the nth differential fusion feature, where n is an integer greater than or equal to 2 and less than or equal to N; The nth spatial depth decoupling convolution layer is used to extract the nth differential fusion feature to obtain the nth spatial depth decoupling feature; Wherein, the multiple feature maps are at least part of N spatial depth decoupled features.

13. An edge computing device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images