Target detection method and device, terminal and storage medium

Abstract

The application provides a target detection method and device, a terminal and a storage medium, and belongs to the technical field of computers. The method is executed by the terminal, and the method comprises the following steps: performing feature extraction on a target image based on a plurality of feature extraction layers of a target detection model to obtain a plurality of first feature maps of different scales, the target image comprising at least one target object, and the target detection model being used for detecting the category and position of the at least one target object; performing fusion on the plurality of first feature maps of different scales based on a feature fusion layer of the target detection model to obtain a first fused feature map, the first fused feature map comprising category features and position features of the at least one target object; and performing decoding on the first fused feature map based on an output layer of the target detection model to obtain a detection result of the target image, the detection result being used for representing the category and position of the at least one target object. The above scheme can successfully detect most target objects in the target image, and improves the accuracy of target detection.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a target detection method, apparatus, terminal and storage medium. Background Technology

[0002] With the development of computer technology, object detection technology is increasingly widely used in intelligent object recognition, smart transportation, and other fields. For example, object detection technology can quickly identify pedestrians, objects, buildings, and other target objects in images, as well as the categories of those objects. However, current object detection technology is highly dependent on image quality. When the target object in an image is small or its edges are blurred, it is difficult to detect the target object or accurately identify its category, resulting in low accuracy. Therefore, improving the accuracy of object detection in these situations is a technical problem that needs to be solved. Summary of the Invention

[0003] This application provides a target detection method, apparatus, terminal, and storage medium, which can improve the accuracy of target detection. The technical solution is as follows:

[0004] According to one aspect of the embodiments of this application, a target detection method is provided, the method comprising:

[0005] Based on multiple feature extraction layers of an object detection model, features are extracted from a target image to obtain first feature maps at multiple scales. The target image includes at least one target object. The object detection model is used to detect the category and location of the at least one target object. The multiple feature extraction layers and the first feature maps at multiple scales are used to... Figure 1 In a one-to-one correspondence, the feature extraction layer includes multiple cascaded first convolutional units, which are used to extract features from the input image through at least one of depthwise convolution, pointwise convolution, and batch normalization.

[0006] Based on the feature fusion layer of the target detection model, the first feature maps of the multiple scales are fused to obtain a first fused feature map, which includes the category features and location features of the at least one target object;

[0007] Based on the output layer of the target detection model, the first fused feature map is decoded to obtain the detection result of the target image, and the detection result is used to represent the category and location of the at least one target object.

[0008] According to another aspect of the embodiments of this application, a target detection device is provided, the device comprising:

[0009] The first extraction module is used to extract features from a target image based on multiple feature extraction layers of a target detection model, obtaining first feature maps at multiple scales. The target image includes at least one target object, and the target detection model is used to detect the category and location of the at least one target object. The multiple feature extraction layers and the first feature maps at multiple scales are then used to extract features from the target image. Figure 1 In a one-to-one correspondence, the feature extraction layer includes multiple cascaded first convolutional units, which are used to extract features from the input image through at least one of depthwise convolution, pointwise convolution, and batch normalization.

[0010] The first fusion module is used to fuse the first feature maps of the multiple scales based on the feature fusion layer of the target detection model to obtain a first fused feature map, wherein the first fused feature map includes the category features and location features of the at least one target object;

[0011] The first decoding module is used to decode the first fused feature map based on the output layer of the target detection model to obtain the detection result of the target image, wherein the detection result is used to represent the category and location of the at least one target object.

[0012] In some embodiments, the first extraction module is used to extract features from the target image based on the initial feature extraction layer of the target detection model to obtain an initial feature map of the target image. The initial feature extraction layer includes a second convolutional unit, and the convolution operation in the second convolutional unit is different from the convolution operation in the first convolutional unit. In the first feature extraction layer of the target detection model, features are extracted from the initial feature map through multiple first convolutional units in the feature extraction layer to obtain a first feature map of the first feature extraction layer. In the i-th feature extraction layer of the target detection model, features are extracted from the first feature map of the (i-1)-th feature extraction layer through multiple first convolutional units in the feature extraction layer to obtain a first feature map of the i-th feature extraction layer, until the first feature map of each feature extraction layer is obtained. The scale of the first feature map is negatively correlated with the number of the corresponding feature extraction layer, where i is an integer greater than 1.

[0013] In some embodiments, the apparatus further includes:

[0014] An edge enhancement module is used to enhance the edges of the original image to obtain the target image. The edge enhancement is used to enhance the edge information of at least one target object in the original image.

[0015] In some embodiments, the feature fusion layer of the target detection model includes multiple fusion units and multiple detection units, wherein the multiple fusion units correspond one-to-one with the multiple feature extraction layers, and the multiple fusion units correspond one-to-one with the multiple detection units;

[0016] The first fusion module includes:

[0017] A feature fusion unit is used to fuse a first feature map of any scale with other first feature maps of different scales based on a fusion unit corresponding to the first feature map, according to the scale of the first feature map, to obtain a first intermediate feature map. The number of channels in the first intermediate feature map is the number of pointwise convolution kernels in the fusion unit. The pointwise convolution kernels are used to perform pointwise convolution on the input feature map so that the number of channels in the input feature map is the same as the number of pointwise convolution kernels.

[0018] The channel separation unit is used to perform channel separation on the first intermediate feature map based on the detection unit corresponding to the fusion unit, so as to obtain a category feature map of at least one channel and a position feature map of at least one channel.

[0019] The splicing unit is used to splice the category feature map of at least one channel and the position feature map of at least one channel in the channel dimension based on the detection unit corresponding to the fusion unit to obtain a second intermediate feature map;

[0020] The stitching unit is also used to stitch multiple second intermediate feature maps in the scale dimension to obtain the first fused feature map.

[0021] In some embodiments, the feature fusion unit is configured to, for any scale of first feature map, perform pointwise convolution on the first feature map at the scale and the first feature maps at other scales respectively, based on the fusion unit corresponding to the first feature map, so that the number of channels of the first feature map at the scale and the first feature maps at other scales are the same, and the number of channels is the same as the number of pointwise convolution kernels in the pointwise convolution; based on the fusion unit, upsample or downsample multiple first feature maps at other scales according to the scale of the first feature map to obtain multiple first feature maps at the scales; and based on the fusion unit, fuse the first feature map at the scale and the multiple first feature maps at the scales to obtain the first intermediate feature map.

[0022] In some embodiments, the stitching unit is configured to, for any second intermediate feature map, convert the second intermediate feature map into a third intermediate feature map if the second intermediate feature map includes three dimensions: channel, width, and height. The third intermediate feature map includes two dimensions: channel and length, where the length is the product of the width and height. Multiple third intermediate feature maps are stitched together along the length dimension to obtain the first fused feature map.

[0023] In some embodiments, the apparatus further includes:

[0024] The second extraction module is used to extract features from the sample image based on multiple feature extraction layers in the target detection model to obtain a second feature map at multiple scales, wherein the sample image includes at least one target object.

[0025] The second fusion module is used to fuse the second feature maps of the multiple scales based on the feature fusion layer of the target detection model to obtain a second fused feature map. The second fused feature map includes the category features and location features of at least one target object in the sample image.

[0026] The second decoding module is used to decode the second fused feature map based on the output layer of the target detection model to obtain the detection result of the sample image. The detection result is used to represent the predicted category and predicted location of at least one target object in the sample image.

[0027] The loss determination module is used to determine the training loss of the target detection model based on the detection results and the labels of the sample images, wherein the labels of the sample images are used to represent the true category and true location of at least one target object in the sample images.

[0028] The model training module is used to train the object detection model based on the training loss.

[0029] In some embodiments, the training loss includes a category loss and a location loss, wherein the category loss is used to represent the difference between the predicted category and the true category of the at least one target object, and the location loss is used to represent the difference between the predicted location and the true location of the at least one target object.

[0030] In some embodiments, the model training module is configured to fuse the first feature maps at multiple scales based on the auxiliary training layer of the object detection model to obtain a third fused feature map; decode the third fused feature map based on the output layer of the object detection model to obtain the auxiliary detection result of the object detection model; determine the auxiliary training loss of the object detection model based on the auxiliary detection result of the sample image and the label of the sample image; and train the object detection model based on the training loss and the auxiliary training loss.

[0031] In some embodiments, the model training module is further configured to perform a first stage of training on the target detection model based on a first sample set, the first sample set including a plurality of original sample images and a first number of augmented sample images, the augmented sample images being obtained by data augmentation of the plurality of original sample images; perform a second stage of training on the target detection model based on a second sample set, the second sample set including the plurality of original sample images and a second number of augmented sample images, the second number being less than the first number; and perform a third stage of training on the target detection model based on the plurality of original sample images.

[0032] According to another aspect of the embodiments of this application, a terminal is provided, the terminal including a processor and a memory; the memory stores at least one piece of program code, the at least one piece of program code being executed by the processor to implement the target detection method as described above.

[0033] According to another aspect of the embodiments of this application, a chip is provided, the chip including programmable logic circuits and / or program instructions, which, when the chip is run on a terminal, is used to implement the target detection method described above.

[0034] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the storage medium storing at least one piece of program code, the at least one piece of program code being executed by a processor to implement the target detection method as described above.

[0035] According to another aspect of the embodiments of this application, a computer program product is provided, which stores at least one piece of program code, the at least one piece of program code being executed by a processor to implement the target detection method described above.

[0036] This application provides a target detection method that extracts features from an image to be detected using multiple feature extraction layers of a target detection model, resulting in feature maps at multiple scales. Since feature maps at different scales can reflect the category and location features of target objects of different sizes, fusing these feature maps to obtain a fused feature map can reflect the category and location features of target objects of different sizes in the image. Then, by decoding the fused feature map, the category and location of target objects of different sizes in the image can be determined. Using this method, most target objects in the target image can be successfully detected, improving the accuracy of target detection. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0039] Figure 2 This is a flowchart of a target detection method provided in an embodiment of this application;

[0040] Figure 3 This is a flowchart of another target detection method provided in the embodiments of this application;

[0041] Figure 4 This is a schematic diagram of a feature extraction layer provided in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the structure of a fusion unit and a detection unit provided in an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of the structure of a first convolutional unit provided in an embodiment of this application;

[0044] Figure 7 This is a schematic diagram of a target detection model provided in an embodiment of this application;

[0045] Figure 8 This application provides a schematic diagram of the structure of a target detection device;

[0046] Figure 9 A schematic diagram of another target detection device provided in this application embodiment;

[0047] Figure 10A structural block diagram of a terminal is provided in the application embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0049] In this article, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0050] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the original images and target images involved in this application were obtained with full authorization.

[0051] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. See also... Figure 1 The implementation environment includes a terminal 101 and a server 102. The terminal 101 can be directly or indirectly connected to the server 102 via wired or wireless communication.

[0052] Optionally, the terminal 101 can be various types of terminals such as mobile phones, desktop computers, laptops, tablets, smartwatches, and in-vehicle terminals. The terminal 101 has an image recognition application installed and running. Users can log in to the application through the terminal 101 and use the intelligent object recognition service provided by the application. For example, if the application is a camera, the user can use the camera to take a picture of the object to be recognized, and the application can identify the category and location of the object in the image. In some embodiments, the application can provide the aforementioned intelligent object recognition service by calling the object detection model in the terminal 101. The application can also be associated with a server 102, which provides backend services such as edge deployment and parameter updates of the object detection model.

[0053] Optionally, server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some embodiments, server 102, through a target monitoring model, undertakes the main computing work in some cases, while terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, while terminal 101 undertakes the main computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0054] Of course, the target detection method provided in this application embodiment can also be applied to other target detection scenarios, and this application embodiment does not limit the application scenarios of the target detection method. For example, in a smart transportation scenario, terminal 101 can be an in-vehicle terminal. Terminal 101 can collect images of the surrounding environment of the vehicle while the vehicle is in motion, and detect objects such as vehicles, pedestrians, and traffic lights appearing in the images through a target detection model. Then, terminal 101 can control the driving state of the vehicle based on the above-mentioned objects and their positions.

[0055] Figure 2 This is a flowchart of a target detection method provided in an embodiment of this application. The method is executed by a terminal; see [link to relevant documentation]. Figure 2 The method includes:

[0056] 201. The terminal performs feature extraction on the target image based on multiple feature extraction layers of the target detection model, obtaining first feature maps at multiple scales. The target image includes at least one target object. The target detection model is used to detect the category and location of at least one target object. The multiple feature extraction layers and the first feature maps at multiple scales... Figure 1 In a one-to-one correspondence, the feature extraction layer includes multiple cascaded first convolutional units, which are used to extract features from the input image through at least one of depthwise convolution, pointwise convolution, and batch normalization.

[0057] In this embodiment, the target image is an image to be detected. The target image includes at least one target object. In response to a user's target detection operation on the target image, the terminal acquires a target detection model and performs detection on the target image using the target detection model. The target detection model is used to detect the category and location of at least one target object in the target image.

[0058] In the process of detecting a target image using an object detection model, the terminal inputs the target image into the model. Multiple cascaded feature extraction layers within the model extract features from the target image, resulting in a feature map output by each layer. Each feature extraction layer includes multiple cascaded convolutional units. The feature extraction operation is implemented using at least one of the following operations within the convolutional units: depthwise convolution, pointwise convolution, and batch normalization.

[0059] Because the number of convolutional units in each feature extraction layer varies, the number of convolutional operations performed during feature extraction also differs within each layer. Therefore, the scale of the feature maps output by each feature extraction layer also differs. The deeper the feature extraction layer, the more convolutional operations its output feature map undergoes, and the smaller the scale of the feature map. By performing feature extraction through multiple feature extraction layers, multiple feature maps of different scales can be obtained. For ease of description, the feature map output by each feature extraction layer will be referred to as the first feature map below.

[0060] 202. The terminal, based on the feature fusion layer of the target detection model, fuses the first feature maps at multiple scales to obtain a first fused feature map. The first fused feature map includes at least one category feature and location feature of the target object.

[0061] In this embodiment, the target detection model further includes a feature fusion layer. The terminal fuses multiple first feature maps at different scales through the feature fusion layer in the target detection model to obtain a first fused feature map. The first fused feature map includes the features contained in the first feature maps at each scale; that is, the first fused feature map includes at least one category feature and a location feature of the target object.

[0062] 203. The terminal decodes the first fusion feature map based on the output layer of the target detection model to obtain the detection result of the target image. The detection result is used to represent the category and location of at least one target object.

[0063] In this embodiment, the target detection model further includes an output layer. The output layer is used to output the detection result of the target image. The terminal inputs the first fused feature into the output layer of the target detection model, and the output layer decodes the category features and location features included in the first fused feature map to obtain the category and location of at least one target object in the target image, that is, to obtain the detection result of the target image.

[0064] In some embodiments, the detection result obtained after decoding can be a multivariate array, such as [k, x, y]. Here, k is the category identifier of the target object, and x and y are the coordinates of the center point of the target object in the target image. The detection result can also be a vector or an annotated target image; this application embodiment does not limit the form of the detection result. For example, in the annotated target image, the location of the target object is marked by a rectangle, and the category of the target object is marked by the color or category identifier of the rectangle.

[0065] This application provides a target detection method that extracts features from an image to be detected using multiple feature extraction layers of a target detection model, resulting in feature maps at multiple scales. Since feature maps at different scales can reflect the category and location features of target objects of different sizes, fusing these feature maps to obtain a fused feature map can reflect the category and location features of target objects of different sizes in the image. Then, by decoding the fused feature map, the category and location of target objects of different sizes in the image can be determined. Using this method, most target objects in the target image can be successfully detected, improving the accuracy of target detection.

[0066] Figure 3 This is a flowchart of another target detection method provided in an embodiment of this application. This method is executed by a terminal; see [link to relevant documentation]. Figure 3 The method includes:

[0067] 301. The terminal performs edge enhancement on the original image to obtain the target image. Edge enhancement is used to enhance the edge information of at least one target object in the original image.

[0068] In this embodiment, the original image is the image to be detected. The original image can be an image captured by a user through a terminal, any frame from a video recorded by the terminal, or an image saved by the terminal; this embodiment does not impose any limitations on this. The original image includes at least one target object. The target object can be various types of objects in the original image, such as people, animals, plants, and buildings. Target detection is used to detect the category and location of at least one target object in the original image.

[0069] In object detection, the image quality of the original image affects the accuracy of object detection. When the image quality of the original image is poor, such as when the edges of the target object are occluded, the edges are blurry, or the boundary between the foreground and background of the target object is unclear, it is difficult to detect the target object in the original image, or it is difficult to accurately detect the category of the target object, resulting in a low accuracy rate for object detection.

[0070] To improve the accuracy of object detection, in response to object detection operations on the original image, the terminal performs edge enhancement on the original image to strengthen the edge contours of target objects. For example, the terminal can use the Sobel operator or the Laplacian operator to enhance the edges of the original image to obtain the target image. The target image can more clearly display the edge contours of different target objects, thus facilitating the terminal to detect the category and location of different target objects. Therefore, performing object detection on the target image by the terminal can improve the accuracy of object detection.

[0071] The target detection operation on the original image can be a long press operation on the original image by the user, an operation of the user uploading the original image at the detection entry, or an operation of the user recognizing the original image through the voice control terminal. The specific operation that triggers target detection is not limited in the embodiments of this application.

[0072] 302. The terminal performs feature extraction on the target image based on multiple feature extraction layers of the target detection model, obtaining first feature maps at multiple scales. The target image includes at least one target object. The target detection model is used to detect the category and location of at least one target object. The multiple feature extraction layers and the first feature maps at multiple scales... Figure 1 In a one-to-one correspondence, the feature extraction layer includes multiple cascaded first convolutional units, which are used to extract features from the input image through at least one of depthwise convolution, pointwise convolution, and batch normalization.

[0073] In this embodiment, after obtaining the target image, the terminal performs target detection on the target image using a target detection model. The target detection model includes multiple feature extraction layers, feature fusion layers, and an output layer. The feature extraction layers extract feature maps from the input image. The feature fusion layers fuse the extracted feature maps to obtain a fused feature map. The output layer decodes the fused feature map to obtain the category and location of at least one target object in the target image.

[0074] During object detection, the terminal inputs the target image into the object detection model. Features are extracted sequentially through multiple cascaded feature extraction layers within the model, resulting in feature maps output by each layer. Specifically, within each feature extraction layer, the terminal extracts features from the input feature map using multiple cascaded convolutional units. These feature extraction operations are implemented using at least one of the following operations within the convolutional units: depthwise convolution, pointwise convolution, and batch normalization. For ease of description, the convolutional unit incorporating at least one of these operations is referred to as the first convolutional unit.

[0075] It should be noted that the number of first convolutional units in different feature extraction layers can be the same or different. This embodiment uses the example of different feature extraction layers having different numbers of first convolutional units for illustration. The more first convolutional units a feature extraction layer includes, the more convolutional operations it performs during feature extraction, and the smaller the scale of the feature map output by that feature extraction layer. Therefore, in this embodiment, by performing feature extraction through the above multiple feature extraction layers, multiple feature maps of different scales can be obtained. For ease of description, the feature map output by the feature extraction layer will be referred to as the first feature map below.

[0076] Figure 4 This is a schematic diagram of a feature extraction layer. To more clearly illustrate the feature extraction process described above, the following section combines... Figure 4 The feature extraction layer shown below explains the process of the terminal extracting features from the target image through multiple feature extraction layers through the following steps (1)-(3).

[0077] First, let's briefly introduce the multiple feature extraction layers in the object detection model. For example... Figure 4 As shown, the object detection model includes one initial feature extraction layer and three feature extraction layers. The initial feature extraction layer includes one second convolutional unit. Each feature extraction layer includes multiple cascaded first convolutional units. The convolution operations in the first and second convolutional units are different. The first convolutional units are used to extract features from the input image using at least one of depthwise convolution, pointwise convolution, and batch normalization. The second convolutional units are used to extract features from the input image using regular convolution.

[0078] The following describes the process by which the terminal extracts features from the target image through the aforementioned multiple feature extraction layers.

[0079] (1) The terminal inputs the target image into the initial feature extraction layer of the target detection model. The target image is then extracted through the second convolution unit in the initial feature extraction layer to obtain the initial feature map of the target image.

[0080] The target image input to the object detection model has four dimensions: N*C*H*W. Here, N represents the number of target images, C represents the number of channels in the target image, H represents the height of the target image, and W represents the width of the target image. For example, if the target image size is 320*320 and the target image includes three channels: R (Red), G (Green), and B (Blue), then the four dimensions of the target image are 1*3*320*320.

[0081] With 3 input channels and 48 output channels in the second convolutional unit, the terminal inputs the target image into the second convolutional unit. The target image is then convolved by the 48 convolutional kernels within the second convolutional unit to obtain an initial feature map. The initial feature map has four dimensions: 1*48*160*160. Each convolutional kernel has a size of 3*3 and a stride of 2. Additionally, before convolution, the terminal can pad the target image with a row of pixels with a value of 0 around its edges using the second convolutional unit. This padding prevents the loss of edge information during convolution, ensuring that edge features are extracted.

[0082] In some embodiments, the initial feature extraction layer further includes an edge enhancement unit. The edge enhancement unit is used to enhance the edges of the input image using the Sobel operator or the Laplacian operator. Therefore, when the initial feature extraction layer includes an edge enhancement unit, the terminal can skip step 301 and directly input the original image into the initial feature extraction layer of the object detection model. The edge enhancement unit in the initial feature extraction layer enhances the edges of the original image to obtain the target image. Then, the terminal extracts features from the target image using the second convolutional unit in the initial feature extraction layer to obtain the initial feature map of the target image.

[0083] (2) The terminal inputs the initial feature map into the first feature extraction layer of the target detection model. In the first feature extraction layer, the terminal performs feature extraction on the initial feature map through at least one of the operations of pointwise convolution, depthwise convolution and batch normalization in multiple first convolutional units to obtain the first feature map of the first feature extraction layer.

[0084] The convolution operations in the multiple first convolutional units can be the same or different, and this application embodiment does not impose any restrictions. For example, the convolution stride in the multiple first convolutional units can be different. The convolution stride of the first first convolutional unit is 2, and the convolution stride of the second first convolutional unit can be 1.

[0085] For example, the first feature extraction layer includes two first convolutional units. Figure 4 A schematic diagram of the structure of a first convolutional unit is also shown. (For example...) Figure 4As shown, the first convolutional unit includes a depthwise convolution part and a pointwise convolution part. The depthwise convolution part consists of depthwise convolution, batch normalization, and an activation function. The pointwise convolution part consists of pointwise convolution, batch normalization, and an activation function. The first convolutional unit has 48 input channels and 48 output channels. After the terminal inputs the initial feature map into the first convolutional unit, the initial feature map undergoes a 3x3 depthwise convolution with a stride of 2 through the 48 convolution kernels of the depthwise convolution part. Then, after batch normalization and activation function processing, the initial feature map undergoes a 1x1 pointwise convolution with a stride of 1 through the 48 convolution kernels of the pointwise convolution part. After another batch normalization and activation function processing, the feature map output by the first convolutional unit is obtained. Then, the terminal inputs this feature map into the second convolutional unit, where similar convolution processing is performed to obtain the first feature map. The first feature map has four dimensions: 1*48*80*80. During the depthwise convolution within the second convolutional unit, the stride of the depthwise convolution is 1.

[0086] It should be noted that, Figure 4 This is merely a schematic diagram of one type of first convolutional unit. In some embodiments, the structure of the first convolutional unit can be other structures, and this application embodiment does not limit this. For example, the structure of the first convolutional unit can be a parallel connection of a depthwise convolution part and a pointwise convolution part, and this parallel structure is then connected in series with feature fusion, batch normalization, and activation functions. Using the above structure, the terminal can perform depthwise convolution and pointwise convolution on the initial feature map in parallel through the first convolutional unit, then fuse the feature maps obtained after convolution, and then process them through batch normalization and activation functions to obtain the output first feature map. In addition, the activation function in the first convolutional unit can be a non-linear activation function such as ReLU (Rectified Linear Unit) or Sigmoid (an activation function), and this application embodiment does not limit this.

[0087] Then, the terminal inputs the first feature map output from the first feature extraction layer into the second feature extraction layer. Since the processing within the second to the nth feature extraction layers is similar, the following description focuses on the feature extraction process performed by the terminal through the i-th feature extraction layer. Here, n is the total number of feature extraction layers, and i is an integer greater than 1.

[0088] (3) The terminal inputs the first feature map output from the (i-1)th layer to the i-th feature extraction layer. In the i-th feature extraction layer, the terminal extracts features from the input first feature map through multiple first convolutional units in the feature extraction layer to obtain the first feature map of the i-th feature extraction layer. Then, the terminal inputs the first feature map to the next feature extraction layer and continues to extract features until the first feature map output by each feature extraction layer is obtained. The scale of the first feature map is negatively correlated with the number of the corresponding feature extraction layer, that is, the deeper the feature extraction layer, the smaller the scale of the first feature map output by that feature extraction layer.

[0089] For example, such as Figure 4 As shown, the object detection model includes three feature extraction layers. The second feature extraction layer contains eight first convolutional units, and the third feature extraction layer contains ten first convolutional units.

[0090] For the second feature extraction layer, except for the first convolutional unit which has 48 input channels and 128 output channels, all other first convolutional units have 128 input channels and 128 output channels.

[0091] For the third feature extraction layer, except for the first convolutional unit which has 128 input channels and 256 output channels, all other first convolutional units have 256 input and output channels.

[0092] Within any first convolutional unit of any feature extraction layer, the number of kernels in a depthwise convolution is the same as the number of output channels of that first convolutional unit, and the number of kernels in a pointwise convolution is also the same as the number of output channels of that first convolutional unit. Therefore, after feature extraction through the above two feature extraction layers, we can obtain the first feature map output by the second feature extraction layer and the first feature map output by the third feature extraction layer. The four dimensions of the first feature map output by the second feature extraction layer are 1*128*40*40. The four dimensions of the first feature map output by the third feature extraction layer are 1*256*20*20.

[0093] In some embodiments, the feature extraction layer may also include an edge enhancement unit. The edge enhancement unit is used to enhance the edges of the input image using the Sobel operator or the Laplacian operator. Therefore, by using edge enhancement units in multiple cascaded feature extraction layers, a Sobel operator pyramid or a Laplacian operator pyramid can be constructed, thereby achieving edge enhancement of the feature map before feature extraction or the feature map after feature extraction within each feature extraction layer, thus improving the accuracy of feature extraction and object detection. Therefore, when the feature extraction layer includes an edge enhancement unit, the terminal may also skip step 301 above and directly input the original image into the object detection model, and achieve edge enhancement of the feature map through multiple edge enhancement units in the object detection model during the feature extraction process.

[0094] 303. The terminal, based on the feature fusion layer of the target detection model, fuses the first feature maps at multiple scales to obtain a first fused feature map, which includes at least one category feature and location feature of the target object.

[0095] In this embodiment, multiple feature extraction layers of the target detection model are connected to a feature fusion layer of the target detection model. In response to obtaining a first feature map output by any feature extraction layer, the terminal inputs the first feature map into the feature fusion layer. Then, the terminal fuses the first feature maps output by the multiple feature extraction layers through the feature fusion layer to obtain a fused feature map.

[0096] In some embodiments, the feature fusion layer of the target detection model includes multiple fusion units and multiple detection units. Each fusion unit corresponds one-to-one with a feature extraction layer, and each fusion unit corresponds one-to-one with a detection unit. Both the fusion units and the detection units are used to fuse feature maps. The process of feature fusion performed by the terminal through the fusion units and detection units is described below using steps (1)-(4).

[0097] First, let me briefly introduce the multiple fusion units and multiple detection units in the object detection model. Figure 5 This is a schematic diagram of a fusion unit and a detection unit. Figure 5 As shown, the target detection model includes three fusion units and three detection units. The three fusion units are connected to three feature extraction layers, and the three detection units are also connected to the three feature extraction layers. The process of the terminal fusing multiple first feature maps using the aforementioned fusion and detection units is described below.

[0098] (1) First, for any scale of first feature map, the terminal inputs the first feature map into the corresponding fusion unit. The fusion unit corresponding to the first feature map refers to the fusion unit connected to the feature extraction layer that outputs the first feature map. Then, the terminal uses this fusion unit to fuse the first feature map at this scale with first feature maps at other scales, according to the scale of the first feature map, to obtain a first intermediate feature map, so that the scale of the first intermediate feature map is the same as the scale of the first feature map. The number of channels in the first intermediate feature map is the number of pointwise convolutional kernels in the fusion unit. Since the number of channels in the first feature maps output by the multiple feature extraction layers is different, to facilitate feature fusion, the terminal uses pointwise convolutional kernels in the fusion unit to perform pointwise convolution on the input multiple first feature maps, so that the number of channels in the multiple first feature maps is the same, and the number of channels in the first feature map is the same as the number of pointwise convolutional kernels. Therefore, the terminal, through the fusion unit, can fuse multiple first feature maps with different scales and different numbers of channels into a first intermediate feature map with a target scale and a target number of channels.

[0099] The following explains the process by which the terminal unifies the scale and number of channels of the feature map through the fusion unit.

[0100] In some embodiments, the terminal can unify the number of channels of multiple first feature maps through pointwise convolution and unify the scale of multiple first feature maps through upsampling or downsampling. For a first feature map of any scale, the terminal performs pointwise convolution on the first feature map of that scale and other first feature maps of different scales through a fusion unit corresponding to the first feature map, so that the number of channels of the first feature map of that scale and other first feature maps of different scales are the same, and the number of channels is the same as the number of pointwise convolution kernels in the pointwise convolution. Then, the terminal uses a fusion unit to upsample or downsample multiple first feature maps of other scales according to the scale of the first feature map, to obtain multiple first feature maps of the same scale. Then, the terminal uses a fusion unit to fuse the multiple first feature maps of the same scale and the same number of channels to obtain a first intermediate feature map.

[0101] For example, the terminal, through fusion unit 1 connected to the first feature extraction layer, first fuses the first feature map output from the second feature extraction layer (dimension 1*128*40*40) and the first feature map output from the third feature extraction layer (dimension 1*256*20*20), and then fuses the fused feature map with the first feature map output from the first feature extraction layer (dimension 1*48*80*80). The specific fusion process is as follows: First, the terminal performs pointwise convolution on the three first feature maps using 96 1*1 pointwise convolution kernels in fusion unit 1, obtaining three first feature maps with 96 channels each. Then, the terminal upsamples the first feature map with dimension 1*96*20*20 through fusion unit 1, obtaining a first feature map with dimension 1*96*40*40. Then, the terminal uses fusion unit 1 to fuse two first feature maps, each with dimensions of 1*96*40*40 (one of which is obtained by performing pointwise convolution on a 1*128*40*40 first feature map, and the other is obtained by first performing pointwise convolution on a 1*256*20*20 feature map and then upsampling it), to obtain an intermediate fused feature map with dimensions of 1*96*40*40. This intermediate fused feature map is the feature map obtained by fusing the first feature map output from the second feature extraction layer and the first feature map output from the third feature extraction layer. Then, the terminal upsamples the intermediate fused feature map of dimension 1*96*40*40 through fusion unit 1 to obtain an intermediate fused feature map of dimension 1*96*80*80. This intermediate fused feature map is then fused with a first feature map of dimension 1*96*80*80 (obtained by pointwise convolution of a first feature map of dimension 1*48*80*80) to obtain the first intermediate feature map output by fusion unit 1. The first intermediate feature map has a dimension of 1*96*80*80.

[0102] The fusion process for the other two fusion units is similar. For example, in fusion unit 2, the first feature maps output from the first and third feature extraction layers are first fused to obtain an intermediate fused feature map. Then, the intermediate fused feature map is fused with the first feature map output from the second feature extraction layer to obtain the first intermediate feature map output by fusion unit 2. The dimension of the first intermediate feature map output by fusion unit 2 is 1*96*40*40. Similarly, the dimension of the first intermediate feature map output by fusion unit 3 is 1*96*20*20.

[0103] (2) The terminal inputs the first intermediate feature map output by the fusion unit to the detection unit connected to the fusion unit. Then, the terminal performs channel separation on the first intermediate feature map according to the detection unit, and divides the first intermediate feature map of multiple channels into at least one channel category feature map and at least one channel position feature map.

[0104] The number of channels in the category feature map can be determined by the number of preset detection categories in the object detection model. These preset detection categories are the categories that the object detection model can detect. For example, with 8 preset detection categories, the object detection model can detect up to 8 categories of target objects in an image. During object detection, for any detected target object, the model determines the probability that the target object belongs to each category. Then, the model identifies the category with the highest probability as the category of the target object. Therefore, when the terminal performs channel separation on the first intermediate feature map through the detection unit, it can separate 8 channels of category feature maps. Each channel corresponds to one category. The category feature map of one channel is used to represent the category features of the target object of the corresponding category.

[0105] In addition, after the terminal separates the category feature map through the detection unit, it can also process the category feature map through the Sigmoid function in the detection unit, so that the pixel value of the pixel in the category feature map is in the range of 0-1, thereby normalizing the category feature map.

[0106] The number of channels in the location feature map can be determined by adjusting the hyperparameters of the target detection model. For example, the number of channels in the location feature map is (1 + reg_max) * 4. Here, reg_max is an adjustable hyperparameter; in this embodiment, reg_max is 7. 4 indicates that the position of the target object in the target image is represented by four values. These four values ​​are the distances between the center point of the target object and the four sides of the rectangular detection box. Therefore, when the detection unit performs channel separation on the first intermediate feature map, it can typically separate location feature maps with 32 channels. Every four location feature maps are used to represent the location features of target objects of the same category.

[0107] (3) Based on the detection unit, the terminal splices at least one channel's category feature map and at least one channel's position feature map in the channel dimension to obtain a second intermediate feature map. The second intermediate feature map includes the category features and position features of the target object.

[0108] For example, after detection unit 1 performs channel separation on the first intermediate feature map to obtain 8-channel category feature maps and 32-channel position feature maps, the terminal uses detection unit 1 to concatenate the category and position feature maps along the channel dimension to obtain a 40-channel second intermediate feature map. Therefore, the dimension of the second intermediate feature map output by detection unit 1 is 1*40*80*80. Similarly, the dimension of the second intermediate feature map output by detection unit 2 is 1*40*40*40, and the dimension of the second intermediate feature map output by detection unit 3 is 1*40*20*20.

[0109] It should be noted that the detection unit may also include multiple cascaded third convolutional units and at least one fourth convolutional unit. The third convolutional unit consists of depthwise convolution, batch normalization, and an activation function. The activation function can be LeakyReLU (a non-linear activation function). The third convolutional unit is used to further extract positional and class features from the first intermediate feature map, thereby improving the accuracy of object detection. The fourth convolutional unit consists of a single convolutional layer. The fourth convolutional unit is used to convolve the feature map, realizing the aforementioned channel separation and channel concatenation process. The fourth convolutional unit has 96 input channels and num_classes + 4*(reg_max + 1) output channels. Here, num_classes is the preset number of detection classes, i.e., the number of channels in the class feature map. 4*(reg_max + 1) is the number of channels in the positional feature map.

[0110] (4) After the terminal obtains the second intermediate feature maps output by multiple detection units, it concatenates the multiple second intermediate feature maps along the scale dimension to obtain the first fused feature map. The scale dimension is the product of the height and width dimensions. In this embodiment, the scale dimension can also be referred to as the length dimension.

[0111] For example, if the dimensions of the second intermediate feature maps output by the three detection units are 1*40*80*80, 1*40*40*40 and 1*40*20*20 respectively, the dimension of the first fused feature map obtained by the terminal by splicing the three second intermediate feature maps is: 1*40*(6400+1600+400)=1*40*8400.

[0112] In some embodiments, before stitching together the second feature maps, the terminal first transforms the scale of the second feature maps. For any second intermediate feature map, if the second intermediate feature map has three dimensions (channel, width, and height), the terminal converts the width and height dimensions into a length dimension to obtain a third intermediate feature map. The third intermediate feature map has two dimensions (channel and length). The length is the product of the width and height. The terminal transforms multiple second intermediate feature maps using the above method to obtain multiple third intermediate feature maps. Then, the terminal stitches together the multiple third intermediate feature maps along the length dimension to obtain a first fused feature map.

[0113] For example, using the method described above, the terminal converts the second intermediate feature maps with dimensions of 1*40*80*80, 1*40*40*40, and 1*40*20*20 into third intermediate feature maps with dimensions of 1*40*6400, 1*40*1600, and 1*40*400, respectively. Then, the terminal concatenates these three third intermediate feature maps along the length dimension to obtain a first fused feature map with a dimension of 1*40*8400.

[0114] 304. The terminal decodes the first fusion feature map based on the output layer of the target detection model to obtain the detection result of the target image. The detection result is used to represent the category and location of at least one target object.

[0115] In this embodiment, the target detection model further includes an output layer. The output layer is used to output the detection result of the target. The terminal inputs the first fused feature into the output layer of the target detection model, and the output layer decodes the category feature and position feature in the first fused feature map to obtain the category and position of at least one target object in the target image, that is, to obtain the detection result of the target image.

[0116] In some embodiments, before the terminal performs decoding through the output layer, the terminal can also transform the dimension of the first fused feature map through the output layer. For example, the terminal transforms the dimension of the first fused feature map, which is 1*40*8400, to 1*6400*40 to facilitate subsequent decoding processing.

[0117] In some embodiments, the detection result of the target image can be in various forms such as vector, multi-element array, or image, and this application embodiment does not limit this. For example, the target detection result can be a ternary array [k,x,y]. Where k is the category identifier of the target object, and x and y are the coordinates of the center point of the target object in the target image. Alternatively, the detection result can also be a quinary array [k,q,r,z,h]. Where k is the category identifier of the target object, and q, r, z, and h are the distances between the center point of the target object and the four sides of the rectangular detection box, respectively. The rectangular detection box is used to select the region where the target object is located in the target image. Alternatively, the detection result can also be an annotated target image. The location of the target object is marked in the annotated target image using a rectangular box, and the category of the target object is marked by the color or category identifier of the rectangular box.

[0118] The above embodiments describe the process of a terminal performing target detection on a target image using a target detection model. The following steps (1)-(5) illustrate the training process of the target detection model, using the terminal as the execution subject.

[0119] (1) The terminal acquires a sample image. The sample image includes at least one target object. The terminal inputs the sample image into the target detection model, and performs feature extraction on the sample image through multiple feature extraction layers in the target detection model to obtain a second feature map at multiple scales.

[0120] In training the object detection model, to improve its ability to extract image features through different convolutional operations, the terminal can design the first convolutional unit of the feature extraction layer with a different structure. For example, Figure 6 This is a schematic diagram of the structure of a first convolutional unit. For example... Figure 6 As shown, within the first convolutional unit, the input feature map is first processed by three parallel branches: pointwise convolution with a 1*1 convolution size, depthwise convolution with a 3*3 convolution size, and batch normalization. Then, the feature maps output from the three branches are sequentially processed by feature fusion and activation functions to obtain intermediate feature maps. Next, two parallel branches are used to process the intermediate feature maps with pointwise convolution with a 1*1 convolution size and batch normalization. Finally, the feature maps output from the two branches are sequentially processed by feature fusion and activation functions to obtain the output feature map of the first convolutional unit. By designing the first convolutional unit with the above structure, the feature extraction layer in the object detection model can accurately extract feature maps with strong expressive power through different convolution operations, improving the accuracy of object detection. Furthermore, when using the object detection model for object detection, the structure of the first convolutional unit can be updated to the above... Figure 4 The structure shown reduces skip connections in the first convolutional unit, decreases the number of parameters in the object detection model, and improves the object detection efficiency. Furthermore, experimental results show that when performing object detection using the Microsoft Cocoa dataset, compared to other object detection models, the object detection model in this embodiment achieves a mAP (mearing accuracy, an evaluation metric for object detection) improvement of more than 2 percentage points at the same detection speed.

[0121] (2) The terminal fuses the second feature maps at multiple scales through the feature fusion layer of the target detection model to obtain a second fused feature map. The second fused feature map includes the category features and location features of at least one target object in the sample image. The specific process of feature fusion is the same as step 303 above, and will not be repeated here.

[0122] (3) The terminal decodes the second fusion feature map through the output layer of the target detection model to obtain the detection result of the sample image. The detection result is used to represent the predicted category and predicted location of at least one target object in the sample image.

[0123] (4) The terminal determines the training loss of the target detection model based on the detection results and labels of the sample images. The labels of the sample images are used to represent the true category and true location of at least one target object in the sample image.

[0124] In some embodiments, the training loss includes a class loss and a location loss. The terminal determines the class loss based on the predicted and true classes of at least one target object. The class loss represents the magnitude of the difference between the predicted and true classes of the at least one target object. The terminal determines the location loss based on the predicted and true locations of at least one target object. The location loss represents the magnitude of the difference between the predicted and true locations of the at least one target object. Both the class loss and the location loss are positively correlated with the magnitude of their respective differences.

[0125] The training loss can be Loss_qfl + w1 * Loss_dfl + w2 * Loss_bbox. Here, Loss_qfl is the classification loss, and Loss_dfl and Loss_bbox are two different location losses. w1 and w2 are the weights of the Loss_dfl and Loss_bbox losses, respectively. w1 and w2 can be preset values, such as w1 = 0.25 and w2 = 2; or w1 = 0.5 and w2 = 1.5. This embodiment does not impose any limitations on these values.

[0126] (5) The terminal trains the object detection model based on the training loss. Specifically, the terminal updates the model parameters of the object detection model based on the training loss to reduce the training loss, thus obtaining an updated object detection model. If the updated object detection model meets the training termination condition, such as the number of training iterations being the target number or the training loss of the object detection model being within the target range, then the updated object detection model is considered the completed object detection model. If the updated object detection model does not meet the training termination condition, the parameters of the object detection model are updated again according to steps (1) to (5) until the updated object detection model meets the training termination condition, thus obtaining a completed object detection model.

[0127] In some embodiments, the terminal can also assist in training the target detection model through an auxiliary training layer. For example... Figure 7 This is a schematic diagram of an object detection model. (For example...) Figure 7As shown, the object detection model also includes an auxiliary training layer. The auxiliary training layer functions similarly to the feature fusion layer described above. The terminal fuses first feature maps at multiple scales using the auxiliary training layer of the object detection model to obtain a third fused feature map. Then, the terminal decodes the third fused feature map using the output layer of the object detection model to obtain the auxiliary detection result of the object detection model. Based on the auxiliary detection result and the label of the sample image, the terminal determines the auxiliary training loss of the object detection model. Then, based on this training loss and the auxiliary training loss, the terminal updates the model parameters of the object detection model until the updated object detection model meets the training termination condition, thus obtaining the trained object detection model.

[0128] The auxiliary training loss can be ALoss_qfl + w3 * ALoss_dfl + w4 * ALoss_bbox. Here, ALoss_qfl is the class loss in the auxiliary training loss. ALoss_dfl and ALoss_bbox are the two position losses in the auxiliary training loss. w3 and w4 are the weights of the ALoss_dfl and ALoss_bbox losses, respectively. w3 and w4 can also be preset values, such as w3 = 0.25 and w4 = 2; or w3 = 0.5 and w4 = 1.5. This embodiment does not impose any limitations on these values.

[0129] It should be noted that this auxiliary training layer is only used to assist in the training of the object detection model. Therefore, after the object detection model has been trained, it may not need to include this auxiliary training layer when performing object detection.

[0130] In some embodiments, the terminal can also train the object detection model through a three-stage process. The training sample sets differ for each training stage. First, the terminal performs a first-stage training on the object detection model using a first sample set. This first sample set includes multiple original sample images and a first number of augmented sample images. The augmented sample images are obtained by augmenting the multiple original sample images using data augmentation methods such as MixUp, Mosaic, and Random Affine. Then, the terminal performs a second-stage training on the object detection model using a second sample set. This second sample set includes multiple original sample images and a second number of augmented sample images. The second number is less than the first number; that is, during the second-stage training, the terminal reduces the proportion of augmented sample images in the training sample set. Finally, the terminal performs a third-stage training on the object detection model using multiple original sample images; that is, during the third-stage training, the terminal trains the object detection model using only the original sample images. By gradually reducing the proportion of augmented sample images in the training sample set during the three-stage training process described above, the accuracy of the object detection model in detecting complex objects can be improved, while the accuracy in detecting simple objects can also be guaranteed. This enables the object detection model to accurately detect objects of varying complexity in the image.

[0131] Furthermore, during the training of the object detection model, an appropriate training sample set can be selected based on the application scenario of object detection. For example, if the application scenario is to identify plant species, the training sample set can be a collection of sample images containing different types of plants. Alternatively, if the application scenario is intelligent transportation, the training sample set can be a collection of sample images containing objects such as pedestrians, traffic lights, vehicles, lane lines, and traffic signs.

[0132] It should be noted that this application embodiment describes the process of training an object detection model on a terminal, with the terminal as the execution entity. In some embodiments, a server can also be the execution entity, training the object detection model through the above process. Then, the server deploys the trained object detection model on the terminal, and the terminal directly calls the trained object detection model to perform object detection. Therefore, the terminal does not need to train the object detection model, thereby reducing the performance requirements of the object detection task on the terminal.

[0133] This application provides a target detection method that extracts features from an image to be detected using multiple feature extraction layers of a target detection model, resulting in feature maps at multiple scales. Since feature maps at different scales can reflect the category and location features of target objects of different sizes, fusing these feature maps to obtain a fused feature map can reflect the category and location features of target objects of different sizes in the image. Then, by decoding the fused feature map, the category and location of target objects of different sizes in the image can be determined. Using this method, most target objects in the target image can be successfully detected, improving the accuracy of target detection.

[0134] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0135] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0136] Figure 8 This is a schematic diagram of the structure of a target detection device provided in an embodiment of this application. See also... Figure 8 The device includes: a first extraction module 801, a first fusion module 802, and a first decoding module 803.

[0137] The first extraction module 801 is used to extract features from a target image based on multiple feature extraction layers of a target detection model, obtaining first feature maps at multiple scales. The target image includes at least one target object, and the target detection model is used to detect the category and location of at least one target object. The multiple feature extraction layers and the first feature maps at multiple scales... Figure 1 In a one-to-one correspondence, the feature extraction layer includes multiple concatenated first convolutional units, which are used to extract features from the input image through at least one of depthwise convolution, pointwise convolution, and batch normalization.

[0138] The first fusion module 802 is used to fuse first feature maps of multiple scales based on the feature fusion layer of the target detection model to obtain a first fused feature map. The first fused feature map includes at least one category feature and position feature of the target object.

[0139] The first decoding module 803 is used to decode the first fused feature map based on the output layer of the target detection model to obtain the detection result of the target image. The detection result is used to represent the category and location of at least one target object.

[0140] In some embodiments, the first extraction module 801 is used to extract features from the target image based on the initial feature extraction layer of the target detection model to obtain an initial feature map of the target image. The initial feature extraction layer includes a second convolutional unit, and the convolution operation in the second convolutional unit is different from the convolution operation in the first convolutional unit. In the first feature extraction layer of the target detection model, features are extracted from the initial feature map through multiple first convolutional units in the feature extraction layer to obtain the first feature map of the first feature extraction layer. In the i-th feature extraction layer of the target detection model, features are extracted from the first feature map of the (i-1)-th feature extraction layer through multiple first convolutional units in the feature extraction layer to obtain the first feature map of the i-th feature extraction layer, until the first feature map of each feature extraction layer is obtained. The scale of the first feature map is negatively correlated with the number of the corresponding feature extraction layer, where i is an integer greater than 1.

[0141] In some embodiments, Figure 9 This is a schematic diagram of another target detection device provided in an embodiment of this application. See also... Figure 9 The device also includes:

[0142] The edge enhancement module 804 is used to enhance the edges of the original image to obtain the target image. The edge enhancement is used to enhance the edge information of at least one target object in the original image.

[0143] In some embodiments, the feature fusion layer of the target detection model includes multiple fusion units and multiple detection units, with each fusion unit corresponding to a multiple feature extraction layer and each detection unit corresponding to a multiple feature extraction layer.

[0144] The first fusion module 802 includes:

[0145] The feature fusion unit 8021 is used to fuse the first feature map at any scale with the first feature map at other scales based on the fusion unit corresponding to the first feature map, according to the scale of the first feature map, to obtain a first intermediate feature map. The number of channels of the first intermediate feature map is the number of pointwise convolution kernels in the fusion unit. The pointwise convolution kernels are used to perform pointwise convolution on the input feature map so that the number of channels of the input feature map is the same as the number of pointwise convolution kernels.

[0146] The channel separation unit 8022 is used to perform channel separation on the first intermediate feature map based on the detection unit corresponding to the fusion unit, so as to obtain a category feature map of at least one channel and a position feature map of at least one channel.

[0147] The splicing unit 8023 is used to splice at least one channel's category feature map and at least one channel's position feature map in the channel dimension based on the detection unit corresponding to the fusion unit, to obtain a second intermediate feature map;

[0148] The splicing unit 8023 is also used to splice multiple second intermediate feature maps in the scale dimension to obtain the first fused feature map.

[0149] In some embodiments, the feature fusion unit 8021 is configured to, for any scale of first feature map, perform pointwise convolution on the first feature map of scale and other scales respectively, based on the fusion unit corresponding to the first feature map, so that the number of channels of the first feature map of scale and other scales are the same, and the number of channels is the same as the number of pointwise convolution kernels in the pointwise convolution; based on the fusion unit, upsample or downsample multiple other scales of first feature maps according to the scale of the first feature map to obtain multiple scales of first feature maps; and based on the fusion unit, fuse the first feature map of scale and the multiple scales of first feature maps to obtain a first intermediate feature map.

[0150] In some embodiments, the splicing unit 8023 is used to convert any second intermediate feature map into a third intermediate feature map when the second intermediate feature map includes three dimensions: channel, width, and height. The third intermediate feature map includes two dimensions: channel and length, and the length is the product of the width and height. Multiple third intermediate feature maps are spliced ​​in the length dimension to obtain a first fused feature map.

[0151] In some embodiments, the apparatus further includes:

[0152] The second extraction module 805 is used to extract features from the sample image based on multiple feature extraction layers in the target detection model to obtain a second feature map at multiple scales. The sample image includes at least one target object.

[0153] The second fusion module 806 is used to fuse the second feature maps at multiple scales based on the feature fusion layer of the target detection model to obtain a second fused feature map. The second fused feature map includes the category features and location features of at least one target object in the sample image.

[0154] The second decoding module 807 is used to decode the second fusion feature map based on the output layer of the target detection model to obtain the detection result of the sample image. The detection result is used to represent the predicted category and predicted location of at least one target object in the sample image.

[0155] The loss determination module 808 is used to determine the training loss of the target detection model based on the detection results and labels of the sample images. The labels of the sample images are used to represent the true category and true location of at least one target object in the sample image.

[0156] Model training module 809 is used to train the object detection model based on training loss.

[0157] In some embodiments, the training loss includes a class loss and a location loss, wherein the class loss is used to represent the difference between the predicted class and the true class of at least one target object, and the location loss is used to represent the difference between the predicted location and the true location of at least one target object.

[0158] In some embodiments, the model training module 809 is used to fuse first feature maps at multiple scales based on the auxiliary training layer of the object detection model to obtain a third fused feature map; decode the third fused feature map based on the output layer of the object detection model to obtain the auxiliary detection result of the object detection model; determine the auxiliary training loss of the object detection model based on the auxiliary detection result of the sample image and the label of the sample image; and train the object detection model based on the training loss and the auxiliary training loss.

[0159] In some embodiments, the model training module 809 is further configured to perform a first stage of training on the object detection model based on a first sample set, the first sample set including multiple original sample images and a first number of augmented sample images, the augmented sample images being obtained by data augmentation of multiple original sample images; perform a second stage of training on the object detection model based on a second sample set, the second sample set including multiple original sample images and a second number of augmented sample images, the second number being less than the first number; and perform a third stage of training on the object detection model based on multiple original sample images.

[0160] This application provides a target detection apparatus that extracts features from an image to be detected using multiple feature extraction layers of a target detection model, resulting in feature maps at multiple scales. Since feature maps at different scales can reflect the category and location features of target objects of different sizes, the fused feature map obtained by fusing feature maps at multiple scales can reflect the category and location features of target objects of different sizes in the image. Then, by decoding the fused feature map, the category and location of target objects of different sizes in the image can be determined. Using this method, most target objects in the target image can be successfully detected, improving the accuracy of target detection.

[0161] It should be noted that the target detection device provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal can be divided into different functional modules to complete all or part of the functions described above. In addition, the target detection device and the target detection method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0162] This application provides a terminal, which includes a processor and a memory; the memory stores at least one piece of program code, which is executed by the processor to implement the target detection method provided in the above-described method embodiments.

[0163] Figure 10 This is a structural block diagram of a terminal provided in an embodiment of this application. In some embodiments, the terminal 1000 is a smartphone, tablet computer, wearable device, or other terminal capable of accessing a wireless local area network as a wireless station. The terminal 1000 in this application includes at least one or more of the following components: a processor 1010, a memory 1020, and at least two wireless links 1030.

[0164] In some embodiments, the processor 1010 includes one or more processing cores. The processor 1010 connects to various parts within the terminal 1000 using various interfaces and lines, and performs various functions and processes data of the terminal 1000 by running or executing program code stored in the memory 1020 and calling data stored in the memory 1020. In some embodiments, the processor 1010 is implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 1010 can integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural-network Processing Unit (NPU), and modem. Specifically, the CPU primarily handles the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the screen; the NPU is used to implement Artificial Intelligence (AI) functions; and the modem is used for wireless communication. It is understandable that the aforementioned modem could also be implemented as a separate chip without being integrated into the processor 1010.

[0165] In some embodiments, the processor 1010 is used to control the operating status of at least two wireless links 1030. Accordingly, the processor 1010 is a processor integrating a Wireless Fidelity (Wi-Fi) chip. This Wi-Fi chip is a chip with dual Wi-Fi processing capabilities. For example, the Wi-Fi chip is a dual-band dual-concurrent (DBDC) chip, or a dual-band simultaneous (DBS) chip, etc.

[0166] In some embodiments, the memory 1020 includes random access memory (RAM), and in some embodiments, the memory 1020 includes read-only memory (ROM). In some embodiments, the memory 1020 includes non-transitory computer-readable storage medium. The memory 1020 can be used to store program code. The memory 1020 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described below, etc.; the data storage area may store data (such as audio data, phonebook, etc.) created according to the use of the terminal 1000.

[0167] In some embodiments, the memory 1020 stores reception schemes for different wireless links 1030 to receive beacon frames, as well as identifiers of access nodes connected to different wireless links 1030, identifiers of wireless links 1030, etc.

[0168] The at least two wireless links 1030 are used to connect different access points (APs). They receive downlink data from the APs. These different access points can be access points within the same router or access points within different routers.

[0169] In some embodiments, the terminal 1000 further includes a display screen. The display screen is a display component used to display a user interface. In some embodiments, the display screen is a touch-enabled display screen, allowing users to perform touch operations on the display screen using fingers, styluses, or any suitable object. In some embodiments, the display screen is typically located on the front panel of the terminal 1000. In some embodiments, the display screen is designed as a full-screen, curved screen, irregularly shaped screen, dual-sided screen, or foldable screen. In some embodiments, the display screen is also designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen, etc., which are not limited in this embodiment.

[0170] In addition, those skilled in the art will understand that the structure of the terminal 1000 shown in the above figures does not constitute a limitation on the terminal 1000. The terminal 1000 may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the terminal 1000 may also include components such as a microphone, speaker, input unit, sensor, audio circuit, module, power supply, and Bluetooth module, which will not be described in detail here.

[0171] This application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by the processor to implement the target detection method shown in the above embodiments.

[0172] This application also provides a chip including programmable logic circuits and / or program instructions, which, when run on a terminal, are used to implement the target detection methods shown in the above embodiments.

[0173] This application also provides a computer program product that stores at least one piece of program code, which is executed by a processor to implement the target detection method shown in the above embodiments.

[0174] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0175] Those skilled in the art will understand that all or part of the steps in the target detection method of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. The above descriptions are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A target detection method, characterized in that, The method includes: Edge enhancement is performed on the original image to obtain the target image, wherein the edge enhancement is used to enhance the edge information of at least one target object in the original image; Based on multiple feature extraction layers of the object detection model, features are extracted from the target image to obtain first feature maps at multiple scales. The target image includes at least one target object. The object detection model is used to detect the category and location of the at least one target object. The multiple feature extraction layers correspond one-to-one with the first feature maps at multiple scales. Each feature extraction layer includes multiple concatenated first convolutional units. The first convolutional units are used to extract features from the input image through at least one operation of depthwise convolution, pointwise convolution, and batch normalization. The feature fusion layer of the object detection model includes multiple fusion units and multiple detection units. The multiple fusion units correspond one-to-one with the multiple feature extraction layers and the multiple fusion units correspond one-to-one with the multiple detection units. For any scale of the first feature map, based on the fusion unit corresponding to the first feature map, the first feature map of the scale and the first feature maps of other scales are fused according to the scale of the first feature map to obtain a first intermediate feature map. The number of channels of the first intermediate feature map is the number of pointwise convolution kernels in the fusion unit. The pointwise convolution kernels are used to perform pointwise convolution on the input feature map so that the number of channels of the input feature map is the same as the number of pointwise convolution kernels. Based on the detection unit corresponding to the fusion unit, the first intermediate feature map is separated into channels to obtain at least one channel's category feature map and at least one channel's position feature map; Based on the detection unit corresponding to the fusion unit, the category feature map of at least one channel and the position feature map of at least one channel are spliced ​​together in the channel dimension to obtain the second intermediate feature map; Multiple second intermediate feature maps are concatenated along the scale dimension to obtain a first fused feature map, which includes the category features and location features of the at least one target object. Based on the output layer of the target detection model, the first fused feature map is decoded to obtain the detection result of the target image, and the detection result is used to represent the category and location of the at least one target object.

2. The method according to claim 1, characterized in that, The multiple feature extraction layers based on the target detection model extract features from the target image to obtain first feature maps at multiple scales, including: Based on the initial feature extraction layer of the target detection model, feature extraction is performed on the target image to obtain the initial feature map of the target image. The initial feature extraction layer includes a second convolutional unit, and the convolution operation in the second convolutional unit is different from the convolution operation in the first convolutional unit. In the first feature extraction layer of the target detection model, features are extracted from the initial feature map through multiple first convolutional units in the feature extraction layer to obtain the first feature map of the first feature extraction layer; In the i-th feature extraction layer of the target detection model, features are extracted from the first feature map of the (i-1)-th feature extraction layer through multiple first convolutional units in the feature extraction layer to obtain the first feature map of the i-th feature extraction layer, and so on until the first feature map of each feature extraction layer is obtained. The scale of the first feature map is negatively correlated with the number of the corresponding feature extraction layer, where i is an integer greater than 1.

3. The method according to claim 1, characterized in that, For any scale of the first feature map, based on the fusion unit corresponding to the first feature map, the first feature map at the first scale is fused with first feature maps at other scales according to the scale of the first feature map to obtain a first intermediate feature map, including: For a first feature map at any scale, based on the fusion unit corresponding to the first feature map, pointwise convolution is performed on the first feature map at the scale and the first feature maps at other scales respectively, so that the number of channels of the first feature map at the scale and the first feature maps at other scales are the same, and the number of channels is the same as the number of pointwise convolution kernels in the pointwise convolution; Based on the fusion unit, according to the scale of the first feature map, multiple other scales of first feature maps are upsampled or downsampled to obtain multiple scales of first feature maps; Based on the fusion unit, the first feature map at the specified scale and the first feature maps at the plurality of specified scales are fused to obtain the first intermediate feature map.

4. The method according to claim 1, characterized in that, The step of concatenating multiple second intermediate feature maps along the scale dimension to obtain the first fused feature map includes: For any second intermediate feature map, if the second intermediate feature map includes three dimensions: channel, width, and height, the second intermediate feature map is transformed into a third intermediate feature map, which includes two dimensions: channel and length, and the length is the product of the width and height. By concatenating multiple third intermediate feature maps along the length dimension, the first fused feature map is obtained.

5. The method according to claim 1, characterized in that, The training process of the target detection model includes: Based on the multiple feature extraction layers in the target detection model, feature extraction is performed on the sample image to obtain a second feature map at multiple scales, wherein the sample image includes at least one target object; Based on the feature fusion layer of the target detection model, the second feature maps of the multiple scales are fused to obtain a second fused feature map, which includes the category features and location features of at least one target object in the sample image. Based on the output layer of the target detection model, the second fused feature map is decoded to obtain the detection result of the sample image. The detection result is used to represent the predicted category and predicted location of at least one target object in the sample image. Based on the detection results and labels of the sample images, the training loss of the target detection model is determined, wherein the labels of the sample images are used to represent the true category and true location of at least one target object in the sample images; The target detection model is trained based on the training loss.

6. The method according to claim 5, characterized in that, The training loss includes a category loss and a location loss. The category loss is used to represent the difference between the predicted category and the true category of the at least one target object, and the location loss is used to represent the difference between the predicted location and the true location of the at least one target object.

7. The method according to claim 5, characterized in that, The step of training the object detection model based on the training loss includes: Based on the auxiliary training layer of the target detection model, the first feature maps of the multiple scales are fused to obtain a third fused feature map; Based on the output layer of the target detection model, the third fusion feature map is decoded to obtain the auxiliary detection result of the target detection model; Based on the auxiliary detection results of the sample images and the labels of the sample images, the auxiliary training loss of the target detection model is determined; The target detection model is trained based on the training loss and the auxiliary training loss.

8. The method according to claim 5, characterized in that, The method further includes: The target detection model is trained in the first stage based on the first sample set, which includes multiple original sample images and a first number of augmented sample images. The augmented sample images are obtained by data augmentation of the multiple original sample images. The target detection model is trained in the second stage based on the second sample set, which includes the plurality of original sample images and a second number of expanded sample images, wherein the second number is less than the first number. The target detection model is trained in the third stage based on the multiple original sample images.

9. A target detection device, characterized in that, The device includes: The first extraction module is used to perform edge enhancement on the original image to obtain a target image. The edge enhancement is used to enhance the edge information of at least one target object in the original image. Based on multiple feature extraction layers of the target detection model, features are extracted from the target image to obtain first feature maps at multiple scales. The target image includes at least one target object. The target detection model is used to detect the category and location of the at least one target object. The multiple feature extraction layers correspond one-to-one with the first feature maps at multiple scales. The feature extraction layer includes multiple concatenated first convolutional units. The first convolutional units are used to extract features from the input image through at least one operation of depthwise convolution, pointwise convolution, and batch normalization. The feature fusion layer of the target detection model includes multiple fusion units and multiple detection units. The multiple fusion units correspond one-to-one with the multiple feature extraction layers and the multiple fusion units correspond one-to-one with the multiple detection units. A first fusion module is configured to, for any scale of a first feature map, based on a fusion unit corresponding to the first feature map, fuse the first feature map at the scale of the first feature map with first feature maps at other scales to obtain a first intermediate feature map. The number of channels in the first intermediate feature map is equal to the number of pointwise convolutional kernels in the fusion unit. The pointwise convolutional kernels are used to perform pointwise convolution on the input feature map so that the number of channels in the input feature map is the same as the number of pointwise convolutional kernels. Based on a detection unit corresponding to the fusion unit, the first intermediate feature map is channel-separated to obtain a category feature map of at least one channel and a position feature map of at least one channel. Based on a detection unit corresponding to the fusion unit, the category feature map of at least one channel and the position feature map of at least one channel are concatenated in the channel dimension to obtain a second intermediate feature map. Multiple second intermediate feature maps are concatenated in the scale dimension to obtain a first fused feature map. The first fused feature map includes the category features and position features of the at least one target object. The first decoding module is used to decode the first fused feature map based on the output layer of the target detection model to obtain the detection result of the target image, wherein the detection result is used to represent the category and location of the at least one target object.

10. A terminal, characterized in that, The terminal includes a processor and a memory; the memory stores at least one piece of program code, which is executed by the processor to implement the target detection method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The storage medium stores at least one piece of program code, which is executed by a processor to implement the target detection method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, The computer program product stores at least one piece of program code, which is executed by a processor to implement the target detection method as described in any one of claims 1 to 8.

Images