Target detection method for reinforcing multi-scale feature extraction, reuse and fusion

Abstract

The application discloses a target detection method for reinforcing multi-scale feature extraction, multiplexing and fusion. It relates to the technical field of artificial intelligence. The method comprises the following steps: in response to a recognition request for a target image, inputting the target image into a target feature extraction network of a target detection model, and outputting N first feature images, wherein the target detection model at least comprises a target feature extraction network, a feature fusion network and a target prediction network, the target detection model comprises a plurality of transmission channels, and the transmission channels are used for transmitting image features of the target image; inputting the N first feature images into the feature fusion network, and outputting M second feature images; and inputting the M second feature images into the target prediction network, and outputting a recognition result of the target image. The application solves the technical problems of complex model structure, high calculation cost, no pertinence to salient features and low target recognition efficiency in the target detection process.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a target detection method that enhances multi-scale feature extraction, reuse, and fusion. Background Technology

[0002] Currently, object detection technology is mainly divided into traditional object detection algorithms, two-stage object detection algorithms, single-stage object detection algorithms, and keypoint-based object detection algorithms. Before 2012, research mainly focused on traditional object detection algorithms. Representative works from this period include the Viola Jones detector (a real-time object detection method that provides object detection rates), the HOG detector (a detection method based on histogram of orientation features), and the DPM detector (a component-based detection method). First, these algorithms use selective windowing to traverse the sample data and generate candidate boxes. Then, feature extraction components are used to extract features from the sample data within the candidate boxes. Finally, a classifier is used for classification. However, these algorithms lack specificity for important target features, thus easily leading to window redundancy and high algorithm complexity.

[0003] Two-stage object detection algorithms are based on the low real-time performance and high detection accuracy of candidate regions. These algorithms first generate candidate regions from the sample data, then extract features from the candidate regions using a convolutional neural network, and finally perform classification. Currently, these algorithms mainly include R-CNN (an algorithm that applies deep learning to object detection), SPPNet (Spatial Pyramid Pooling Convolutional Network), Fast R-CNN, Faster R-CNN, HyperNet, FPN, Mask R-CNN, and TridentNet. Among these, R-CNN, SPPNet, Fast R-CNN, Faster R-CNN, HyperNet, FPN, Mask R-CNN, and TridentNet represent different types of two-stage object detection algorithms within this technology.

[0004] Single-stage object detection algorithms integrate object feature extraction, object classification, and candidate box regression, thus achieving end-to-end object detection. Currently, these algorithms mainly include the YOLO series, SSD series, MobileNet series, ShuffleNet series, RetinaNet, EfficientDet, and Swin Transformer (YOLO series, SSD series, MobileNet series, ShuffleNet series, RetinaNet, EfficientDet, and Swin Transformer are different types of single-stage object detection algorithms within related technologies). Single-stage object detection algorithms generally outperform two-stage object detection algorithms in real-time performance and have higher detection accuracy, thus finding widespread application in many situations.

[0005] Keypoint-based object detection algorithms essentially replace the candidate box generation process with the detection and matching of keypoints on the target, thereby eliminating problems such as sample imbalance caused by candidate boxes. These algorithms first treat the process of finding the target's center point as target keypoint evaluation, and then use keypoint regression strategies to adjust the target's position, angle, and pose attributes. Currently, keypoint-based object detection algorithms are mainly based on the CornerNet series, CenterNet, and ExtremeNet. However, these algorithms are complex and difficult to achieve the requirements of lightweight design and real-time performance.

[0006] Currently, the main challenges hindering further improvement of deep learning-based object detection algorithms are complex model structures, an increase in hyperparameters, complex optimization processes, and uneven distribution of sample data.

[0007] Currently, there are many types of backbone feature extraction networks with complex structures. Expanding the depth and width of backbone feature extraction networks can easily increase the computational load of the algorithm and make the algorithm model difficult to deploy, resulting in low efficiency in identifying and classifying targets in images. For example, after financial institutions acquire images through image acquisition equipment, they need to perform target identification. However, due to the complex structure of existing target detection algorithms, problems such as low image recognition efficiency, serious waste of system resources, and difficulty in quickly carrying out other tasks of financial institutions can be encountered.

[0008] Furthermore, multi-scale feature fusion in related technologies can easily lead to the flow of non-saliency features in the neck feature fusion network, causing the algorithm model to reuse gradient information repeatedly, resulting in unnecessary consumption of computational resources.

[0009] Furthermore, multi-scale feature fusion strategies stack or weight features with large differences in an averaging manner, thus failing to effectively distinguish between salient and non-salient features, resulting in significant differences between input and output features.

[0010] There is currently no effective solution to the above problems. Summary of the Invention

[0011] This invention provides a target detection method that enhances multi-scale feature extraction, reuse, and fusion to address the technical problems of complex model structures, high computational costs, and lack of specificity for salient features in target detection algorithms, which lead to low target recognition efficiency during the target detection process.

[0012] According to one aspect of the present invention, a target detection method with enhanced multi-scale feature extraction, reuse, and fusion is provided, comprising: responding to a target image recognition request, inputting the target image into a target feature extraction network of a target detection model, and outputting N first feature images, wherein the target detection model includes at least: the target feature extraction network, a feature fusion network, and a target prediction network, and the target detection model includes: multiple transmission channels, the transmission channels being used to transmit image features of the target image, where N is an integer greater than 1; inputting the N first feature images into the feature fusion network, and outputting M second feature images, wherein the feature fusion network is used to perform multi-scale feature fusion on the N first feature images, where M is an integer greater than 1; inputting the M second feature images into the target prediction network, and outputting a recognition result of the target image, wherein the recognition result includes a classification result of the target in the target image.

[0013] Furthermore, the target feature extraction network includes at least: multiple first feature extraction modules and multiple dimension adjustment modules. Each dimension adjustment module includes at least: a convolutional layer, a batch normalization layer, and an activation function layer. The target image is input into the target feature extraction network of the target detection model, and N first feature images are output. This includes: adjusting the dimensions of the target image through each dimension adjustment module, and extracting features from the dimension-adjusted target image through each first feature extraction module to obtain N first feature images.

[0014] Further, each of the first feature extraction modules includes at least: multiple dimension adjustment submodules and a residual submodule. The residual submodule includes multiple residual units. Each dimension adjustment submodule has the same structure as the dimension adjustment module. The first feature extraction module extracts features from the dimension-adjusted target image to obtain N first feature images. This includes: receiving a third feature image through the first feature extraction module, inputting the third feature image into the first dimension adjustment submodule through a first transmission channel, inputting a fourth feature image output by the first dimension adjustment submodule into the residual submodule, and outputting a first feature layer. The first feature extraction module is one of multiple first feature extraction modules, and the third feature image is obtained after processing by multiple first feature extraction modules and multiple... The target image is processed by one or more modules of the dimension adjustment modules, wherein the first dimension adjustment submodule is one of the multiple dimension adjustment submodules; in the first transmission channel, the first feature layer is convolved and then input into the stacking unit; in the second transmission channel, the third feature image is convolved and then input into the stacking unit, and the feature data received by the stacking unit is stacked to obtain a second feature layer; the second feature layer is subjected to target processing to obtain a target feature image, wherein the target processing includes at least one of the following: normalization processing, activation function processing, feature dimension adjustment processing, and the target feature image is one of N first feature images; based on the multiple target feature images, N first feature images are determined.

[0015] Furthermore, each residual unit includes at least: multiple dimension adjustment subunits and a weighting subunit. The fourth feature image output by the first dimension adjustment submodule is input into the residual submodule to output a first feature layer. This includes: performing channel adjustment on the fourth feature image based on a first transmission subchannel and the multiple dimension adjustment subunits to obtain a third feature layer, and inputting the third feature layer into the weighting subunit; inputting the fourth feature image into the weighting subunit through a second transmission subchannel; and performing weighted processing on the third feature layer and the fourth feature image through the weighting subunit to output the first feature layer.

[0016] Further, the feature fusion network includes at least: multiple second feature extraction modules, multiple feature fusion modules, and multiple feature fusion layers. Multiple residual units in the first feature extraction module are replaced with multiple dimension adjustment units as the second feature extraction modules. N first feature images have different image sizes. The N first feature images are input into the feature fusion network, and M second feature images are output. This includes: In the first feature fusion layer, the first feature image of a first size is input into the first feature fusion module, and the output of the first feature fusion module is input into the first extraction module to output a first image. The first feature fusion module is used to fuse the image features transmitted through the first feature flow path with the first feature image of the first size. The transmission direction of the image features transmitted through the first feature flow path is from the last feature fusion layer to the first feature fusion layer. The first extraction module is one of multiple second feature extraction modules. The first extraction module is used to transmit the output of the first feature extraction module to the last feature fusion layer via the second feature flow path. The image features transmitted through the second feature flow path... The input direction is from the first feature fusion layer to the last feature fusion layer; in the second feature fusion layer, based on the third and fourth transmission channels, the first feature image of the second size is fused with the image features transmitted through the first feature flow path, and the result of feature fusion is fused with the image features transmitted through the second feature flow path to determine the second image; in the last feature fusion layer, based on the fifth and sixth transmission channels, the first feature image of the third size is fused with the features transmitted through the second feature flow path after pyramid pooling to determine the third image; based on the first image, the second image, and the third image, M second feature images are determined.

[0017] Further, in the second feature fusion layer, based on the third and fourth transmission channels, the first feature image of the second size is fused with the image features transmitted through the first feature flow path, and the fused result is fused with the image features transmitted through the second feature flow path to determine the second image. This includes: in the second feature fusion layer, transmitting the first feature image of the second size to the second feature fusion module through the third transmission channel, performing multi-scale fusion with the image features transmitted through the first feature flow path to obtain a first fused feature; inputting the first fused feature into the second extraction module to obtain a second fused feature, wherein the second extraction module is one of multiple second feature extraction modules; transmitting the first feature image of the second size to the third feature fusion module through the fourth transmission channel, performing multi-scale feature fusion with the features transmitted through the second feature flow path and the second fused feature to determine a third fused feature; inputting the third fused feature into the third extraction module to output the second image, wherein the third extraction module is one of multiple second feature extraction modules.

[0018] Furthermore, after performing pyramid pooling on the first feature image of the third size, the method further includes: transmitting the result obtained after performing pyramid pooling on the first feature image of the third size to the first feature fusion layer through the first feature flow path.

[0019] Further, the target detection model is determined as follows: Pre-trained weights and training samples are obtained, and the training samples are divided into a training set, a validation set, and a test set. The training samples include multiple images, each with a bounding box labeled with the target. An initial detection model is trained based on the pre-trained weights and the training set, and cross-validation is used during training to verify whether the initial detection model has converged using the validation set. The initial detection model is an untrained model. If the initial detection model converges, the target detection model is determined, and the detection accuracy of the initial detection model is tested based on the weight file and the test set. The weight file stores multiple weights of the target detection model.

[0020] Furthermore, the target detection method that enhances multi-scale feature extraction, reuse, and fusion also includes: during the training of the initial detection model, receiving the fused feature map output by the feature fusion network through the target prediction network; dividing the fused feature map into multiple grids, wherein the fused feature map is marked with the bounding boxes of the target, and each grid includes multiple anchor boxes; finding the target grid in the multiple grids where the bounding boxes in the fused feature map are located; determining the prediction box based on the intersection-union ratio (IU) of each anchor box in the target grid with the bounding boxes of the target; and determining the detection result of the initial detection model through the prediction box.

[0021] Further, the step of determining the predicted bounding box based on the intersection-union ratio (IU) of each anchor box in the target grid with the labeled bounding box of the target includes: calculating the IU of each anchor box in the target grid with the labeled bounding box of the target to obtain multiple ratios; taking the anchor box associated with the largest ratio among the multiple ratios as the target anchor box, wherein the target anchor box is used to detect the target on the fused feature map; and updating the position parameters of the target anchor box to obtain the predicted bounding box.

[0022] Furthermore, the loss functions used when training the initial detection model include at least: a classification loss function, a confidence loss function, and a localization loss function.

[0023] According to another aspect of the present invention, a target detection apparatus for enhancing multi-scale feature extraction, reuse, and fusion is also provided, comprising: a first processing unit, configured to respond to a target image recognition request, input the target image into a target feature extraction network of a target detection model, and output N first feature images, wherein the target detection model includes at least: the target feature extraction network, a feature fusion network, and a target prediction network, and the target detection model includes: multiple transmission channels, the transmission channels being used to transmit image features of the target image, where N is an integer greater than 1; a second processing unit, configured to input the N first feature images into the feature fusion network and output M second feature images, wherein the feature fusion network is used to perform multi-scale feature fusion on the N first feature images, where M is an integer greater than 1; and a third processing unit, configured to input the M second feature images into the target prediction network and output a recognition result of the target image, wherein the recognition result includes a classification result of the target in the target image.

[0024] Furthermore, the target feature extraction network includes at least: multiple first feature extraction modules and multiple dimension adjustment modules. Each dimension adjustment module includes at least: a convolutional layer, a batch normalization layer, and an activation function layer. The first processing unit includes: a first processing subunit, used to perform dimension adjustment on the target image through each dimension adjustment module, and to extract features from the dimension-adjusted target image through each first feature extraction module to obtain N first feature images.

[0025] Further, each of the first feature extraction modules includes at least: multiple dimension adjustment sub-modules and a residual sub-module. The residual sub-module includes multiple residual units. Each dimension adjustment sub-module has the same structure as the dimension adjustment module. The processing sub-unit includes: a first processing module, used to receive a third feature image through the first feature extraction module, input the third feature image into the first dimension adjustment sub-module through a first transmission channel, input the fourth feature image output by the first dimension adjustment sub-module into the residual sub-module, and output a first feature layer. The first feature extraction module is one of multiple first feature extraction modules, and the third feature image is the target image processed by one or more of the multiple first feature extraction modules and the multiple dimension adjustment modules. The dimension adjustment submodule is one of the plurality of dimension adjustment submodules; the second processing module is used to perform convolution processing on the first feature layer in the first transmission channel and then input it into the stacking unit; the third processing module is used to perform convolution processing on the third feature image in the second transmission channel and then input it into the stacking unit, and perform stacking processing on the feature data received by the stacking unit to obtain the second feature layer; the fourth processing module is used to perform target processing on the second feature layer to obtain a target feature image, wherein the target processing includes at least one of the following: normalization processing, activation function processing, feature dimension adjustment processing, and the target feature image is one of N first feature images; the determination module is used to determine N first feature images based on the plurality of target feature images.

[0026] Furthermore, each residual unit includes at least: multiple dimension adjustment subunits, a weighting subunit, and the first processing module includes: an adjustment submodule, used to perform channel adjustment on the fourth feature image based on a first transmission subchannel and the multiple dimension adjustment subunits to obtain a third feature layer, and input the third feature layer into the weighting subunit; a first input module, used to input the fourth feature image into the weighting subunit through a second transmission subchannel; and a weighting module, used to perform weighted processing on the third feature layer and the fourth feature image through the weighting subunit, and output the first feature layer.

[0027] Further, the feature fusion network includes at least: multiple second feature extraction modules, multiple feature fusion modules, and multiple feature fusion layers. Multiple residual units in the first feature extraction module are replaced with multiple dimension adjustment units as the second feature extraction modules. The N first feature images have different image sizes. The second processing unit includes: a second processing subunit, used to input the first feature image of a first size into the first feature fusion module in the first feature fusion layer, and input the output of the first feature fusion module into the first extraction module to output a first image. The first feature fusion module is used to fuse the image features transmitted through the first feature flow path with the first feature image of the first size. The transmission direction of the image features transmitted through the first feature flow path is from the last feature fusion layer to the first feature fusion layer. The first extraction module is one of multiple second feature extraction modules. The first extraction module is used to transmit the output of the first feature extraction module to the last feature fusion layer via the second feature flow path. The image feature transmission direction of the second feature flow path is from the first feature fusion layer to the last feature fusion layer. The feature fusion layer extends to the last feature fusion layer; a third processing subunit is configured to, in the second feature fusion layer, based on the third and fourth transmission channels, perform feature fusion on the first feature image of the second size and the image features transmitted through the first feature flow path, and fuse the result of feature fusion with the image features transmitted through the second feature flow path to determine the second image; a fourth processing subunit is configured to, in the last feature fusion layer, based on the fifth and sixth transmission channels, perform feature fusion on the first feature image of the third size after pyramid pooling operation with the features transmitted through the second feature flow path to determine the third image; a determination subunit is configured to determine M second feature images based on the first image, the second image, and the third image.

[0028] Further, the third processing subunit includes: a first fusion module, used to transmit the first feature image of the second size to the second feature fusion module through the third transmission channel in the second feature fusion layer, and perform multi-scale fusion with the image features transmitted through the first feature flow path to obtain a first fused feature; an input module, used to input the first fused feature into a second extraction module to obtain a second fused feature, wherein the second extraction module is one of a plurality of second feature extraction modules; a second fusion module, used to transmit the first feature image of the second size to the third feature fusion module through a fourth transmission channel, and perform multi-scale feature fusion with the features transmitted through the second feature flow path and the second fused feature to determine a third fused feature; and an input / output module, used to input the third fused feature into the third extraction module and output the second image, wherein the third extraction module is one of a plurality of second feature extraction modules.

[0029] Furthermore, the second processing unit further includes a transmission subunit, used to transmit the result obtained after the pyramid pooling operation on the first feature image of the third size to the first feature fusion layer through the first feature flow path after the pyramid pooling operation on the first feature image of the third size.

[0030] Further, the target detection model is determined by the following units: a fourth processing unit, used to acquire pre-trained weights and training samples, and divide the training samples into a training set, a validation set, and a test set, wherein the training samples include: multiple images, each image having a bounding box labeled with the target; a training unit, used to train the initial detection model based on the pre-trained weights and the training set, and to use cross-validation during the training process to verify whether the initial detection model has converged through the validation set, wherein the initial detection model is an untrained model; and a testing unit, used to determine the target detection model if the initial detection model has converged, and to test the detection accuracy of the initial detection model based on the weight file and the test set, wherein the weight file is used to store multiple weights of the target detection model.

[0031] Furthermore, the target detection method that enhances multi-scale feature extraction, reuse, and fusion also includes: a receiving subunit, used to receive the fused feature map output by the feature fusion network through the target prediction network during the training of the initial detection model; a partitioning subunit, used to divide the fused feature map into multiple grids, wherein the fused feature map is marked with the bounding boxes of the target, and each grid includes multiple anchor boxes; a searching subunit, used to search for the target grid in the multiple grids where the bounding boxes in the fused feature map are located; a first determining subunit, used to determine the prediction box based on the intersection-union ratio (IoU) between each anchor box in the target grid and the bounding box of the target; and a second determining subunit, used to determine the detection result of the initial detection model through the prediction box.

[0032] Further, the first determining subunit includes: a calculation module, used to calculate the intersection-union ratio (IUU) of each anchor box in the target grid with the labeled box of the target, to obtain multiple ratios; an anchor box processing module, used to associate the anchor box with the largest ratio among the multiple ratios as the target anchor box, wherein the target anchor box is used to detect the target on the fused feature map; and an update module, used to update the position parameters of the target anchor box to obtain the predicted box.

[0033] Furthermore, the loss functions used when training the initial detection model include at least: a classification loss function, a confidence loss function, and a localization loss function.

[0034] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the enhanced multi-scale feature extraction, reuse and fusion target detection method of any one of the above-described embodiments by executing the executable instructions.

[0035] According to another aspect of the present invention, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer program controls the device where the computer-readable storage medium is located to execute the enhanced multi-scale feature extraction, reuse and fusion target detection method described above when the computer program is running.

[0036] In this invention, in response to a request to recognize a target image, the target image is input into the target feature extraction network of a target detection model, which outputs N first feature images. The target detection model includes at least the target feature extraction network, a feature fusion network, and a target prediction network. The target detection model includes multiple transmission channels for transmitting image features of the target image, where N is an integer greater than 1. The N first feature images are input into the feature fusion network, which outputs M second feature images. The feature fusion network performs multi-scale feature fusion on the N first feature images, where M is an integer greater than 1. The M second feature images are input into the target prediction network, which outputs the recognition result of the target image, where the recognition result includes a classification result of the target in the target image. This solves the technical problem of low target recognition efficiency in target detection processes due to complex model structures, high computational costs, and lack of specificity for salient features in target detection algorithms. In this invention, a target detection model containing multiple transmission channels is used to classify and identify targets in a target image. This avoids the situation in related technologies where the detection model has a complex structure and slow feature flow speed, resulting in low recognition efficiency. As a result, the technical effect of the target detection model in classifying and identifying targets in an image is improved. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0038] Figure 1 This is a flowchart of an optional target detection method for enhancing multi-scale feature extraction, reuse and fusion according to an embodiment of the present invention;

[0039] Figure 2 This is a model structure diagram of an optional target detection model according to an embodiment of the present invention;

[0040] Figure 3 This is a structural diagram of an optional ECSPDarkNet1-X module according to an embodiment of the present invention;

[0041] Figure 4 This is a structural diagram of an optional ECSPDarkNet2-X module according to an embodiment of the present invention;

[0042] Figure 5 This is a flowchart of an optional neck feature fusion network multi-scale feature fusion process according to an embodiment of the present invention;

[0043] Figure 6This is a flowchart of an optional model training method according to an embodiment of the present invention;

[0044] Figure 7 This is a flowchart of an optional model test according to an embodiment of the present invention;

[0045] Figure 8 This is a flowchart of three loss function change curves using an optional cross-validation training strategy according to an embodiment of the present invention;

[0046] Figure 9 This is a flowchart of an optional anchor frame to prediction frame adjustment process according to an embodiment of the present invention;

[0047] Figure 10 This is a flowchart of an optional prediction box regression process according to an embodiment of the present invention;

[0048] Figure 11 This is a schematic diagram of an optional target detection device for enhancing multi-scale feature extraction, reuse and fusion according to an embodiment of the present invention;

[0049] Figure 12 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0052] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, image feature data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0053] This invention can be applied to various software products, control systems, and client-side systems (including but not limited to mobile clients and PCs) of financial institutions for visual inspection. Taking software products as an example, the visual inspection system installed on the mobile client can realize real-time monitoring of bank branch foot traffic, bank visitor analysis, anti-fraud and anti-counterfeiting identification of banknotes, detection of abnormal behavior of bank visitors, bank branch monitoring and alarm systems, bank card recognition, and bank visitor facial recognition for relevant business content of financial institutions (including but not limited to: transfer, wealth management, funds, payment, account inquiry, advertising, recommendation, etc.).

[0054] Example 1

[0055] According to an embodiment of the present invention, an optional method embodiment for enhancing multi-scale feature extraction, reuse and fusion of target detection is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0056] Figure 1 This is a flowchart of an optional target detection method for enhancing multi-scale feature extraction, reuse, and fusion according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0057] Step S101: In response to the recognition request for the target image, the target image is input into the target feature extraction network of the target detection model, and N first feature images are output. The target detection model includes at least: a target feature extraction network, a feature fusion network, and a target prediction network. The target detection model includes: multiple transmission channels, which are used to transmit the image features of the target image. N is an integer greater than 1.

[0058] Figure 2 This is a model structure diagram of an optional target detection model according to an embodiment of the present invention, such as... Figure 2As shown, the target detection model described above can be composed of four modules: an input image component (for receiving the target image to be identified), a backbone feature extraction network (corresponding to the target feature extraction network mentioned above), a neck feature fusion network (corresponding to the feature fusion network mentioned above), and a target prediction network. The size of the input target image can be within a preset range, for example, between 0.5 and 1.5 inches. (640 Within the range of 640), in one optional manner, the size of the input target image can be 416. 416. The backbone feature extraction network described above can be used to handle multi-scale shallow feature extraction.

[0059] In this embodiment, by inputting the target image into the target feature extraction network of the target detection model, N first feature images can be output by the target feature extraction network. In order to avoid the situation where the model structure is complex in related technologies, resulting in low feature flow efficiency in the model, the above-mentioned target detection model can include multiple transmission channels to transmit image features, thereby improving the flow speed of image features in the target detection model, so as to improve the efficiency of the target detection model in recognizing and classifying targets in the image.

[0060] Step S102: Input N first feature images into the feature fusion network and output M second feature images. The feature fusion network is used to perform multi-scale feature fusion on the N first feature images, and M is an integer greater than 1.

[0061] The aforementioned feature fusion network can perform multi-scale fusion of the feature images output by the target feature extraction network to obtain multiple second feature images (corresponding to the M second feature images mentioned above). For example, the neck feature fusion network (corresponding to the aforementioned feature fusion network) can perform multi-scale fusion of multi-scale features output from the backbone feature extraction network.

[0062] Step S103: Input M second feature images into the target prediction network and output the recognition result of the target image, wherein the recognition result includes the classification result of the target in the target image.

[0063] In this embodiment, the M second feature images output by the feature fusion network can be input into the target prediction network. The target prediction network predicts and judges the attributes of the target in the target image to identify whether the target image contains a target and the category of the target.

[0064] The target detection method for enhancing multi-scale feature extraction, reuse, and fusion in target images in this embodiment can be used in image acquisition equipment of financial institutions to identify and classify targets in images acquired by financial institutions, thereby improving the processing efficiency of image data in financial institutions.

[0065] It should be noted that the target detection model in this embodiment can be applied to the visual inspection system of financial institutions, such as bank visual inspection systems, to improve the online real-time monitoring security capabilities of bank branches. Specifically, this embodiment can be applied to scenarios such as real-time monitoring of people flow in bank branches, bank visitor analysis, anti-fraud and anti-counterfeiting identification of banknotes, detection of abnormal behavior of bank visitors, bank branch monitoring and alarm systems, bank card recognition, and bank visitor facial recognition.

[0066] In the target detection model of this embodiment, feature information can be integrated through statistical analysis of target geometric features to achieve the effect of target detection. The primary task of this target detection model is to locate targets in the sample data, then process the feature information using a feature extraction and fusion module, and finally classify the target information. Based on artificial intelligence technology, this embodiment can also be widely applied in fields such as smart finance, intelligent transportation, smart healthcare, intelligent security, target tracking, and industrial inspection.

[0067] Through the above steps, in this embodiment, a target detection model with multiple transmission channels is used to classify and identify targets in the target image. This avoids the problems of complex detection models and slow feature flow in related technologies, which lead to low recognition efficiency. Therefore, the technical effect of the target detection model in classifying and identifying targets in the image is improved. Furthermore, it solves the technical problems of complex model structures, high computational costs, and lack of specificity for salient features in target detection algorithms, which result in low target recognition efficiency during the target detection process.

[0068] The basic computational process of the target detection algorithm in this embodiment is explained below:

[0069] In this embodiment, feature extraction can be performed using ordinary convolution operations, as shown in formula (1). Wherein, x For input features, w For convolution kernel, s For output features.

[0070] (1)

[0071] In addition, the feature dimension needs to be determined between the input features and the output features. In this embodiment, the feature size can be calculated using formulas (2) and (3).

[0072] (2)

[0073] (3)

[0074] in, P For filling,S For step size, ( H , W ) represents the height and width of the input feature, FH , FW ) represents the height and width of the convolution kernel, OH , OW The input feature's height and width are denoted as (0.5~1.5). In this embodiment, the input feature's size can be between 0.5 and 1.5. (640 Within the range of 640), a maximum downsampling operation of 32 times is sufficient. Typically, the kernel size can be 1. 1, 3 3.

[0075] The forward and backward propagation processes in the target detection algorithm of this embodiment are explained below:

[0076] Forward propagation essentially involves extracting features through convolutional operations within the network and then using optimization strategies to propagate these features forward. Backward propagation, on the other hand, involves learning and updating the gradients of each weight and bias of the convolutional kernel.

[0077] The forward propagation process of the convolutional layer in the target detection algorithm of this embodiment is explained below: Let M be the (l-1)th output feature map, x j l-1 Input features for the current layer, x j l The forward propagation process of the convolutional layer, which outputs features for the current layer, is shown in equations (4) to (5). w For convolution kernel, b For bias, f() This is the activation function.

[0078] (4)

[0079] (5)

[0080] The backpropagation process of the convolutional layer in the target detection algorithm of this embodiment is explained below:

[0081] For the l layer gradient By performing backpropagation, we can obtain the first... l The gradient of layer -1 is denoted as As shown in formula (6). If upsampling is not used during backpropagation, then the upsampling can be performed. UpSampling() neglect.

[0082] (6)

[0083] Current convolutional layer pairs w The gradient result is the gradient of the deviation between the input and output of this layer, which is then convolved. Other convolution kernels are similar, as shown in equation (7). The bias... b It is a vector; therefore, by summing the error gradient terms of each convolution kernel, we can obtain the value about... b The error vector is shown in Equation (8). Finally, the gradient optimization algorithm is used to update the parameters, as shown in Equation (9). w The update process, b The update process is similar. Among them, J Let be the target loss function. This is the learning rate.

[0084] (7)

[0085] (8)

[0086] (9)

[0087] The forward propagation process of the pooling layer in the target detection algorithm of this embodiment is explained below:

[0088] Similar to convolutional layers, let the first layer be denoted as the second layer. l The layer is a pooling layer, and its forward propagation process is shown in Equations (10) to (11).

[0089] in, DowmSampling() This is a downsampling operation.

[0090] (10)

[0091] (11)

[0092] The backpropagation process of the pooling layer in the target detection algorithm of this embodiment is explained below:

[0093] Record No. l The gradient of the -1 pooling layer is Thus, the back propagation process of the pooling layer is obtained as shown in formula (12).

[0094] (12)

[0095] The following describes the model optimization algorithm process in the target detection algorithm of this embodiment:

[0096] In this embodiment, the Adam gradient optimization algorithm can be used to update the parameters. The important function of this algorithm is to perform moving averages and bias correction on the gradient. Its pseudocode execution process is shown below:

[0097] The pseudocode for the Adam gradient optimization algorithm in this embodiment is as follows:

[0098] For learning rate Learnable parameters and Perform initialization;

[0099] right m and v Set the smoothness constant and ;

[0100] initialization , , ;

[0101] while When it does not converge

[0102] ;

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] end while

[0110] The algorithm first calculates... t Gradient at next time Then, first-order moment estimates are calculated for each gradient. and second-order moment estimation Then, bias corrections are applied to the first-order moment estimate and the second-order moment estimate respectively to obtain... and Finally, the learnable parameters Update.

[0111] Optionally, the target feature extraction network includes at least: multiple first feature extraction modules and multiple dimension adjustment modules. Each dimension adjustment module includes at least: a convolutional layer, a batch normalization layer, and an activation function layer. The target image is input into the target feature extraction network of the target detection model, and N first feature images are output. This includes: adjusting the dimensions of the target image through each dimension adjustment module, and extracting features from the dimension-adjusted target image through each first feature extraction module to obtain N first feature images.

[0112] The backbone feature extraction network (corresponding to the target feature extraction network mentioned above) is described below using an optional method, such as... Figure 2 As shown, the backbone feature extraction network can include four ECSPDarkNet1-X modules (corresponding to the multiple first feature extraction modules mentioned above), such as... Figure 2 The backbone feature extraction networks, including ECSPDarkNet1-4, ECSPDarkNet1-9, ECSPDarkNet1-12, and ECSPDarkNet1-4, can also include five CBS modules (corresponding to the aforementioned multiple dimensionality adjustment modules). The ECSPDarkNet1-X module can be used to enhance multi-scale feature extraction while reducing model parameters, while the CBS modules can be used to adjust the dimensionality information between the input and output features of each layer. Figure 2 As shown, the CBS module can include ordinary convolution (Conv) (corresponding to the convolutional layer mentioned above), batch normalization (BN) (corresponding to the batch normalization layer mentioned above), and SiLU activation function (corresponding to the activation function layer mentioned above). Ordinary convolution (Conv) can be used for channel integration, batch normalization (BN) can be used to balance nonlinear feature flows while accelerating model training, and the SiLU activation function can be used to introduce nonlinear feature factors, improving the deep neural network's ability to express nonlinear features.

[0113] like Figure 2 As shown, the target feature map described above can be generated by sequentially inputting two 3D models. After the CBS module of step 3, it goes through an ECSPDarkNet1-4 module for processing, and then through a 3-step CBS module. After processing by the CBS module (3), the result is input into the ECSPDarkNet1-9 module. The output of the ECSPDarkNet1-9 module (one of the N first feature images mentioned above) is then input into the neck feature fusion network. Alternatively, the output of the ECSPDarkNet1-9 module can be processed by a 3-bit CBS module. After processing by the 3-bit CBS module, the result is input into the ECSPDarkNet1-12 module. The output of the ECSPDarkNet1-12 module (one of the aforementioned N first feature images) can be input into the neck feature fusion network. Alternatively, the output of the ECSPDarkNet1-12 module can be processed through a 3-bit CBS module. After processing by the CBS module 3, the result is input into another ECSPDarkNet1-4 module. The output of the ECSPDarkNet1-4 module (one of the above N first feature images) can be input into the neck feature fusion network, thereby achieving the technical effect of improving the ability of deep neural networks to express nonlinear features of the model.

[0114] Optionally, each first feature extraction module includes at least: multiple dimension adjustment submodules and a residual submodule. The residual submodule includes multiple residual units. Each dimension adjustment submodule has the same structure as the dimension adjustment module. Features are extracted from the dimension-adjusted target image through each first feature extraction module to obtain N first feature images. This includes: receiving a third feature image through the first feature extraction module, inputting the third feature image into the first dimension adjustment submodule through a first transmission channel, inputting the fourth feature image output by the first dimension adjustment submodule into the residual submodule, and outputting a first feature layer. The first feature extraction module is one of multiple first feature extraction modules, and the third feature image is the result of multiple first feature extraction modules and multiple... The target image is processed by one or more modules of the dimension adjustment module, and the first dimension adjustment submodule is one of the multiple dimension adjustment submodules; in the first transmission channel, the first feature layer is convolved and then input into the stacking unit; in the second transmission channel, the third feature image is convolved and then input into the stacking unit, and the feature data received by the stacking unit is stacked to obtain the second feature layer; the second feature layer is subjected to target processing to obtain the target feature image, wherein the target processing includes at least one of the following: normalization processing, activation function processing, feature dimension adjustment processing, and the target feature image is one of N first feature images; based on multiple target feature images, N first feature images are determined.

[0115] Figure 3 This is a structural diagram of an optional ECSPDarkNet1-X module according to an embodiment of the present invention, as shown below. Figure 3 As shown, the above feature extraction module (such as...) Figure 3 The ECSPDarkNet1-X module can include multiple dimension-adjusting submodules (such as...). Figure 3 CBS in the middle), residual submodules (such as Figure 3 The X residual units in the model can also include convolutional modules (such as...). Figure 3Conv in the middle), stacked modules (such as Figure 3 Concat in (and normalization modules such as) Figure 3 BN in (e.g., activation function modules) Figure 3 In addition to LeakyReLU, each CBS can also include: Conv (convolution), BN (normalization), and SiLU (activation function).

[0116] First, the input features (corresponding to the third feature image) of each ECSPDarkNet1-X module (corresponding to the first feature extraction module mentioned above) are divided into two parts by channel. (Corresponding to the first transmission channel mentioned above) is the backbone branch of the ECSPDarkNEt1-X module. It is first processed by the CBS module (corresponding to the first dimension adjustment submodule mentioned above) before entering the ResdualUnit residual module (corresponding to the residual submodule mentioned above). The ResdualUnit residual module outputs the feature layer (corresponding to the first feature layer mentioned above). For the cross-connection edge (corresponding to the second transmission channel mentioned above), and will The output after processing will be the feature layer (i.e., the feature layer obtained after convolution processing the first feature layer) and... The feature layer obtained after the third feature image is processed by ordinary convolution Conv (i.e., the feature layer obtained after convolution processing of the third feature image) is stacked or processed by channel through the Concat module (corresponding to the stacking unit mentioned above) to enrich the multi-scale feature information fusion. Then, the batch normalization (BN) operation (corresponding to the normalization processing mentioned above) is used to ensure the consistency of features, and the LeakyReLU activation function (corresponding to the activation function processing mentioned above) is used to nonlinearize the features. Finally, the CBS module is used to adjust the scale of the features (feature dimension adjustment processing) and it is used as the input of the next module.

[0117] The convolution processing described above can be ordinary convolution processing. As shown in equations (13) to (15), this is the forward propagation process of the ECSPDarkNet1-X module, and equations (16) to (18) are the backward propagation process. Where w is the weight parameter, x is the input feature, " "This is a convolution operation." The output features of the feature layer after being processed by the ordinary convolution operation Conv. The weighted output features of X residual modules For feature stacking, J Let be the objective function. g For gradient.

[0118] (13)

[0119] (14)

[0120] (15)

[0121] (16)

[0122] (17)

[0123] (18)

[0124] Such as the backpropagation process, (Second transmission channel) and (First transmission channel) The gradients of each of the two branches do not contain the gradients of the other branch, meaning there is no gradient reuse. Furthermore, features are propagated across different network branches by the ECSPDarkNet1-X module, reducing model training parameters while mitigating redundant gradient injection into the network. This allows the model to extract more effective features, thus improving the feature extraction efficiency of the object detection model.

[0125] Optionally, each residual unit includes at least: multiple dimension adjustment subunits and a weighting subunit. The fourth feature image output from the first dimension adjustment submodule is input into the residual submodule to output a first feature layer. This includes: adjusting the fourth feature image based on a first transmission subchannel and multiple dimension adjustment subunits to obtain a third feature layer, and inputting the third feature layer into the weighting subunit; inputting the fourth feature image into the weighting subunit through a second transmission subchannel; and performing weighted processing on the third feature layer and the fourth feature image through the weighting subunit to output the first feature layer.

[0126] like Figure 3 As shown, each residual unit in the above residual submodule may include: multiple dimension adjustment subunits (such as...) Figure 3 The two CBSs in the middle), weighted sub-units (such as Figure 3 (Add in the middle).

[0127] In this embodiment, the first branch (i.e., the first transmission sub-channel) in the residual module (i.e., the residual unit) first uses a convolution kernel with a size of 1. The CBS module (dimension adjustment subunit) of size 1 performs channel adjustment, and then a convolution kernel of size 3 is used. The CBS module (dimension adjustment sub-unit) in section 3 enhances feature extraction. It's worth noting that another edge in the residual module can serve as a residual edge (i.e., the second transmission sub-channel). Finally, to increase feature information without changing the dimension of the feature layer, the Add module (weighting sub-unit) can be used for weighting operations, outputting the first feature layer. The aforementioned first and second transmission sub-channels can be used to transmit the image features transmitted in the residual unit. This feature flow through the first and second transmission sub-channels improves the efficiency of feature flow in the object detection model, achieving the technical effect of improving the processing efficiency of the object detection model in image recognition.

[0128] Optionally, the feature fusion network includes at least: multiple second feature extraction modules, multiple feature fusion modules, and multiple feature fusion layers. Multiple residual units in the first feature extraction module are replaced with multiple dimension adjustment units as the second feature extraction modules. N first feature images have different image sizes. The N first feature images are input into the feature fusion network, and M second feature images are output. This includes: in the first feature fusion layer, the first feature image of a first size is input into the first feature fusion module, and the output of the first feature fusion module is input into the first extraction module to output a first image. The first feature fusion module is used to fuse the image features transmitted through the first feature flow path with the first feature image of the first size. The transmission direction of the image features transmitted through the first feature flow path is from the last feature fusion layer to the first feature fusion layer. The first extraction module consists of multiple second feature extraction modules. One of the modules, the first extraction module, is used to transmit the output of the first feature extraction module to the last feature fusion layer via the second feature flow path. The image feature transmission direction of the second feature flow path is from the first feature fusion layer to the last feature fusion layer. In the second feature fusion layer, based on the third and fourth transmission channels, the first feature image of the second size is fused with the image features transmitted by the first feature flow path, and the fused result is fused with the image features transmitted by the second feature flow path to determine the second image. In the last feature fusion layer, based on the fifth and sixth transmission channels, the first feature image of the third size is fused with the features transmitted by the second feature flow path after pyramid pooling to determine the third image. Based on the first image, the second image, and the third image, M second feature images are determined.

[0129] like Figure 2As shown, the ECSPDarkNet2-X module (corresponding to the second feature extraction module mentioned above) can be applied to the neck feature fusion network (corresponding to the feature fusion network mentioned above). In related technologies, gradient reuse is one of the main reasons for increasing model inference cost. Therefore, in this embodiment, the ECSPDarkNet2-X module is proposed for use in the neck feature fusion network to improve the efficiency of multi-scale feature fusion while accelerating the flow of multi-scale features in the network. Figure 4 This is a structural diagram of an optional ECSPDarkNet2-X module according to an embodiment of the present invention, as shown below. Figure 4 As shown, unlike ECSPDarkNet1-X, ECSPDarkNet2-X uses X convolutional kernels with sizes of 1... 1 and 3 The purpose of replacing the CBS (corresponding to the multiple dimension adjustment units mentioned above) with 3 is to reduce feature computation and speed up feature flow.

[0130] Similar to the ECSPDarkNet1-X module, this embodiment, in order to accelerate feature flow and reduce model parameters, still employs cross-link edge operations (multiple transmission channels) in the ECSPDarkNet2-X module to reduce the model's utilization of repeated gradients. The forward propagation process of the ECSPDarkNet2-X module is shown in equations (19) to (21), and the backward propagation process is shown in equations (22) to (24). Let's denote... and For cross-connecting edge branches and backbone branches, where, w For weight parameters, x For input features, "This is a convolution operation." Let X be the output characteristics of the combined CBS modules. The output features of the feature layer after being processed by the ordinary convolution operation Conv. For feature stacking, J Let be the objective function. g For gradient.

[0131] (19)

[0132] (20)

[0133] (twenty one)

[0134] (twenty two)

[0135] (twenty three)

[0136] (twenty four)

[0137] As can be seen from the backpropagation process, the gradients of the backbone branches and cross-link edges are integrated separately. When processing gradients, only the gradients belonging to each individual are updated. This indicates that the use of the ECSPDarkNet2-X module can reduce the blocking of the network by repeated gradients and enhance the multi-scale fusion of features in the neck feature fusion network from bottom to top, from top to bottom, and with lateral connections. This can alleviate the consumption of computing power and accelerate the flow of features in the network.

[0138] like Figure 5 The diagram illustrates the multi-scale feature fusion process of the neck feature fusion network. The neck feature fusion network utilizes four NFPN modules (corresponding to the aforementioned feature fusion modules) and four ECSPDarkNet2-X modules (corresponding to the aforementioned second feature extraction modules). The neck feature fusion network may also include an SPPF module (i.e., a pyramid pooling operation module), such as... Figure 2 As shown, the SPPF module described above may include: two CBSs, a Concat (stacked module), and a 5-bit module. Maxpool (maximum pooling module) of 5, a 9 9 Maxpool and a 13 The network consists of a 13-dimensional Maxpool, where CBS can be composed of a Conv (ordinary convolution), BN (normalization), and SiLU (activation function). Since single-input and intermediate operations without feature fusion have little effect on feature fusion, in this embodiment, these operations are first removed from the neck feature fusion network. Then, cross-connection edges (transmission channels) are added between input and output features to reduce model computational consumption while ensuring the fusion of more effective features. Furthermore, the fusion of shallow and deep features is one of the main reasons for increasing model size. Therefore, in the neck feature fusion network of this embodiment, an NFPN module can be used to set learnable weight parameters for each input feature, allowing the model to learn and distinguish the importance of features. Then, an ECSPDarkNet2-X module is used after the NFPN module to accelerate the propagation of more effective multi-scale features in the network.

[0139] like Figure 5 They can be recorded separately. (Corresponding to the first feature image of the first size mentioned above) (corresponding to the first feature image of the second size mentioned above) and (The first feature image corresponding to the third size mentioned above) represents the feature maps at three scales from the backbone feature extraction network, and their corresponding output features are denoted as follows: (Corresponding to the first image above) (Corresponding to the second image above) and (Corresponding to the third image above), the features are, in order, a small target feature map, a medium target feature map, and a large target feature map. In this embodiment, the ECSPDarkNet2-X module can be denoted as... The output of the NFPN module is denoted as , , UpSampling This is an upsampling operation.

[0140] As shown in formulas (25) to (31), this describes the process of generating feature maps for small, medium, and large targets. and For learnable weight parameters, We set the value to 0.0001 to prevent division by zero. Analysis shows that NFPN can be used to set learnable parameters for multi-scale features, then perform multi-scale fusion, and finally process the data through convolution as the input for the next layer.

[0141] from Figure 5 Analysis shows that, These are shallow features from the backbone feature extraction network, which serve as the features passed down from the top-down path and the small target input features of the neck feature fusion network, respectively. As shown in equations (25) to (26), in During the formation process, the NFPN module will Features from the bottom-up path (corresponding to the first feature flow path mentioned above) in the neck feature fusion network are fused at multiple scales. Finally, the fused features are processed by the ECSPDarkNet2-X module (corresponding to the first extraction module mentioned above) to form a small target detection feature map. (i.e., the first image).

[0142] (25)

[0143] (26)

[0144] From the analysis of formulas (27) to (29), we can see that... Features are passed to the top-down path and neck feature fusion networks, respectively. After processing along a top-down path, the result is... The path for transmitting features into the neck feature fusion network is through two lateral channels (the third and fourth transmission channels) across the connecting edges. In the formation process (corresponding to the second image above), firstly Multi-scale features, sequentially transmitted from the bottom-up path (corresponding to the first feature flow path mentioned above) and the top-down path (corresponding to the second feature flow path mentioned above) in the neck feature fusion network, are fused at multiple scales through two NFPN modules, thereby forming... and ,Then and The propagation continues along the bottom-up and top-down paths in the neck feature fusion network, respectively, and finally... After processing by the ECSPDarkNet2-X module, the mid-target detection feature map is obtained. .

[0145] (27)

[0146] (28)

[0147] (29)

[0148] The formation process (corresponding to the third image above) is shown in formulas (30) to (31). First, After processing by the Spatial Pyramid Pooling (SPPF) module, features are sequentially transferred in the neck feature fusion network via a bottom-up path and across connection edge channels (the fifth and sixth transmission channels). The features are passed through the lateral connection channel to the neck feature fusion network, and then fused with the multi-scale features passed down from the top-down path in the neck feature fusion network (corresponding to the second feature flow path mentioned above) using the NFPN module. Finally, the features are processed by the ECSPDarkNet2-X module (the third extraction module) to obtain the large target detection feature map (corresponding to the third image mentioned above). .

[0149] (30)

[0150] (31)

[0151] Optionally, in the second feature fusion layer, based on the third and fourth transmission channels, the first feature image of the second size is fused with the image features transmitted through the first feature flow path, and the fused result is fused with the image features transmitted through the second feature flow path to determine the second image. This includes: in the second feature fusion layer, transmitting the first feature image of the second size to the second feature fusion module through the third transmission channel, performing multi-scale fusion with the image features transmitted through the first feature flow path to obtain a first fused feature; inputting the first fused feature into the second extraction module to obtain a second fused feature, wherein the second extraction module is one of multiple second feature extraction modules; transmitting the first feature image of the second size to the third feature fusion module through the fourth transmission channel, performing multi-scale feature fusion with the features transmitted through the second feature flow path and the second fused feature to determine a third fused feature; inputting the third fused feature into the third extraction module to output the second image, wherein the third extraction module is one of multiple second feature extraction modules.

[0152] From the analysis of formulas (27) to (29), we can see that... Features are passed to the top-down path and neck feature fusion networks, respectively. After processing along a top-down path, the result is... The path for transmitting features into the neck feature fusion network is through two lateral channels (the third and fourth transmission channels) across the connecting edges. In the formation process (corresponding to the second image above), firstly Multi-scale features, sequentially transmitted from the bottom-up path (corresponding to the first feature flow path mentioned above) and the top-down path (corresponding to the second feature flow path mentioned above) in the neck feature fusion network, are fused at multiple scales through two NFPN modules, thereby forming... and ,Then and The propagation continues along the bottom-up path (first feature flow path) and the top-down path (second feature flow path) in the neck feature fusion network, respectively. After processing by the ECSPDarkNet-X module, the mid-target detection feature map is obtained. This achieves the technical effect of improving the fusion efficiency of multi-scale feature fusion.

[0153] Optionally, after performing pyramid pooling on the first feature image of the third size, the method further includes: transmitting the result obtained after performing pyramid pooling on the first feature image of the third size to the first feature fusion layer through the first feature flow path.

[0154] In this implementation, The formation process (corresponding to the third image above) is shown in formulas (30) to (31). First, After processing by the Spatial Pyramid Pooling (SPPF) module, features are sequentially transferred in the neck feature fusion network through a bottom-up path (corresponding to the first feature flow path mentioned above) and cross-connection edge channels (the fifth and sixth transmission channels). The features are passed through the lateral connection channel to the neck feature fusion network, and then fused with the multi-scale features passed down from the top-down path in the neck feature fusion network (corresponding to the second feature flow path mentioned above) using the NFPN module. Finally, the features are processed by the ECSPDarkNet2-X module (the third extraction module) to obtain the large target detection feature map (corresponding to the third image mentioned above). The feature flow rate in the neck feature fusion network is improved by transmitting the feature to the SPPF module in the first feature flow path through the lateral connection channel, or through the bottom-up path.

[0155] like Figure 5 As shown, the bottom-up path in the neck feature fusion network can be SPPF->1 1CBS->UpSampling->NFPN->ECSDarkNet2-4->1 The top-down path in the neck feature fusion network can be CBS -> UpSampling -> NFPN; it can be ECS DarkNet2-4 -> 3. 3CBS->NFPN->ECSDarkNet2-4->3 3's CBS->NFPN->ECSDarkNet2-4.

[0156] Optionally, the object detection model is determined as follows: Pre-trained weights and training samples are obtained, and the training samples are divided into a training set, a validation set, and a test set. The training samples include multiple images, each with a bounding box labeled with the target. An initial detection model is trained based on the pre-trained weights and the training set, and cross-validation is used during training to verify whether the initial detection model has converged using the validation set. The initial detection model is an untrained model. If the initial detection model converges, the object detection model is determined, and its detection accuracy is tested based on the weight file and the test set. The weight file stores multiple weights of the object detection model.

[0157] The model training process in this embodiment will be described below using an optional method:

[0158] (1) Setting up the environment for the target detection algorithm (corresponding to the target detection model mentioned above).

[0159] In this embodiment, Pychrm and Anaconda can be used as script editing tools, Python 3.7 can be used as the scripting language, the GPU parallel computing tool can be RTX 2070 Super, the GPU accelerator can be CUDA 10.1, and Logitech Brio 500 can be used as the sample collector.

[0160] (2) Create an image sample dataset.

[0161] In this embodiment, the target detection algorithm can use the VOC format to create the dataset. First, a VOC file can be created in the folder as the dataset storage source. Then, the LabelImg tool is used to annotate each target in each image of the dataset with a ground truth bounding box (corresponding to the annotation box mentioned above), and a corresponding XML file is generated to store the target's source file path, center point, width, height, and other attributes. Finally, Annotation and Image files are created under the VOC file to store the XML file and the annotated image, respectively.

[0162] Once the dataset (corresponding to the training samples mentioned above) is created, it can first be divided into training, validation, and test sets. Then, under the Annotation folder, create Training-Annotation, Val-Annotation, and Test-Annotation files to store the XML files for the training, validation, and test sets, respectively. Finally, under the Image folder, create Training-Image, Val-Image, and Test-Image files to store the image samples for the training, validation, and test sets, respectively. Typically, the random partitioning ratio of the training, validation, and test sets can be a preset ratio, such as 0.6:0.2:0.2.

[0163] (3) Model training.

[0164] Model training process as follows Figure 6 As shown, after the environment setup and dataset creation are complete, the model is trained. First, the pre-trained weight PT file is loaded to shorten training time and improve model detection accuracy; the final trained weight parameters are stored in the PT file. Then, the model configuration file is modified to determine the number of target classes and the input image training size. Finally, the hyperparameters required for model training are adjusted to determine the model training and optimization strategies. Cross-validation training is then performed using a validation set during the training process.

[0165] (4) Model testing and evaluation.

[0166] Model testing process as follows Figure 7 As shown, the final weight PT file will be generated after model training is complete. First, load the weight PT file and modify the model test configuration file. Then, set the batch size and adjust the input size of the test images. Finally, the confidence threshold can be set to 0.5 and... IOU A threshold of 0.5 is used to filter predicted bounding boxes, and the predicted bounding box with the highest confidence is output as the detection result after the final test.

[0167] To verify the effectiveness of the model, this embodiment evaluates the model on a test set, which can be done using accuracy. P Recall rate R Mean accuracy AP and mean precision mAP Evaluation metrics were used to analyze the model, and finally, visualization analysis and ablation experiments were used to verify the effectiveness of the model.

[0168] As shown in formula (32), the accuracy is... P .in, TP This indicates that the sample was actually a positive example, and the test result was also a positive example; FP This indicates that the sample was actually a negative example, but the test result was a positive example; TP + FP This represents the positive examples detected by the model.

[0169] Therefore, accuracy represents the model's ability to identify true cases.

[0170] (32)

[0171] As shown in formula (33), this represents the recall rate. R .in, FN This indicates that the sample was actually a negative example, and the test result was also a negative example; FP + FN This represents all positive examples in the test set. Therefore, recall characterizes the model's ability to detect actual positive examples as positive examples.

[0172] (33)

[0173] As shown in formula (34), this is the mean accuracy. AP It represents the average precision as the recall varies from 0 to 1, which is the model's detection accuracy for various categories.

[0174] (34)

[0175] As shown in formula (35), this represents the mean of the average precision. mAP .in, N This represents the total number of all targets across all categories, i.e., the total number of true bounding boxes across all categories. mAP This characterizes the model's ability to detect all categories.

[0176] (35)

[0177] like Figure 8 The figure shows the loss function variation curves for three different training strategies using cross-validation in this embodiment. The training epochs are 200. Analysis shows that the model in this embodiment trains smoothly on the PASCAL VOC 2012 dataset and eventually reaches a stable value.

[0178] Additionally, the PR curve of the model in this embodiment on the PASCAL VOC 2012 test set can be used to determine the performance of the object detection model in this embodiment. IOU When the value is 0.5, mAP The optimal value of 0.841 can be obtained, which means that the target detection model in this embodiment has high detection accuracy for the entire PASCAL VOC 2012 (a target detection dataset) dataset.

[0179] Optionally, the target detection method that enhances multi-scale feature extraction, reuse, and fusion further includes: during the training of the initial detection model, receiving the fused feature map output by the feature fusion network through the target prediction network; dividing the fused feature map into multiple grids, wherein the fused feature map is marked with the target's bounding box, and each grid includes multiple anchor boxes; finding the target grid in the multiple grids where the bounding box in the fused feature map is located; determining the prediction box based on the intersection-union ratio (IU) between each anchor box in the target grid and the target's bounding box; and determining the detection result of the initial detection model based on the prediction box.

[0180] In this embodiment, the target prediction network can generate , and Three types of feature maps. These feature maps are divided into three categories. There are 1 grid, and each grid generates B anchor boxes. When the center point of the ground truth bounding box of a target falls within a grid point, the target is matched with the ground truth bounding box (corresponding to the labeled box mentioned above) in the B anchor boxes generated by that grid. IOU The largest anchor box is responsible for detecting the target. The network obtains predicted boxes by updating the anchor box position parameters. Finally, the predicted box that is closest to the ground truth box is selected as the detection result by the confidence score and the NMS algorithm. In this embodiment, all predicted boxes can be compared with the ground truth boxes. IOU The largest predicted bounding box is used as a positive sample, along with the image background and all boxes between it and the ground truth bounding boxes. IOU Anchor frames with a value less than 0.5 are considered negative samples.

[0181] Optionally, the step of determining the predicted bounding box based on the intersection-union ratio (IU) of each anchor box in the target grid with the target's labeled bounding box includes: calculating the IU of each anchor box in the target grid with the target's labeled bounding box to obtain multiple ratios; using the anchor box associated with the largest ratio among the multiple ratios as the target anchor box, wherein the target anchor box is used to detect the target on the fused feature map; and updating the position parameters of the target anchor box to obtain the predicted bounding box.

[0182] During the training of the initial model, this embodiment can generate [the desired result] in the target prediction network. , and Three types of feature maps. These feature maps can be divided into: There are 1 grid, and each grid generates B anchor boxes. When the center point of the ground truth bounding box of a target falls within a grid point, the target is matched with the ground truth bounding box (corresponding to the labeled box mentioned above) in the B anchor boxes generated by that grid. IOU The anchor box with the highest intersection-over-union (IoU) ratio is responsible for detecting the target. The network obtains predicted boxes by updating the anchor box position parameters. Finally, the predicted box that is closest to the ground truth box is selected as the detection result by the confidence score and the NMS algorithm. In addition, in this embodiment, all predicted boxes can be compared with the ground truth boxes. IOU The largest predicted bounding box is used as a positive sample, along with the image background and all boxes between it and the ground truth bounding boxes. IOU Anchor frames with an intersection-to-union ratio (CRO) less than 0.5 are considered negative samples.

[0183] Where 75 is the product of 3 and 25, where 3 generates 3 anchor boxes per grid, 25 is the sum of 20, 4, and 1, where 20 is the number of classes in the PASCAL VOC dataset, and 4 represents the four location parameters of the predicted box. , , and 1 represents the confidence score. , represents the confidence level that the target within the current prediction box belongs to a certain category, as shown in formula (36). This represents the confidence level of the j-th predicted bounding box within the i-th grid point, when the grid point contains the target. It is 1 if it is true, otherwise it is 0. This represents the probability that an object within a grid point belongs to a certain category. This represents the overlap rate between the true bounding boxes and the predicted bounding boxes.

[0184] (36)

[0185] like Figure 9As shown, this is the process of adjusting the anchor frame to the prediction frame. The specific process is shown in formulas (37) to (40).

[0186] exist Figure 9 In this diagram, each grid point has a width and height of 1. Dashed boxes represent anchor frames, and solid boxes represent prediction frames. The width of the anchor frame is... Gao Wei The normalized center point, width, and height of the prediction box are respectively... , and ; for sigmoid Logical constraint functions are used to... and The constraint is between 0 and 1, therefore and This is the offset of the center point of the prediction box relative to the top left corner of the grid. (1,1); , , and The parameters that need to be continuously learned and updated in the target prediction network are the center point, width, and height of the predicted bounding box relative to the feature map. Furthermore, the coefficients 2 and 0.5 are primarily used to eliminate grid sensitivity and fix the relative offset within the range of (-0.5, 1.5).

[0187] (37)

[0188] (38)

[0189] (39)

[0190] (40)

[0191] By adjusting the position parameters of the anchor boxes, the prediction closest to the true bounding box is determined, and thus the detection result is determined. This achieves the technical effect of improving the image recognition accuracy of the target detection model.

[0192] Optionally, the loss function used when training the initial detection model may include at least: a classification loss function, a confidence loss function, and a localization loss function.

[0193] In this embodiment, the loss function of the corresponding object detection framework (or object detection model) It can be determined by the classification loss function. Target confidence loss function (Corresponding to the confidence loss function and localization loss function mentioned above) Composition, loss function As shown in formula (41).

[0194] As shown in formula (42), the classification error can be expressed as the binary cross-entropy function as the classification loss function, that is, when the first... i The first grid j When an anchor box is responsible for detecting a specific object, the predicted bounding box generated by that anchor box is used to calculate the classification loss. The feature map is divided into... There are grids, and the number of target categories is . c , Used to determine whether it is a positive sample, i.e., the first... i The first grid generated by the first j Anchor frame and the real frame IOU At its maximum, this anchor frame is responsible for target detection. =1, otherwise =0, Indicates the first i The corresponding grid number is the first j The target category in the anchor box is the first c The probability of class targets, Indicates the first i The corresponding grid number is the first j Is the target in the anchor box the first one? c If it is a target class, then at this time =1, otherwise =0.

[0195] (41)

[0196] (42)

[0197] As shown in formula (43), the target confidence loss function can be composed of two sets of binary cross-entropy functions. The first set is the positive sample confidence loss function, and the second set is the negative sample confidence loss function. If the confidence of the positive sample prediction box is closer to 1, then the positive sample confidence loss function value is closer to 0, which means that the confidence level of the prediction box being a certain type of target is greater. Used to balance the distribution of positive and negative samples Indicates the first i The corresponding grid number is the first j If the target of each anchor box is a negative sample, then... =1, otherwise =0, Indicates the first i The corresponding grid number is the first j Does each anchor box handle target detection? If so, then... =1, otherwise =0, Indicates the first i The corresponding grid number is the first j The predicted value generated after the anchor frames are finally formed into a prediction frame.

[0198] (43)

[0199] As shown in formula (44), the localization loss function is derived from the overlap loss function. Center distance loss function and width and height loss function The composition is as follows: Overlap loss function and center distance loss function are used to eliminate convergence speed issues caused by neglecting the overlap area and center point distance, thus optimizing the loss function in the direction of increasing overlap area, as shown in formulas (45) and (46), respectively. The intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box. Let Euclidean distance be the center point of the predicted bounding box and the ground truth bounding box. c It is the diagonal length of the smallest bounding box between the predicted box and the ground truth box.

[0200] (44)

[0201] (45)

[0202] (46)

[0203] Figure 10 A schematic diagram of an optional prediction box regression process according to an embodiment of the present invention, in Figure 10 middle, pred For the prediction box, truth The dashed box represents the true bounding box, while the dashed box represents the smallest outer bounding box. The center point of the prediction box, The center point of the ground truth bounding box is the center point of the ground truth bounding box of the target in each image. The shaded area is the area where the predicted bounding box overlaps with the ground truth bounding box. w and h These are the width and height of the predicted bounding box, respectively. and The width and height of the actual bounding box. and The width and height of the minimum bounding box.

[0204] To prevent the aspect ratio of the predicted bounding box from becoming non-linear with respect to the ground truth bounding box (i.e., to allow the width and height of the predicted bounding box to increase or decrease simultaneously), this embodiment employs a width-height loss function. This minimizes the difference in width and height between the predicted bounding box and the ground truth bounding box, thereby accelerating model convergence and achieving more accurate localization. (Width and height loss function) As shown in formula (47). Wherein, This is the difference in width between the predicted bounding box and the ground truth bounding box. This represents the difference in height between the predicted bounding box and the ground truth bounding box.

[0205] (47)

[0206] Furthermore, since there are more samples with poor image quality than samples with high image quality during training, and the poor image quality samples will produce a greater error affecting model optimization, resulting in an imbalance of training samples during training, this embodiment resets the weights of the localization loss function, as shown in formula (48). Taking 0.5, the final localization loss function is shown in formula (49).

[0207] (48)

[0208] (49)

[0209] The loss function in this embodiment achieves the technical effect of improving the detection accuracy of the target detection model.

[0210] As shown in Table 1, the target detection model in this embodiment was compared with the more mainstream target detection algorithms on the PASCAL VOC 2012 (target detection dataset) test set in terms of test time and frame rate. FPS and mAP@.5 (i.e. IOU When set to 0.5, the average value of all images in each category is... AP (Average accuracy) comparison. Analysis shows that this embodiment achieved the minimum test time for a single image, and the frame rate... FPS The score reached 98, indicating that this embodiment meets the real-time requirements. Furthermore, mAP @.5 also reached the optimal value, demonstrating the effectiveness of the object detection model in this embodiment. In Table 1, YOLOv4, YOLOv3, SSD, Faster R-CNN, TridentNet, RetinaNet, CornerNet, ExtremeNet, and CenterNet represent different types of deep learning-based detection models in related technologies.

[0211] Table 1

[0212]

[0213] As shown in Table 2, this embodiment also uses ablation experiments to verify the effectiveness of some of the proposed strategies. Analysis shows that the average accuracy in the basic model without the three strategies is [missing information]. APAverage recall rate AR and mAP @.5 reached the lowest value, but the experiments in groups 2, 4, 7 and 8 demonstrated the feasibility and effectiveness of the three strategies of this invention.

[0214] Table 2

[0215]

[0216] Example 2

[0217] Embodiment 2 of this application provides an optional target detection device that enhances multi-scale feature extraction, reuse and fusion, wherein each implementation unit in the recognition device corresponds to each implementation step in Embodiment 1.

[0218] Figure 11 This is a schematic diagram of an optional target detection device for enhancing multi-scale feature extraction, reuse, and fusion according to an embodiment of the present invention, as shown below. Figure 11 As shown, the identification device includes a first processing unit 111, a second processing unit 112, and a third processing unit 113.

[0219] Specifically, the first processing unit 111 is used to respond to the recognition request of the target image, input the target image into the target feature extraction network of the target detection model, and output N first feature images. The target detection model includes at least: a target feature extraction network, a feature fusion network and a target prediction network. The target detection model includes: multiple transmission channels, which are used to transmit the image features of the target image, and N is an integer greater than 1.

[0220] The second processing unit 112 is used to input N first feature images into a feature fusion network and output M second feature images, wherein the feature fusion network is used to perform multi-scale feature fusion on the N first feature images, and M is an integer greater than 1;

[0221] The third processing unit 113 is used to input M second feature images into the target prediction network and output the recognition result of the target image, wherein the recognition result includes the classification result of the target in the target image.

[0222] In the enhanced multi-scale feature extraction, reuse, and fusion target detection device provided in Embodiment 2 of this application, the first processing unit 111 responds to the recognition request of the target image, inputs the target image into the target feature extraction network of the target detection model, and outputs N first feature images. The target detection model includes at least a target feature extraction network, a feature fusion network, and a target prediction network. The target detection model includes multiple transmission channels for transmitting image features of the target image, where N is an integer greater than 1. The second processing unit 112 inputs the N first feature images into the feature fusion network, outputting M second feature images. The feature fusion network performs multi-scale feature fusion on the N first feature images, where M is an integer greater than 1. The third processing unit 113 inputs the M second feature images into the target prediction network, outputting the recognition result of the target image. The recognition result includes the classification result of the target in the target image. This solves the technical problem of low target recognition efficiency in the target detection process due to complex model structure, high computational cost, and lack of specificity for salient features in target detection algorithms. In this embodiment, the target detection model with multiple transmission channels is used to classify and identify targets in the target image, avoiding the situation in related technologies where the detection model has a complex structure and slow feature flow speed, resulting in low recognition efficiency. This improves the technical effect of the target detection model in classifying and identifying targets in the image.

[0223] Optionally, in the target detection device for enhanced multi-scale feature extraction, reuse and fusion provided in Embodiment 2 of this application, the target feature extraction network includes at least: multiple first feature extraction modules and multiple dimension adjustment modules. Each dimension adjustment module includes at least: a convolutional layer, a batch normalization layer and an activation function layer. The first processing unit includes: a first processing subunit, used to perform dimension adjustment on the target image through each dimension adjustment module and to extract features from the dimension-adjusted target image through each first feature extraction module to obtain N first feature images.

[0224] Optionally, in the enhanced multi-scale feature extraction, reuse, and fusion target detection device provided in Embodiment 2 of this application, each first feature extraction module includes at least: multiple dimension adjustment sub-modules and residual sub-modules. The residual sub-module includes multiple residual units. Each dimension adjustment sub-module has the same structure as the dimension adjustment module. The processing sub-unit includes: a first processing module, used to receive a third feature image through the first feature extraction module, input the third feature image into the first dimension adjustment sub-module through a first transmission channel, input the fourth feature image output by the first dimension adjustment sub-module into the residual sub-module, and output a first feature layer. The first feature extraction module is one of multiple first feature extraction modules, and the third feature image is one or more of the images processed by multiple first feature extraction modules and multiple dimension adjustment modules. The target image processed by each module includes: a first dimension adjustment submodule, which is one of multiple dimension adjustment submodules; a second processing module, which performs convolution processing on the first feature layer in the first transmission channel and then inputs it into the stacking unit; a third processing module, which performs convolution processing on the third feature image in the second transmission channel and then inputs it into the stacking unit, and performs stacking processing on the feature data received by the stacking unit to obtain a second feature layer; a fourth processing module, which performs target processing on the second feature layer to obtain a target feature image, wherein the target processing includes at least one of the following: normalization processing, activation function processing, feature dimension adjustment processing, and the target feature image is one of N first feature images; and a determination module, which determines N first feature images based on multiple target feature images.

[0225] Optionally, in the enhanced multi-scale feature extraction, reuse, and fusion target detection device provided in Embodiment 2 of this application, each residual unit includes at least: multiple dimension adjustment sub-units and weighting sub-units. The first processing module includes: an adjustment sub-module, used to perform channel adjustment on the fourth feature image based on the first transmission sub-channel and the multiple dimension adjustment sub-units to obtain a third feature layer, and input the third feature layer into the weighting sub-unit; a first input module, used to input the fourth feature image into the weighting sub-unit through the second transmission sub-channel; and a weighting module, used to perform weighted processing on the third feature layer and the fourth feature image through the weighting sub-unit, and output the first feature layer.

[0226] Optionally, in the enhanced multi-scale feature extraction, reuse, and fusion target detection device provided in Embodiment 2 of this application, the feature fusion network includes at least: multiple second feature extraction modules, multiple feature fusion modules, and multiple feature fusion layers. Multiple residual units in the first feature extraction module are replaced with multiple dimension adjustment units as the second feature extraction modules. The N first feature images have different image sizes. The second processing unit includes: a second processing subunit, used to input the first feature image of the first size into the first feature fusion module in the first feature fusion layer, and input the output of the first feature fusion module into the first extraction module to output the first image. The first feature fusion module is used to fuse the image features transmitted through the first feature flow path with the first feature image of the first size. The transmission direction of the image features transmitted through the first feature flow path is from the last feature fusion layer to the first feature fusion layer. The first extraction module is one of multiple second feature extraction modules. The first extraction module is used to transmit the output of the first feature extraction module to the last feature fusion layer via the second feature flow path. The image feature transmission direction of the second feature flow path is from the first feature fusion layer to the last feature fusion layer. The first feature fusion layer is connected to the last feature fusion layer; the third processing subunit is used in the second feature fusion layer to perform feature fusion of the first feature image of the second size with the image features transmitted through the first feature flow path based on the third and fourth transmission channels, and to fuse the result of feature fusion with the image features transmitted through the second feature flow path to determine the second image; the fourth processing subunit is used in the last feature fusion layer to perform feature fusion of the first feature image of the third size with the features transmitted through the second feature flow path based on the fifth and sixth transmission channels to determine the third image; the determination subunit is used to determine M second feature images based on the first image, the second image and the third image.

[0227] Optionally, in the enhanced multi-scale feature extraction, reuse, and fusion target detection device provided in Embodiment 2 of this application, the third processing subunit includes: a first fusion module, used to transmit a first feature image of a second size to the second feature fusion module through a third transmission channel in the second feature fusion layer, and perform multi-scale fusion with the image features transmitted through the first feature flow path to obtain a first fused feature; an input module, used to input the first fused feature into a second extraction module to obtain a second fused feature, wherein the second extraction module is one of a plurality of second feature extraction modules; a second fusion module, used to transmit the first feature image of a second size to the third feature fusion module through a fourth transmission channel, and perform multi-scale feature fusion with the features transmitted through the second feature flow path and the second fused feature to determine a third fused feature; and an input / output module, used to input the third fused feature into the third extraction module and output a second image, wherein the third extraction module is one of a plurality of second feature extraction modules.

[0228] Optionally, in the target detection device for enhanced multi-scale feature extraction, reuse and fusion provided in Embodiment 2 of this application, the second processing unit further includes: a transmission subunit, used to transmit the result obtained after the pyramid pooling operation of the first feature image of the third size to the first feature fusion layer through the first feature flow path after the pyramid pooling operation of the first feature image of the third size.

[0229] Optionally, in the enhanced multi-scale feature extraction, reuse, and fusion target detection device provided in Embodiment 2 of this application, the target detection model is determined by the following units: a fourth processing unit, used to acquire pre-trained weights and training samples, and divide the training samples into a training set, a validation set, and a test set, wherein the training samples include: multiple images, each image having a bounding box labeled with the target; a training unit, used to train the initial detection model based on the pre-trained weights and the training set, and to use cross-validation during the training process to verify whether the initial detection model has converged through the validation set, wherein the initial detection model is an untrained model; and a testing unit, used to determine the target detection model when the initial detection model has converged, and to test the detection accuracy of the initial detection model based on the weight file and the test set, wherein the weight file is used to store multiple weights of the target detection model.

[0230] Optionally, in the enhanced multi-scale feature extraction, reuse, and fusion target detection apparatus provided in Embodiment 2 of this application, the enhanced multi-scale feature extraction, reuse, and fusion target detection method further includes: a receiving subunit, used to receive the fused feature map output by the feature fusion network through the target prediction network during the training process of the initial detection model; a partitioning subunit, used to divide the fused feature map into multiple grids, wherein the fused feature map is marked with the target's bounding box, and each grid includes multiple anchor boxes; a searching subunit, used to search for the target grid in the fused feature map where the bounding box is located; a first determining subunit, used to determine the prediction box based on the intersection-union ratio of each anchor box in the target grid and the target's bounding box; and a second determining subunit, used to determine the detection result of the initial detection model through the prediction box.

[0231] Optionally, in the target detection device for enhanced multi-scale feature extraction, reuse and fusion provided in Embodiment 2 of this application, the first determining subunit includes: a calculation module, used to calculate the intersection-union ratio (IUU) of each anchor box in the target grid with the target's labeled box, to obtain multiple ratios; an anchor box processing module, used to associate the anchor box with the largest ratio among the multiple ratios as the target anchor box, wherein the target anchor box is used to detect the target on the fused feature map; and an update module, used to update the position parameters of the target anchor box to obtain a predicted box.

[0232] Optionally, in the target detection device for enhanced multi-scale feature extraction, reuse and fusion provided in Embodiment 2 of this application, the loss function used when training the initial detection model includes at least: a classification loss function, a confidence loss function and a localization loss function.

[0233] The aforementioned enhanced multi-scale feature extraction, reuse and fusion target detection device may also include a processor and a memory. The first processing unit 111, the second processing unit 112 and the third processing unit 113 are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0234] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, a target detection model with multiple transmission channels can be used to classify and identify targets in an image. This avoids the problems of complex detection models and slow feature flow in related technologies, which lead to low target recognition efficiency, thereby improving the technical effectiveness of the target detection model in classifying and recognizing targets.

[0235] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0236] According to another aspect of the present invention, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the enhanced multi-scale feature extraction, reuse and fusion target detection method of any one of the above-mentioned methods by executing the executable instructions.

[0237] According to another aspect of the present invention, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer program controls the device where the computer-readable storage medium is located to execute the enhanced multi-scale feature extraction, reuse and fusion target detection method described above when the computer program is running.

[0238] Figure 12 This is a schematic diagram of an electronic device according to an embodiment of the present invention, such as... Figure 12 As shown, this embodiment of the invention provides an electronic device 120, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the enhanced multi-scale feature extraction, reuse, and fusion target detection method described above.

[0239] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0240] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0241] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0242] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0243] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0244] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0245] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A target detection method that enhances multi-scale feature extraction, reuse, and fusion, characterized in that, include: In response to a request to recognize a target image, the target image is input into the target feature extraction network of the target detection model, and N first feature images are output. The target detection model includes at least the target feature extraction network, the feature fusion network, and the target prediction network. The target detection model includes multiple transmission channels for transmitting the image features of the target image, where N is an integer greater than 1. The target feature extraction network includes at least: multiple first feature extraction modules and multiple dimension adjustment modules. Each dimension adjustment module includes at least: a convolutional layer, a batch normalization layer, and an activation function layer. The target image is input into the target feature extraction network of the target detection model, and N first feature images are output. This includes: adjusting the dimensions of the target image through each dimension adjustment module, and extracting features from the dimension-adjusted target image through each first feature extraction module to obtain N first feature images. Each first feature extraction module includes at least: multiple dimension adjustment submodules and a residual submodule. The residual submodule includes multiple residual units. Each dimension adjustment submodule has the same structure as the dimension adjustment module. The first feature extraction module extracts features from the dimension-adjusted target image to obtain N first feature images. This includes: receiving a third feature image through the first feature extraction module, inputting the third feature image into the first dimension adjustment submodule through a first transmission channel, inputting a fourth feature image output by the first dimension adjustment submodule into the residual submodule, and outputting a first feature layer. The first feature extraction module is one of multiple first feature extraction modules, and the third feature image is the result of processing multiple first feature extraction modules and multiple residual units. The target image is processed by one or more modules of the dimension adjustment module, wherein the first dimension adjustment submodule is one of the plurality of dimension adjustment submodules; in the first transmission channel, the first feature layer is convolved and then input into the stacking unit; in the second transmission channel, the third feature image is convolved and then input into the stacking unit, and the feature data received by the stacking unit is stacked to obtain a second feature layer; the second feature layer is subjected to target processing to obtain a target feature image, wherein the target processing includes at least one of the following: normalization processing, activation function processing, feature dimension adjustment processing, and the target feature image is one of N first feature images; based on the plurality of target feature images, N first feature images are determined; N first feature images are input into the feature fusion network, and M second feature images are output. The feature fusion network is used to perform multi-scale feature fusion on the N first feature images, and M is an integer greater than 1. The M second feature images are input into the target prediction network, and the target image recognition result is output, wherein the recognition result includes the classification result of the target in the target image.

2. The detection method according to claim 1, characterized in that, Each residual unit includes at least: multiple dimension adjustment subunits, a weighting subunit, and inputs the fourth feature image output by the first dimension adjustment submodule into the residual submodule to output a first feature layer, including: Based on the first transmission sub-channel and the multiple dimension adjustment sub-units, the fourth feature image is channel-adjusted to obtain the third feature layer, and the third feature layer is input into the weighting sub-unit. The fourth feature image is input into the weighting sub-unit through the second transmission sub-channel; The weighting subunit performs weighted processing on the third feature layer and the fourth feature image to output the first feature layer.

3. The detection method according to claim 1, characterized in that, The feature fusion network includes at least: multiple second feature extraction modules, multiple feature fusion modules, and multiple feature fusion layers. Multiple residual units in the first feature extraction modules are replaced with multiple dimension adjustment units as the second feature extraction modules. N first feature images have different image sizes. The N first feature images are input into the feature fusion network, and M second feature images are output, including: In the first feature fusion layer, the first feature image of the first size is input to the first feature fusion module, and the output of the first feature fusion module is input to the first extraction module to output the first image. The first feature fusion module is used to fuse the image features transmitted by the first feature flow path with the first feature image of the first size. The transmission direction of the image features transmitted by the first feature flow path is from the last feature fusion layer to the first feature fusion layer. The first extraction module is one of a plurality of second feature extraction modules. The first extraction module is used to transmit the output of the first feature extraction module to the last feature fusion layer through the second feature flow path. The transmission direction of the image features of the second feature flow path is from the first feature fusion layer to the last feature fusion layer. In the second feature fusion layer, based on the third and fourth transmission channels, the first feature image of the second size is fused with the image features transmitted through the first feature flow path, and the result of feature fusion is fused with the image features transmitted through the second feature flow path to determine the second image; In the last feature fusion layer, based on the fifth and sixth transmission channels, the first feature image of the third size is fused with the features transmitted through the second feature flow path after pyramid pooling to determine the third image. Based on the first image, the second image, and the third image, M second feature images are determined.

4. The detection method according to claim 3, characterized in that, In the second feature fusion layer, based on the third and fourth transmission channels, the first feature image of the second size is fused with the image features transmitted through the first feature flow path, and the fused result is fused with the image features transmitted through the second feature flow path to determine the second image, including: In the second feature fusion layer, the first feature image of the second size is transmitted to the second feature fusion module through the third transmission channel, and multi-scale fusion is performed with the image features transmitted through the first feature flow path to obtain the first fused feature; The first fused feature is input into the second extraction module to obtain the second fused feature, wherein the second extraction module is one of a plurality of second feature extraction modules; The first feature image of the second size is transmitted to the third feature fusion module through the fourth transmission channel. It is then fused with the features transmitted through the second feature flow path and the second fused feature to determine the third fused feature. The third fusion feature is input into the third extraction module, and the second image is output. The third extraction module is one of a plurality of second feature extraction modules.

5. The detection method according to claim 3, characterized in that, After performing pyramid pooling on the first feature image of the third size, the method further includes: The result obtained by performing pyramid pooling on the first feature image of the third size is transmitted to the first feature fusion layer through the first feature flow path.

6. The detection method according to claim 1, characterized in that, The target detection model is determined in the following way: Obtain pre-trained weights and training samples, and divide the training samples into a training set, a validation set, and a test set. The training samples include multiple images, each of which has a bounding box labeled with the target. The initial detection model is trained based on the pre-trained weights and the training set, and cross-validation is used during the training process to verify whether the initial detection model has converged through the validation set. The initial detection model is an untrained model. If the initial detection model converges, the target detection model is determined, and the detection accuracy of the initial detection model is tested based on the weight file and the test set, wherein the weight file is used to store multiple weights of the target detection model.

7. The detection method according to claim 6, characterized in that, The method further includes: During the training of the initial detection model, the target prediction network receives the fused feature map output by the feature fusion network; The fused feature map is divided into multiple grids, wherein the fused feature map is marked with the bounding boxes of the target, and each grid includes multiple anchor boxes; Locate the target grid containing the bounding box in the fused feature map among the plurality of grids; The predicted bounding box is determined based on the intersection-union ratio (IUU) between each anchor box in the target mesh and the labeled bounding box of the target. The detection result of the initial detection model is determined by the prediction box.

8. The detection method according to claim 7, characterized in that, The step of determining the predicted bounding box based on the intersection-union ratio (IoU) of each anchor box in the target mesh with the labeled bounding box of the target includes: Calculate the intersection-union ratio (IUU) between each anchor box in the target mesh and the labeled box of the target to obtain multiple ratio values; The anchor box associated with the largest ratio among the plurality of ratios is used as the target anchor box, wherein the target anchor box is used to detect the target on the fused feature map; Update the position parameters of the target anchor box to obtain the predicted box.

9. The detection method according to claim 6, characterized in that, The loss functions used when training the initial detection model include at least: a classification loss function, a confidence loss function, and a localization loss function.

10. A target detection device that enhances multi-scale feature extraction, reuse, and fusion, characterized in that, include: The first processing unit is used to respond to the recognition request of the target image, input the target image into the target feature extraction network of the target detection model, and output N first feature images. The target detection model includes at least the target feature extraction network, the feature fusion network, and the target prediction network. The target detection model includes multiple transmission channels, which are used to transmit the image features of the target image, where N is an integer greater than 1. The target feature extraction network includes at least: multiple first feature extraction modules and multiple dimension adjustment modules. Each dimension adjustment module includes at least: a convolutional layer, a batch normalization layer and an activation function layer. The first processing unit includes: a first processing subunit, which is used to perform dimension adjustment on the target image through each dimension adjustment module and to extract features from the dimension-adjusted target image through each first feature extraction module to obtain N first feature images. Each of the first feature extraction modules includes at least: multiple dimension adjustment submodules and a residual submodule. The residual submodule includes multiple residual units. Each dimension adjustment submodule has the same structure as the dimension adjustment module. The processing submodule includes: a first processing module, configured to receive a third feature image through the first feature extraction module, input the third feature image into the first dimension adjustment submodule through a first transmission channel, input a fourth feature image output by the first dimension adjustment submodule into the residual submodule, and output a first feature layer. The first feature extraction module is one of multiple first feature extraction modules, and the third feature image is the target image processed by one or more of the multiple first feature extraction modules and the multiple dimension adjustment modules. The first processing module is one of multiple dimension adjustment sub-modules; the second processing module is used to perform convolution processing on the first feature layer in the first transmission channel and then input it into the stacking unit; the third processing module is used to perform convolution processing on the third feature image in the second transmission channel and then input it into the stacking unit, and perform stacking processing on the feature data received by the stacking unit to obtain the second feature layer; the fourth processing module is used to perform target processing on the second feature layer to obtain a target feature image, wherein the target processing includes at least one of the following: normalization processing, activation function processing, feature dimension adjustment processing, and the target feature image is one of N first feature images; the determination module is used to determine N first feature images based on multiple target feature images. The second processing unit is used to input N first feature images into the feature fusion network and output M second feature images, wherein the feature fusion network is used to perform multi-scale feature fusion on the N first feature images, and M is an integer greater than 1; The third processing unit is used to input the M second feature images into the target prediction network and output the recognition result of the target image, wherein the recognition result includes the classification result of the target in the target image.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the target detection method for enhanced multi-scale feature extraction, reuse and fusion as described in any one of claims 1 to 9.

12. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the enhanced multi-scale feature extraction, reuse and fusion target detection method according to any one of claims 1 to 9.

Images