A target detection method, device, apparatus and storage medium

Abstract

The application discloses a target detection method and device, equipment and a storage medium. The method comprises the following steps: inputting different scale multi-level image features of a to-be-detected roadside image into different scale multi-level feature fusion layers of a multi-direction feature fusion network for multi-direction feature fusion to obtain target fusion image features. Each level feature fusion layer comprises an input node, an intermediate node and an output node. The image features of the corresponding scale output by each level input node are input into the same level intermediate node, the same level output node and the adjacent level intermediate node. The initial fusion image features output by each level intermediate node are input into the same level output node, the next level intermediate node and the next level output node. The fusion image features output by each level output node are input into the previous level output node. Target detection is performed based on the target fusion image features to obtain a target detection result. The technical scheme provided by the application can improve the detection accuracy of small targets in roadside images.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a target detection method, apparatus, device, and storage medium. Background Technology

[0002] With the development of the intelligent transportation industry, perception technologies using sensors such as image processing and LiDAR have been widely adopted in roadside intelligent transportation. Roadside traffic scenarios are complex, especially with targets occluding each other between different traffic participants and targets being too small to detect. Therefore, network models need to have good detection capabilities for small targets.

[0003] Currently, mainstream target detection methods cannot effectively extract high-resolution feature map information, and they also cannot fully fuse multiple feature information at different scales during the feature fusion stage. As a result, the fused features obtained in the end cannot fully express the semantic information in the image to be detected, making it difficult to guarantee the detection accuracy of small roadside targets. Summary of the Invention

[0004] This application provides a target detection method, apparatus, device, and storage medium, which can improve the accuracy of target fusion image feature representation of roadside images to be detected, thereby improving the detection accuracy of small targets in roadside images. The technical solution of this application is as follows:

[0005] On the one hand, a target detection method is provided, the method comprising:

[0006] Acquire the roadside image to be detected;

[0007] The roadside image to be detected is input into a U-shaped image feature extraction network for multi-scale feature extraction to obtain multi-level image features at different scales.

[0008] The multi-level image features are respectively input into multi-level feature fusion layers of different scales in a multi-directional feature fusion network for multi-directional feature fusion to obtain target fused image features. Each level of the feature fusion layer includes an input node, an intermediate node, and an output node. The image features of the corresponding scale output by the input node of each level are respectively input into the intermediate node of the same level, the output node of the same level, and the intermediate node of the adjacent level. The initial fused image features output by the intermediate node of each level are respectively input into the output node of the same level, the intermediate node of the next level, and the output node of the next level. The fused image features output by the output node of each level are input into the output node of the previous level.

[0009] The target fused image features are input into the target detection network for target detection to obtain the target detection result.

[0010] On the other hand, a target detection device is provided, the device comprising:

[0011] The roadside image acquisition module is used to acquire the roadside image to be detected.

[0012] The multi-scale feature extraction module is used to input the roadside image to be detected into a U-shaped image feature extraction network for multi-scale feature extraction to obtain multi-level image features at different scales.

[0013] A multi-directional feature fusion module is used to input the multi-level image features into multi-level feature fusion layers of different scales in a multi-directional feature fusion network to perform multi-directional feature fusion and obtain target fused image features. Each feature fusion layer includes an input node, an intermediate node, and an output node. The image features of the corresponding scale output by the input node of each level are respectively input to the intermediate node of the same level, the output node of the same level, and the intermediate node of the adjacent level. The initial fused image features output by the intermediate node of each level are respectively input to the output node of the same level, the intermediate node of the next level, and the output node of the next level. The fused image features output by the output node of each level are input to the output node of the previous level.

[0014] The target detection module is used to input the target fused image features into the target detection network for target detection and obtain the target detection result.

[0015] On the other hand, a target detection device is provided, the device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the target detection method as described above.

[0016] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the target detection method as described above.

[0017] The target detection method, apparatus, equipment, and storage medium provided in this application have the following technical advantages:

[0018] By utilizing the technical solution provided in this application, the roadside image to be detected is input into a U-shaped image feature extraction network for multi-scale feature extraction. This avoids the direct output of deep feature information in the low-resolution branch, allowing deep feature information to flow into the high-resolution module. This yields multi-level image features at different scales, including high-resolution image features, thus improving the semantic expression of the image feature extraction network. Then, the multi-level image features are input into multi-level feature fusion layers at different scales in the multi-directional feature fusion network for multi-directional feature fusion. This allows for full interaction between deep and shallow feature information, and full fusion of detailed information at different resolutions with high-level semantic information. This improves the accuracy of the target fused image features in representing the roadside image to be detected, thereby enhancing the detection accuracy of small targets in the roadside image. Attached Figure Description

[0019] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart of a target detection method provided in an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the structure of an image feature extraction network provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of a process provided in this application embodiment of which inputs the roadside image to be detected into a U-shaped image feature extraction network to perform multi-scale feature extraction and obtain multi-level image features at different scales;

[0023] Figure 4 This is a schematic diagram of a process provided in this application embodiment to extract downsampled image features by using a downsampled short-time dense connection layer at each level to extract downsampled features from the input features input to itself.

[0024] Figure 5 This is a schematic diagram of the structure of a short-term dense connection module provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of a process provided in this application embodiment to extract upsampled image features by upsampling the input features input to itself through each level of upsampling short-time dense connection layer;

[0026] Figure 7This is a schematic diagram of the structure of a multi-directional feature fusion network provided in an embodiment of this application;

[0027] Figure 8 This is a flowchart illustrating an anchor frame size optimization method provided in an embodiment of this application;

[0028] Figure 9 This application provides a schematic diagram of the process for jointly training the above-mentioned image feature extraction network, the above-mentioned multi-directional feature fusion network and the above-mentioned object detection network.

[0029] Figure 10 This is a schematic diagram of the structure of a roadside small target detection network provided in an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of a target detection method apparatus provided in an embodiment of this application;

[0031] Figure 12 This is a schematic diagram of the structure of a target point detection device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0033] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0034] The following describes a target detection method provided by an embodiment of this application. Figure 1This is a flowchart illustrating a target detection method provided in an embodiment of this application. It should be noted that this specification provides the operational steps of the method as described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual systems or products, the methods can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment) as shown in the embodiments or accompanying drawings. Specifically, as... Figure 1 As shown, the above method may include:

[0035] S101, acquire the roadside image to be detected.

[0036] In the embodiments of this specification, the roadside image to be detected can be a traffic scene image collected by a roadside device. Specifically, the roadside device can be an environmental sensing device deployed at the end of the road, for example, a roadside camera.

[0037] In one specific embodiment, the traffic scene image can be an image containing traffic scene targets. Specifically, traffic scene targets can include, but are not limited to, non-motorized vehicles, pedestrians, trucks, cars, etc.

[0038] In one specific embodiment, the initial roadside image can be scaled based on the input size of the image feature extraction network to obtain a roadside image to be detected that conforms to the input size.

[0039] S102, the roadside image to be detected is input into the U-shaped image feature extraction network for multi-scale feature extraction to obtain multi-level image features at different scales.

[0040] In the embodiments of this specification, the U-shaped image feature extraction network can be used to extract multi-scale features from the roadside image to be detected, and output multi-level image features at different scales.

[0041] In the embodiments of this specification, each scale of the multi-level image features described above can represent the context fusion information of the corresponding scale. In a specific embodiment, the multi-level image features at different scales can be represented as multi-level feature maps of different resolutions.

[0042] In a specific embodiment, such as Figure 2 As shown, the U-shaped image feature extraction network may include: an initial feature extraction layer, a series of multi-level downsampling short-time dense connection layers of different scales, and a series of multi-level upsampling short-time dense connection layers of different scales.

[0043] In a specific embodiment, the aforementioned multi-level downsampling short-time dense connection layer can be multiple downsampling short-time dense connection layers with successively decreasing corresponding image feature scales and successively increasing corresponding image feature depths. Specifically, the image feature scale here can be the feature map resolution, and the image feature depth here can be the feature map channel dimension, i.e., the number of feature map channels. Optionally, the successively decreasing image feature scale can be the feature map resolution being halved successively, and the successively increasing image feature depth can be the number of feature map channels being doubled successively. In one example, the feature map parameters output by each of the aforementioned multi-level downsampling short-time dense connection layers can be as follows: 320×320×32, 160×160×64, 80×80×128, 40×40×256, 20×20×512.

[0044] In a specific embodiment, the aforementioned multi-level upsampling short-time dense connection layer can be multiple upsampling short-time dense connection layers with sequentially increasing corresponding image feature scales and sequentially decreasing corresponding image feature depths. Specifically, the image feature scale here can be the feature map resolution, and the image feature depth here can be the feature map channel dimension, i.e., the number of feature map channels. Optionally, the sequential increase in image feature scale can be a sequential doubling of feature map resolution, and the sequential decrease in image feature depth can be a sequential halving of the number of feature map channels. In one example, the feature map parameters output by each of the aforementioned multi-level upsampling short-time dense connection layers can be as follows: 20×20×512, 40×40×256, 80×80×128, 160×160×64, 320×320×32.

[0045] Specifically, Figure 2 The U-shaped image feature extraction network shown is only an illustrative network structure provided in this application. The U-shaped image feature extraction network provided in this application may also include: U-net (a U-shaped convolutional network), etc., and this application is not limited to this.

[0046] In a specific embodiment, such as Figure 3 As shown, the above-mentioned image feature extraction network, which inputs the roadside image to be detected into a U-shaped structure, performs multi-scale feature extraction to obtain multi-level image features at different scales, which may include:

[0047] S301, the initial feature extraction layer extracts features from the roadside image to be detected to obtain the initial image features.

[0048] Specifically, the initial feature extraction layer can perform preliminary feature extraction on the high-resolution roadside image to be detected, obtaining initial image features. This allows the subsequent downsampling short-time dense connection layer to perform downsampling and multi-dimensional feature extraction based on the initial image features, thereby effectively learning the image features.

[0049] In a specific embodiment, the initial feature extraction layer may include an initial downsampling layer, an initial convolutional layer, and an initial feature mapping layer. Specifically, the initial downsampling layer may perform downsampling processing on the roadside image to be detected to obtain an initial downsampled image; the initial convolutional layer may perform preliminary convolution processing on the initial downsampled image to obtain image convolutional features; and the initial feature mapping layer may perform feature mapping processing on the image convolutional features based on an activation function to obtain initial image features.

[0050] In one example, the initial downsampling layer can be a 2x downsampling layer, the initial convolutional layer can be a 3×3 convolutional layer, and the activation function in the initial feature mapping layer can be a silu function.

[0051] S302, the input features input to itself are downsampled and extracted through each downsampled short-time dense connection layer to obtain downsampled image features. The input features of the first downsampled short-time dense connection layer connected to the initial feature extraction layer are the initial image features, and the input features of the second to last downsampled short-time dense connection layers are the downsampled image features output by the previous downsampled short-time dense connection layer.

[0052] Specifically, each downsampling short-time dense connection layer can downsample the input features input to itself and extract multi-dimensional features, thereby obtaining image features with lower resolution and deeper depth compared to its own input features.

[0053] In one specific embodiment, each downsampling short-time dense connection layer may include: a downsampling layer, a first short-time dense connection layer, a first feature integration layer, a first regularization layer, and a first feature mapping layer, such as... Figure 4 As shown, the downsampled image features obtained by extracting downsampled features from the input features input to itself through each downsampled short-time dense connection layer can include:

[0054] S401, by downsampling the input features input to itself through each downsampling layer, the first image features are obtained.

[0055] Specifically, the first image feature is a feature map obtained by downsampling the input features input to itself through each downsampling layer.

[0056] Specifically, the input features of the first-level downsampling layer are the initial image features, and the input features of the second to the last level downsampling layers are the downsampled image features output by the previous level downsampling short-time dense connection layer.

[0057] In one example, each downsampling layer can be a 2x downsampling layer.

[0058] In an optional embodiment, when the initial feature extraction layer includes an initial downsampling layer, in order to ensure that the first-level downsampling short-time dense connection layer connected to the initial feature extraction layer can learn high-resolution information, the first-level downsampling short-time dense connection layer no longer performs downsampling operations. That is, the first-level downsampling short-time dense connection layer only includes a first short-time dense connection layer, a first feature integration layer, a first regularization layer, and a first feature mapping layer.

[0059] S402 extracts multi-dimensional features from the first image features input to itself through the first short-term dense connection layer at each level to obtain the second image features.

[0060] Specifically, the second image feature is a feature map obtained by performing multi-dimensional feature extraction on the first image feature input to itself through the first short-time dense connection layer at each level.

[0061] Specifically, the first short-time dense connection layer here refers to a network layer formed by multiple short-time dense connection modules connected in series. The multiple short-time dense connection modules have the same structure. Each short-time dense connection module contains multiple operation blocks. The number of channels in the first to the second to last operation blocks of each operation block in the short-time dense connection module is in a geometric sequence. The number of channels in the last operation block is the same as that in the second to last operation block. Specifically, from the first operation block (i.e., the short-time dense connection module) to the second to last operation block, the number of channels in the kernel is half of the number of channels in the previous feature block. The number of channels in the kernel of the last operation block is consistent with the number of channels in the second to last kernel.

[0062] In one example, in the short-time dense connection module, the kernel size of the first operation block is 1x1, and its main function is to reduce the dimension of the input features to half of the output dimension. Except for the first operation block, the kernels of the remaining operation blocks are all 3x3.

[0063] See Figure 5 , Figure 5 This is a schematic diagram of a short-time dense connection module provided in an embodiment of this application. Specifically, the short-time dense connection module may include: four operation blocks connected in sequence, a global average pooling layer, and a feature concatenation layer. Each operation block includes: a convolutional layer, a regularization layer, and a feature mapping layer. The first to third operation blocks can reduce the number of channels of the output feature map by half while maintaining the same output resolution of the feature map. Only the number of channels of the feature map in the fourth operation block is consistent with that of the third operation block. After the fourth operation block, the number of channels of the feature map is reduced to 1 / 8 of the initial input feature map. After the output feature map of the first operation block is processed by global average pooling, it is fused with the output feature maps of the second to fourth operation blocks through the feature concatenation layer to obtain a multi-dimensional feature map.

[0064] Specifically, Figure 5 The module structure of the short-term dense connection module shown is only an illustrative module structure provided in this application, and this application is not limited thereto.

[0065] S403, through the first feature integration layer of each level, performs feature integration processing on the second image features input to itself to obtain the third image features.

[0066] Specifically, the third image feature is a feature map obtained by integrating the second image features input to itself through the first feature integration layer at each level.

[0067] In one example, the first feature integration layer at each level can be a 3×3 convolutional layer.

[0068] S404, through the first regularization layer of each level, performs regularization processing on the third image features input to itself to obtain the fourth image features.

[0069] Specifically, the fourth image feature is a feature map obtained by regularizing the third image feature input to itself through the first regularization layer at each level.

[0070] In one example, the first regularization layer at each level can be a BN batch regularization layer.

[0071] S405, the fourth image features input to itself are processed by the first feature mapping layer at each level to obtain the downsampled image features.

[0072] Specifically, the downsampled image features are obtained by performing feature mapping on the fourth image features input to itself through the activation function in the first feature mapping layer of each level.

[0073] In one example, the activation function in the first feature mapping layer of each level can be a silu function.

[0074] Optionally, an SPP (Spatial Pyramid Pooling) layer can be set between the first short-term dense connection layer and the first feature integration layer of the last-level downsampled short-term dense connection layer to expand the receptive field of the network model.

[0075] As can be seen from the above embodiments, the number of channels for deep features in the short-time dense connection module is small, while the number of channels for shallow features is large. As the number of network layers increases, the number of feature channels gradually decreases, thus reducing the computational load. At the same time, since the output of the short-time dense connection module integrates the output features of multiple operation blocks, it can extract multi-dimensional information while avoiding the problem of information redundancy caused by too many feature channels. By performing feature extraction on the low-resolution branch through the downsampling short-time dense connection layer containing the short-time dense connection module, rich deep semantic features can be extracted.

[0076] S303, upsampling features are extracted from the input features input to itself through each upsampling short-time dense connection layer to obtain upsampling image features. The input features of the first upsampling short-time dense connection layer connected to the last downsampling short-time dense connection layer are: downsampling image features output by the downsampling short-time dense connection layer at the same scale as itself. The input features of the second to last upsampling short-time dense connection layers include: upsampling image features output by the previous upsampling short-time dense connection layer and downsampling image features output by the downsampling short-time dense connection layer at the same scale as itself.

[0077] Specifically, each upsampling short-time dense connection layer can upsample the input features input to itself and extract multi-dimensional features, thereby obtaining image features with higher resolution and shallower depth compared to its own input features.

[0078] In a specific embodiment, each upsampling short-time dense connection layer may include: a feature concatenation layer, an upsampling layer, a second short-time dense connection layer, a second feature integration layer, a second regularization layer, and a second feature mapping layer, such as... Figure 6 As shown, the above-described upsampled image features obtained by extracting upsampled features from the input features input to itself through each level of upsampled short-time dense connection layer can include:

[0079] S601, through each feature stitching layer, performs feature stitching processing on the input features input to itself to obtain the fifth image feature.

[0080] Specifically, the fifth image feature is a feature map obtained by upsampling the input features input to itself through each feature splicing layer.

[0081] Specifically, the input features of the feature stitching layer in the second to last level upsampled short-time dense connection layers include: the upsampled image features output by the previous level upsampled short-time dense connection layer, and the downsampled image features output by the downsampled short-time dense connection layer at the same scale as itself; while the input features of the first level upsampled short-time dense connection layer only include the downsampled image features output by the downsampled short-time dense connection layer at the same scale as itself. Therefore, the feature stitching layer can be omitted in the first level upsampled short-time dense connection layer.

[0082] Specifically, a tiered structure can be set between downsampling short-time dense connection layers and upsampling short-time dense connection layers at the same scale. In a specific embodiment, the first feature mapping layer and the second feature concatenation layer at the same scale are connected, so that the U-shaped image feature extraction network can utilize both shallow and deep semantic information at the same time, avoiding feature loss due to scale changes (resolution changes).

[0083] S602, by upsampling the fifth image feature input to itself through each upsampling layer, the sixth image feature is obtained.

[0084] Specifically, the sixth image feature is a feature map obtained by upsampling the fifth image feature input to itself through each upsampling layer.

[0085] In one example, each upsampling layer can be twice the number of upsampling layers.

[0086] In an optional embodiment, in order to ensure that the first-level upsampled short-time dense connection layer and the last-level downsampled short-time dense connection layer connected to it are on the same scale, the first-level upsampled short-time dense connection layer no longer performs upsampling operations. That is, the first-level upsampled short-time dense connection layer only contains the second short-time dense connection layer, the second feature integration layer, the second regularization layer, and the second feature mapping layer.

[0087] S603 extracts the sixth image features input to itself from the second short-time dense connection layer at each level to obtain the seventh image features.

[0088] Specifically, the seventh image feature is a feature map obtained by performing multi-dimensional feature extraction on the sixth image feature input to itself through the second short-time dense connection layer at each level.

[0089] Specifically, the second short-time dense connection layer here has a similar structure to the first short-time dense connection layer mentioned above. For details on the second short-time dense connection layer, please refer to the details on the first short-time dense connection layer in step S402 above, which will not be repeated here.

[0090] S604, through the second feature integration layer at each level, performs feature integration processing on the seventh image features input to itself to obtain the eighth image features.

[0091] Specifically, the eighth image feature is a feature map obtained by integrating the seventh image features input to itself through the second feature integration layer at each level.

[0092] In one example, each of the above-mentioned second feature integration layers can be a 3×3 convolutional layer.

[0093] S605, through the second regularization layer at each level, performs regularization processing on the eighth image feature input to itself to obtain the ninth image feature.

[0094] Specifically, the ninth image feature is a feature map obtained by regularizing the eighth image feature input to itself through each level of the second regularization layer.

[0095] In one example, each of the above-mentioned second regularization layers can be a BN batch regularization layer.

[0096] S606, through the second feature mapping layer at each level, performs feature mapping processing on the ninth image features input to itself to obtain upsampled image features.

[0097] Specifically, the upsampled image features are obtained by performing feature mapping on the ninth image features input to itself through the activation function in each level of the second feature mapping layer.

[0098] In one example, the activation function in each of the second feature mapping layers described above can be a silu function.

[0099] As can be seen from the above embodiments, feature extraction is performed on the high-resolution branch by using an upsampled short-time dense connection layer containing a short-time dense connection module, and a hop structure is set between the upsampled short-time dense connection layer and the downsampled short-time dense connection layer at the same resolution. This allows the U-shaped image feature extraction network to utilize both shallow and deep semantic information simultaneously, avoiding feature loss due to scale changes.

[0100] S304 uses the upsampled image features output by each of the multi-level upsampled short-time dense connection layers as multi-level image features.

[0101] As can be seen from the above embodiments, inputting the roadside image to be detected into the U-shaped image feature extraction network for multi-scale feature extraction can avoid the direct output of deep feature information in the low-resolution branch, so that the deep feature information can flow into the high-resolution module to obtain multi-level image features of different scales, including high-resolution image features, thereby improving the semantic expression of the image feature extraction network.

[0102] S103, multi-level image features are input into multi-level feature fusion layers of different scales in the multi-directional feature fusion network to perform multi-directional feature fusion and obtain target fused image features. Each feature fusion layer includes an input node, an intermediate node, and an output node. The image features of the corresponding scale output by each input node are input into the same-level intermediate node, the same-level output node, and the adjacent-level intermediate node, respectively. The initial fused image features output by each intermediate node are input into the same-level output node, the next-level intermediate node, and the next-level output node, respectively. The fused image features output by each output node are input into the previous-level output node.

[0103] In the embodiments of this specification, the multi-directional feature fusion network can be used to perform multi-directional feature fusion on multi-level image features. Specifically, the multi-directional feature fusion network can be used to perform feature fusion between adjacent levels of each level of image features.

[0104] Specifically, a multi-directional feature fusion network may include: multi-level feature fusion layers at different scales, with the scale of each multi-level feature fusion layer increasing progressively in a top-down order. The scale of each level of image features in the multi-level image features corresponds one-to-one with the scale of each level of the feature fusion layer. Each level of feature fusion layer may include an input node, intermediate nodes, and an output node.

[0105] Specifically, each input node can be used to pass image features of its corresponding scale to itself. In one specific embodiment, each input node can be connected to a second feature mapping layer of the corresponding scale in the image feature extraction network.

[0106] Specifically, each intermediate node performs feature fusion on the input features input to itself to obtain preliminary fused image features. The input features of the first-level (top-level) intermediate node include: image features of the corresponding scale output by the input nodes of the same level and the next level. The input features of the last-level (bottom-level) intermediate node include: image features of the corresponding scale output by the input nodes of the same level and the previous level, as well as the preliminary fused image features output by the intermediate node of the previous level. The input features of the second-level to penultimate-level (intermediate layer) intermediate nodes include: image features of the corresponding scale output by the input nodes of the same level, the previous level and the next level, as well as the preliminary fused image features output by the intermediate node of the previous level.

[0107] Specifically, each output node performs feature fusion on the input features input to itself to obtain fused image features. The input features of the first-level (top-level) output node include: image features of the corresponding scale output by the input node at the same level, preliminary fused image features output by the intermediate nodes at the same and next levels, and fused image features output by the next-level output node. The input features of the last-level (bottom-level) output node include: image features of the corresponding scale output by the input node at the same level, and preliminary fused image features output by the intermediate nodes at the same and previous levels. The input features of the second to penultimate-level (intermediate layer) output nodes include: image features of the corresponding scale output by the input node at the same level, preliminary fused image features output by the intermediate nodes at the same, previous, and next levels, and fused image features output by the next-level output node.

[0108] In one specific embodiment, intermediate nodes can represent the first convolution operation, and output nodes can represent the second convolution operation.

[0109] Taking a multi-level feature fusion layer with 5 levels in a multi-directional feature fusion network as an example, Figure 7 This is a schematic diagram of the structure of a multi-directional feature fusion network provided in an embodiment of this application. For details, please refer to... Figure 7 Each feature fusion layer can include input nodes, intermediate nodes, and output nodes. To improve the feature fusion level, a cross-level connection structure is set between input nodes and output nodes at the same level, and an X-shaped connection structure is added to the network so that all nodes except the input nodes are connected to the surrounding nodes. Through this X-shaped connection structure, the multi-directional feature fusion network can fully exchange deep and shallow information, thereby effectively expressing the fused feature information.

[0110] In the embodiments of this specification, the target fused image features can be image features obtained by multi-directional fusion processing of shallow and deep features of different resolutions. Specifically, the target fused image features can be represented as a target fused feature map.

[0111] In one specific embodiment, the target fusion image features can be a fusion feature map output by the target-scale feature fusion layer, wherein the target-scale feature fusion layer can be determined from the multi-level feature fusion layer based on the target size of the anchor box in the target detection network.

[0112] As can be seen from the above embodiments, a cross-level connection structure is set between input nodes and output nodes at the same level, and an X-shaped connection structure is added to the network, so that all nodes except the input nodes are connected to the surrounding nodes. Through this X-shaped connection structure, the multi-directional feature fusion network can fully carry out deep and shallow information interaction, thereby effectively expressing the fused feature information.

[0113] S104, Input the target fused image features into the target detection network to perform target detection and obtain the target detection result.

[0114] In the embodiments of this specification, the target detection result can characterize the detection result obtained by the target detection network after performing target detection on the target fused image features. Specifically, the target detection result may include: the detection category of the target in the roadside image to be detected, the confidence level of the detection category, and the detection box location information.

[0115] In practical applications, object detection networks can determine the number of output heads through adaptive algorithms. Currently, mainstream object detection networks mainly use 3 or 5 output heads, with each output head corresponding to a specific anchor box size. As the number of output heads increases, the positive and negative sample allocation stage of the algorithm needs to generate more anchor boxes, and each anchor box then generates a target, significantly increasing the computational load. Simultaneously, during the inference stage, more output heads will output more predicted targets, leading to a longer processing time for the NMS (non-maximum suppression) stage. In roadside scenarios, traffic participant targets occupy relatively small pixels, but the number of targets is large, and the computational resources of edge devices are limited. Therefore, it is necessary to optimize the output heads to improve the computational efficiency of the object detection network. This application provides an anchor box size optimization method based on mean shift, which optimizes the anchor box size to improve the output heads.

[0116] In a specific embodiment, such as Figure 8 As shown, the above anchor frame size optimization method may include:

[0117] S801, obtain the size information of multiple sample annotation boxes.

[0118] Specifically, the size information of multiple sample bounding boxes can be the size information of the sample bounding boxes of each sample roadside image in the sample roadside image set. The sample roadside image set can be a set of roadside images pre-collected for network model training. The sample bounding boxes can be the bounding boxes obtained by pre-annotating the sample targets in the sample roadside images.

[0119] In one specific embodiment, the sample annotation box size information may include: the height, width, and area of ​​the sample annotation box.

[0120] S802, perform mean-shift clustering on the size information of multiple sample bounding boxes to obtain at least one set of bounding box sizes and the cluster size information corresponding to each of the at least one set of bounding box sizes.

[0121] Specifically, the annotation box size set can be a set of annotation box size information of samples with similar sizes obtained after clustering multiple sample annotation box size information. The cluster size information can be the size information corresponding to the cluster center of the annotation box size set.

[0122] In a specific embodiment, the multiple sample bounding box size information may include: the area of ​​multiple sample bounding boxes. The above-mentioned mean-shift clustering of the multiple sample bounding box size information to obtain at least one set of bounding box sizes and the cluster size information corresponding to each of the at least one set of bounding box sizes may include:

[0123] 1) Using the area of ​​the sample bounding box as the sample point, and the area x of the sample bounding box being the height × width of the sample bounding box, construct the sample set as: X={x1,x2,x3,…,x n}, and randomly select a sample point from the sample set as the starting center point for mean drift.

[0124] 2) Generate a circular region with the starting center point as the center and h as the radius. Based on formula (1), determine all sample points in the circular region in the sample set, and assign all sample points in the circular region to the same initial annotation box size set.

[0125] S h | (x) =(y|(xy)(yx) T ≤h 2 ) Formula (1)

[0126] Where x represents the selected starting center point, y represents other sample points in the sample set besides the starting center point, and S h | (a) This represents all sample points within a circular area of ​​radius h. The radius h can be selected based on the distribution of the sample annotation area and can be either 5 or 10.

[0127] 3) Based on the above starting center point x, calculate the m sample points x in the initial annotation box size set according to formula (2). i Mean shift vector of (i = 1, ..., m);

[0128]

[0129] Where G(x) is a unit kernel function, h represents the radius of the current circular region, and x i (i = 1, ..., m) represent sample points in the initial bounding box size set, x represents the initial center point, and ω(x) represents the initial center point. i The value represents the weight of a sample point. The closer a sample point is to the center point, the greater its influence on the direction of mean shift, and the higher its weight.

[0130] 4) According to the iterative method shown in formula (3), the initial center point is moved along the mean drift direction to obtain the moved center point.

[0131] x t+1 =x t +M h (x t ) Formula (3)

[0132] Where t represents the current iteration round.

[0133] 5) Based on the moved center point, repeat steps 2) to 4) until the magnitude of the current mean drift vector is less than the preset threshold. Integrate the multiple initial annotation box size sets in the iteration process to obtain the annotation box size set. Use the current center point as the cluster center of the annotation box size set and use the sample annotation box area corresponding to the cluster center as the cluster size information.

[0134] Specifically, the preset threshold can be determined based on empirical values, and optionally, the preset threshold can be 1.

[0135] 6) If the distance between the cluster center of the current set of annotation box sizes and the cluster center of other existing sets of annotation box sizes is less than a preset threshold when the iteration is completed, the two sets of annotation box sizes can be merged.

[0136] 7) Repeat steps 1) to 5) until all sample points in the sample set have been added to the annotation box size set.

[0137] S803, based on at least one set of annotation box dimensions and clustering dimension information, generate annotation box size distribution information.

[0138] Specifically, the dimension distribution information of the annotation boxes can characterize the distribution of dimension information of multiple annotation boxes.

[0139] S804, Based on the annotation box size distribution information, determine the target annotation box size set from at least one annotation box size set.

[0140] In one specific embodiment, the target bounding box size set can be a set of bounding box sizes in at least one bounding box size set whose corresponding sample proportion information meets a preset ratio condition.

[0141] Specifically, the sample proportion information can be the ratio of the number of samples in the corresponding set of annotation box sizes to the number of samples in multiple sets of annotation box sizes. The preset ratio condition can be a pre-set ratio condition used to filter the set of annotation box sizes; the preset ratio condition can be preset based on the filtering requirements of the set of annotation box sizes and the optimization requirements of the anchor boxes in actual applications. Optionally, the preset ratio condition can be that the sample proportion information is greater than 90%.

[0142] In one specific embodiment, if the sum of the sample proportion information of multiple annotation box size sets in at least one annotation box size set meets a preset proportion condition, the multiple annotation box size sets can be used as the target annotation box size set.

[0143] S805, based on the clustering size information corresponding to the target annotation box size set, determines the target size anchor box from multiple preset size anchor boxes.

[0144] Specifically, among multiple anchor frames of preset sizes, the anchor frame of the preset size that is closest to the cluster size information corresponding to the target annotation frame size set can be used as the anchor frame of the target size.

[0145] Specifically, multiple preset anchor boxes can be set according to the scale of each of the multi-level feature fusion layers and the output resolution of the output head in the object detection network, with each preset size corresponding one-to-one with the scale of each of the multi-level feature fusion layers. In a specific embodiment, the scale of each of the multi-level feature fusion layers can be the resolution of each of the multi-level feature fusion layers.

[0146] In an optional embodiment, the preset dimensions of the anchor frame may include a preset height and a preset width.

[0147] For example Figure 7Taking the 5-level feature fusion layer as an example, the target detection network can include 5 initial output heads. Assuming that the resolutions of the 5-level feature fusion layers are 20×20, 40×40, 80×80, 160×160, and 320×320 respectively, and the output resolution of the initial output head is 640×640, then the preset anchor box size mapped to 640x640 for each initial output head can be 5×5, 10×10, 20×20, 40×40, and 80×80 respectively. Therefore, the anchor box distribution is preset to 5 classes. Calculate the number of samples for each of the at least one set of bounding box sizes. When the number of samples in the current n (n < 5) bounding box size sets accounts for 90% of the number of bounding box size information in the multiple samples, determine at most two anchor boxes in the five anchor boxes whose preset size is closest to the n cluster size information based on the cluster size information corresponding to each of the n bounding box size sets. Then retain the initial output heads corresponding to the at most two anchor boxes in the five initial output heads and delete the other initial output heads from the target detection network.

[0148] Optionally, if the input size of the image feature extraction network is updated, or the preset anchor box size of the initial output head is updated, the output head to be retained can be re-determined based on the sample roadside images and the mean-shift clustering algorithm.

[0149] In a specific embodiment, a target-scale feature fusion layer can be pre-determined from multi-level feature fusion layers of different sizes based on the anchor frame of the target size. Correspondingly, after inputting multi-level image features into multi-level preset feature fusion layers of different sizes in the multi-directional feature fusion network for multi-directional feature fusion, only the target-scale feature fusion layer outputs a fused feature map to the target detection network. This fused feature map is then used as the target fused image feature output by the multi-directional feature fusion network. Accordingly, the above-mentioned inputting the target fused image feature into the target detection network for target detection, and obtaining the target detection result, can include:

[0150] The fused feature map is input into the target detection network, and the target detection is performed on the fused feature map based on the anchor boxes corresponding to the target size, so as to obtain the target detection result;

[0151] Among them, the target size anchor box is determined from multiple anchor boxes of preset size based on the size distribution information of the annotation boxes corresponding to the sample roadside image set; the size distribution information of the annotation boxes is obtained by mean-shift clustering of the size information of multiple sample annotation boxes; the size information of multiple sample annotation boxes is the size information of the sample annotation boxes of each sample roadside image in the sample roadside image set.

[0152] As can be seen from the above embodiments, by performing mean-shift clustering on the size information of multiple sample bounding boxes in advance to obtain the size distribution information of the bounding boxes, and by determining the target size anchor box from multiple anchor boxes of preset sizes based on the size distribution information of the bounding boxes, the output head corresponding to the target size anchor box can be retained even when the number of initial output heads is large, thereby reducing the amount of anchor box calculation and optimizing the inference speed. This improves the detection efficiency while ensuring the detection accuracy of small roadside targets.

[0153] See Figure 9 This application provides a network training method for jointly training the above-mentioned image feature extraction network, the above-mentioned multi-directional feature fusion network, and the above-mentioned object detection network. Specifically, as shown in the embodiments, Figure 9 As shown, the method may include:

[0154] S901, obtain the roadside image to be trained and the corresponding labeled detection results of the roadside image to be trained.

[0155] In practical applications, training data can be determined before network training. Specifically, in this embodiment, roadside images containing labeled detection results can be obtained as training data.

[0156] Specifically, the annotation detection results can be the detection information of pre-annotated sample targets in the roadside image to be trained. The annotation detection results can include: the annotation category of the sample target and the location information of the annotation box.

[0157] In an optional embodiment, the sample detection image can be a 3-channel image with a resolution of 640×640.

[0158] S902, the roadside image to be trained is input into a U-shaped preset image feature extraction network for multi-scale feature extraction, and multi-level predicted image features at different scales are obtained.

[0159] S903, the multi-level predicted image features are respectively input into the multi-level preset feature fusion layers of different scales in the preset multi-directional feature fusion network to perform multi-directional feature fusion to obtain predicted fused image features. Each preset feature fusion layer includes a preset input node, a preset intermediate node, and a preset output node. The predicted image features of the corresponding scale output by each preset input node are respectively input into the same-level preset intermediate node, the same-level preset output node, and the adjacent-level preset intermediate node. The initial fused image features output by each preset intermediate node are respectively input into the same-level preset output node, the next-level preset intermediate node, and the next-level preset output node. The fused image features output by each preset output node are input into the previous-level preset output node.

[0160] S904: Input the predicted fused image features into the preset target detection network for target detection to obtain the predicted detection result.

[0161] S905 determines the loss information of the detection results based on the labeled detection results and the predicted detection results.

[0162] S906, based on the loss information of the detection results, train a preset image feature extraction network, a preset multi-directional feature fusion network, and a preset target detection network to obtain the image feature extraction network, the multi-directional feature fusion network, and the target detection network.

[0163] In an optional embodiment, the annotation detection result may include annotation category labels and bounding box location labels, and the prediction detection result may include predicted category labels and predicted bounding box location labels. Correspondingly, the detection result loss information may include: category label loss and location label loss; the category label loss can characterize the difference between the annotation category label and the predicted category label, and the location label loss can characterize the difference between the bounding box location label and the predicted bounding box location label.

[0164] Accordingly, the above-mentioned determination of detection result loss information based on labeled detection results and predicted detection results may include: determining category label loss based on labeled category labels and predicted category labels; and determining location label loss based on labeled bounding box location labels and predicted bounding box location labels.

[0165] In one specific embodiment, determining the category label loss based on the labeled category label and the predicted category label may include: determining the category label loss between the labeled category label and the predicted category label based on a preset category loss function. Here, the preset category loss function may include, but is not limited to, the cross-entropy loss function, the logistic loss function, the exponential loss function, etc.

[0166] In a specific embodiment, the above-mentioned determination of location label loss based on the bounding box location label and the predicted box location label may include: determining the location label loss between the bounding box location label and the predicted box location label based on a preset bounding box loss function.

[0167] In an optional embodiment, based on the detection result loss information, a preset image feature extraction network, a preset multi-directional feature fusion network, and a preset target detection network are trained to obtain the image feature extraction network, the multi-directional feature fusion network, and the target detection network, which may include:

[0168] S9061, based on the loss information of the detection results, update the network parameters of the preset image feature extraction network, the network parameters of the preset multi-directional feature fusion network, and the network parameters of the preset target detection network;

[0169] S9062, based on the updated preset image feature extraction network, the updated preset multi-directional feature fusion network and the updated preset target detection network, repeat the target detection iterative operation of steps S902, S903, S904, S905 and S9061 until the target detection convergence condition is met.

[0170] S9063, the preset image feature extraction network, preset multi-directional feature fusion network and preset target detection network obtained under the condition of target detection convergence are respectively used as image feature extraction network, multi-directional feature fusion network and target detection network.

[0171] In an optional embodiment, the aforementioned convergence condition for target detection can be that the number of training iterations reaches a preset number of training iterations. Optionally, the convergence condition for target detection can also be that the loss information of the current detection result is less than a specified threshold. In the embodiments of this specification, the preset number of training iterations and the specified threshold can be preset in conjunction with the training speed and accuracy of the network in practical applications.

[0172] As can be seen from the above examples, while improving model accuracy and training efficiency, the generalization ability of the network can also be improved.

[0173] See Figure 10 , Figure 10 This is a schematic diagram of a roadside small target detection network provided in an embodiment of this application. Specifically, the roadside small target detection network may include, for example: Figure 2 The U-shaped image feature extraction network shown, such as Figure 7 The multi-directional feature fusion network shown is the target detection network after output head optimization.

[0174] In roadside scenarios, traffic participants occupy small pixel areas, are numerous, and are prone to occlusion. Edge devices have limited computing resources, requiring target detection algorithms to possess both fast detection speed and high accuracy. Therefore, it is necessary to balance the detection accuracy and computational efficiency of target detection algorithms, through methods such as... Figure 10 The roadside small target detection network shown can detect small targets in roadside scenes, improving the network's detection accuracy while reducing network computation, optimizing inference speed, and improving detection efficiency.

[0175] As can be seen from the embodiments in the above specification, by using the technical solution provided in the embodiments of this specification, the roadside image to be detected is input into a U-shaped image feature extraction network for multi-scale feature extraction. This avoids the direct output of deep feature information in the low-resolution branch, allowing deep feature information to flow into the high-resolution module, resulting in multi-level image features at different scales, including high-resolution image features. This improves the semantic expression of the image feature extraction network. Then, the multi-level image features are input into multi-level feature fusion layers at different scales in the multi-directional feature fusion network for multi-directional feature fusion. Cross-level connection structures are set between the input nodes and output nodes of the same level in the multi-directional feature fusion network, and an X-shaped connection structure is added to the network, so that in addition to the input... All nodes outside the target node are interconnected with surrounding nodes. This X-shaped connection structure allows for full interaction between deep and shallow feature information, and full fusion of detailed information at different resolutions with high-level semantic information. This improves the accuracy of the target fusion image features in representing the roadside image to be detected. Then, mean-shift clustering is performed on the size information of multiple sample bounding boxes to obtain the size distribution information of the bounding boxes. Based on the size distribution information of the bounding boxes, the target size anchor box is determined from multiple anchor boxes of preset sizes. This allows the output head corresponding to the target size anchor box to be retained even when the number of initial output heads is large, reducing the amount of anchor box computation and optimizing the inference speed. This can improve the detection accuracy of small targets in roadside images while improving detection efficiency.

[0176] This application provides a target detection device, such as... Figure 11 As shown, the above-mentioned device may include:

[0177] The roadside image acquisition module 1110 is used to acquire the roadside image to be detected.

[0178] The multi-scale feature extraction module 1120 is used to input the roadside image to be detected into the U-shaped image feature extraction network for multi-scale feature extraction to obtain multi-level image features at different scales.

[0179] The multi-directional feature fusion module 1130 is used to input multi-level image features into multi-level feature fusion layers of different scales in the multi-directional feature fusion network to perform multi-directional feature fusion and obtain target fused image features. Each feature fusion layer includes an input node, an intermediate node, and an output node. The image features of the corresponding scale output by each input node are input to the same-level intermediate node, the same-level output node, and the adjacent-level intermediate node, respectively. The initial fused image features output by each intermediate node are input to the same-level output node, the next-level intermediate node, and the next-level output node, respectively. The fused image features output by each output node are input to the previous-level output node.

[0180] The target detection module 1140 is used to input the target fused image features into the target detection network for target detection and obtain the target detection result.

[0181] In a specific embodiment, the image feature extraction network may include: an initial feature extraction layer, sequentially connected multi-level downsampling short-time dense connection layers of different scales, and sequentially connected multi-level upsampling short-time dense connection layers of different scales. The multi-scale feature extraction module 1120 includes:

[0182] The feature extraction unit is used to extract features from the roadside image to be detected through the initial feature extraction layer to obtain initial image features;

[0183] The downsampling feature extraction unit is used to extract downsampling features from the input features input to itself through each downsampling short-time dense connection layer to obtain downsampling image features. The input features of the first downsampling short-time dense connection layer connected to the initial feature extraction layer are the initial image features, and the input features of the second to last downsampling short-time dense connection layers are the downsampling image features output by the previous downsampling short-time dense connection layer.

[0184] The upsampling feature extraction unit is used to extract upsampling features from the input features input to itself through each upsampling short-time dense connection layer to obtain upsampling image features. The input features of the first upsampling short-time dense connection layer connected to the last downsampling short-time dense connection layer are: downsampling image features output by the downsampling short-time dense connection layer at the same scale as itself. The input features of the second to last upsampling short-time dense connection layers include: upsampling image features output by the previous upsampling short-time dense connection layer and downsampling image features output by the downsampling short-time dense connection layer at the same scale as itself.

[0185] The multi-level image feature unit is used to take the upsampled image features output by each of the multi-level upsampled short-time dense connection layers as multi-level image features.

[0186] In a specific embodiment, each downsampling short-time dense connection layer may include: a downsampling layer, a first short-time dense connection layer, a first feature integration layer, a first regularization layer, and a first feature mapping layer. The downsampling feature extraction unit may include:

[0187] The downsampling unit is used to downsample the input features input to itself through each downsampling layer to obtain the first image features;

[0188] The first multi-dimensional feature extraction unit is used to extract multi-dimensional features from the first image features input to itself through each level of the first short-time dense connection layer to obtain the second image features.

[0189] The first feature integration unit is used to perform feature integration processing on the second image features input to itself through each level of the first feature integration layer to obtain the third image features;

[0190] The first regularization unit is used to perform regularization processing on the third image features input to itself through each first regularization layer to obtain the fourth image features;

[0191] The first feature mapping unit is used to perform feature mapping processing on the fourth image features input to itself through each first feature mapping layer to obtain downsampled image features.

[0192] In a specific embodiment, each upsampling short-time dense connection layer may include: a feature concatenation layer, an upsampling layer, a second short-time dense connection layer, a second feature integration layer, a second regularization layer, and a second feature mapping layer. The upsampling feature extraction unit may include:

[0193] The feature stitching unit is used to perform feature stitching processing on the input features input to itself through each feature stitching layer to obtain the fifth image feature;

[0194] The upsampling unit is used to upsample the input features input to itself through each upsampling layer to obtain the sixth image feature;

[0195] The second multi-dimensional feature extraction unit is used to extract the sixth image features input to itself through each level of the second short-time dense connection layer to obtain the seventh image features.

[0196] The second feature integration unit is used to perform feature integration processing on the seventh image features input to itself through each level of the second feature integration layer to obtain the eighth image features;

[0197] The second regularization unit is used to perform regularization processing on the eighth image features input to itself through each level of the second regularization layer to obtain the ninth image features;

[0198] The second feature mapping unit is used to perform feature mapping processing on the ninth image features input to itself through each level of the second feature mapping layer to obtain upsampled image features.

[0199] In one specific embodiment, the target fusion image features are the fused feature maps output by the feature fusion layer at the target scale, and the target detection module 1140 includes:

[0200] The target detection unit is used to input the fused feature map into the target detection network, perform target detection on the fused feature map based on the anchor box corresponding to the target size, and obtain the target detection result;

[0201] Among them, the target size anchor box is determined from multiple anchor boxes of preset size based on the size distribution information of the annotation boxes corresponding to the sample roadside image set; the size distribution information of the annotation boxes is obtained by mean-shift clustering of the size information of multiple sample annotation boxes; the size information of multiple sample annotation boxes is the size information of the sample annotation boxes of each sample roadside image in the sample roadside image set.

[0202] In one specific embodiment, the anchor frame of the above-mentioned target size can be determined by the following means:

[0203] The sample annotation box size information acquisition module is used to acquire the size information of multiple sample annotation boxes;

[0204] The mean-shift clustering module is used to perform mean-shift clustering on the size information of multiple sample bounding boxes to obtain at least one set of bounding box sizes and the cluster size information corresponding to each set of bounding box sizes.

[0205] The annotation box size distribution information module is used to generate annotation box size distribution information based on at least one set of annotation box sizes and clustered size information;

[0206] The target annotation box size set determination module is used to determine the target annotation box size set from at least one annotation box size set based on annotation box size distribution information;

[0207] The anchor frame determination module is used to determine the target size anchor frame from multiple preset size anchor frames based on the cluster size information corresponding to the target annotation frame size set.

[0208] In the embodiments described in this specification, the above-mentioned apparatus may further include:

[0209] The annotation detection result acquisition module is used to acquire the roadside image to be trained and the annotation detection results corresponding to the roadside image to be trained;

[0210] The multi-level predicted image feature module is used to input the roadside image to be trained into a U-shaped preset image feature extraction network for multi-scale feature extraction, and obtain multi-level predicted image features at different scales.

[0211] The predictive fusion image feature module is used to input multi-level predicted image features into multi-level preset feature fusion layers of different scales in a preset multi-directional feature fusion network to perform multi-directional feature fusion and obtain predicted fusion image features. Each preset feature fusion layer includes a preset input node, a preset intermediate node, and a preset output node. The predicted image features of the corresponding scale output by each preset input node are input into the same-level preset intermediate node, the same-level preset output node, and the adjacent-level preset intermediate node, respectively. The initial fusion image features output by each preset intermediate node are input into the same-level preset output node, the next-level preset intermediate node, and the next-level preset output node, respectively. The fusion image features output by each preset output node are input into the previous-level preset output node.

[0212] The prediction detection result module is used to input the predicted fused image features into a preset target detection network for target detection and obtain the prediction detection result.

[0213] The detection result loss information determination module is used to determine the detection result loss information based on the labeled detection results and the predicted detection results.

[0214] The network training module is used to train a preset image feature extraction network, a preset multi-directional feature fusion network, and a preset target detection network based on the loss information of the detection results, so as to obtain the image feature extraction network, the multi-directional feature fusion network, and the target detection network.

[0215] It should be noted that the apparatus in the device embodiment and the method embodiment are based on the same inventive concept.

[0216] This application provides a target detection device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the target detection method provided in the above method embodiments.

[0217] Furthermore, Figure 12 A schematic diagram of the hardware structure of a target detection device for implementing the target detection method provided in the embodiments of this application is shown. The target detection device can participate in or include the target detection apparatus provided in the embodiments of this application. Figure 12As shown, the target detection device 120 may include one or more processors 1202 (shown as 1202a, 1202b, ..., 1202n in the figure) 1202 (processor 1202 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 1204 for storing data, and a transmission device 1206 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 12 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, the target detection device 120 may also include a... Figure 12 The more or fewer components shown, or having the same Figure 12 The different configurations shown.

[0218] It should be noted that the aforementioned one or more processors 1202 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element within the target detection device 120 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0219] The memory 1204 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the target detection method described in the embodiments of this application. The processor 1202 executes various functional applications and data processing by running the software programs and modules stored in the memory 1204, thereby realizing the target detection method described above. The memory 1204 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1204 may further include memory remotely located relative to the processor 1202, and these remote memories can be connected to the target detection device 120 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0220] The transmission device 1206 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the target detection device 120. In one example, the transmission device 1206 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In one embodiment, the transmission device 1206 may be a radio frequency (RF) module for wireless communication with the Internet.

[0221] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the target detection device 120 (or mobile device).

[0222] This application embodiment also provides a storage medium, which can be disposed in a server to store at least one instruction or at least one program related to implementing a target detection method in the method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the target detection method provided in the above method embodiment.

[0223] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0224] As can be seen from the embodiments of the target detection method, apparatus, device, or storage medium provided in this application, by using the technical solution provided in the embodiments of this specification to input the roadside image to be detected into a U-shaped image feature extraction network for multi-scale feature extraction, deep feature information can be avoided from being directly output in the low-resolution branch, allowing deep feature information to flow into the high-resolution module, obtaining multi-level image features at different scales, including high-resolution image features, thus improving the semantic expression of the image feature extraction network. Then, the multi-level image features are respectively input into multi-level feature fusion layers at different scales in the multi-directional feature fusion network for multi-directional feature fusion. Cross-level connection structures are set between the input nodes and output nodes of the same level in the multi-directional feature fusion network, and an X-shaped connection is added to the network. The connection structure allows all nodes except the input node to be interconnected with their surrounding nodes. This X-shaped connection structure enables deep feature information to fully interact with shallow information, and detailed information at different resolutions to be fully integrated with high-level semantic information. This improves the accuracy of the target fusion image features in representing the roadside image to be detected. Then, mean-shift clustering is performed on the size information of multiple sample bounding boxes to obtain the size distribution information of the bounding boxes. Based on the size distribution information of the bounding boxes, the target size anchor box is determined from multiple anchor boxes of preset sizes. Even when there are a large number of initial output heads, the output head corresponding to the target size anchor box can be retained, reducing the amount of anchor box computation and optimizing the inference speed. This can improve the detection accuracy of small targets in roadside images while improving detection efficiency.

[0225] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0226] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0227] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0228] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A target detection method characterized by, The method includes: Acquire the roadside image to be detected; The roadside image to be detected is input into a U-shaped image feature extraction network for multi-scale feature extraction to obtain multi-level image features at different scales. The multi-level image features are respectively input into multi-level feature fusion layers of different scales in a multi-directional feature fusion network for multi-directional feature fusion to obtain target fused image features. Each level of the feature fusion layer includes an input node, an intermediate node, and an output node. The image features of the corresponding scale output by the input node of each level are respectively input into the intermediate node of the same level, the output node of the same level, and the intermediate node of the adjacent level. The initial fused image features output by the intermediate node of each level are respectively input into the output node of the same level, the intermediate node of the next level, and the output node of the adjacent level. The fused image features output by the output node of each level are input into the output node of the previous level. The target fused image features are input into the target detection network for target detection to obtain the target detection result; The image feature extraction network includes: an initial feature extraction layer, sequentially connected multi-level downsampling short-time dense connection layers at different scales, and sequentially connected multi-level upsampling short-time dense connection layers at different scales. The step of inputting the roadside image to be detected into the U-shaped image feature extraction network for multi-scale feature extraction to obtain multi-level image features at different scales includes: The initial image features are obtained by extracting features from the roadside image to be detected through the initial feature extraction layer. The input features input to itself are downsampled and extracted by each downsampled short-time dense connection layer to obtain downsampled image features. The input features of the first downsampled short-time dense connection layer connected to the initial feature extraction layer are the initial image features. The input features of the second to last downsampled short-time dense connection layers are the downsampled image features output by the previous downsampled short-time dense connection layer. Upsampled image features are obtained by upsampling the input features input to itself through each level of the upsampled short-time dense connection layer. The input features of the first level of the upsampled short-time dense connection layer connected to the last level of the downsampled short-time dense connection layer are: the downsampled image features output by the downsampled short-time dense connection layer at the same scale as itself. The input features of the second to last level of the upsampled short-time dense connection layer include: the upsampled image features output by the previous level of the upsampled short-time dense connection layer, and the downsampled image features output by the downsampled short-time dense connection layer at the same scale as itself. The upsampled image features output by each of the multi-level upsampled short-time dense connection layers are used as the multi-level image features.

2. The method of claim 1, wherein, Each level of the downsampled short-time dense connection layer includes: a downsampling layer, a first short-time dense connection layer, a first feature integration layer, a first regularization layer, and a first feature mapping layer. The downsampled image features obtained by extracting downsampled features from the input features input to itself through each level of the downsampled short-time dense connection layer include: The input features are downsampled at each downsampling layer to obtain the first image features. The first image features are obtained by performing multi-dimensional feature extraction on the first image features input to itself through the first short-time dense connection layer at each level; The first feature integration layer at each level performs feature integration processing on the second image features input to itself to obtain the third image features; The third image features input to itself are regularized by the first regularization layer at each level to obtain the fourth image features; The downsampled image features are obtained by performing feature mapping processing on the fourth image features input to itself through the first feature mapping layer at each level.

3. The method of claim 1, wherein, Each level of the upsampled short-time dense connection layer includes: a feature concatenation layer, an upsampling layer, a second short-time dense connection layer, a second feature integration layer, a second regularization layer, and a second feature mapping layer. The upsampled image features obtained by upsampling the input features input to itself through each level of the upsampled short-time dense connection layer include: The input features input to itself are processed by each of the feature stitching layers to obtain the fifth image feature. The sixth image feature is obtained by upsampling the fifth image feature input to itself through each upsampling layer. The sixth image features are extracted from the input sixth image features through the second short-time dense connection layer at each level to obtain the seventh image features; The second feature integration layer at each level performs feature integration processing on the seventh image features input to itself to obtain the eighth image features; The eighth image feature is obtained by performing regularization processing on the input eighth image feature through the second regularization layer at each level; The upsampled image features are obtained by performing feature mapping processing on the ninth image features input to itself through each level of the second feature mapping layer.

4. The method according to claim 1, characterized in that, The target fused image features are fused feature maps output by the feature fusion layer at the target scale. The step of inputting the target fused image features into a target detection network for target detection, and obtaining the target detection results, includes: The fused feature map is input into the target detection network, and the target detection is performed on the fused feature map based on the anchor boxes of the target size corresponding to the target scale to obtain the target detection result; The target size anchor frame is determined from multiple anchor frames of preset size based on the size distribution information of the annotation frames corresponding to the sample roadside image set; the size distribution information of the annotation frames is obtained by mean-shift clustering of the size information of multiple sample annotation frames; the size information of the multiple sample annotation frames refers to the size information of the sample annotation frames of each sample roadside image in the sample roadside image set.

5. The method of claim 4, wherein, The method further includes: Obtain the size information of the multiple sample annotation boxes; Mean-shift clustering is performed on the size information of the multiple sample bounding boxes to obtain at least one set of bounding box sizes and cluster size information corresponding to each of the at least one set of bounding box sizes. Based on the at least one set of bounding box sizes and the clustering size information, the bounding box size distribution information is generated; Based on the annotation box size distribution information, a target annotation box size set is determined from the at least one annotation box size set; Based on the clustering size information corresponding to the target annotation box size set, the anchor box of the target size is determined from the anchor boxes of the plurality of preset sizes.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain the roadside image to be trained and the corresponding labeled detection results of the roadside image to be trained; The roadside image to be trained is input into a U-shaped preset image feature extraction network for multi-scale feature extraction to obtain multi-level predicted image features at different scales. The multi-level predicted image features are respectively input into multi-level preset feature fusion layers of different scales in a preset multi-directional feature fusion network for multi-directional feature fusion to obtain predicted fused image features. Each preset feature fusion layer includes a preset input node, a preset intermediate node, and a preset output node. The predicted image features of the corresponding scale output by the preset input node of each level are respectively input into the preset intermediate node of the same level, the preset output node of the same level, and the preset intermediate node of the adjacent level. The initial fused image features output by the preset intermediate node of each level are respectively input into the preset output node of the same level, the preset intermediate node of the next level, and the preset output node of the adjacent level. The fused image features output by the preset output node of each level are input into the preset output node of the previous level. The predicted fused image features are input into a preset target detection network for target detection to obtain the predicted detection result; Based on the labeled detection results and the predicted detection results, the detection result loss information is determined; Based on the loss information of the detection results, the preset image feature extraction network, the preset multi-directional feature fusion network, and the preset target detection network are trained to obtain the image feature extraction network, the multi-directional feature fusion network, and the target detection network.

7. A target detection apparatus characterized by comprising: The device includes: The roadside image acquisition module is used to acquire the roadside image to be detected. A multi-scale feature extraction module is used to input the roadside image to be detected into a U-shaped image feature extraction network for multi-scale feature extraction, obtaining multi-level image features at different scales. The image feature extraction network includes: an initial feature extraction layer, sequentially connected multi-level downsampling short-time dense connection layers at different scales, and sequentially connected multi-level upsampling short-time dense connection layers at different scales. The initial feature extraction layer extracts features from the roadside image to be detected to obtain initial image features. Each level of the downsampling short-time dense connection layer performs downsampling feature extraction on the input features input to itself to obtain downsampling image features. The input features of the first level of the downsampling short-time dense connection layer connected to the initial feature extraction layer are the initial image features, and the input features of the second to last levels of the downsampling short-time dense connection layer are the self-sampled features. The downsampled image features are output by the downsampled short-time dense connection layer above the current level. Upsampled image features are obtained by extracting upsampled features from the input features input to the current level through each upsampled short-time dense connection layer. Specifically, the input features of the first upsampled short-time dense connection layer connected to the last downsampled short-time dense connection layer are: downsampled image features output by the downsampled short-time dense connection layer at the same scale as itself. The input features of the second to last upsampled short-time dense connection layers include: the upsampled image features output by the upsampled short-time dense connection layer above the current level, and the downsampled image features output by the downsampled short-time dense connection layer at the same scale as itself. The upsampled image features output by each of the multiple upsampled short-time dense connection layers are used as the multi-level image features. A multi-directional feature fusion module is used to input the multi-level image features into multi-level feature fusion layers of different scales in a multi-directional feature fusion network to perform multi-directional feature fusion and obtain target fused image features. Each feature fusion layer includes an input node, an intermediate node, and an output node. The image features of the corresponding scale output by the input node of each level are respectively input to the intermediate node of the same level, the output node of the same level, and the intermediate node of the adjacent level. The initial fused image features output by the intermediate node of each level are respectively input to the output node of the same level, the intermediate node of the next level, and the output node of the adjacent level. The fused image features output by the output node of each level are input to the output node of the previous level. The target detection module is used to input the target fused image features into the target detection network for target detection and obtain the target detection result.

8. A target detection device, characterized by The device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the target detection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the target detection method as described in any one of claims 1 to 6.

Images