Multi-target detection method, model training method, edge computing device and medium

Abstract

The invention belongs to the field of computer vision, and discloses a multi-target detection method, a multi-target detection model training method, an edge computing device and a medium, and the multi-target detection model training method comprises the steps: extracting a first-class feature map; respectively inputting the first type of feature map into a first target detection module, a multi-layer feature fusion module and a second target detection module; calculating a target loss function value based on the obtained target detection result and the target labeling result; and adjusting network parameters of the to-be-trained model until a preset model training convergence condition is met, and obtaining a multi-target detection model. According to the method, the generalization ability of the model is improved, and the driving safety is enhanced.

Description

[Technical field]

[0001] The present invention relates to the field of computer vision, and in particular to a multi-target detection method and model training method, edge computing equipment and medium. [Background technology]

[0002] With the rapid development of intelligent transportation systems and intelligent driving technologies, the ability of vehicles to accurately perceive their surroundings has become crucial. This perception ability is the basis for vehicles to achieve autonomous navigation, obstacle avoidance, and compliance with traffic rules. In this field, multi-target detection technology plays a core role, which requires the simultaneous identification and location of multiple different targets, including but not limited to lane lines, pedestrians, non-motor vehicles, motor vehicles, and obstacles. The accurate detection of these targets is essential for safe driving and decision-making of vehicles.

[0003] However, traditional target detection methods often only focus on a single type of detection task, such as detecting only traffic participants or only obstacles / markers, while ignoring other important targets in the environment. The limitation of this method is that it cannot fully understand and respond to complex traffic environments, which limits its adaptability and robustness. In changing road conditions and complex traffic flows, single target detection models are difficult to provide comprehensive situational awareness, which may lead to deviations in the decision-making of intelligent driving systems and even cause safety accidents. [Summary of the invention]

[0004] In order to solve one or more defects in the prior art, the present invention provides a multi-target detection method and model training method, edge computing device and medium.

[0005] In one aspect, the present invention provides a method for training a multi-target detection model, comprising:

[0006] Input the training image as an input image to a backbone feature extraction module included in the to-be-trained model, so as to extract a plurality of first-category feature maps corresponding to the training image based on the original feature map corresponding to the training image; wherein the number of channels of the plurality of first-category feature maps is different; the number of channels of the first-category feature maps is greater than the number of channels of the original feature map corresponding to the training image, and the number of features in the first-category feature map is greater than the number of features in the original feature map corresponding to the training image;

[0007] Inputting at least one of the first-type feature maps into a first target detection module included in the model to be trained to detect a first target of a predetermined type and obtain a first target detection result;

[0008] Inputting at least two of the first-type feature maps into a multi-layer feature fusion module included in the model to be trained, so as to perform multi-layer feature fusion on at least two of the first-type feature maps to obtain a plurality of second-type feature maps; the feature information in the second-type feature map is more than the feature information in the first-type feature map;

[0009] Inputting the plurality of the second-type feature maps into a second target detection module included in the model to be trained to detect a second target of a predetermined type and obtain a second target detection result;

[0010] Calculating a first target loss function value representing a first target detection loss based on the first target detection result and a first target annotation result of the training image acquired in advance; calculating a second target loss function value representing a second target detection loss based on the second target detection result and a second target annotation result of the training image acquired in advance;

[0011] The network parameters of the model to be trained are adjusted according to the first objective loss function value and the second objective loss function value until a predetermined model training convergence condition is met to obtain a trained multi-target detection model.

[0012] Optionally, extracting a plurality of first-category feature maps corresponding to the training image based on the original feature map corresponding to the training image includes:

[0013] For the original feature map of the training image, an offset maximum pooling downsampling process is used to extract the first type of feature map of the training image; wherein the number of offset maximum pooling downsampling processes used to obtain different first type feature maps is different; the offset maximum pooling downsampling process includes:

[0014] Cutting and filling different positions of the original feature map of the training image to obtain a plurality of different offset feature maps;

[0015] For each of the offset feature maps, performing a maximum pooling downsampling process on the offset feature map to obtain a downsampling result;

[0016] The multiple downsampling results are concatenated to obtain the first type of feature map.

[0017] Optionally, the plurality of first-type feature maps include a first feature map and a second feature map;

[0018] The method of extracting a first type of feature map of the training image by using offset maximum pooling downsampling processing from the original feature map of the training image comprises:

[0019] For the original feature map of the training image, the offset maximum pooling downsampling process is adopted to extract the first feature map of the training image;

[0020] Performing residual linking processing on the first feature map to obtain a first intermediate feature map; the first intermediate feature map has the same size as the first feature map;

[0021] The first intermediate feature map and the first feature map are subjected to channel merging processing to obtain the second feature map.

[0022] Optionally, the plurality of the second-type feature maps include a third feature map and a fourth feature map;

[0023] The step of performing multi-layer feature fusion on at least two of the first-type feature maps to obtain a plurality of second-type feature maps comprises:

[0024] Performing convolution processing on the first feature map to obtain the third feature map; the size of the third feature map is consistent with the size of the first feature map;

[0025] Performing residual link processing and convolution processing on the third feature map to obtain the fourth feature map, and the size of the fourth feature map is consistent with the size of the second feature map.

[0026] Optionally, the plurality of first-type feature maps further include a fifth feature map, and the plurality of second-type feature maps further include a sixth feature map;

[0027] After performing residual link processing and convolution processing on the third feature map to obtain the fourth feature map, the method further includes:

[0028] Channel addition processing is performed on the fourth feature map and the second feature map, and channel merging processing is performed on the channel addition processing result and the fifth feature map to obtain the sixth feature map.

[0029] Optionally, detecting a second target of a predetermined type to obtain a second target detection result includes:

[0030] Obtaining an expected number of channels; the expected number of channels is related to the number of preset target detection categories;

[0031] Performing convolution processing on at least two of the second-type feature maps respectively to obtain at least two second target detection feature maps having the same number of channels as the expected number of channels;

[0032] The second target detection result is determined based on at least two of the second target detection feature maps.

[0033] Optionally, detecting a first target of a predetermined type to obtain a first target detection result includes:

[0034] Performing convolution processing and residual link processing on at least one of the first-type feature maps to obtain a first target detection feature map having the same number of channels as a preset number of first targets;

[0035] The first target detection result is determined according to the first target detection feature map.

[0036] Optionally, before calculating a first target loss function value representing a first target detection loss based on the first target detection result and a first target annotation result of the training image acquired in advance, the method further includes:

[0037] Perform global detection on the first target detection feature map according to the global detection function:

[0038] P i,j =f(X i,j ),i∈[1,(H / 8)],j∈[1,L]

[0039] The optimization function of the global detection function is:

[0040]

[0041] Confirm the first objective structural constraint loss:

[0042]

[0043] Confirm that the first objective loss function is:

[0044] L total =αL str +βL class ,str:α+β=1

[0045] Where X is the first target detection feature map, X i,j is the jth channel in the i-th row of the first target detection feature map; f is the detection discriminant function, P i,j is the detection vector of the jth channel in the i-th row of the first target detection feature map, P i,j+1 is the detection vector of the j+1th channel in the i-th row of the first target detection feature map, ||P i,j -P i,j+1 ||2 represents the calculation of the second norm, H / 8 is the height of the first target detection feature map, L is the preset number of first targets, and L class is the category loss, L ce is the cross entropy loss, T i,j is the one-hot encoding value, L str is the first objective structure constraint loss, L total is the first objective loss function.

[0046] The present invention also provides a multi-target detection method, comprising inputting an image to be detected into a multi-target detection model trained by a training method of a multi-target detection model as described in any of the above items, and obtaining a first target detection result and a second target detection result of the image to be detected output by the multi-target detection model.

[0047] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute any of the multi-target detection methods described above.

[0048] The present invention also provides an edge computing device, comprising:

[0049] at least one processor;

[0050] and at least one memory in communication with the processor, wherein:

[0051] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute any of the multi-target detection methods described above.

[0052] The training method of the multi-target detection model provided by the embodiment of the present invention includes inputting a training image as an input image into a backbone feature extraction module included in the model to be trained, so as to extract multiple first-category feature maps corresponding to the training image based on the original feature map corresponding to the training image; inputting at least one first-category feature map into a first target detection module included in the model to be trained, so as to detect a first target of a predetermined type and obtain a first target detection result; inputting at least two first-category feature maps into a multi-layer feature fusion module included in the model to be trained, so as to perform multi-layer feature fusion on at least two first-category feature maps and obtain multiple second-category feature maps; inputting multiple second-category feature maps into a second target detection module included in the model to be trained, so as to detect a second target of a predetermined type and obtain a second target detection result; calculating a first target loss function value for representing the first target detection loss based on the first target detection result and the first target annotation result of the training image obtained in advance; calculating a second target loss function value for representing the second target detection loss based on the second target detection result and the second target annotation result of the training image obtained in advance; adjusting the network parameters of the model to be trained according to the first target loss function value and the second target loss function value until the predetermined model training convergence condition is met, thereby obtaining a trained multi-target detection model. The present invention significantly improves the detection accuracy of the first target and the second target of the predetermined type by extracting multi-level feature maps and fusing feature information at different levels, while enhancing the robustness of the model in complex traffic environments. The computational efficiency is optimized, allowing the model to process data quickly while maintaining high accuracy, which is crucial for intelligent driving systems that require real-time response. In addition, the present invention reduces the hardware requirements for model deployment and the complexity of development and maintenance by reducing computing resource consumption and simplifying the training process. By simultaneously optimizing different types of first target detection tasks and second target detection tasks through a unified training framework, the generalization ability of the model is improved, enabling it to adapt to different traffic scenarios, which is conducive to enhancing the safety of intelligent driving and providing strong technical support for the development of intelligent transportation systems.

Brief Description of the Drawings

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 A flowchart of a method for training a multi-target detection model provided by an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the structure of a multi-target detection model provided by an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of the structure of an edge computing device provided by an embodiment of the present invention. [Specific implementation method]

[0057] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0058] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0060] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0061] The inventors found in the process of practicing the embodiments of the present invention that traditional target detection methods can often only focus on a single type of detection task, such as only detecting traffic participants or only detecting obstacles / markers, while ignoring other important targets in the environment. The limitation of this method is that it cannot fully understand and respond to complex traffic environments, thereby limiting the adaptability and robustness of the target detection system. In changing road conditions and complex traffic flows, the model of single target detection is difficult to provide comprehensive situational awareness, which may lead to deviations in the decision-making of the intelligent driving system and even cause safety accidents. Therefore, an embodiment of the present invention provides a training method for a multi-target detection model to solve one or more of the above-mentioned technical problems. Among them, the multi-target may include multiple targets of different types, such as a first target and a second target of different types, and the type of the target may include but is not limited to at least one of the following: lane lines, pedestrians, motor vehicles, non-motor vehicles, traffic signs, traffic lights, obstacles, etc.

[0062] Please refer to Figure 1 , is a flow chart of a training method for a multi-target detection model provided by an embodiment of the present invention, comprising the following steps:

[0063] Step S01: input a training image as an input image into a backbone feature extraction module included in a model to be trained, so as to extract a plurality of first-category feature maps corresponding to the training image based on an original feature map corresponding to the training image.

[0064] In this embodiment, the number of channels of multiple first-category feature maps is different, the number of channels of the first-category feature maps is greater than the number of channels of the original feature maps corresponding to the training images, and the feature types in the first-category feature maps are more than the feature types of the original feature maps of the training images.

[0065] In this embodiment, the training image is first input into the backbone feature extraction module of the model, which is responsible for extracting multiple first-class feature maps from the original feature map. The number of channels of these feature maps is not only greater than that of the original feature map, but also the number of channels of each feature map is different, so that the types of features contained in the first-class feature map are richer. The purpose of this is to capture more detailed and diverse information in the training image, enhance the model's ability to recognize different types of targets, and provide a more comprehensive and in-depth feature representation for subsequent multi-target detection tasks, thereby improving the detection accuracy and robustness of the model.

[0066] In some embodiments, as mentioned in step S01, extracting a plurality of first-category feature maps corresponding to the training image based on the original feature map corresponding to the training image may specifically include:

[0067] For the original feature map of the training image, the offset maximum pooling downsampling process is used to extract the first type of feature map of the training image.

[0068] Among them, the number of times of offset maximum pooling downsampling processing used to obtain different first-category feature maps is different.

[0069] The process of the offset maximum pooling downsampling processing may specifically include the following steps:

[0070] Step S11, cutting and filling different positions in the original feature map of the training image to obtain a plurality of different offset feature maps.

[0071] In this embodiment, by simulating observations from different perspectives and positions, the model's ability to capture local features in the image is enhanced. In order to simulate observation perspectives from different positions so that the model can capture the features of the target from multiple angles and improve the model's adaptability to changes in target shape, size, and position, this embodiment applies windows of different sizes and positions to the original feature map for cropping, and performs appropriate padding on the edges of the cropped image to maintain the size consistency of the feature map, thereby generating multiple offset feature maps. In this way, the model can learn a more comprehensive feature representation, thereby improving the accuracy and robustness of detection in practical applications.

[0072] In some embodiments, the elements of all channels in the leftmost column of the original feature map may be cut off first, and a column of zero elements in which all channels are zero elements is added to the right side of the original feature map to obtain a first offset feature map;

[0073] The elements of all channels in the two leftmost columns of the original feature map are cut off, and two columns of zero elements in all channels are added to the right of the original feature map to obtain the second offset feature map;

[0074] The elements of all channels in the three leftmost columns of the original feature map are cut off, and three columns of zero elements in all channels are added to the right side of the original feature map to obtain the third offset feature map.

[0075] Step S12: For each offset feature map, perform maximum pooling downsampling processing on the offset feature map to obtain a downsampling result.

[0076] In this embodiment, for each offset feature map, a maximum pooling downsampling process is performed, which is mainly used to reduce the spatial dimension of the feature map while retaining the most important information. The maximum pooling operation effectively reduces the amount of data, reduces the computational complexity of the model, and reduces the risk of overfitting to a certain extent by selecting the maximum value in the local area of ​​the feature map as the representative of the area. In addition, maximum pooling also helps the model to obtain translation invariance, that is, the model remains unchanged for small changes in the position of the object in the input image, allowing the model to still recognize the target when the object position is slightly offset. In this way, the model can focus more on the significant features in the image rather than the slight changes in the details, thereby improving the model's ability to recognize targets in the image.

[0077] In some embodiments, maximum pooling convolution processing may be performed on the first offset feature map, the second offset feature map, and the third offset feature map, respectively, to obtain a first downsampling result, a second downsampling result, and a third downsampling result, respectively.

[0078] Step S13, concatenating multiple downsampling results to obtain a first type of feature map.

[0079] In this embodiment, the downsampling results of different offset feature maps after the maximum pooling operation are merged into a single feature map to increase the number of channels and information richness of the feature map. The purpose of this is to integrate the information of different perspectives and scales captured in each offset feature map, so as to provide a more comprehensive image feature representation for the model. The spliced ​​feature map not only contains the local details in the original feature map, but also incorporates the global context information obtained after different offsets and downsampling processes, which helps the model make more accurate judgments in subsequent first target and second target detection tasks. The splicing operation effectively utilizes the idea of ​​spatial pyramid pooling and enhances the model's perception of features of different scales to adapt to multi-target detection tasks in complex and changeable traffic scenes.

[0080] Here, the size of the original feature map A1 is H1×W1×C1. The convolution kernel size used in the maximum pooling convolution is 3×3, and the convolution step is 2. Among them, H1 is the height of the original feature map A1, W1 is the width of the original feature map A1, and C1 is the number of channels of the original feature map A1. The processing process of the offset maximum pooling downsampling process includes:

[0081] 1) Cut off the elements of all channels in the leftmost column of the original feature map A1, and fill all channels in the rightmost column with 0 elements to obtain the first offset feature map; use the maximum pooling convolution to process the first offset feature map to obtain the first downsampling result K1; the size of K1 is (H1 / 2)×(W1 / 2)×C1;

[0082] 2) Cut off the elements of all channels in the two leftmost columns of the original feature map A1, and fill all channels in the two rightmost columns with 0 elements to obtain a second offset feature map; use the maximum pooling convolution to process the second offset feature map to obtain a second downsampling result K2; the size of the second downsampling result K2 is (H1 / 2)×(W1 / 2)×C1;

[0083] 3) The elements of all channels in the leftmost three columns of the original feature map A1 are cut off, and 0 elements are added to all channels in the rightmost three columns to obtain a third offset feature map; the third offset feature map is processed using the maximum pooling convolution to obtain a third downsampling result K3; the size of the third downsampling result K3 is (H1 / 2)×(W1 / 2)×C1;

[0084] 4) The original feature map A1 is processed by a maximum pooling convolution to obtain a fourth downsampling result K4; the size of the fourth downsampling result K4 is (H1 / 2)×(W1 / 2)×C1;

[0085] 5) The first downsampling result K1, the second downsampling result K2, the third downsampling result K3, and the fourth downsampling result K4 are spliced ​​in the channel direction to obtain a spliced ​​feature map K5, whose size is (H1 / 2)×(W1 / 2)×(4×C1);

[0086] 6) Perform convolution processing on the concatenated feature map K5 to obtain a feature map K6; the size of the feature map K6 is (H1 / 2)×(W1 / 2)×(4×C1).

[0087] Based on the above technical solution, this embodiment first crops and fills different positions of the original feature map to generate multiple offset feature maps, then performs a maximum pooling operation on each offset feature map to downsample and extract key features, and finally splices these downsampling results to form multiple first-class feature maps with different numbers of channels and rich features. That is, by changing the number and position of the offset of the pooling operation, this embodiment can capture features of different scales and different regions in the image, enhance the spatial sensitivity of the model to the target, and thus improve the accuracy and robustness of the detection. This enables the model to understand the training image from multiple angles and multiple scales, and capture more comprehensive feature information to cope with complex and changing traffic environments.

[0088] Based on the above embodiments, in some embodiments, the plurality of first-category feature maps may include a first feature map and a second feature map.

[0089] On this basis, the original feature map of the training image mentioned above is subjected to the offset maximum pooling downsampling process to extract the first type of feature map of the training image, which may specifically include the following steps:

[0090] Step S21, for the original feature map of the training image, an offset maximum pooling downsampling process is adopted to extract the first feature map of the training image.

[0091] In some embodiments, different first feature maps of the training image can be obtained by performing different times of offset maximum pooling downsampling processing on the original feature map. For example, the original feature map T1 can be subjected to four times of offset maximum pooling downsampling processing to obtain four different first feature maps: T2, T3, T4, and T5.

[0092] Step S22, performing residual linking processing on the first feature map to obtain a first intermediate feature map.

[0093] In this embodiment, the first intermediate feature map is the same size as the first feature map. Residual links (also known as skip connections) allow signals in the network to bypass some layers and pass directly, which can solve the gradient vanishing problem in deep network training because the direct path ensures that the gradient propagates back unimpeded. In this way, the network can learn the identity mapping, that is, directly pass the input features to the subsequent layers, which helps the network to remain stable during training and can increase the depth of the network without causing performance degradation. In this embodiment, the first intermediate feature map is the same size as the first feature map, which makes the residual link processing process not only transfer features, but also maintain the consistency of spatial dimensions, which helps to more effectively fuse feature information later, thereby improving the model's ability to learn image features and multi-target detection performance.

[0094] In some embodiments, the processing process of residual link processing may specifically be:

[0095] The first feature map is subjected to three consecutive convolution processes, and the results of the three convolution processes are concatenated in the channel direction to obtain the first intermediate feature map.

[0096] Step S23: perform channel merging processing on the first intermediate feature map and the first feature map to obtain a second feature map.

[0097] In this embodiment, by splicing two feature maps in the channel dimension, the dimension of the feature can be increased, thereby fusing information from different levels. The reason for this is that the first intermediate feature map contains information of the same size as the first feature map after residual link processing, but the expression of the feature is different. Through channel merging, the second feature map can have unique information in two sets of feature maps at the same time, enhancing the expressiveness of the feature. This fusion method helps the model to better utilize features at different levels in subsequent tasks and improve the ability to recognize and locate targets in images.

[0098] The following is an example of the calculation process of the backbone feature extraction module, taking the input original image as a three-channel RGB image. Assuming the size of the sample image is H×W×C, the calculation process includes:

[0099] 1) After the sample image is processed by convolution, the number of channels of the original image is changed to N, which is recorded as feature map T1;

[0100] 2) Based on step 1, the feature map T1 is subjected to offset maximum pooling downsampling processing; the size of the obtained second feature map is (H / 2)×(W / 2)×(4×N), which is recorded as feature map T2;

[0101] 3) Based on step 2, the feature map T2 is downsampled by offset maximum pooling to obtain a feature map with a size of (H / 4)×(W / 4)×(16×N), which is recorded as feature map T3;

[0102] 4) Based on step 3, the feature map T3 is downsampled by offset maximum pooling to obtain a feature map with a size of (H / 8)×(W / 8)×(64×N), which is recorded as feature map T4;

[0103] 5) Based on step 4, the feature map T4 is downsampled by offset maximum pooling to obtain a feature map with a size of (H / 16)×(W / 16)×(256×N), which is recorded as feature map T5;

[0104] 6) Based on step 5, the feature map T5 is processed by deep feature fusion to obtain a feature map with a size of (H / 16)×(W / 16)×(512×N), which is recorded as feature map T6;

[0105] Among them, the processing process of deep feature fusion processing includes:

[0106] ①For a certain feature layer, the feature map is recorded as Ic1;

[0107] ②Perform convolution processing on the feature map Ic1 to obtain the feature map Ic2;

[0108] ③ Perform residual link processing on the feature map Ic2 at least once to obtain the feature map Ic3;

[0109] ④ Perform convolution processing on the feature map Ic1 to obtain the feature map Ic4;

[0110] ⑤ Concatenate the feature map Ic3 and the feature map Ic4 in the channel direction to obtain the feature map Ic5;

[0111] ⑥Perform convolution processing on the feature map Ic5 to obtain the feature map Ic6.

[0112] Deep feature fusion processing can make full use of feature information at different levels, including original features and features that have undergone different processing steps, thereby improving the model's understanding of details and context in the image, enhancing the model's generalization ability and robustness to adapt to multi-target detection tasks in complex and changeable traffic scenes.

[0113] 7) Based on step 6, the feature map T6 is downsampled by offset maximum pooling to obtain a feature map with a size of (H / 32)×(W / 32)×(2048×N), which is recorded as feature map T7;

[0114] 8) Based on step 7, the feature map T7 is convolved to obtain a feature map with a size of (H / 32)×(W / 32)×(128×N), which is recorded as feature map T8;

[0115] The feature maps T4 and T5 are determined as the first feature maps, and the feature map T8 is determined as the second feature map.

[0116] Based on the above technical solution, this embodiment extracts the first feature map by applying offset maximum pooling downsampling processing to the original feature map of the training image, which helps to capture the features of different regions and scales in the image. By performing residual link processing on the first feature map, a first intermediate feature map of the same size as the first feature map is generated, which helps to alleviate the problem of gradient disappearance or explosion in deep network training and enhance the learning ability of the model. Finally, by merging the first intermediate feature map with the first feature map through channels, a second feature map is obtained. This merging operation realizes feature fusion and enhances the amount of information in the feature map, so that the model can comprehensively utilize image information from different angles and scales, and improve the recognition ability and detection accuracy of the target. This method aims to build a richer feature representation by combining the advantages of different processing steps to adapt to complex and changing traffic environments and improve the performance of intelligent driving systems.

[0117] In some embodiments, the backbone feature extraction module may also be composed of a series of convolutional layers, activation functions, and pooling layers for extracting low-level and high-level features of the sample image, wherein:

[0118] The convolutional layer applies multiple convolutional filters to extract local features such as edges and textures in the sample image.

[0119] Activation functions are used to add nonlinearity and help the model learn more complex feature representations.

[0120] The pooling layer reduces the spatial dimension of the feature map through pooling processing (such as maximum pooling or average pooling) while increasing the invariance to image displacement.

[0121] In the convolutional network of the backbone feature extraction module, each layer produces a set of first-class feature maps that capture abstract representations of the input image at different levels, where the first-class feature maps of lower layers usually contain more spatial details, while the first-class feature maps of higher layers contain more semantic information. For example, the initial layer may focus on edge detection, while higher layers may recognize more complex shapes and object parts.

[0122] In some embodiments, the backbone feature extraction module extracts useful feature information from the input image and converts the information into a first-class feature map, which is then used for further analysis and processing, which may specifically include:

[0123] Convolution operation: The feature extraction process is first performed through a convolution layer, where each convolution kernel is responsible for extracting specific features in the image, such as edges, corners, or more abstract texture information.

[0124] Activation function: The output of the convolutional layer usually passes through an activation function, such as ReLU (Rectified Linear Unit), to introduce non-linearity, which helps the model learn more complex feature representations.

[0125] Pooling operation: In order to reduce the spatial dimension of features, reduce the number of parameters and computational complexity, a pooling layer such as maximum pooling or average pooling is usually applied after the convolutional layer.

[0126] Multi-scale feature extraction: Extract features from convolutional layers at different levels to capture feature information at different scales.

[0127] Batch Normalization: Used to speed up the training process and reduce internal covariate shift by normalizing the output of the layer.

[0128] Feature fusion: Feature fusion techniques, such as skip connections, are used at different stages of the backbone network to combine low-level features with high-level features to enhance the feature expression capabilities of the model.

[0129] Output feature map: The backbone feature extraction module outputs a set of first-class feature maps, which contain compressed and abstract representations of the input image and provide a basis for subsequent target detection or other tasks.

[0130] In some embodiments, the process of acquiring the training image may be:

[0131] Image acquisition: Collect a large amount of image data from different traffic scenes, which can cover various weather, lighting and traffic conditions.

[0132] Data preprocessing: Preprocess the collected images, such as adjusting the image size, normalizing pixel values, etc., to meet the input requirements of the model.

[0133] Step S02: input at least one of the first-type feature maps into a first target detection module included in the model to be trained to detect a first target of a predetermined type and obtain a first target detection result.

[0134] In this embodiment, at least one first-class feature map is input into the first target detection module included in the model to be trained in order to use these first-class feature maps that have been processed at multiple levels to detect a specific type of first target, such as a lane line. This step captures the key visual information in the image through the extracted feature map, so that the model can identify and locate the first target and obtain the first target detection result, so that the model can focus on a specific type of detection task, such as lane line detection, thereby improving the accuracy and efficiency of detection. In this way, the model can provide accurate lane position information for intelligent driving vehicles, helping the vehicle stay in the correct lane and avoid lane deviation or collision.

[0135] In some embodiments, the first target detection module is used to identify and locate lane lines from the output results of the backbone feature extraction module, and a specific algorithm, such as Hough transform or a segmentation network based on deep learning, can be used to detect the boundary of the road. Its workflow is as follows: receiving the first type of feature map output by the backbone feature extraction module, identifying the position of the lane line through a specific lane detection algorithm, and converting it into a representation of pixel coordinates or a lane position annotation box.

[0136] In some embodiments, the first target of a predetermined type is detected in step S02 to obtain a first target detection result, which may specifically be:

[0137] Performing convolution processing and residual link processing on at least one first-category feature map to obtain a first target detection feature map having the same number of channels as a preset number of first targets;

[0138] A first target detection result is determined according to the first target detection feature map.

[0139] In this embodiment, by inputting at least one first-class feature map into the first target detection module of the model to be trained, the model performs convolution processing and residual link processing on these feature maps to extract and strengthen features related to the first-class target. These operations can enhance the model's ability to recognize lane line features, and ultimately obtain a first target detection feature map whose number of channels matches the number of first targets expected to be detected. Each channel corresponds to a lane line, so that the model can accurately locate and identify each lane line separately. Based on this first target detection feature map, the model can determine the first target detection result, that is, the position and state of the lane line, which is crucial for path planning and safe driving of intelligent driving vehicles. Such detection results provide the vehicle with the necessary environmental information, enabling it to follow traffic rules, stay in the correct lane, and avoid traffic accidents.

[0140] The following takes the first type of feature map as the second feature map T8 as an example for explanation, and the size of the second feature map T8 is (H / 32)×(W / 32)×(128×N):

[0141] The feature map T8 is subjected to two convolution processes, residual link processing, and one convolution process in sequence to obtain the first target detection feature map O4 of the preset number of lane lines. The size of O4 is (H / 8)×(W / 8)×L, where L is the preset number of first targets, that is, the number of lane lines. In practice, since the vehicle only pays attention to the lane it is in and the surrounding adjacent lanes during driving, the number of lane lines L can be 2, 4, or 6 to facilitate lane line detection.

[0142] Step S03: input at least two first-category feature maps into a multi-layer feature fusion module included in the model to be trained, so as to perform multi-layer feature fusion on the at least two first-category feature maps to obtain a plurality of second-category feature maps.

[0143] In this embodiment, the feature information in the second type of feature map is more than that in the first type of feature map. At least two first type feature maps are input into the multi-layer feature fusion module of the model to be trained, and multi-layer feature fusion is performed to obtain multiple second type feature maps. This process aims to integrate feature information of different levels and scales to enrich the feature representation of the model. In this way, the second type of feature map contains more feature information than the first type of feature map. This is because multi-layer feature fusion can integrate low-level detail information and high-level semantic information to enhance the model's comprehensive understanding of the image content. The reason for this is that the second type of feature map that integrates multi-scale features can provide the model with more comprehensive contextual information, which can help the model better identify and locate targets and improve the accuracy and robustness of detection.

[0144] In some embodiments, the plurality of second-type feature maps may include a third feature map and a fourth feature map;

[0145] On this basis, as mentioned in step S03, multi-layer feature fusion is performed on at least two first-type feature maps to obtain multiple second-type feature maps, which may specifically include the following steps:

[0146] Step S31, performing convolution processing on the first feature map to obtain a third feature map.

[0147] In this embodiment, the size of the third feature map is consistent with the size of the first feature map. The original feature map is processed by the convolution layer to extract higher-level features while maintaining the spatial resolution of the feature map, which is crucial for subsequent feature fusion and second target detection. Keeping the size of the feature map consistent can ensure the spatial alignment of different feature maps during feature fusion, so that the fused feature map can more accurately reflect the spatial structure and detail information of the original image, which is important for improving the model's ability to spatially locate targets in the image and the accuracy of feature recognition.

[0148] Step S32, performing residual linking processing and convolution processing on the third feature map to obtain a fourth feature map.

[0149] In this embodiment, the size of the fourth feature map is consistent with the size of the second feature map. By keeping the size of the fourth feature map consistent with the second feature map, the consistency of the spatial resolution of the feature map can be ensured, which not only improves the expressiveness of the features, but also helps the network learn richer and more robust feature representations, thereby improving the performance and generalization ability of the model in complex scenarios.

[0150] Based on the above technical solution, this embodiment obtains a third feature map of the same size by convolution processing on the first feature map, which can retain the important spatial information in the original feature map. By performing residual link processing and convolution processing on the third feature map, a fourth feature map of the same size as the second feature map is obtained, which not only helps to capture deeper features, but also alleviates the gradient vanishing problem in deep networks through residual links, allowing the network to learn more effectively. Such a multi-layer feature fusion method enables the model to capture rich feature information at different levels, enhance the model's understanding of image details and context, thereby improving the performance of tasks such as target detection and lane line detection, and improving the accuracy and robustness of multi-target detection in complex and changing traffic environments.

[0151] On the basis of the above embodiment, in some embodiments, the plurality of first-type feature maps may further include a fifth feature map, and the plurality of second-type feature maps may further include a sixth feature map;

[0152] After executing step S32, performing residual link processing and convolution processing on the third feature map to obtain a fourth feature map, the following steps may be further performed:

[0153] Step S33, performing channel addition processing on the fourth feature map and the second feature map, and performing channel merging processing on the channel addition processing result and the fifth feature map to obtain a sixth feature map.

[0154] In this embodiment, the fourth feature map and the second feature map are subjected to channel addition processing, and the channel addition processing result and the fifth feature map are subjected to channel merging processing to obtain the sixth feature map, which is a feature fusion strategy in deep learning. This strategy increases the amount of information of the feature by channel addition, so that the amount of information in each dimension increases, while channel merging increases the number of features, thereby enriching the dimension of the feature. The reason for this is that channel addition can help the model fuse information from different feature maps in each channel and enhance the expressive power of the feature, while channel merging expands the number of features, so that the model can capture more diverse features and improve the recognition and positioning capabilities of targets in the image. This feature fusion method is crucial to improving the performance and generalization ability of the model in complex scenarios, especially in tasks such as target detection and lane line detection, which enables the model to better utilize feature information at different levels and from different sources, thereby improving the accuracy and robustness of detection.

[0155] Please refer to Figure 2 , is a schematic diagram of the structure of a target and lane fusion detection model provided by an embodiment of the present invention. Figure 2 As shown, taking the first characteristic map T4 and the fifth characteristic map T5 as an example, the size of the first characteristic map T4 is (H / 8)×(W / 8)×(64×N), the size of the fifth characteristic map T5 is (H / 16)×(W / 16)×(256×N), and the acquisition process of the third characteristic map, the fourth characteristic map and the sixth characteristic map may include:

[0156] 1) The first feature map T4 is convolved to obtain a feature map with a size of (H / 8)×(W / 8)×(32×N), which is recorded as the third feature map T9;

[0157] 2) The third feature map T9 is convolved to obtain a feature map with a size of (H / 16)×(W / 16)×(64×N);

[0158] 3) Perform residual link processing based on step 2);

[0159] 4) Based on step 3), a feature map with a size of (H / 32)×(W / 32)×(128×N) is obtained through convolution processing, which is recorded as the fourth feature map T10.

[0160] 5) The second feature map T8 and the fourth feature map T10 are channel-added to obtain a feature map of size (H / 32)×(W / 32)×(256×N), which is recorded as feature map T11;

[0161] 5) Perform convolution processing on the feature map T11 to obtain a feature map of size (H / 16)×(W / 16)×(256×N), which is recorded as feature map T13;

[0162] 6) The feature map T13 and the fifth feature map T5 are subjected to channel merging processing, and the size of the feature map obtained is (H / 16)×(W / 16)×(256×N), which is recorded as feature map T14;

[0163] 7) Based on step 6), a feature map with a size of (H / 32)×(W / 32)×(256×N) is obtained through convolution processing, which is recorded as the sixth feature map T12.

[0164] Step S04: input multiple second-category feature maps into a second target detection module included in the model to be trained to detect a second target of a predetermined type and obtain a second target detection result.

[0165] In this embodiment, multiple second-class feature maps are input into the second target detection module of the model to be trained in order to use the second-class feature map that has been processed by multi-layer feature fusion to detect the second target of the predetermined type in the image and obtain the second target detection result. The purpose of this step is to use the richer and more comprehensive feature information in the second-class feature map to improve the detection accuracy of specific targets. In this way, the model can more effectively identify and locate the second target in the image, such as vehicles, pedestrians or other traffic participants. The reason for this is that the second-class feature map provides a more comprehensive image representation than a single feature map by fusing features at different levels and from different sources, which helps the model make more accurate judgments in complex and changeable traffic environments and improves the accuracy of multi-target detection tasks.

[0166] In some embodiments, the step S04 mentioned above, detecting a second target of a predetermined type and obtaining a second target detection result may specifically include the following steps:

[0167] Step S41, obtaining the expected number of channels.

[0168] In this embodiment, the expected number of channels is related to the number of preset target detection categories. Obtaining the expected number of channels, i.e. determining the number of channels related to the preset number of target detection categories, is an important step in the design of the target detection network. This step involves assigning an independent channel to each target category, so the expected number of channels directly corresponds to the total number of target categories that need to be identified. The reason for this is that each channel can be specifically used to predict the existence of a specific category of targets and their attributes (such as position, size, etc.), so that the network can output a probability distribution for each category, thereby realizing multi-category target detection. This design allows the model to generate prediction results for each category for each pixel at the output layer, improving the flexibility and accuracy of detection.

[0169] Step S42: performing convolution processing on at least two second-category feature maps respectively to obtain at least two second target detection feature maps having the same number of channels as the expected number of channels.

[0170] In this embodiment, a dedicated channel is generated for each target category through a customized convolution operation, so that the number of channels of each output feature map matches the number of target categories. The reason for this is that each channel can specifically extract and respond to target features of a specific category, so that the model can generate an independent probability distribution map for each category, which is crucial for subsequent target classification and positioning. In this way, the model can more accurately identify and locate multiple target categories in the image, improving the accuracy and efficiency of target detection.

[0171] Step S43, determining a second target detection result according to at least two second target detection feature maps.

[0172] In this embodiment, multiple feature maps obtained through convolution processing are used to comprehensively judge and identify the second target in the image, such as vehicles, pedestrians and other traffic participants. The existence, category and location of the target are determined by analyzing the activation pattern and response intensity in these feature maps. The reason for this is that a single feature map may not provide enough information to accurately identify all targets, especially when the target features are not obvious or there are occlusions and complex backgrounds. By integrating the information of multiple feature maps, the model can use features of different scales and levels to improve the accuracy and robustness of detection, thereby more effectively identifying and locating targets in complex traffic environments.

[0173] like Figure 2 As shown, taking the second type of feature map as the third feature map T9, the fourth feature map T10 and the sixth feature map T12 as an example, the calculation process of the second target detection result is described:

[0174] 1) Let the number of target recognition categories be m;

[0175] 2) Perform convolution processing on the third feature map T9 to obtain a feature map with a size of (H / 8)×(W / 8)×([m+5]×3), which is recorded as the second target detection feature map O1;

[0176] 3) Perform a convolution operation on the fourth feature map T10 to obtain a feature map with a size of (H / 16)×(W / 16)×([m+5]×3), which is recorded as the second target detection feature map O2;

[0177] 4) Perform a convolution operation on the sixth feature map T12 to obtain a feature map with a size of (H / 32)×(W / 32)×([m+5]×3), which is recorded as the second target detection feature map O3;

[0178] The second target detection results at different scales are determined according to the second target detection feature maps O1, O2, and O3.

[0179] Based on the above technical solution, this embodiment sets the expected number of channels for each target detection category, and performs convolution processing on at least two second-category feature maps to obtain second target detection feature maps that match the expected number of channels, thereby achieving accurate detection of a predetermined type of second target. This embodiment can extract features in a customized manner for each target category, so that the feature map can be more focused on targets of the corresponding category, thereby improving the accuracy of target detection and category differentiation. In addition, by determining the second target detection result based on at least two such second target detection feature maps, the model can integrate the information of multiple feature maps and enhance the ability to recognize targets.

[0180] Step S05, based on the first target detection result and the first target labeling result of the pre-acquired training image, a first target loss function value representing the first target detection loss is calculated; based on the second target detection result and the second target labeling result of the pre-acquired training image, a second target loss function value representing the second target detection loss is calculated.

[0181] In this embodiment, the accuracy of the model prediction is measured by comparing the output of the model (the first target detection result and the second target detection result) with the actual labeled data (the first target labeling result and the second target labeling result), and calculating the loss function value. The loss function can quantify the difference between the model prediction and the actual labeling, and provide a direction for improvement of the model. By minimizing these loss values, the model can learn how to more accurately detect and identify targets in the image, including lane lines, vehicles, pedestrians, etc. The optimization process of the loss function ensures that the model continuously adjusts parameters during the training process to better adapt to the training data, thereby achieving more accurate target detection in practical applications.

[0182] In some embodiments, before executing step S05, and calculating the first target loss function value representing the first target detection loss based on the first target detection result and the first target annotation result of the pre-acquired training image, the following steps may also be performed:

[0183] Perform global detection on the first target detection feature map according to the global detection function:

[0184] P i,j =f(X i,j ),i∈[1,(H / 8)],j∈[1,L]

[0185] The optimization function of the global detection function is:

[0186]

[0187] Confirm the first objective structural constraint loss:

[0188]

[0189] Confirm that the first objective loss function is:

[0190] L total =αL str +βL class ,str:α+β=1

[0191] Among them, X is the first target detection feature map, X i,j is the jth channel in the i-th row of the first target detection feature map; f is the detection discriminant function, P i,j is the detection vector of the jth channel in the i-th row of the first target detection feature map, P i,j+1 is the detection vector of the j+1th channel in the i-th row of the first target detection feature map, ||P i,j -P i,j+1 ||2 represents the calculation of the second norm, H / 8 is the height of the first target detection feature map, L is the preset number of first targets, and L class is the category loss, L ce is the cross entropy loss, T i,j is the one-hot encoding value, L str is the first objective structure constraint loss, L total is the first objective loss function. Among them, the meaning of the lane line structure constraint loss indicates that the detection results of the lane lines of adjacent rows are relatively small.

[0192] In some embodiments, the first loss function may also include but is not limited to: mean square error loss function, mean absolute error loss function, Huber loss function, etc. According to the selected first loss function, the loss value between the lane position detection result and the lane marking result is calculated.

[0193] In some embodiments, the second loss function may include:

[0194] Classification loss function: measures the difference between the predicted category and the true category, such as the cross entropy loss function.

[0195] Bounding box regression loss function: measures the difference between the predicted bounding box and the true bounding box, such as Smooth L1 Loss or IoU Loss (Intersection over Union Loss).

[0196] After calculating the classification loss and bounding box regression loss between the target detection result and the target annotation result, they are merged into a total loss value as the second loss function value.

[0197] In some embodiments, the first target annotation result may be the location information of the lane line marked in the sample image. These annotations may be represented in the form of curves or polygons, showing the boundaries and directions of the lanes. The first target annotation result is used to train the lane position detection module. By learning these annotations, the model can identify and predict the location of the lane line, thereby assisting the intelligent driving vehicle to stay in the correct lane.

[0198] The second target annotation result may be the location and category information of the target object manually or automatically marked in the sample image. These annotations are usually expressed in the form of bounding boxes, which may contain category labels (such as "vehicle", "pedestrian", etc.). The target annotation results are used to train the target detection module. By comparing the bounding boxes predicted by the model with the actually annotated bounding boxes, the model can learn how to more accurately identify and locate the objects in the image.

[0199] Based on the above technical solution, this embodiment can provide a clear feedback signal for the model to guide the model to adjust parameters during the training process to reduce prediction errors. By minimizing these loss function values, the model can learn how to more accurately detect the first target (such as lane lines) and the second target (such as vehicles, pedestrians, etc.), thereby improving the accuracy of target detection. At the same time, the generalization ability of the model is also enhanced, so that it can better adapt to different detection tasks and environmental conditions in practical applications.

[0200] Step S06, adjusting the network parameters of the model to be trained according to the first objective loss function value and the second objective loss function value until the predetermined model training convergence condition is met to obtain a trained multi-target detection model.

[0201] In this embodiment, the performance of the model on a specific task is evaluated by calculating the loss function value, and the network parameters are adjusted by gradient descent or other optimization algorithms according to these loss values, with the aim of reducing prediction errors and improving the accuracy of the model. The purpose is that by continuously iterating and optimizing the network parameters, the model can learn how to better extract features from the input data and accurately predict the location and category of the first target (such as lane lines) and the second target (such as vehicles and pedestrians). Ultimately, when the model meets the predetermined convergence conditions, it can be considered that the model has been trained. At this time, the multi-target detection model can handle multiple detection tasks at the same time, with high accuracy and generalization ability, which is crucial to achieving efficient and reliable intelligent driving and intelligent transportation systems.

[0202] The present invention designs the second target detection module and the first target detection module based on the output results of the backbone feature extraction module. By sharing the backbone feature extraction module, the model avoids multiple independent feature extractions on the same input image, reduces the amount of calculation and resource consumption, and greatly improves the model efficiency.

[0203] In this embodiment, the first objective loss function value and the second objective loss function value can be combined into a total loss value to optimize the lane position detection task and the target detection task at the same time. The total loss value is then used for back propagation to calculate the gradient of each parameter. Finally, the parameters of the model are updated according to the gradient and the preset learning rate to reduce the prediction error. In this way, the model continuously learns how to reduce the errors of lane position and target detection during the training process, and finally achieves a higher detection accuracy, and finally obtains a multi-target detection model, which can provide richer and more accurate environmental perception information for intelligent driving.

[0204] Based on the above technical solution, the present invention significantly improves the detection accuracy of the first target and the second target of the predetermined type by extracting multi-level feature maps and fusing feature information at different levels, while enhancing the robustness of the model in complex traffic environments. The computational efficiency is optimized, allowing the model to process data quickly while maintaining high accuracy, which is crucial for intelligent driving systems that require real-time response. In addition, the present invention reduces the hardware requirements for model deployment and the complexity of development and maintenance by reducing computing resource consumption and simplifying the training process. By simultaneously optimizing different types of first target detection tasks and second target detection tasks through a unified training framework, the generalization ability of the model is improved, enabling it to adapt to different traffic scenarios, which is conducive to enhancing the safety of intelligent driving.

[0205] Based on the above embodiments, the present invention also provides a multi-target detection method, including: inputting an image to be detected into a multi-target detection model trained by the training method of the multi-target detection model described in any of the above embodiments, and obtaining a first target detection result and a second target detection result of the image to be detected output by the multi-target detection model.

[0206] The multi-target detection method provided by the present invention can realize efficient and accurate simultaneous detection of multiple targets in the image by inputting the image to be detected into the multi-target detection model trained by the above-mentioned training method, and obtaining the first target detection result and the second target detection result output by the model. By processing multiple detection tasks through a unified model, the computing cost is reduced and the processing speed is improved. At the same time, since the model has learned how to integrate the characteristics of different targets during training, it can provide more robust detection results, especially in complex traffic environments. In addition, the method can provide real-time and accurate environmental perception information for intelligent driving systems and intelligent traffic management systems, enhancing the safety and reliability of the system.

[0207] On the basis of the above embodiments, the present invention also provides a target and lane position detection method using a multi-target detection model training method such as any of the above, and a multi-target detection model is obtained using a multi-target detection model training method such as any of the above; the target image to be predicted is input into the multi-target detection model, and the multi-target detection model outputs a target image annotated with a target annotation box and a lane position annotation box.

[0208] The present invention also provides an edge computing device, including a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the training method of the multi-target detection model as described in the figure are implemented.

[0209] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the following is implemented: Figure 1 The steps of the training method of the multi-target detection model.

[0210] The embodiment of the present invention also provides a computer program product, which, when executed by a processor, implements the following Figure 1 The steps of the training method of the multi-target detection model.

[0211] The following specific reference Figure 3, which shows a schematic diagram of the structure of the edge computing device 700 suitable for implementing the embodiment of the present invention. The edge computing device 700 in the embodiment of the present invention may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), eye trackers, AR / VR / MR smart glasses, wearable devices, etc., and fixed terminals such as digital TVs, desktop computers, etc. The edge computing device shown is only an example and should not bring any limitations to the functions and occupancy range of the embodiments of the present invention.

[0212] like Figure 3 As shown, the edge computing device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 710, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 720 or a program loaded from a storage device 780 to a random access memory (RAM) 730. In RAM730, various programs and data required for the operation of the edge computing device are also stored. The processing device 710, ROM720, and RAM730 are connected to each other via a bus 740. An input / output (I / O) interface 750 is also connected to the bus 740.

[0213] Typically, the following devices may be connected to the I / O interface 750: input devices 760 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 770 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 780 including, for example, a tape, a hard disk, etc.; and communication devices 790. The communication device 790 may allow the edge computing device 700 to communicate wirelessly or wired with other devices to exchange data. Although an edge computing device 700 having various devices is shown, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or provided alternatively.

[0214] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the training method of the multi-target detection model of the embodiment of the present invention are executed.

[0215] It should be noted that the computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0216] In some embodiments, the user end and the management end may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0217] The above-mentioned computer-readable medium may be included in the above-mentioned edge computing device; or it may exist independently without being assembled into the edge computing device.

[0218] The computer-readable medium carries one or more programs. When the one or more programs are executed by the edge computing device, the edge computing device:

[0219] Input the training image as an input image to a backbone feature extraction module included in the to-be-trained model, so as to extract a plurality of first-category feature maps corresponding to the training image based on the original feature map corresponding to the training image; wherein the number of channels of the plurality of first-category feature maps is different; the number of channels of the first-category feature maps is greater than the number of channels of the original feature map corresponding to the training image, and the number of features in the first-category feature map is greater than the number of features in the original feature map corresponding to the training image;

[0220] Inputting at least one of the first-type feature maps into a first target detection module included in the model to be trained to detect a first target of a predetermined type and obtain a first target detection result;

[0221] Inputting at least two of the first-type feature maps into a multi-layer feature fusion module included in the model to be trained, so as to perform multi-layer feature fusion on at least two of the first-type feature maps to obtain a plurality of second-type feature maps; the feature information in the second-type feature map is more than the feature information in the first-type feature map;

[0222] Inputting the plurality of the second-type feature maps into a second target detection module included in the model to be trained to detect a second target of a predetermined type and obtain a second target detection result;

[0223] Calculating a first target loss function value representing a first target detection loss based on the first target detection result and a first target annotation result of the training image acquired in advance; calculating a second target loss function value representing a second target detection loss based on the second target detection result and a second target annotation result of the training image acquired in advance;

[0224] The network parameters of the model to be trained are adjusted according to the first objective loss function value and the second objective loss function value until a predetermined model training convergence condition is met to obtain a trained multi-target detection model.

[0225] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0226] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0227] The units involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a unit does not limit the unit itself in some cases.

[0228] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0229] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0230] According to one or more embodiments of the present invention, the present invention provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the training method of the multi-target detection model as described in any one of the embodiments of the present invention.

[0231] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

[0232] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0233] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A training method for a multi-target detection model, characterized in that: include: Inputting the training image as an input image into a backbone feature extraction module included in the to-be-trained model, so as to extract a plurality of first-category feature maps corresponding to the training image based on the original feature map corresponding to the training image; wherein the plurality of first-category feature maps have different numbers of channels; the number of channels of the first-category feature maps is greater than the number of channels of the original feature map corresponding to the training image, and the number of features in the first-category feature map is greater than the number of features in the original feature map corresponding to the training image; Inputting at least one of the first-type feature maps into a first target detection module included in the model to be trained to detect a first target of a predetermined type and obtain a first target detection result; Inputting at least two of the first-type feature maps into a multi-layer feature fusion module included in the model to be trained, so as to perform multi-layer feature fusion on at least two of the first-type feature maps to obtain a plurality of second-type feature maps; the feature information in the second-type feature map is more than the feature information in the first-type feature map; Inputting the plurality of the second-type feature maps into a second target detection module included in the model to be trained to detect a second target of a predetermined type and obtain a second target detection result; Calculating a first target loss function value representing a first target detection loss based on the first target detection result and a first target annotation result of the training image acquired in advance; calculating a second target loss function value representing a second target detection loss based on the second target detection result and a second target annotation result of the training image acquired in advance; The network parameters of the model to be trained are adjusted according to the first objective loss function value and the second objective loss function value until a predetermined model training convergence condition is met to obtain a trained multi-target detection model.

2. The method according to claim 1, characterized in that The extracting a plurality of first-category feature maps corresponding to the training image based on the original feature map corresponding to the training image comprises: For the original feature map of the training image, an offset maximum pooling downsampling process is used to extract the first type of feature map of the training image; wherein the number of offset maximum pooling downsampling processes used to obtain different first type feature maps is different; the offset maximum pooling downsampling process includes: Cutting and filling different positions of the original feature map of the training image to obtain a plurality of different offset feature maps; For each of the offset feature maps, performing a maximum pooling downsampling process on the offset feature map to obtain a downsampling result; The multiple downsampling results are concatenated to obtain the first type of feature map.

3. The method according to claim 2, characterized in that The plurality of first-type feature maps include a first feature map and a second feature map; The method of extracting a first type of feature map of the training image by using offset maximum pooling downsampling processing from the original feature map of the training image comprises: For the original feature map of the training image, the offset maximum pooling downsampling process is adopted to extract the first feature map of the training image; Performing residual linking processing on the first feature map to obtain a first intermediate feature map; the first intermediate feature map has the same size as the first feature map; The first intermediate feature map and the first feature map are subjected to channel merging processing to obtain the second feature map.

4. The method according to claim 3, characterized in that: The plurality of the second-type feature maps include a third feature map and a fourth feature map; The step of performing multi-layer feature fusion on at least two of the first-type feature maps to obtain a plurality of second-type feature maps comprises: Performing convolution processing on the first feature map to obtain the third feature map; the size of the third feature map is consistent with the size of the first feature map; Performing residual link processing and convolution processing on the third feature map to obtain the fourth feature map, and the size of the fourth feature map is consistent with the size of the second feature map.

5. The method according to claim 4, characterized in that The plurality of first-type feature maps further include a fifth feature map, and the plurality of second-type feature maps further include a sixth feature map; After performing residual link processing and convolution processing on the third feature map to obtain the fourth feature map, the method further includes: Channel addition processing is performed on the fourth feature map and the second feature map, and channel merging processing is performed on the channel addition processing result and the fifth feature map to obtain the sixth feature map.

6. The method according to claim 1, characterized in that The detecting of a second target of a predetermined type to obtain a second target detection result includes: obtaining an expected number of channels; the expected number of channels is related to the number of preset target detection categories; performing convolution processing on at least two of the second-category feature maps respectively to obtain at least two second target detection feature maps with the same number of channels as the expected number of channels; determining the second target detection result according to the at least two second target detection feature maps; and / or, The detecting of a first target of a predetermined type and obtaining a first target detection result includes: performing convolution processing and residual link processing on at least one feature map of the first type to obtain a first target detection feature map with the same number of channels as a preset number of first targets; and determining the first target detection result based on the first target detection feature map.

7. The method according to claim 6, characterized in that Before calculating a first target loss function value representing a first target detection loss based on the first target detection result and a first target annotation result of the training image acquired in advance, the method further includes: Perform global detection on the first target detection feature map according to the global detection function: P i,j =f(X i,j ),i∈[1,(H / 8)],j∈[1,L] The optimization function of the global detection function is: Confirm the first objective structural constraint loss: Confirm that the first objective loss function is: L total =αL str +βL class ,str:a+b=1 Where X is the first target detection feature map, X i,j is the jth channel in the i-th row of the first target detection feature map; f is the detection discriminant function, P i,j is the detection vector of the jth channel in the i-th row of the first target detection feature map, P i,j+1 is the detection vector of the j+1th channel in the i-th row of the first target detection feature map, ||P i,j -P i,j+1 ||2 represents the calculation of the second norm, H / 8 is the height of the first target detection feature map, L is the preset number of first targets, and L class is the category loss, L ce is the cross entropy loss, T i,j is the one-hot encoding value, L str is the first objective structure constraint loss, L total is the first objective loss function.

8. A multi-target detection method, characterized in that: include: The image to be detected is input into the multi-target detection model trained by the training method of the multi-target detection model as described in any one of claims 1 to 7, and the first target detection result and the second target detection result of the image to be detected are obtained by the multi-target detection model output.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the method according to any one of claims 1 to 7.

10. An edge computing device, characterized in that: include: at least one processor; and at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 7 by calling the program instructions.

Images