Target detection method and device based on adaptive image mixed data enhancement

Abstract

The application relates to a target detection method and device based on adaptive image mixing data enhancement, wherein the method comprises the following steps: acquiring a plurality of original image pairs through a training data set; extracting a feature map pair of each original image pair through a first shared network, and acquiring a mixed image of the feature map pair by using an adaptive image mixing network; calculating a first loss function of the mixed image through the first shared network, and optimizing the adaptive image mixing network in combination with an auxiliary loss function, so as to optimize the mixed image; optimizing a second shared network through the original image pair and the optimized mixed image, updating the first shared network parameters, training the optimized second shared network in combination with the optimized adaptive image mixing network, and generating a target detection model to perform a target detection operation. Therefore, the problems that the adaptive image mixing data enhancement method is insufficient in the amount of collected and labeled data in actual application, and greatly limits the application of the target detection model are solved.

Description

Technical Field

[0001] This application relates to the field of image data enhancement technology, and in particular to a target detection method and apparatus based on adaptive image hybrid data enhancement. Background Technology

[0002] Object detection is one of the most fundamental and important applications in computer vision. Given an image, the object detection task aims to identify target objects in the image and return their corresponding bounding boxes for object localization. Object detection has been widely applied in fields such as autonomous driving, video surveillance, and medical image analysis. Although object detection has broad application potential, the limited amount of collected and labeled data is often a significant bottleneck restricting the application of object detection methods in practice.

[0003] To address the challenges of data scarcity and high data collection costs, image data augmentation methods are widely used in various tasks to enhance model generalization capabilities. Image data augmentation increases the quantity and diversity of training samples by transforming and expanding the original data. In recent years, image fusion methods have demonstrated more robust performance and achieved success in image classification tasks, attracting increasing attention. These methods primarily enhance the diversity of the original dataset by merging image pairs or different regions of images. Adaptive image fusion data augmentation strategies provide an effective approach to generating mixed image samples. The augmented data can be considered as extracted from a neighboring distribution that approximates the true distribution; therefore, the augmented dataset typically contains a more comprehensive feature representation of the real data.

[0004] However, adaptive image fusion data augmentation methods are mainly applied to image classification tasks, and can usually only handle images containing only a single object category. Therefore, applying adaptive image fusion data augmentation methods to object detection tasks still faces serious challenges that urgently need to be addressed. Summary of the Invention

[0005] This application provides a target detection method and apparatus based on adaptive image fusion data augmentation to solve the problem that the amount of data collected and labeled by adaptive image fusion data augmentation methods in practical applications is insufficient, which greatly limits the application of target detection models.

[0006] The first aspect of this application provides a target detection method based on adaptive image fusion data enhancement, applied in the online training phase, comprising the following steps: constructing a training dataset of the target object, and obtaining multiple pairs of original images through the training dataset; extracting feature map pairs of each of the multiple original image pairs through a first shared network in a preset Siamese network, and inputting the feature map pairs into an adaptive image fusion network in the preset Siamese network to obtain a fused image corresponding to the feature map pairs; calculating a first loss function corresponding to the fused image based on the first shared network, and optimizing the adaptive image fusion network through the first loss function and a preset auxiliary loss function, and optimizing the fused image using the optimized adaptive image fusion network; optimizing a second shared network in the preset Siamese network through the original image pairs and the optimized fused image, and updating the parameters of the first shared network using an exponential moving average strategy, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image fusion network to generate a target detection model, and performing target detection operations through the target detection model in the online detection phase.

[0007] Optionally, in one embodiment of this application, the step of inputting the feature map pair into the adaptive image mixing network in the preset Siamese network to obtain the mixed image corresponding to the feature map pair includes: embedding a preset mixing ratio into the feature map pair to obtain an embedded feature map pair, and calculating a similarity matrix using the embedded feature map pair; generating an image mixing pixel-level mask based on the adaptive image mixing network and the embedded feature map pair, and through the similarity matrix and a preset upsampling function; and mixing the original image pair according to the image mixing pixel-level mask to obtain the mixed image.

[0008] Optionally, in one embodiment of this application, the step of calculating the first loss function corresponding to the mixed image based on the first shared network includes: inputting the mixed image into the first shared network, and extracting the first average mixing ratio of the target object in the mixed image from the image mixing pixel-level mask according to a preset mixing weight strategy; and calculating the first loss function based on the first average mixing ratio of the target object.

[0009] Optionally, in one embodiment of this application, the step of optimizing the second shared network in the preset Siamese network using the original image pair and the optimized hybrid image, and updating the parameters of the first shared network using an exponential moving average strategy, includes: inputting the original image pair into the second shared network to calculate a second loss function corresponding to the original image pair using the second shared network; inputting the optimized hybrid image into the second shared network, and extracting a second average mixing ratio of the target object in the optimized hybrid image according to the mixing weight strategy, to calculate a third loss function corresponding to the optimized hybrid image using the second shared network and the second average mixing ratio; optimizing the second shared network using the second loss function and the third loss function, and updating the parameters of the optimized second shared network to obtain optimized parameters; and updating the parameters of the first shared network based on the exponential moving average strategy and the optimized parameters.

[0010] Optionally, in one embodiment of this application, the mathematical expression of the auxiliary loss function is:

[0011]

[0012] Wherein, λ represents the mixing ratio; λ j s represents the mean of the target mixing ratio; i,h,w This represents the image's pixel-level blending mask; ∈ is a constant term.

[0013] A second aspect of this application provides a target detection method based on adaptive image fusion data enhancement, applied in the online detection stage, comprising the following steps: acquiring image data of the target to be tested; inputting the image data into a pre-trained target detection model to output the detection result of the target to be tested, wherein the target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image fusion network in a preset Siamese network, and in combination with a preset training dataset.

[0014] A third aspect of this application provides a target detection device based on adaptive image fusion data enhancement, applied in the online training phase, comprising: a first acquisition module for constructing a training dataset of target objects and acquiring multiple pairs of original images through the training dataset; an extraction module for extracting feature map pairs of each of the multiple pairs of original images through a first shared network in a preset Siamese network, and inputting the feature map pairs into an adaptive image fusion network in the preset Siamese network to obtain a fused image corresponding to the feature map pairs; an optimization module for calculating a first loss function corresponding to the fused image based on the first shared network, optimizing the adaptive image fusion network through the first loss function and a preset auxiliary loss function, and optimizing the fused image using the optimized adaptive image fusion network; and a training module for optimizing a second shared network in the preset Siamese network through the original image pairs and the optimized fused image, updating the parameters of the first shared network using an exponential moving average strategy, training the optimized second shared network according to the updated parameters and the optimized adaptive image fusion network to generate a target detection model, and performing target detection operations through the target detection model in the online detection phase.

[0015] Optionally, in one embodiment of this application, the extraction module includes: an embedding unit, configured to embed a preset mixing ratio into the feature map pair to obtain an embedded feature map pair, and to calculate a similarity matrix using the embedded feature map pair; a generation unit, configured to generate an image mixing pixel-level mask based on the adaptive image mixing network and the embedded feature map pair, and through the similarity matrix and a preset upsampling function; and a mixing unit, configured to mix the original image pair according to the image mixing pixel-level mask to obtain the mixed image.

[0016] Optionally, in one embodiment of this application, the optimization module includes: an input unit, configured to input the mixed image into the first shared network and extract the first average mixing ratio of the target object in the mixed image from the image mixed pixel-level mask according to a preset mixing weight strategy; and a first calculation unit, configured to calculate the first loss function based on the first average mixing ratio of the target object.

[0017] Optionally, in one embodiment of this application, the training module includes: a second calculation unit, configured to input the original image pair into the second shared network to calculate a second loss function corresponding to the original image pair through the second shared network; a third calculation unit, configured to input the optimized mixed image into the second shared network, and extract a second average mixing ratio of the target object in the optimized mixed image according to the mixing weight strategy, so as to calculate a third loss function corresponding to the optimized mixed image through the second shared network and the second average mixing ratio; a first update unit, configured to optimize the second shared network through the second loss function and the third loss function, and update the parameters of the optimized second shared network to obtain optimized parameters; and a second update unit, configured to update the parameters of the first shared network based on the exponential moving average strategy and the optimized parameters.

[0018] Optionally, in one embodiment of this application, the mathematical expression of the auxiliary loss function is:

[0019]

[0020] Wherein, λ represents the mixing ratio; λ j s represents the mean of the target mixing ratio; i,h,w This represents the image's pixel-level blending mask; ∈ is a constant term.

[0021] The fourth aspect of this application provides a target detection device based on adaptive image fusion data enhancement, applied in the online detection stage, comprising: a second acquisition module for acquiring image data of a target to be tested; and a detection module for inputting the image data into a pre-trained target detection model to output the detection result of the target to be tested, wherein the target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image fusion network in a preset Siamese network, and in combination with a preset training dataset.

[0022] A fifth aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method based on adaptive image fusion data enhancement as described in the above embodiments.

[0023] A sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described target detection method based on adaptive image fusion data enhancement.

[0024] Therefore, the embodiments of this application have the following beneficial effects:

[0025] The embodiments of this application construct a training dataset for the target object and obtain multiple pairs of original images from the training dataset. Feature map pairs are extracted from each of the multiple original image pairs using a first shared network in a preset Siamese network. These feature map pairs are then input into an adaptive image mixing network within the preset Siamese network to obtain a mixed image corresponding to the feature map pairs. Based on the first shared network, a first loss function corresponding to the mixed image is calculated. The adaptive image mixing network is then optimized using the first loss function and a preset auxiliary loss function, and the optimized image is further optimized using the optimized adaptive image mixing network. A second shared network in the preset Siamese network is optimized using the original image pairs and the optimized mixed image. The parameters of the first shared network are updated using an exponential moving average strategy. The optimized second shared network is then trained based on the updated parameters and the optimized adaptive image mixing network to generate a target detection model. During the online detection phase, the target detection model performs target detection operations. This application effectively expands the quantity and diversity of the original dataset from a data augmentation perspective, effectively improving the performance and generalization of the target detection model. Therefore, it solves the problem that the adaptive image mixing data augmentation method suffers from insufficient data collection and annotation in practical applications, which greatly limits the application of the target detection model.

[0026] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0027] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 This is a flowchart of a target detection method based on adaptive image fusion data augmentation applied in the online training phase, according to an embodiment of this application.

[0029] Figure 2 A schematic diagram of an automated optimized mixed sample generation data augmentation framework is provided for one embodiment of this application;

[0030] Figure 3 A schematic diagram of the execution logic of an adaptive image mixing network provided for one embodiment of this application;

[0031] Figure 4 A schematic diagram of an image blending effect provided for one embodiment of this application;

[0032] Figure 5 A logical diagram illustrating a target-level hybrid weighting strategy provided in one embodiment of this application;

[0033] Figure 6 A schematic diagram of the logical architecture of a target detection method based on adaptive image fusion data enhancement, provided for one embodiment of this application;

[0034] Figure 7 This is a flowchart of a target detection method based on adaptive image fusion data enhancement applied in the online detection stage according to an embodiment of this application;

[0035] Figure 8 This is an example diagram of a target detection device based on adaptive image fusion data augmentation applied in the online training phase according to an embodiment of this application;

[0036] Figure 9 This is an example diagram of a target detection device based on adaptive image fusion data enhancement applied in the online detection stage according to an embodiment of this application;

[0037] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0038] Among them, 10-target detection device based on adaptive image hybrid data enhancement applied to the online training stage, 20-target detection device based on adaptive image hybrid data enhancement applied to the online detection stage; 101-first acquisition module, 102-extraction module, 103-optimization module, 104-training module; 201-second acquisition module, 202-detection module; 1001-memory, 1002-processor, 1003-communication interface. Detailed Implementation

[0039] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0040] The following describes a target detection method and apparatus based on adaptive image fusion data enhancement according to embodiments of this application with reference to the accompanying drawings. Addressing the problems mentioned in the background art, this application provides a target detection method based on adaptive image fusion data enhancement. In this method, a training dataset of the target object is constructed, and multiple original image pairs are obtained through the training dataset. Feature map pairs of each original image pair are extracted through a first shared network in a preset Siamese network, and the feature map pairs are input into an adaptive image fusion network in the preset Siamese network to obtain a fused image corresponding to the feature map pairs. Based on the first shared network, a first loss function corresponding to the fused image is calculated, and the adaptive image fusion network is optimized using the first loss function and a preset auxiliary loss function. The fused image is then optimized using the optimized adaptive image fusion network. A second shared network in the preset Siamese network is optimized using the original image pairs and the optimized fused image, and the parameters of the first shared network are updated using an exponential moving average strategy. The optimized second shared network is trained based on the updated parameters and the optimized adaptive image fusion network to generate a target detection model. During the online detection phase, the target detection operation is performed using the target detection model. This application effectively expands the quantity and diversity of the original dataset from a data augmentation perspective, thereby significantly improving the performance and generalization of the object detection model. This solves the problem that adaptive image fusion data augmentation methods often suffer from insufficient collected and labeled data in practical applications, which greatly limits the application of object detection models.

[0041] Specifically, Figure 1 This is a flowchart illustrating an adaptive image fusion data augmentation-based target detection method applied during the online training phase, as provided in an embodiment of this application.

[0042] like Figure 1 As shown, the target detection method based on adaptive image fusion data enhancement includes the following steps:

[0043] In step S101, a training dataset for the target object is constructed, and multiple pairs of original images are obtained through the training dataset.

[0044] The embodiments of this application first initialize the parameters of each network model in a preset automatic optimized mixed sample generation data augmentation framework. This framework adopts a Siamese network structure, including network k (i.e., the first shared network) and network q (i.e., the second shared network). The two networks share the same structure and initial parameters. In addition, the framework also includes an adaptive image mixing network, which can use a random initialization strategy for parameter initialization. Network q is mainly used for object detection, while the more stable network k is connected to the adaptive image mixing network for adaptive image mixing, such as... Figure 2 As shown, this enables the automatic optimization of mixed sample generation for data enhancement in the field of target detection.

[0045] Subsequently, embodiments of this application also require the construction of a training dataset for the target object, and the random selection of two training images from the training dataset as original image pairs, thereby providing reliable data support for the training of subsequent networks such as network q and adaptive image hybrid networks.

[0046] In step S102, feature map pairs of each original image pair are extracted from multiple original image pairs through the first shared network in the preset Siamese network, and the feature map pairs are input into the adaptive image mixing network in the preset Siamese network to obtain the mixed image corresponding to the feature map pairs.

[0047] After acquiring multiple original image pairs, the embodiments of this application further require inputting each original image pair into the backbone network of network k, so as to extract the feature map of each original image pair through the backbone network in network k, provide basic feature representation for subsequent tasks, and input the extracted feature map into the adaptive image mixing network to obtain the mixed image corresponding to the feature map pair.

[0048] Optionally, in one embodiment of this application, the feature map pair is input into an adaptive image mixing network in a preset Siamese network to obtain a mixed image corresponding to the feature map pair, including: embedding a preset mixing ratio into the feature map pair to obtain an embedded feature map pair, and calculating a similarity matrix using the embedded feature map pair; generating an image mixing pixel-level mask based on the adaptive image mixing network and the embedded feature map pair, and through the similarity matrix and a preset upsampling function; and mixing the original image pair according to the image mixing pixel-level mask to obtain a mixed image.

[0049] It should be noted that embodiments of this application can utilize an adaptive image blending module. Based on image sample pairs Feature map pairs obtained from (i.e., original image pairs) The preset blending ratio λ predicts pixel-level mask pairs (s) used to generate image blending. i ,s j ), to be used to generate a blended image. Wherein, mask s u The value in represents the image. The blending ratio at the corresponding position, mask s j The value in represents the image. The mixing ratio at the corresponding position.

[0050] Specifically, in adaptive image mixing networks In this application, the embodiments can first be implemented in a simple and effective manner. Embed λ into the l-th layer feature map to obtain λ-embedding-based features. and (i.e., embedding feature map pairs) and using them as The input; secondly, according to and Output generated image with pixel-level mask s i Used for blending of embedded feature map pairs. Based on symmetry, x j mask s j via s j =1-s i Obtained through calculation. The generated pixel-level mask pairs (s) i ,s j This is used to blend original image pairs to obtain a blended image. The mathematical expression for this process is:

[0051]

[0052]

[0053] Here, ⊙ represents element-wise multiplication. Indicates according to The resulting blended image.

[0054] Image blending pixel-level mask s includes The corresponding mask s i ,s j Obviously, s i ,s j It exhibits symmetry, meaning the sum of the two masks used to generate the mixed sample equals 1, as shown in the following equation:

[0055] s i +s j =1

[0056] This application embodiment obtains an image mixing pixel-level mask, namely The process is as follows:

[0057] First, embodiments of this application can calculate the similarity matrix P through a cross-attention mechanism, as shown in the following equation:

[0058]

[0059] Among them, W P This represents a shared linear transformation matrix (1×1 convolution). Represents matrix multiplication. Normalization factor;

[0060] Secondly, embodiments of this application can utilize adaptive image mixing networks. s is calculated using a pairwise similarity matrix P and an upsampling function U(·). i As shown in the following formula:

[0061]

[0062] Among them, W Z σ is the linear transformation matrix; σ is the Sigmoid activation function used to probabilize the mask; s i To obtain a mask with the same size H×W as the sample image; by embedding P with the value Multiply, choose x i,λ Relative to x j,1-λ The discriminative features, and based on the symmetry strategy, x j mask s j It can be accessed via s j =1-s i Calculated.

[0063] The process of obtaining image mixed pixel-level masks according to embodiments of this application will be described below with reference to the accompanying drawings.

[0064] Figure 3 This is a schematic diagram illustrating the execution logic of an adaptive image mixing network. For example... Figure 3 As shown, the steps for obtaining the image blending pixel-level mask through an adaptive image blending network in this embodiment are as follows:

[0065] 1. Embed the obtained λ into the feature map of the l-th layer. and Perform a 1×1 convolution operation, and the output is obtained. and

[0066] 2. and Perform matrix multiplication and normalize the results using the softmax function to obtain the similarity matrix P;

[0067] 3. Feature map of λ embedded into the features of layer l Performing a 1×1 convolution operation alone, the output is the value embedding.

[0068] 4. Combine the similarity matrix P and The result of multiplication is probabilized using a Sigmoid activation function and then upsampled to obtain an image blending pixel-level mask s with the same size as the sample image. i .

[0069] Therefore, the embodiments of this application obtain a pixel-level mask for image mixing based on feature map pairs and mixing ratio through an adaptive image mixing network, thereby generating a mixed image and providing a reliable foundation for the training of the corresponding model.

[0070] In step S103, based on the first shared network, the first loss function corresponding to the mixed image is calculated, and the adaptive image mixing network is optimized by the first loss function and the preset auxiliary loss function, and the mixed image is optimized by the optimized adaptive image mixing network.

[0071] Furthermore, embodiments of this application can input the generated hybrid image into network k to calculate the target detection loss function (i.e., the first loss function) of the hybrid image through network k, and optimize the adaptive image mixing network through backpropagation to optimize the hybrid image through the optimized adaptive image mixing network, thereby improving the quality of the subsequently generated hybrid image.

[0072] Optionally, in one embodiment of this application, the mathematical expression of the auxiliary loss function is:

[0073]

[0074] Where λ represents the mixing ratio; λ j s represents the mean of the target mixing ratio; i,h,w Represents a pixel-level mask for image blending; ∈ is a constant term.

[0075] In actual implementation, in order to make the adaptive image mixing network Able to learn in the early stages of training To establish the correspondence between λ and object level, and to avoid semantic inconsistencies caused by image-level mixing and object detection as object-level tasks in adaptive image mixing networks, embodiments of this application propose an auxiliary loss function for object-level adaptive image mixing networks.

[0076] Specifically, embodiments of this application adjust the loss constraint range from the image level to the object level, and utilize a small amount of background information to make the adaptive image fusion network more sensitive to the distinction between foreground targets and background, thereby enabling the adaptive image fusion network to focus more on learning the proportional relationships at the object level. The mathematical expression of the auxiliary loss function is as follows:

[0077]

[0078] Where λ represents the randomly generated mixing ratio; λ j This represents the mean of the target blending ratio extracted from the pixel-level mask of the generated image; s i,h,w Represents the pixel-level mask of the image generated by the adaptive image mixing network, ∈ = 0.1.

[0079] Therefore, embodiments of this application ensure that the output image blends pixel-level masks by setting an auxiliary loss function. i To preserve the feature regions of each object to the greatest extent, the performance of the object detection model is further improved.

[0080] It is understood that embodiments of this application generate blended images from the target level to ensure that the key features of the target are preserved as accurately as possible, such as Figure 4 As shown, the key to this process is to avoid generating mixed samples that may mislead the target detection task, thereby enabling them to become useful training samples. This helps the target detection model learn the key features of the target and ultimately improve the performance of the model. The mixed image generation method in this application not only has accuracy but also helps ensure that the target detection model can more effectively capture and understand the features of the target during the learning process, providing a reliable foundation for improving the performance of the model.

[0081] Optionally, in one embodiment of this application, calculating a first loss function corresponding to a blended image based on a first shared network includes: inputting the blended image into the first shared network, and extracting the first average blending ratio of the target object in the blended image from the image blending pixel-level mask according to a preset blending weight strategy; and calculating the first loss function based on the first average blending ratio of the target object.

[0082] It should be noted that after obtaining the mixed image, the embodiments of this application can input the mixed image into network k, use network k to calculate the target detection loss function (i.e., the first loss function) of the mixed image, and feed it back to the adaptive image mixing network through backpropagation, thereby optimizing the quality of the generated mixed image.

[0083] As one possible approach, when calculating the first loss function of the blended image, embodiments of this application design a target-level blending weight strategy. Specifically, in embodiments of this application, the loss weight coefficient for each target is obtained by extracting the average blending ratio (i.e., the first average blending ratio) of that target from the generated image blending pixel-level mask, such as... Figure 5 As shown, and used to calculate object detection, classification, and regression losses, the mathematical expression of the first loss function for the mixed images is as follows:

[0084]

[0085] Where i represents the index of the target to be predicted; λ i The mixed weights represent the real targets that are closest to the predicted targets; and These represent the classification and regression losses for a single target, respectively.

[0086] Therefore, the embodiments of this application calculate the average mixing ratio of each target from the generated image mixed pixel-level mask, and use the obtained average mixing ratio as the loss weight of each target when calculating the target detection classification and regression loss of the mixed image. Thus, by introducing target-level mixing weights, adaptive mixing of target detection labels is achieved, which provides strong support for improving the target detection performance of mixed images.

[0087] In step S104, the second shared network in the preset Siamese network is optimized by using the original image pairs and the optimized hybrid image, and the parameters of the first shared network are updated using an exponential moving average strategy. The optimized second shared network is trained based on the updated parameters and the optimized adaptive image hybrid network to generate an object detection model. In the online detection stage, the object detection operation is performed through the object detection model.

[0088] Furthermore, embodiments of this application can jointly train network q (i.e., the second shared network) using the original image pairs and the optimized hybrid image to calculate the target detection loss function (i.e., the second loss function) for the original image pairs and the target detection loss function (i.e., the third loss function) for the optimized hybrid image, respectively. Simultaneously, network q updates its network parameters through backpropagation. To avoid feature collapse, embodiments of this application can use an exponential moving average strategy to update the parameters of network k, so as to train the optimized network q based on the updated parameters and the optimized adaptive image hybrid network, thereby obtaining the final target detection model.

[0089] Optionally, in one embodiment of this application, optimizing the second shared network in a preset Siamese network using the original image pairs and the optimized hybrid image, and updating the parameters of the first shared network using an exponential moving average strategy, includes: inputting the original image pairs into the second shared network to calculate a second loss function corresponding to the original image pairs; inputting the optimized hybrid image into the second shared network, and extracting a second average mixing ratio of the target object in the optimized hybrid image according to a mixing weight strategy, to calculate a third loss function corresponding to the optimized hybrid image using the second shared network and the second average mixing ratio; optimizing the second shared network using the second loss function and the third loss function, and updating the parameters of the optimized second shared network to obtain optimized parameters; and updating the parameters of the first shared network based on the exponential moving average strategy and the optimized parameters.

[0090] It should be noted that the embodiments of this application train the input network q with the original image using a conventional object detection network and calculate the corresponding object detection network loss (i.e., the second loss function) so that the backbone network can provide stable feature mapping in the early stage, thereby accelerating the convergence speed of the optimized adaptive image mixing network.

[0091] In the embodiments of this application, the main loss function of the target detection network includes two parts: the target detection anchor box regression loss and the classification loss, as shown in the following formula:

[0092]

[0093] Where i represents the index of the target to be predicted; and Let represent the classification and regression losses for the predicted target, respectively, and sum the classification and regression losses for all targets.

[0094] Furthermore, embodiments of this application also require inputting the optimized hybrid image into network q to expand the number and diversity of training samples, and calculating the optimized hybrid image object detection loss function (i.e., the third loss function), which has the same mathematical expression as the first loss function mentioned above, thereby improving the performance of the model object detection.

[0095] In summary, the embodiments of this application employ an adaptive mixed sample generation data augmentation method in the field of object detection, and design an end-to-end trainable automatic optimized mixed sample generation data augmentation framework for object detection. This framework generates adaptive pixel-level masks, which are then used to generate mixed samples to assist model training based on the original data sample pairs, thereby improving the performance of the object detection model. Since the label types for object detection and classification tasks are inconsistent, label mixing strategies used for classification tasks are no longer applicable to object detection tasks. The embodiments of this application utilize an object-level mixing weight strategy to achieve adaptive mixing of object detection labels tailored to the characteristics of the object detection task. Furthermore, to ensure that the feature regions of multiple objects are preserved during the learning of image data mixing ratios, and to avoid interference with the original features of the objects due to mixing, the embodiments of this application improve the adaptive image mixing network. By adjusting the constraints from the entire image to primarily targeting the objects to be detected, the adaptive image mixing network focuses more on the detected objects, and the output mixed mask preserves the feature regions of each object as much as possible, thereby further improving the performance of the object detection model.

[0096] It is understood that, in order to directly improve the performance of the object detection model from the perspective of expanding data diversity, the embodiments of this application divide the model training process into two sub-tasks: a hybrid image optimization task and an object detection task, such as... Figure 6As shown, the image fusion task is regarded as an auxiliary task to the object detection task, thereby solving the problems of the separate fusion data augmentation strategy and introducing a more flexible collaborative optimization. In addition, in the traditional image fusion strategy, the optimization of the image fusion task and the object detection task are separated, which can easily lead to the generated fusion samples misleading the object detection task. However, the embodiments of this application combine the above two tasks in a mutually beneficial way, thereby effectively ensuring that the fusion samples play an effective role in the object detection task.

[0097] The target detection method based on adaptive image fusion data augmentation proposed in this application, applied to the online training stage, involves constructing a training dataset of the target object and obtaining multiple pairs of original images from the training dataset. Feature map pairs are extracted from each of the multiple original image pairs using a first shared network in a pre-defined Siamese network. These feature map pairs are then input into an adaptive image fusion network within the pre-defined Siamese network to obtain a fused image corresponding to the feature map pairs. Based on the first shared network, a first loss function corresponding to the fused image is calculated. The adaptive image fusion network is then optimized using the first loss function and a pre-defined auxiliary loss function, and the fused image is further optimized using the optimized adaptive image fusion network. A second shared network in the pre-defined Siamese network is optimized using the original image pairs and the optimized fused image. The parameters of the first shared network are updated using an exponential moving average strategy. The optimized second shared network is then trained based on the updated parameters and the optimized adaptive image fusion network to generate a target detection model. During the online detection stage, the target detection model is used to perform target detection operations. This application effectively expands the quantity and diversity of the original dataset from a data augmentation perspective, effectively improving the performance and generalization of the target detection model.

[0098] Figure 7 This is a flowchart illustrating a target detection method based on adaptive image fusion data augmentation applied in the online detection stage, as provided in an embodiment of this application.

[0099] like Figure 7 As shown, the target detection method based on adaptive image fusion data enhancement includes the following steps:

[0100] In step S701, image data of the target to be tested is acquired.

[0101] In step S702, image data is input into a pre-trained target detection model to output the detection result of the target to be tested. The target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image hybrid network in a preset Siamese network, combined with a preset training dataset.

[0102] Therefore, embodiments of this application input the original image to be detected into a pre-trained object detection model, thereby identifying the predicted object detection box and category on the original image and performing visualization output.

[0103] The target detection method based on adaptive image fusion data augmentation proposed in this application, applied to the online detection stage, acquires image data of the target to be detected; inputs the image data into a pre-trained target detection model to output the detection result of the target to be detected. The target detection model is obtained by training a pre-set second shared network based on a first shared network and an adaptive image fusion network in a pre-set Siamese network, combined with a pre-set training dataset. This application effectively expands the quantity and diversity of the original dataset from the perspective of data augmentation, effectively improving the performance and generalization of the target detection model.

[0104] Secondly, the target detection apparatus based on adaptive image fusion data enhancement proposed according to the embodiments of this application will be described with reference to the accompanying drawings.

[0105] Figure 8 This is a block diagram of an adaptive image fusion data augmentation-based target detection device applied in the online training phase according to an embodiment of this application.

[0106] like Figure 8 As shown, the target detection device 10 based on adaptive image fusion data enhancement includes: a first acquisition module 101, an extraction module 102, an optimization module 103, and a training module 104.

[0107] The first acquisition module 101 is used to construct a training dataset for the target object and acquire multiple pairs of original images through the training dataset.

[0108] The extraction module 102 is used to extract feature map pairs of each original image pair from multiple original image pairs through the first shared network in the preset Siamese network, and input the feature map pairs into the adaptive image mixing network in the preset Siamese network to obtain the mixed image corresponding to the feature map pairs.

[0109] The optimization module 103 is used to calculate the first loss function corresponding to the mixed image based on the first shared network, optimize the adaptive image mixing network through the first loss function and the preset auxiliary loss function, and optimize the mixed image using the optimized adaptive image mixing network.

[0110] The training module 104 is used to optimize the second shared network in the preset Siamese network using the original image pairs and the optimized hybrid image, and to update the parameters of the first shared network using an exponential moving average strategy, so as to train the optimized second shared network based on the updated parameters and the optimized adaptive image hybrid network to generate an object detection model, and to perform object detection operations through the object detection model in the online detection stage.

[0111] Optionally, in one embodiment of this application, the extraction module 102 includes an embedding unit, a generation unit, and a mixing unit.

[0112] The embedding unit is used to embed a preset mixing ratio into a feature map pair to obtain an embedded feature map pair, and to calculate a similarity matrix using the embedded feature map pair.

[0113] The generation unit is used to generate a pixel-level mask for image mixing based on an adaptive image mixing network and embedded feature map pairs, and through a similarity matrix and a preset upsampling function.

[0114] The mixing unit is used to mix the original image pairs according to the image mixing pixel-level mask to obtain a mixed image.

[0115] Optionally, in one embodiment of this application, the optimization module 103 includes an input unit and a first calculation unit.

[0116] The input unit is used to input the blended image into the first shared network and extract the first average blending ratio of the target object in the blended image from the image blending pixel-level mask according to a preset blending weight strategy.

[0117] The first computing unit is used to calculate the first loss function based on the first average mixing ratio of the target object.

[0118] Optionally, in one embodiment of this application, the training module 104 includes: a second computing unit, a third computing unit, and an update unit.

[0119] The second computing unit is used to input the original image pairs into the second shared network to calculate the second loss function corresponding to the original image pairs through the second shared network.

[0120] The third computational unit is used to input the optimized blended image into the second shared network, and extract the second average blending ratio of the target object in the optimized blended image according to the blending weight strategy, so as to calculate the third loss function corresponding to the optimized blended image through the second shared network and the second average blending ratio.

[0121] The first update unit is used to optimize the second shared network using the second loss function and the third loss function, and update the parameters of the optimized second shared network to obtain the optimized parameters.

[0122] The second update unit is used to update the parameters of the first shared network based on the exponential moving average strategy and optimized parameters.

[0123] Optionally, in one embodiment of this application, the mathematical expression of the auxiliary loss function is:

[0124]

[0125] Where λ represents the mixing ratio; λ j s represents the mean of the target mixing ratio; i,h,w Represents a pixel-level mask for image blending; ∈ is a constant term.

[0126] The target detection device based on adaptive image fusion data augmentation proposed in the online training phase according to the embodiments of this application includes a first acquisition module for constructing a training dataset of target objects and acquiring multiple pairs of original images through the training dataset; an extraction module for extracting feature map pairs of each original image pair from the multiple original image pairs through a first shared network in a preset Siamese network, and inputting the feature map pairs into an adaptive image fusion network in the preset Siamese network to obtain a fused image corresponding to the feature map pairs; an optimization module for calculating a first loss function corresponding to the fused image based on the first shared network, optimizing the adaptive image fusion network through the first loss function and a preset auxiliary loss function, and optimizing the fused image using the optimized adaptive image fusion network; and a training module for optimizing a second shared network in the preset Siamese network through the original image pairs and the optimized fused image, updating the parameters of the first shared network using an exponential moving average strategy, training the optimized second shared network based on the updated parameters and the optimized adaptive image fusion network to generate a target detection model, and performing target detection operations through the target detection model in the online detection phase. This application effectively expands the quantity and diversity of the original dataset from the perspective of data augmentation, effectively improving the performance and generalization of the target detection model.

[0127] Figure 9 This is a block diagram illustrating an online pre-stage target detection device based on adaptive image fusion data enhancement, as described in this application.

[0128] like Figure 9 As shown, the target detection device 20 based on adaptive image fusion data enhancement applied in the online detection stage includes: a second acquisition module 201 and a detection module 202.

[0129] The second acquisition module 201 is used to acquire image data of the target to be tested.

[0130] The detection module 202 is used to input image data into a pre-trained target detection model to output the detection result of the target to be tested. The target detection model is obtained by training a pre-set second shared network based on a first shared network and an adaptive image mixing network in a pre-set Siamese network, combined with a pre-set training dataset.

[0131] It should be noted that the foregoing explanation of the target detection method based on adaptive image fusion data enhancement also applies to the target detection device based on adaptive image fusion data enhancement in this embodiment, and will not be repeated here.

[0132] The target detection device based on adaptive image fusion data augmentation proposed in the online detection stage according to the embodiments of this application includes a second acquisition module for acquiring image data of the target to be detected; and a detection module for inputting the image data into a pre-trained target detection model to output the detection result of the target to be detected. The target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image fusion network in a preset Siamese network, combined with a preset training dataset. This application effectively expands the quantity and diversity of the original dataset from the perspective of data augmentation, effectively improving the performance and generalization of the target detection model.

[0133] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0134] The memory 1001, the processor 1002, and the computer program stored on the memory 1001 and capable of running on the processor 1002.

[0135] When the processor 1002 executes the program, it implements the target detection method based on adaptive image fusion data enhancement provided in the above embodiments.

[0136] Furthermore, electronic devices also include:

[0137] Communication interface 1003 is used for communication between memory 1001 and processor 1002.

[0138] The memory 1001 is used to store computer programs that can run on the processor 1002.

[0139] The memory 1001 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0140] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, then the communication interface 1003, memory 1001, and processor 1002 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized into address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0141] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.

[0142] The processor 1002 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0143] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described target detection method based on adaptive image fusion data enhancement.

[0144] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0145] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0146] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0148] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0149] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0151] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A target detection method based on adaptive image fusion data augmentation, characterized in that, When applied to the online training phase, it includes the following steps: Construct a training dataset for the target object, and obtain multiple pairs of original images using the training dataset; Feature map pairs of each original image pair are extracted from the multiple original image pairs by the first shared network in the preset twin network, and the feature map pairs are input into the adaptive image mixing network in the preset twin network to obtain the mixed image corresponding to the feature map pairs; Based on the first shared network, a first loss function corresponding to the mixed image is calculated, and the adaptive image mixing network is optimized by the first loss function and a preset auxiliary loss function. The optimized adaptive image mixing network is then used to optimize the mixed image. The second shared network in the preset Siamese network is optimized by the original image pair and the optimized hybrid image, and the parameters of the first shared network are updated by an exponential moving average strategy. The optimized second shared network is trained according to the updated parameters and the optimized adaptive image hybrid network to generate an object detection model. In the online detection stage, the object detection operation is performed by the object detection model. The step of inputting the feature map pair into the adaptive image mixing network in the preset Siamese network to obtain the mixed image corresponding to the feature map pair includes: A preset mixing ratio is embedded into the feature map pair to obtain an embedded feature map pair, and a similarity matrix is ​​calculated using the embedded feature map pair; Based on the adaptive image mixing network and the embedded feature map pairs, and through the similarity matrix and a preset upsampling function, an image mixing pixel-level mask is generated; The original image pairs are blended according to the image blending pixel-level mask to obtain the blended image; The step of calculating the first loss function corresponding to the hybrid image based on the first shared network includes: The blended image is input into the first shared network, and the first average blending ratio of the target object in the blended image is extracted from the image blending pixel-level mask according to the preset blending weight strategy; The first loss function is calculated based on the first average mixing ratio of the target object; The mathematical expression for the auxiliary loss function is: in, This indicates the mixing ratio; Indicates the mean of the target mixing ratio; This represents the image's pixel-level mask; This is a constant term.

2. The method according to claim 1, characterized in that, The step of optimizing the second shared network in the preset twin network using the original image pairs and the optimized hybrid image, and updating the parameters of the first shared network using an exponential moving average strategy, includes: The original image pairs are input into the second shared network to calculate the second loss function corresponding to the original image pairs. The optimized blended image is input into the second shared network, and the second average blending ratio of the target object in the optimized blended image is extracted according to the blending weight strategy, so as to calculate the third loss function corresponding to the optimized blended image through the second shared network and the second average blending ratio; The second shared network is optimized using the second and third loss functions, and the parameters of the optimized second shared network are updated to obtain the optimized parameters. The parameters of the first shared network are updated based on the exponential moving average strategy and the optimized parameters.

3. A target detection method based on adaptive image fusion data augmentation, employing the target detection method based on adaptive image fusion data augmentation applied during the online training phase as described in any one of claims 1-2, characterized in that, When applied to the online detection phase, it includes the following steps: Acquire image data of the target object; The image data is input into a pre-trained target detection model to output the detection result of the target to be tested. The target detection model is obtained by training a pre-set second shared network based on a first shared network and an adaptive image hybrid network in a pre-set Siamese network, combined with a pre-set training dataset.

4. A target detection device based on adaptive image fusion data enhancement, characterized in that, Applied to the online training phase to achieve the target detection method based on adaptive image fusion data augmentation applied to the online training phase as described in any one of claims 1-2, comprising: The first acquisition module is used to construct a training dataset of the target object and acquire multiple pairs of original images through the training dataset; The extraction module is used to extract feature map pairs of each of the plurality of original image pairs through a first shared network in a preset Siamese network, and input the feature map pairs into an adaptive image mixing network in the preset Siamese network to obtain a mixed image corresponding to the feature map pairs; An optimization module is used to calculate a first loss function corresponding to the mixed image based on the first shared network, optimize the adaptive image mixing network through the first loss function and a preset auxiliary loss function, and optimize the mixed image using the optimized adaptive image mixing network. The training module is used to optimize the second shared network in the preset Siamese network using the original image pairs and the optimized hybrid image, and to update the parameters of the first shared network using an exponential moving average strategy, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image hybrid network to generate an object detection model, and to perform object detection operation through the object detection model in the online detection stage; The extraction module includes: An embedding unit is used to embed a preset mixing ratio into the feature map pair to obtain an embedded feature map pair, and to calculate a similarity matrix using the embedded feature map pair; The generation unit is used to generate an image mixing pixel-level mask based on the adaptive image mixing network and the embedded feature map pair, and through the similarity matrix and a preset upsampling function; A mixing unit is configured to mix the original image pairs according to the image mixing pixel-level mask to obtain the mixed image; The optimization module includes: The input unit is used to input the mixed image into the first shared network and extract the first average mixing ratio of the target object in the mixed image from the image mixing pixel-level mask according to a preset mixing weight strategy; The first calculation unit is used to calculate the first loss function based on the first average mixing ratio of the target object; The mathematical expression for the auxiliary loss function is: in, This indicates the mixing ratio; Indicates the mean of the target mixing ratio; This represents the image's pixel-level mask; This is a constant term.

5. A target detection device based on adaptive image fusion data enhancement, characterized in that, Applied to the online detection stage to achieve the target detection method based on adaptive image fusion data augmentation applied to the online detection stage as described in claim 3, comprising: The second acquisition module is used to acquire image data of the target to be tested; The detection module is used to input the image data into a pre-trained target detection model to output the detection result of the target to be tested. The target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image hybrid network in a preset Siamese network, combined with a preset training dataset.

6. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the target detection method based on adaptive image fusion data augmentation as described in any one of claims 1-2 or 3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the target detection method based on adaptive image fusion data augmentation as described in any one of claims 1-2 or 3.

Images