Target detection method and device based on adaptive image mixed data enhancement

By using twin networks and adaptive image mixing networks in the object detection task to generate mixed images and optimize network parameters, the problem of insufficient data volume in object detection is solved, and the performance and generalization of the model are improved sex.

CN119068228AActive Publication Date: 2024-12-03WUHAN UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410877750.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-02
Publication Date
2024-12-03
Estimated Expiration
2044-07-02

AI Technical Summary

Technical Problem

The adaptive image mixed data enhancement method is difficult to apply in object detection tasks, and insufficient data volume limits the application of the model.

Method used

By constructing the training data set of the target object, the feature images are extracted using the twin network and inputting them into the adaptive image mixing network to generate mixed images, the loss function is calculated to optimize network parameters, the exponential moving average strategy is used to update network parameters, and the object detection model is generated.

Benefits of technology

It effectively expands the number and diversity of data sets, improves the performance and generalization of the object detection model, and solves the problem of insufficient data volume.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068228A_ABST
    Figure CN119068228A_ABST
Patent Text Reader

Abstract

The invention relates to a target detection method and device based on adaptive image mixed data enhancement, and the method comprises the steps: obtaining a plurality of original image pairs through a training data set; extracting a feature map pair of each original image pair through a first shared network, and obtaining a mixed image of the feature map pairs by using a self-adaptive image mixing network; calculating a first loss function of the hybrid image through the first shared network, and optimizing the adaptive image hybrid network in combination with an auxiliary loss function so as to optimize the hybrid image; and optimizing the second shared network through the original image pair and the optimized mixed image, updating the parameters of the first shared network, training the optimized second shared network in combination with the optimized self-adaptive image mixed network, and generating a target detection model to execute target detection operation. Therefore, the problem that the application of a target detection model is greatly limited due to the fact that the amount of collected and labeled data is insufficient in the actual application of an adaptive image mixed data enhancement method is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of image data enhancement, and in particular to a target detection method and device based on adaptive image hybrid data enhancement. Background Art

[0002] Object detection is one of the most basic and important applications in the field of computer vision. For a given image, the object detection task aims to identify the target object in the image and return the corresponding bounding box for locating the target. Object detection has been widely used in fields such as autonomous driving, video surveillance, and medical image analysis. Although object detection has a wide range of application potential, in practical applications, the limited amount of collected and labeled data is often one of the important bottlenecks restricting the application of object detection methods.

[0003] In order to solve the problems of data scarcity and high data collection costs, image data augmentation methods are currently widely used in various tasks as a method to enhance the generalization ability of models. Image data augmentation increases the number and diversity of training samples by transforming and expanding the original data. In recent years, methods that use image mixing for data augmentation have shown more robust performance and have achieved success in image classification tasks, attracting more and more attention. These methods mainly improve the diversity of the original dataset by mixing image pairs or mixing different regions of images; adaptive image mixing data augmentation strategies provide an effective way to generate image mixing samples. The enhanced data can be regarded as extracted from an adjacent distribution close to the true distribution, so the enhanced dataset usually contains a more comprehensive feature representation of the real data.

[0004] However, adaptive image mixture data augmentation methods are mainly used in image classification tasks and can usually only process images containing only a single target category. Therefore, applying adaptive image mixture data augmentation methods to target detection tasks still faces severe challenges that need to be solved urgently. Summary of the invention

[0005] The present application provides a target detection method and device based on adaptive image hybrid data enhancement to solve the problem that the amount of data collected and annotated in practical applications of the adaptive image hybrid data enhancement method is insufficient, which greatly limits the application of the target detection model.

[0006] The first aspect of the present application provides a target detection method based on adaptive image mixing data enhancement, which is applied to the online training stage and includes the following steps: constructing a training data set of the target object, and obtaining multiple original image pairs through the training data set; extracting the feature map pair of each original image pair in the multiple original image pairs through the first shared network in the preset twin network, and inputting the feature map pair into the adaptive image mixing network in the preset twin network to obtain the mixed image corresponding to the feature map pair; based on the first shared network, calculating the first loss function corresponding to the mixed image, and optimizing the adaptive image mixing network through the first loss function and the preset auxiliary loss function, and optimizing the mixed image using the optimized adaptive image mixing network; optimizing the second shared network in the preset twin network through the original image pairs and the optimized mixed image, and updating the parameters of the first shared network using the exponential moving average strategy, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image mixing network to generate a target detection model, and in the online detection stage, performing target detection operations through the target detection model.

[0007] Optionally, in one embodiment of the present application, the feature map pair is input into the adaptive image mixing network in the preset twin network to obtain a mixed image corresponding to the feature map pair, including: embedding a preset mixing ratio into the feature map pair to obtain an embedded feature map pair, and using the embedded feature map pair to calculate a similarity matrix; based on the adaptive image mixing network and the embedded feature map pair, and through the similarity matrix and a preset upsampling function, to generate an image mixing pixel-level mask; and mixing the original image pair according to the image mixing pixel-level mask to obtain the mixed image.

[0008] Optionally, in one embodiment of the present application, calculating the first loss function corresponding to the mixed image based on the first shared network includes: inputting the mixed image into the first shared network, and extracting the first average mixing ratio of the target object in the mixed image from the image mixed pixel-level mask according to a preset mixing weight strategy; calculating the first loss function based on the first average mixing ratio of the target object.

[0009] Optionally, in one embodiment of the present application, the second shared network in the preset twin network is optimized by the original image pair and the optimized mixed image, and the parameters of the first shared network are updated by the exponential moving average strategy, including: inputting the original image pair into the second shared network to calculate the second loss function corresponding to the original image pair through the second shared network; inputting the optimized mixed image into the second shared network, and extracting the second average mixing ratio of the target object in the optimized mixed image according to the mixing weight strategy, so as to calculate the third loss function corresponding to the optimized mixed image through the second shared network and the second average mixing ratio; optimizing the second shared network through the second loss function and the third loss function, and updating the parameters of the optimized second shared network to obtain optimized parameters; updating the parameters of the first shared network based on the exponential moving average strategy and the optimized parameters.

[0010] Optionally, in one embodiment of the present application, the mathematical expression of the auxiliary loss function is:

[0011]

[0012] Wherein, λ represents the mixing ratio; λ j represents the mean of the target mixture ratio; s i,h,w represents the mixed pixel-level mask of the image; ∈ is a constant term.

[0013] The second aspect of the present application provides a target detection method based on adaptive image mixed data enhancement, which is applied to the online detection stage, and includes the following steps: acquiring image data of the target to be detected; inputting the image data into a pre-trained target detection model to output the detection result of the target to be detected, wherein the target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image mixed network in a preset twin network, and in combination with a preset training data set.

[0014] The third aspect of the present application is an embodiment that provides a target detection device based on adaptive image mixing data enhancement, which is applied to the online training stage, including: a first acquisition module, used to construct a training data set of the target object, and obtain multiple original image pairs through the training data set; an extraction module, used to extract the feature map pair of each original image pair in the multiple original image pairs through the first shared network in the preset twin network, and input the feature map pair into the adaptive image mixing network in the preset twin network to obtain the mixed image corresponding to the feature map pair; an optimization module, used to calculate the first loss function corresponding to the mixed image based on the first shared network, and optimize the adaptive image mixing network through the first loss function and the preset auxiliary loss function, and optimize the mixed image using the optimized adaptive image mixing network; a training module, used to optimize the second shared network in the preset twin network through the original image pair and the optimized mixed image, and update the parameters of the first shared network using the exponential moving average strategy, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image mixing network, so as to generate a target detection model, and in the online detection stage, perform target detection operations through the target detection model.

[0015] Optionally, in one embodiment of the present application, the extraction module includes: an embedding unit, used to embed a preset mixing ratio into the feature map pair to obtain an embedded feature map pair, and use the embedded feature map pair to calculate a similarity matrix; a generation unit, used to generate an image mixing pixel-level mask based on the adaptive image mixing network and the embedded feature map pair, and through the similarity matrix and a preset upsampling function; a mixing unit, used to mix the original image pair according to the image mixing pixel-level mask to obtain the mixed image.

[0016] Optionally, in one embodiment of the present application, the optimization module includes: an input unit, used to input the mixed image into the first shared network, and extract a first average mixing ratio of the target object in the mixed image from the image mixed pixel-level mask according to a preset mixing weight strategy; a first calculation unit, used to calculate the first loss function based on the first average mixing ratio of the target object.

[0017] Optionally, in one embodiment of the present application, the training module includes: a second computing unit, used to input the original image pair into the second shared network, so as to calculate the second loss function corresponding to the original image pair through the second shared network; a third computing unit, used to input the optimized mixed image into the second shared network, and extract the second average mixing ratio of the target object in the optimized mixed image according to the mixing weight strategy, so as to calculate the third loss function corresponding to the optimized mixed image through the second shared network and the second average mixing ratio; a first updating unit, used to optimize the second shared network through the second loss function and the third loss function, and update the parameters of the optimized second shared network to obtain optimized parameters; a second updating unit, used to update the parameters of the first shared network based on the exponential moving average strategy and the optimized parameters.

[0018] Optionally, in one embodiment of the present application, the mathematical expression of the auxiliary loss function is:

[0019]

[0020] Wherein, λ represents the mixing ratio; λ j represents the mean of the target mixture ratio; s i,h,w represents the mixed pixel-level mask of the image; ∈ is a constant term.

[0021] The fourth aspect of the present application provides a target detection device based on adaptive image mixed data enhancement, which is applied to the online detection stage, including: a second acquisition module, used to acquire image data of the target to be detected; a detection module, used to input the image data into a pre-trained target detection model to output the detection result of the target to be detected, wherein the target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image mixed network in a preset twin network, and in combination with a preset training data set.

[0022] The fifth aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method based on adaptive image hybrid data enhancement as described in the above embodiment.

[0023] The sixth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, it implements the above-mentioned target detection method based on adaptive image hybrid data enhancement.

[0024] Therefore, the embodiments of the present application have the following beneficial effects:

[0025] The embodiment of the present application can construct a training data set of the target object, and obtain multiple original image pairs through the training data set; extract the feature map pair of each original image pair in the multiple original image pairs through the first shared network in the preset twin network, and input the feature map pair into the adaptive image hybrid network in the preset twin network to obtain the hybrid image corresponding to the feature map pair; based on the first shared network, calculate the first loss function corresponding to the hybrid image, and optimize the adaptive image hybrid network through the first loss function and the preset auxiliary loss function, and optimize the hybrid image using the optimized adaptive image hybrid network; optimize the second shared network in the preset twin network through the original image pair and the optimized hybrid image, and use the exponential moving average strategy to update the parameters of the first shared network, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image hybrid network, so as to generate a target detection model, and perform the target detection operation through the target detection model in the online detection stage. The present application effectively expands the number and diversity of original data sets from the perspective of data enhancement, and effectively improves the performance and generalization of the target detection model. Thus, the problem that the amount of data collected and annotated by the adaptive image hybrid data enhancement method in practical applications is insufficient, which greatly limits the application of the target detection model, etc.

[0026] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A flowchart of a target detection method based on adaptive image hybrid data enhancement applied to an online training stage provided according to an embodiment of the present application;

[0029] Figure 2 A schematic diagram of a data enhancement framework for automatically optimizing mixed sample generation provided by an embodiment of the present application;

[0030] Figure 3 A schematic diagram of an execution logic of an adaptive image mixing network provided for one embodiment of the present application;

[0031] Figure 4 A schematic diagram of an image mixing effect provided for an embodiment of the present application;

[0032] Figure 5 A logic diagram of a target-level hybrid weight strategy provided for one embodiment of the present application;

[0033] Figure 6 A schematic diagram of the logical architecture of a target detection method based on adaptive image hybrid data enhancement provided for one embodiment of the present application;

[0034] Figure 7 A flowchart of a target detection method based on adaptive image hybrid data enhancement applied to an online detection stage provided according to an embodiment of the present application;

[0035] Figure 8 This is an example diagram of an object detection device based on adaptive image hybrid data enhancement applied in an online training phase according to an embodiment of the present application;

[0036] Fig. 9 is an example diagram of an object detection device based on adaptive image hybrid data enhancement applied in an online detection stage according to an embodiment of the present application;

[0037] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0038] Among them, 10-a target detection device based on adaptive image hybrid data enhancement applied in the online training stage, 20-a target detection device based on adaptive image hybrid data enhancement applied in the online detection stage; 101-a first acquisition module, 102-an extraction module, 103-an optimization module, 104-a training module; 201-a second acquisition module, 202-a detection module; 1001-a memory, 1002-a processor, 1003-a communication interface. DETAILED DESCRIPTION

[0039] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0040] The following describes the target detection method and device based on adaptive image hybrid data enhancement of the embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides a target detection method based on adaptive image hybrid data enhancement, in which a training data set of the target object is constructed, and multiple original image pairs are obtained through the training data set; the feature map pairs of each original image pair in the multiple original image pairs are extracted through the first shared network in the preset twin network, and the feature map pairs are input into the adaptive image hybrid network in the preset twin network to obtain the mixed image corresponding to the feature map pair; based on the first shared network, the first loss function corresponding to the mixed image is calculated, and the adaptive image hybrid network is optimized by the first loss function and the preset auxiliary loss function, and the optimized adaptive image hybrid network is used to optimize the mixed image; the second shared network in the preset twin network is optimized by the original image pair and the optimized hybrid image, and the parameters of the first shared network are updated by the exponential moving average strategy, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image hybrid network, so as to generate a target detection model, and in the online detection stage, the target detection operation is performed by the target detection model. This application effectively expands the quantity and diversity of the original data set from the perspective of data enhancement, and effectively improves the performance and generalization of the target detection model. As a result, it solves the problem that the amount of data collected and annotated in practical applications of the adaptive image hybrid data enhancement method is insufficient, which greatly limits the application of the target detection model.

[0041] Specifically, Figure 1 A flowchart of a target detection method based on adaptive image hybrid data enhancement applied in an online training phase provided in an embodiment of the present application.

[0042] like Figure 1 As shown, the target detection method based on adaptive image hybrid data enhancement includes the following steps:

[0043] In step S101, a training data set of a target object is constructed, and a plurality of original image pairs are obtained through the training data set.

[0044] The embodiment of the present application can first initialize the parameters of each network model in the preset automatic optimization mixed sample generation data enhancement framework. The framework adopts a twin network structure, including a network k (i.e., a first shared network) and a network q (i.e., a second shared network). The two networks share the same structure and initial parameters. In addition, the framework also includes an adaptive image mixing network, which can use a random initialization strategy for parameter initialization operation, wherein the network q is mainly used for target detection, and the more stable network k is connected to the adaptive image mixing network for adaptive image mixing, such as Figure 2 As shown, automatic optimization of mixed sample generation data enhancement in the field of target detection is achieved.

[0045] Afterwards, the embodiment of the present application also needs to construct a training data set of the target object, and randomly extract two training images from the training data set as original image pairs, so as to provide reliable data support for the training of subsequent network q and adaptive image mixing network, etc.

[0046] In step S102, a feature map pair of each original image pair in a plurality of original image pairs is extracted through a first shared network in a preset twin network, and the feature map pair is input into an adaptive image mixing network in the preset twin network to obtain a mixed image corresponding to the feature map pair.

[0047] After obtaining multiple original image pairs, further, the embodiment of the present application needs to input each original image pair into the backbone network of network k, so as to extract the feature map of each original image pair through the backbone network in network k, provide basic feature representation for subsequent tasks, and input the extracted feature map into the adaptive image mixing network to obtain the mixed image corresponding to the feature map pair.

[0048] Optionally, in one embodiment of the present application, the feature map pair is input into an adaptive image mixing network in a preset twin network to obtain a mixed image corresponding to the feature map pair, including: embedding a preset mixing ratio into the feature map pair to obtain an embedded feature map pair, and using the embedded feature map pair to calculate a similarity matrix; based on the adaptive image mixing network and the embedded feature map pair, and through the similarity matrix and a preset upsampling function, to generate an image mixing pixel-level mask; mixing the original image pair according to the image mixing pixel-level mask to obtain a mixed image.

[0049] It should be noted that the embodiment of the present application can be implemented by an adaptive image mixing module. According to the image samples (i.e. original image pair) to obtain the feature map pair and the preset mixing ratio λ to predict the pixel-level mask pair (s i ,s j ) to generate a mixed image. Among them, the mask s u The values ​​in represent the image The mixing ratio at the corresponding position, mask s j The values ​​in represent the image The mixing ratio at the corresponding position.

[0050] Specifically, in the adaptive image mixture network In the embodiment of the present application, firstly, a simple and effective method can be used to Embed λ into the lth layer feature map to obtain the λ-embedded and (i.e., embedding feature map pairs), and use it as input; secondly, according to and Output generated image mixed pixel-level mask s i is used to embed a mixture of feature map pairs. According to the symmetry, x j Masks j By s j =1-s i The generated pixel-level mask pair (s i ,s j ) is used to mix the original image pair to obtain a mixed image. The mathematical expression of this process is:

[0051]

[0052]

[0053] Among them, ⊙ represents element-by-element multiplication, According to The resulting mixed image.

[0054] The image mixed pixel-level mask s includes The corresponding mask s i ,s j . Apparently, s i ,s j It has symmetry, that is, the sum of the two masks used to generate mixed samples is equal to 1, as shown in the following formula:

[0055] s i +s j =1

[0056] The embodiment of the present application obtains an image mixed pixel-level mask, that is, The process is as follows:

[0057] First, the embodiment of the present application can calculate the similarity matrix P through the cross attention mechanism, as shown in the following formula:

[0058]

[0059] Among them, W P represents a shared linear transformation matrix (1×1 convolution), represents matrix multiplication, is the normalization factor;

[0060] Secondly, the embodiment of the present application can be realized through an adaptive image mixing network The pairwise similarity matrix P and upsampling function U(·) are used to calculate s i , as shown below:

[0061]

[0062] Among them, W Z is the linear transformation matrix; σ is the Sigmoid activation function, which is used to make the mask probabilistic; s i To obtain a mask that is consistent with the sample image size H×W; by embedding P and the value Multiply, choose x i,λ Relative to x j,1-λ The discriminant features of x j Masks j Available through j =1-s i Calculated.

[0063] The following describes the process of obtaining an image mixed pixel-level mask in an embodiment of the present application in conjunction with the accompanying drawings.

[0064] Figure 3 Figure 1 is a schematic diagram of the execution logic of the adaptive image mixing network. Figure 3 As shown, the steps of obtaining the image mixed pixel-level mask through the adaptive image mixing network in the embodiment of the present application are as follows:

[0065] 1. Embed the obtained λ into the feature map of the lth layer feature and Perform a 1×1 convolution operation and get the output and

[0066] 2. and Perform matrix multiplication and use the softmax function to normalize the result to obtain the similarity matrix P;

[0067] 3. λ is embedded into the feature map of the lth layer feature Perform a 1×1 convolution operation separately and output the value embedding

[0068] 4. The similarity matrix P and The result is probabilized by the Sigmoid activation function and upsampled to obtain an image mixed pixel-level mask s that is consistent with the size of the sample image. i .

[0069] Therefore, the embodiment of the present application obtains an image mixing pixel-level mask according to feature map pairs and mixing ratios through an adaptive image mixing network, thereby generating a mixed image, providing a reliable basis for the subsequent training of corresponding models.

[0070] In step S103, based on the first shared network, a first loss function corresponding to the mixed image is calculated, and the adaptive image mixing network is optimized by the first loss function and a preset auxiliary loss function, and the optimized adaptive image mixing network is used to optimize the mixed image.

[0071] Furthermore, the embodiments of the present application can input the generated mixed image into the network k to calculate the target detection loss function (i.e., the first loss function) of the mixed image through the network k, and optimize the adaptive image mixing network through back propagation to optimize the mixed image through the optimized adaptive image mixing network, thereby improving the quality of the subsequently generated mixed image.

[0072] Optionally, in one embodiment of the present application, the mathematical expression of the auxiliary loss function is:

[0073]

[0074] Where, λ represents the mixing ratio; λ j represents the mean of the target mixture ratio; s i,h,w Represents the image mixed pixel-level mask; ∈ is a constant term.

[0075] In the actual implementation, in order to make the adaptive image mixing network Can learn early in training The corresponding relationship between λ and λ is achieved while avoiding the semantic inconsistency problem caused by the adaptive image mixing network mixing the image level and the target detection as the object level task. The embodiment of the present application proposes an auxiliary loss function for the adaptive image mixing network at the object level.

[0076] Specifically, the embodiment of the present application adjusts the loss constraint range from the image level to the object level, and uses a small amount of background information to make the adaptive image mixing network more sensitive to the distinction between the foreground target and the background, so that the adaptive image mixing network is more focused on learning the proportional relationship at the object level. The mathematical expression of the auxiliary loss function is as follows:

[0077]

[0078] Among them, λ represents the randomly generated mixing ratio; λ j represents the mean value of the target mixing ratio extracted from the generated image mixed pixel-level mask; s i,h,w represents the image pixel-level mask generated by the adaptive image mixture network, ∈ = 0.1.

[0079] Therefore, the embodiment of the present application sets an auxiliary loss function to ensure that the output image mixed pixel-level mask s i The characteristic areas of each object are retained to the greatest extent possible to further improve the performance of the target detection model.

[0080] It can be understood that the embodiments of the present application generate a mixed image from the target level to ensure that the key features of the target are preserved as accurately as possible, such as Figure 4 As shown, the key to this process is to prevent the generated mixed samples from misleading the target detection task, so that they can become beneficial training samples, help the target detection model learn the key features of the target, and ultimately improve the performance level of the model; the mixed image generation method of the embodiment of the present application is not only accurate, but also helps to ensure that the target detection model can more effectively capture and understand the characteristics of the target during the learning process, providing a reliable foundation for improving the performance of the model.

[0081] Optionally, in one embodiment of the present application, based on the first shared network, a first loss function corresponding to the mixed image is calculated, including: inputting the mixed image into the first shared network, and extracting a first average mixing ratio of the target object in the mixed image from the image mixed pixel-level mask according to a preset mixing weight strategy; based on the first average mixing ratio of the target object, calculating the first loss function.

[0082] It should be noted that after obtaining the mixed image, the embodiment of the present application can input the mixed image into the network k, and use the network k to calculate the target detection loss function (i.e., the first loss function) of the mixed image, and feed it back to the adaptive image mixing network through back propagation, thereby optimizing the quality of the generated mixed image.

[0083] As an implementable manner, when calculating the first loss function of the mixed image, the embodiment of the present application designs a target-level mixed weight strategy. Specifically, in the embodiment of the present application, the loss weight coefficient of each target is obtained by extracting the average mixed ratio of the target (i.e., the first average mixed ratio) from the generated image mixed pixel-level mask, such as Figure 5 As shown, and used to calculate the target detection classification and regression losses, the mathematical expression of the first loss function of the mixed image is as follows:

[0084]

[0085] Among them, i represents the index of the predicted target; λ i The mixture weight representing the true target closest to the predicted target; and denote the classification and regression losses of a single target, respectively.

[0086] Therefore, the embodiments of the present application calculate the average mixing ratio of each target from the generated image mixed pixel-level mask, and use the obtained average mixing ratio as the loss weight of each target when calculating the target detection classification and regression loss of the mixed image, thereby introducing the target-level mixing weight to achieve adaptive mixing of target detection labels, providing strong support for improving the target detection performance of mixed images.

[0087] In step S104, the second shared network in the preset twin network is optimized by the original image pair and the optimized hybrid image, and the parameters of the first shared network are updated by using the exponential moving average strategy to train the optimized second shared network according to the updated parameters and the optimized adaptive image hybrid network to generate a target detection model, and in the online detection stage, the target detection operation is performed through the target detection model.

[0088] Furthermore, the embodiments of the present application can use the original image pair and the optimized mixed image to jointly train the network q (i.e., the second shared network) to respectively calculate the target detection loss function of the original image pair (i.e., the second loss function) and the target detection loss function of the optimized mixed image (i.e., the third loss function); at the same time, the network q updates the network parameters through back propagation, and to avoid feature collapse, the embodiments of the present application can adopt an exponential moving average strategy to update the parameters of the network k, so as to train the optimized network q according to the updated parameters and the optimized adaptive image mixed network, thereby obtaining the final target detection model.

[0089] Optionally, in one embodiment of the present application, the second shared network in the preset twin network is optimized by the original image pair and the optimized mixed image, and the parameters of the first shared network are updated by the exponential moving average strategy, including: inputting the original image pair into the second shared network to calculate the second loss function corresponding to the original image pair through the second shared network; inputting the optimized mixed image into the second shared network, and extracting the second average mixing ratio of the target object in the optimized mixed image according to the mixing weight strategy, so as to calculate the third loss function corresponding to the optimized mixed image through the second shared network and the second average mixing ratio; optimizing the second shared network through the second loss function and the third loss function, and updating the parameters of the optimized second shared network to obtain optimized parameters; updating the parameters of the first shared network based on the exponential moving average strategy and the optimized parameters.

[0090] It should be noted that the embodiment of the present application performs conventional target detection network training on the original image input network q and calculates the corresponding target detection network loss (i.e., the second loss function) so that the backbone network can provide stable feature mapping in the early stage, thereby accelerating the convergence speed of the optimized adaptive image hybrid network.

[0091] In an embodiment of the present application, the main loss function of the target detection network includes two parts: the detection anchor box regression loss and the classification loss of the target, as shown in the following formula:

[0092]

[0093] Among them, i represents the index of the predicted target; and Denote the classification and regression losses of the predicted targets respectively, and sum the classification and regression losses of all targets.

[0094] In addition, the embodiments of the present application also need to input the optimized mixed image into the network q to expand the number and diversity of training samples, and calculate the optimized mixed image target detection loss function (i.e., the third loss function), which has the same mathematical expression as the above-mentioned first loss function, thereby improving the performance of the model target detection.

[0095] In summary, the embodiments of the present application use a data enhancement method for adaptive mixed sample generation in the field of target detection, and design an end-to-end trainable automatically optimized mixed sample generation data enhancement framework for the field of target detection, which realizes the generation of adaptive pixel-level masks, and uses the masks to generate mixed sample auxiliary models for training based on the original data sample pairs, thereby improving the performance of the target detection model; due to the inconsistency of the label types of target detection and classification tasks, the label mixing strategy used for classification tasks is no longer applicable to target detection tasks. The embodiments of the present application use a target-level mixed weight strategy to achieve adaptive mixing of target detection labels based on the characteristics of the target detection task; in addition, in order to ensure that the feature areas of multiple objects are retained when learning the image data mixing ratio, and to avoid interference with the original features of the target due to mixing, the embodiments of the present application improve the adaptive image mixing network, by adjusting the constraints from the entire image to mainly for the objects to be detected, so that the adaptive image mixing network pays more attention to the detected objects, and the output mixed mask retains the feature areas of each object as much as possible, thereby further improving the performance of the target detection model.

[0096] It can be understood that in order to directly improve the performance of the target detection model from the perspective of data diversity expansion, the embodiment of the present application divides the model training process into two subtasks: a hybrid image optimization task and a target detection task. Figure 6As shown, the image mixing task is regarded as an auxiliary task of the target detection task, thereby solving the problems existing in the separate mixed data enhancement strategy and introducing a more flexible collaborative optimization; in addition, in the traditional image mixing strategy, the optimization of the image mixing task and the target detection task are separated from each other, which easily leads to the generated mixed samples misleading the target detection task. The embodiments of the present application combine the above two tasks in a mutually beneficial way, thereby effectively ensuring that the mixed samples play an effective role in the target detection task.

[0097] According to the target detection method based on adaptive image hybrid data enhancement applied to the online training stage proposed in the embodiment of the present application, a training data set of the target object is constructed, and multiple original image pairs are obtained through the training data set; the feature map pairs of each original image pair in the multiple original image pairs are extracted through the first shared network in the preset twin network, and the feature map pairs are input into the adaptive image hybrid network in the preset twin network to obtain the hybrid image corresponding to the feature map pair; based on the first shared network, the first loss function corresponding to the hybrid image is calculated, and the adaptive image hybrid network is optimized through the first loss function and the preset auxiliary loss function, and the optimized adaptive image hybrid network is used to optimize the hybrid image; the second shared network in the preset twin network is optimized through the original image pairs and the optimized hybrid image, and the parameters of the first shared network are updated using the exponential moving average strategy, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image hybrid network, so as to generate a target detection model, and in the online detection stage, the target detection operation is performed through the target detection model. The present application effectively expands the number and diversity of original data sets from the perspective of data enhancement, and effectively improves the performance and generalization of the target detection model.

[0098] Figure 7 A flowchart of a target detection method based on adaptive image hybrid data enhancement applied in an online detection stage provided in an embodiment of the present application.

[0099] like Figure 7 As shown, the target detection method based on adaptive image hybrid data enhancement includes the following steps:

[0100] In step S701, image data of the target to be measured is obtained.

[0101] In step S702, the image data is input into a pre-trained target detection model to output the detection result of the target to be detected, wherein the target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image hybrid network in a preset twin network and in combination with a preset training data set.

[0102] Therefore, the embodiments of the present application input the original image to be detected into a pre-trained target detection model, thereby identifying the predicted target detection box and category on the original image and performing a visual output.

[0103] According to the target detection method based on adaptive image hybrid data enhancement applied to the online detection stage proposed in the embodiment of the present application, the image data of the target to be detected is obtained; the image data is input into a pre-trained target detection model to output the detection result of the target to be detected, wherein the target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image hybrid network in a preset twin network, and in combination with a preset training data set. The present application effectively expands the number and diversity of the original data set from the perspective of data enhancement, and effectively improves the performance and generalization of the target detection model.

[0104] Secondly, the target detection device based on adaptive image hybrid data enhancement proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0105] Figure 8 It is a block diagram of a target detection device based on adaptive image hybrid data enhancement applied in an online training phase according to an embodiment of the present application.

[0106] like Figure 8 As shown, the target detection device 10 based on adaptive image hybrid data enhancement includes: a first acquisition module 101, an extraction module 102, an optimization module 103 and a training module 104.

[0107] The first acquisition module 101 is used to construct a training data set of the target object and acquire a plurality of original image pairs through the training data set.

[0108] The extraction module 102 is used to extract the feature map pair of each original image pair in multiple original image pairs through the first shared network in the preset twin network, and input the feature map pair into the adaptive image mixing network in the preset twin network to obtain the mixed image corresponding to the feature map pair.

[0109] The optimization module 103 is used to calculate the first loss function corresponding to the mixed image based on the first shared network, optimize the adaptive image mixing network through the first loss function and the preset auxiliary loss function, and optimize the mixed image using the optimized adaptive image mixing network.

[0110] The training module 104 is used to optimize the second shared network in the preset twin network through the original image pair and the optimized hybrid image, and adopt the exponential moving average strategy to update the parameters of the first shared network, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image hybrid network to generate a target detection model, and in the online detection stage, perform the target detection operation through the target detection model.

[0111] Optionally, in one embodiment of the present application, the extraction module 102 includes: an embedding unit, a generation unit and a mixing unit.

[0112] The embedding unit is used to embed a preset mixing ratio into the feature map pair to obtain an embedded feature map pair, and calculate a similarity matrix using the embedded feature map pair.

[0113] The generation unit is used to generate an image mixed pixel-level mask based on an adaptive image mixing network and an embedded feature map pair, and through a similarity matrix and a preset upsampling function.

[0114] The mixing unit is used to mix the original image pair according to the image mixing pixel level mask to obtain a mixed image.

[0115] Optionally, in one embodiment of the present application, the optimization module 103 includes: an input unit and a first calculation unit.

[0116] Among them, the input unit is used to input the mixed image into the first shared network, and extract the first average mixing ratio of the target object in the mixed image from the image mixed pixel-level mask according to a preset mixing weight strategy.

[0117] The first calculation unit is used to calculate a first loss function based on a first average mixing ratio of the target object.

[0118] Optionally, in one embodiment of the present application, the training module 104 includes: a second computing unit, a third computing unit and an updating unit.

[0119] The second computing unit is used to input the original image pair into the second shared network to calculate the second loss function corresponding to the original image pair through the second shared network.

[0120] The third computing unit is used to input the optimized mixed image into the second shared network, and extract the second average mixing ratio of the target object in the optimized mixed image according to the mixing weight strategy, so as to calculate the third loss function corresponding to the optimized mixed image through the second shared network and the second average mixing ratio.

[0121] The first updating unit is used to optimize the second shared network through the second loss function and the third loss function, and update the parameters of the optimized second shared network to obtain optimized parameters.

[0122] The second updating unit is used to update the parameters of the first sharing network based on the exponential moving average strategy and the optimization parameters.

[0123] Optionally, in one embodiment of the present application, the mathematical expression of the auxiliary loss function is:

[0124]

[0125] Where, λ represents the mixing ratio; λ j represents the mean of the target mixture ratio; s i,h,w Represents the image mixed pixel-level mask; ∈ is a constant term.

[0126] According to the target detection device based on adaptive image hybrid data enhancement applied to the online training stage proposed in the embodiment of the present application, it includes a first acquisition module for constructing a training data set of the target object, and obtaining multiple original image pairs through the training data set; an extraction module for extracting feature map pairs of each original image pair in multiple original image pairs through the first shared network in the preset twin network, and inputting the feature map pairs into the adaptive image hybrid network in the preset twin network to obtain the hybrid image corresponding to the feature map pair; an optimization module for calculating the first loss function corresponding to the hybrid image based on the first shared network, and optimizing the adaptive image hybrid network through the first loss function and the preset auxiliary loss function, and optimizing the hybrid image using the optimized adaptive image hybrid network; a training module for optimizing the second shared network in the preset twin network through the original image pair and the optimized hybrid image, and updating the parameters of the first shared network using the exponential moving average strategy, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image hybrid network, so as to generate a target detection model, and in the online detection stage, perform the target detection operation through the target detection model. The present application effectively expands the number and diversity of original data sets from the perspective of data enhancement, and effectively improves the performance and generalization of the target detection model.

[0127] Fig. 9 It is a block diagram of a target detection device based on adaptive image hybrid data enhancement applied to an online pre-stage according to an embodiment of the present application.

[0128] like Fig. 9 As shown, the target detection device 20 based on adaptive image hybrid data enhancement applied in the online detection stage includes: a second acquisition module 201 and a detection module 202.

[0129] The second acquisition module 201 is used to acquire image data of the target to be measured.

[0130] The detection module 202 is used to input image data into a pre-trained target detection model to output the detection result of the target to be detected, wherein the target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image hybrid network in a preset twin network and in combination with a preset training data set.

[0131] It should be noted that the aforementioned explanation of the embodiment of the target detection method based on adaptive image mixed data enhancement is also applicable to the target detection device based on adaptive image mixed data enhancement of this embodiment, which will not be repeated here.

[0132] According to the target detection device based on adaptive image hybrid data enhancement applied to the online detection stage proposed in the embodiment of the present application, it includes a second acquisition module for acquiring image data of the target to be detected; a detection module for inputting the image data into a pre-trained target detection model to output the detection result of the target to be detected, wherein the target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image hybrid network in a preset twin network, and in combination with a preset training data set. The present application effectively expands the quantity and diversity of the original data set from the perspective of data enhancement, and effectively improves the performance and generalization of the target detection model.

[0133] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0134] A memory 1001 , a processor 1002 , and a computer program stored in the memory 1001 and executable on the processor 1002 .

[0135] When the processor 1002 executes the program, the target detection method based on adaptive image hybrid data enhancement provided in the above embodiment is implemented.

[0136] Furthermore, the electronic device further comprises:

[0137] The communication interface 1003 is used for communication between the memory 1001 and the processor 1002 .

[0138] The memory 1001 is used to store computer programs that can be executed on the processor 1002 .

[0139] The memory 1001 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0140] If the memory 1001, the processor 1002 and the communication interface 1003 are implemented independently, the communication interface 1003, the memory 1001 and the processor 1002 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0141] Optionally, in a specific implementation, if the memory 1001, the processor 1002 and the communication interface 1003 are integrated on a chip, the memory 1001, the processor 1002 and the communication interface 1003 can communicate with each other through an internal interface.

[0142] The processor 1002 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0143] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned target detection method based on adaptive image hybrid data enhancement.

[0144] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0145] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0148] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0151] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A target detection method based on adaptive image hybrid data enhancement, characterized in that: Applied in the online training phase, it includes the following steps: Constructing a training data set of the target object, and obtaining a plurality of original image pairs through the training data set; Extracting a feature map pair of each original image pair in the plurality of original image pairs through a first shared network in a preset twin network, and inputting the feature map pair into an adaptive image mixing network in the preset twin network to obtain a mixed image corresponding to the feature map pair; Based on the first shared network, a first loss function corresponding to the mixed image is calculated, and the adaptive image mixing network is optimized by the first loss function and a preset auxiliary loss function, and the optimized adaptive image mixing network is used to optimize the mixed image; The second shared network in the preset twin network is optimized by the original image pair and the optimized mixed image, and the parameters of the first shared network are updated by using the exponential moving average strategy, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image mixed network to generate a target detection model, and in the online detection stage, the target detection operation is performed through the target detection model.

2. The method according to claim 1, characterized in that: The step of inputting the feature map pair into the adaptive image mixing network in the preset twin network to obtain a mixed image corresponding to the feature map pair includes: Embedding a preset mixing ratio into the feature map pair to obtain an embedded feature map pair, and calculating a similarity matrix using the embedded feature map pair; Based on the adaptive image mixing network and the embedded feature map pair, and through the similarity matrix and a preset upsampling function, an image mixing pixel-level mask is generated; The pair of original images are mixed according to the image mixing pixel level mask to obtain the mixed image.

3. The method according to claim 1, characterized in that The calculating, based on the first shared network, a first loss function corresponding to the mixed image includes: Inputting the mixed image into the first shared network, and extracting a first average mixing ratio of the target object in the mixed image from the image mixed pixel-level mask according to a preset mixing weight strategy; The first loss function is calculated based on a first average mixing ratio of the target object.

4. The method according to claim 1, characterized in that: The optimizing the second shared network in the preset twin network by using the original image pair and the optimized mixed image, and updating the parameters of the first shared network by using an exponential moving average strategy, comprises: Inputting the original image pair into the second shared network to calculate a second loss function corresponding to the original image pair through the second shared network; Inputting the optimized mixed image into the second shared network, and extracting a second average mixed ratio of the target object in the optimized mixed image according to the mixed weight strategy, so as to calculate a third loss function corresponding to the optimized mixed image through the second shared network and the second average mixed ratio; Optimizing the second shared network by using the second loss function and the third loss function, and updating parameters of the optimized second shared network to obtain optimized parameters; Based on the exponential moving average strategy and the optimization parameters, parameters of the first sharing network are updated.

5. The method according to claim 1, characterized in that: The mathematical expression of the auxiliary loss function is: Wherein, λ represents the mixing ratio; λ j represents the mean of the target mixture ratio; s i,h,w represents the mixed pixel-level mask of the image; ∈ is a constant term.

6. A target detection method based on adaptive image hybrid data enhancement, characterized in that: Applied to the online detection stage, it includes the following steps: Acquire image data of the target to be measured; The image data is input into a pre-trained target detection model to output the detection result of the target to be detected, wherein the target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image hybrid network in a preset twin network and in combination with a preset training data set.

7. An object detection device based on adaptive image hybrid data enhancement, characterized in that: Applied in the online training phase, including: A first acquisition module, used to construct a training data set of the target object and acquire a plurality of original image pairs through the training data set; An extraction module, configured to extract a feature map pair of each of the plurality of original image pairs through a first shared network in a preset twin network, and input the feature map pair into an adaptive image mixing network in the preset twin network to obtain a mixed image corresponding to the feature map pair; an optimization module, configured to calculate a first loss function corresponding to the mixed image based on the first shared network, optimize the adaptive image mixing network by using the first loss function and a preset auxiliary loss function, and optimize the mixed image using the optimized adaptive image mixing network; A training module is used to optimize the second shared network in the preset twin network through the original image pair and the optimized mixed image, and update the parameters of the first shared network using an exponential moving average strategy, so as to train the optimized second shared network according to the updated parameters and the optimized adaptive image mixed network to generate a target detection model, and in the online detection stage, perform target detection operations through the target detection model.

8. An object detection device based on adaptive image hybrid data enhancement, characterized in that: Applied in the online detection stage, including: The second acquisition module is used to acquire image data of the target to be measured; A detection module is used to input the image data into a pre-trained target detection model to output the detection result of the target to be detected, wherein the target detection model is obtained by training a preset second shared network based on a first shared network and an adaptive image hybrid network in a preset twin network and in combination with a preset training data set.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method based on adaptive image hybrid data enhancement as described in any one of claims 1 to 5 or 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the target detection method based on adaptive image hybrid data enhancement as described in any one of claims 1-5 or 6.

Citation Information

Patent Citations

  • Target segmentation method, system and equipment based on improved attention and loss function

    CN113988179A

  • Multi-vision mixed task learning method based on attention mechanism

    CN115690510A

  • Target segmentation model training method and target segmentation method

    CN116503592A

  • Annotating unlabeled images using convolutional neural networks

    US20220261599A1