A Real-Time Intelligent Hidden Object Removal Method

Abstract

This invention belongs to the field of target information hidden object removal technology and discloses a real-time intelligent target hidden object removal method, including the following steps: Step 1 is dataset collection and annotation, including target dataset collection and scene dataset collection; Step 2 is model training, including target segmentation model training and target hidden object removal model training; Step 3 is real-time target hidden object removal, including image acquisition, single-frame target segmentation, updating the image storage library, target hidden object removal, and image display. This invention intelligently identifies and efficiently removes sensitive targets from the video stream transmitted between video acquisition and terminal storage, effectively reducing the appearance of sensitive targets on the Internet. Furthermore, in terms of technical implementation, it adopts a real-time segmentation to generate a mask, followed by hidden object removal, effectively solving the problem that current hidden object removal algorithms heavily rely on manual annotation, greatly reducing manpower and material resources.

Description

Technical Field

[0001] This invention belongs to the field of target information concealment technology, specifically relating to a real-time intelligent target concealment method. Background Technology

[0002] In today's world, information transmission on the internet is extremely fast. People often post pictures or videos that interest them online, which can lead to confidential or private information being exposed to the public and impacting certain groups. Therefore, how to use technology to process image data in real time during the period from image acquisition to storage, to conceal sensitive targets and reduce their spread on the internet, is a pressing issue that needs to be addressed.

[0003] Currently, existing target hidden surface removal algorithms are generally referred to as image inpainting algorithms. They are based on pre-annotated images and masks of the regions to be repaired to repair images or video sequences. Among them, video inpainting often requires annotating the regions to be removed frame by frame in the video sequence images, which is a very tedious, time-consuming and labor-intensive process. Summary of the Invention

[0004] The present invention aims to provide a real-time intelligent target concealment method, and aims to solve the technical problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a real-time intelligent target concealment method, comprising the following steps:

[0006] Step 1: Dataset collection and annotation, including target dataset collection and scene dataset collection;

[0007] Step two, model training, including target segmentation model training and target hidden object removal model training;

[0008] Step 3, real-time target culling, includes image acquisition, single-frame target segmentation, updating the image database, target culling, and image display.

[0009] In another preferred embodiment of the present invention, the target dataset collection in step one is composed of selected publicly available images, videos and self-shot data to form a sensitive target dataset, and the target segmentation model is prompted to distinguish between sensitive targets and ordinary targets, and then the dataset is labeled using annotation software.

[0010] In another preferred embodiment of the present invention, the scene dataset collection in step one is achieved by collecting various scene datasets from the public dataset Plances2.

[0011] In another preferred embodiment of the present invention, the target segmentation model training in step two is based on the target dataset labeled in step one. A target segmentation network and a specific loss function are constructed, and the target segmentation model is trained. When the loss function no longer decreases, a segmentation model capable of effectively distinguishing between sensitive targets and ordinary targets is obtained.

[0012] In another preferred embodiment of the present invention, the target segmentation model uses MobileNetV2 as the backbone feature extraction network, which includes two parts: an encoder and a decoder. The encoder uses dilated convolutions with different dilation rates to extract features from the input image in parallel, and then merges the features. The extracted features are compressed using a 1x1 convolutional layer.

[0013] The decoder first performs channel adjustment on the input image using a 1x1 convolutional layer, and then stacks it with the upsampled result of the effective feature layer after dilated convolution. After stacking, it performs two depthwise separable convolutional blocks and upsampling to obtain the final segmented image.

[0014] In another preferred embodiment of the present invention, the specific loss function of the target segmentation model is the cross-entropy loss L. CE And similarity metric Dice loss L Dice The specific calculation formulas for the two loss functions are as follows:

[0015]

[0016]

[0017] In another preferred embodiment of the present invention, the target occlusion model training in step two is based on the scene dataset collected in step one. A target occlusion network and a specific loss function are constructed, and the target occlusion model is trained by randomly generating masks. When the loss function no longer decreases, a target occlusion model that can effectively hide the target to be occluded and simultaneously complete the actual scene is obtained.

[0018] In another preferred embodiment of the present invention, the network architecture of the target hidden image removal model includes a generator and a discriminator. For the input sequence of images, the generator first uses the original image and the mask image sequence as input to perform coarse reconstruction of the missing regions. Then, the coarsely filled image is passed to a two-branch fine extraction network for refinement to reconstruct the hidden image.

[0019] The discriminator is based on the SN-GANs network structure. It extracts the feature statistics of Markov blocks through six stacked convolutional layers, and spectral normalization is applied to each standard convolutional layer.

[0020] In another preferred embodiment of the present invention, the specific loss function of the target hidden object elimination model includes the generator loss function L. G and discriminator loss function L D The specific calculation formulas for the two loss functions are as follows:

[0021]

[0022]

[0023] In another preferred embodiment of the present invention, the image acquisition in step three is performed by a real-time camera, and the first frame image is used as input; single-frame target segmentation is to use a trained target segmentation model to segment the input single-frame image to obtain a sensitive target mask image, and then merge the input original image and the segmented mask image for output.

[0024] Updating the image repository involves storing images from consecutive video frames. When the number of images stored in the repository is less than 3, new input images are stored. When the number of images stored in the repository is equal to 3, the process proceeds directly to the next step. When the number of images stored in the repository is greater than 3, new input images are stored in the repository while the oldest image stored in the repository is removed, ensuring that the number of images stored in the repository is always maintained at 3.

[0025] Target culling is based on three consecutive frames of input images and their corresponding segmentation masks. The target culling model trained in step two is used to culle sensitive targets in the current image frame.

[0026] The image display step shows the blanking image of the current frame and then proceeds to acquire the next frame image.

[0027] In summary, this invention addresses the issue of the concealment of sensitive targets on the current internet by proposing a real-time intelligent hidden target removal algorithm for time-sensitive targets in road surveillance and personal photography devices. It employs an innovative approach to intelligently identify and efficiently remove sensitive targets, effectively reducing the large number of sensitive targets appearing on the internet. Furthermore, in terms of technical implementation, it utilizes real-time segmentation to generate masks before removal, effectively solving the problem of current hidden target removal algorithms heavily relying on manual annotation, thus significantly reducing manpower and material resources.

[0028] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0029] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0030] Figure 1This is a flowchart of an embodiment of this application.

[0031] Figure 2 This is the target segmentation model framework in step two of the embodiments of this application.

[0032] Figure 3 This is the target hidden object elimination model framework in step two of the embodiments of this application.

[0033] Figure 4 These are schematic diagrams of segmentation masks, hidden line removal, and original images from the embodiments of this application. Detailed Implementation

[0034] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0035] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "vertical", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0036] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0037] This invention provides a real-time intelligent target concealment method, specifically using a military-sensitive target as an example for illustration. Figure 1 As shown, it includes the following steps:

[0038] Step 1: Dataset collection and annotation, including target dataset collection and scene dataset collection.

[0039] Target Dataset Collection: A military-sensitive target dataset was created using publicly available online images and videos, along with some self-taken data. Based on some existing public datasets (such as DAVIS and VOC), a non-military target dataset was constructed to enable the target segmentation model to accurately distinguish between military-sensitive targets and ordinary targets. The dataset was then labeled using annotation software, with labels primarily including military personnel, civilians, military vehicles, cars, and trucks.

[0040] Scene Dataset Collection: Various scene datasets were collected from the public dataset Places2. In this embodiment, Places2 contains more than 10 million scene images of various types, including more than 400 unique scenes, which are consistent with the frequency of scenes in the real world.

[0041] Step two, model training, includes training the target segmentation model and the target hidden object removal model.

[0042] Target segmentation model training: Based on the target dataset labeled in step one, construct a target segmentation network and a specific loss function, train the target segmentation model until the loss function no longer decreases, and obtain a segmentation model that can effectively distinguish between military sensitive targets and ordinary targets.

[0043] The target segmentation network uses MobileNetV2 as the backbone feature extraction network, consisting of an encoder and a decoder. The encoder is as follows: Figure 2 As shown, dilated convolutions with different dilation rates are used to extract features from the input image in parallel. Then, the features are merged and the extracted features are compressed using a 1x1 convolutional layer.

[0044] The decoder first performs channel adjustment on the input image using a 1x1 convolutional layer, and then stacks it with the upsampled result of the effective feature layer after dilated convolution. After stacking, it performs two depthwise separable convolutional blocks and upsampling to obtain the final segmented image.

[0045] A specific loss function uses cross-entropy loss L CE And similarity metric Dice loss L Dice The specific calculation formulas for the two loss functions are as follows:

[0046]

[0047]

[0048] Among them, y i,k Let the true label of the i-th element be k,p i,k Let X represent the probability that the predicted value of the i-th sample is k, N be the total number of samples, K be the total number of labels, and X and Y represent the real samples and generated samples of the segmented image, respectively.

[0049] Training the target hidden object removal model: Based on the scene dataset collected in step one, construct a target hidden object removal network and a specific loss function. Train the target hidden object removal model by randomly generating masks until the loss function no longer decreases. This will result in a target hidden object removal model that can effectively hide the target to be hidden and complete the actual scene.

[0050] The network architecture of the target hidden object elimination model is as follows: Figure 3 As shown, it includes a generator and a discriminator. For the input sequence of images, the generator includes a two-stage feature extraction process. The first stage is a coarse extraction stage, in which the network takes the original image and the mask image sequence as input and performs coarse reconstruction of the missing regions. Then, the coarsely filled image is passed to the two-branch fine extraction network for refinement, and finally the hidden image is reconstructed.

[0051] The discriminator is based on the SN-GANs (spectral-normalized) network structure. It extracts the feature statistics of Markov blocks through six stacked convolutional layers. At the same time, spectral normalization is applied to each standard convolutional layer to further improve the stability of model training.

[0052] The specific loss functions of the target hidden object removal model include the generator loss function L. G and discriminator loss function L D The specific calculation formulas for the two loss functions are as follows:

[0053]

[0054]

[0055] Where: D sn This represents the spectral-normalized discriminator, and G is the image generation network.

[0056] Step 3, real-time target culling, includes image acquisition, single-frame target segmentation, updating the image database, target culling, and image display.

[0057] Image acquisition: Images are acquired through a real-time camera, and the first frame is used as input for further processing.

[0058] Single-frame target segmentation: The target segmentation model trained in step two is used to segment the input single-frame image to obtain a sensitive target mask image. The original input image and the segmented mask image are then merged and output for further processing.

[0059] Update the image repository: Store images from consecutive video frames for subsequent target removal tasks. Updating the image repository involves three cases: first, if the image repository contains fewer than 3 images, store the newly input images; second, if the image repository contains 3 images, proceed directly to the next step; third, if the image repository contains more than 3 images, store the newly input images in the repository while removing the oldest images stored there, ensuring the image repository size is always maintained at 3.

[0060] Target removal: Based on the input three consecutive frame sequence images and their corresponding segmentation masks, the target removal model trained in step two is used to remove the hidden objects in the current image frame;

[0061] Image display: Displays the blanking image of the current frame and proceeds to acquire the image of the next frame.

[0062] The technical solution provided by this invention was tested using videos containing military targets obtained from the internet, such as... Figure 4 As shown, by comparing with the original image, it is possible to see the segmented mask image and the removal of military-sensitive targets.

[0063] In summary, this invention can identify labeled sensitive targets through an intelligent hidden target removal algorithm, and then use real-time segmentation to generate masks before removing hidden targets. This effectively solves the problem that current hidden target removal algorithms rely heavily on manual labeling, and greatly reduces manpower and material resources.

[0064] In the description of this specification, references to terms such as "preferred embodiment," "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0065] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A real-time intelligent target concealment method, characterized in that, Includes the following steps: Step 1: Dataset collection and annotation, including target dataset collection and scene dataset collection; The target dataset is collected by selecting publicly available images, videos and self-shot data to form a sensitive target dataset, and prompting the target segmentation model to distinguish between sensitive targets and ordinary targets. Then, the dataset is labeled using annotation software. Step 2, model training, including target segmentation model training and target hidden object removal model training; the target segmentation model uses MobileNetV2 as the backbone feature extraction network, which includes two parts: encoder and decoder. The encoder uses dilated convolutions with different dilation rates to extract features from the input image in parallel, and then merges the above features. The whole system uses a 1X1 convolutional layer to compress the extracted features. The decoder first performs channel adjustment on the input image using a 1x1 convolutional layer, and then stacks it with the upsampled result of the effective feature layer after dilated convolution. After stacking, it performs two depthwise separable convolutional blocks and upsampling to obtain the final segmented image. The target concealment model training is based on the scene dataset collected in step one. The target concealment network and a specific loss function are constructed. The target concealment model is trained by randomly generating masks until the loss function no longer decreases. This results in a target concealment model that can effectively hide the target to be concealed and complete the actual scene. Step 3, real-time target culling, includes image acquisition, single-frame target segmentation, updating the image database, target culling, and image display. Image acquisition is performed using a real-time camera, with the first frame image as input. Single-frame target segmentation uses a trained target segmentation model to segment the input single-frame image into a sensitive target mask image, and then merges the original input image with the segmented mask image for output. Updating the image repository involves storing images from consecutive video frames. When the number of images stored in the repository is less than 3, new input images are stored. When the number of images stored in the repository is equal to 3, the process proceeds directly to the next step. When the number of images stored in the repository is greater than 3, new input images are stored in the repository while the oldest image stored in the repository is removed, ensuring that the number of images stored in the repository is always maintained at 3. Target culling is based on three consecutive frames of input images and their corresponding segmentation masks. The target culling model trained in step two is used to culle sensitive targets in the current image frame. The image display step shows the blanking image of the current frame and then proceeds to acquire the image of the next frame.

2. The real-time intelligent target concealment method according to claim 1, characterized in that: The scene dataset collection in step one involves collecting various scene datasets from the public dataset Plances2.

3. The real-time intelligent target concealment method according to claim 2, characterized in that: The target segmentation model training in step two is based on the target dataset labeled in step one. The target segmentation network and a specific loss function are constructed to train the target segmentation model.

4. The real-time intelligent target concealment method according to claim 3, characterized in that: The specific loss function of the target segmentation model uses cross-entropy loss. And similarity metric Dice loss The specific calculation formulas for the two loss functions are as follows: 。 5. The real-time intelligent target concealment method according to claim 4, characterized in that: The network architecture of the target hidden image removal model includes a generator and a discriminator. For the input sequence of images, the generator first takes the original image and the mask image sequence as input and performs coarse reconstruction of the missing regions. Then, the coarsely filled image is passed to a two-branch fine extraction network for refinement and reconstruction of the hidden image. The discriminator is based on the SN-GANs network structure. It extracts the feature statistics of Markov blocks through six stacked convolutional layers, and spectral normalization is applied to each standard convolutional layer.

6. The real-time intelligent target concealment method according to claim 5, characterized in that: The specific loss function of the target hidden object removal model includes the generator loss function. and discriminator loss function The specific calculation formulas for the two loss functions are as follows: 。

Images