Terminal device for image processing, image processing method, and program product

Abstract

The application provides a terminal device for image processing, an image processing method and a program product. The terminal device comprises a memory and a processor. The processor executes computer execution instructions stored in the memory to implement the following image processing method: determining a size parameter of a unit block based on depth information of an image target region; and performing image disturbance processing on the target region using a plurality of unit blocks with the size parameter. The size of the unit block is adaptively configured based on the depth, which improves the accuracy of determining the size of the unit block during image disturbance processing such as mosaicking, replacement and the like, and balances the interference processing effect and processing efficiency.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a terminal device, image processing method and program product for image processing. Background Technology

[0002] In the field of image editing and privacy protection, it is common to perform image perturbation processing on specific areas of an image (such as faces, body parts, license plates, etc.), such as adding mosaics or replacing content.

[0003] In existing technologies, when applying mosaic effects to images, fixed-size mosaic blocks are often used to blur the target area. This involves dividing the target area into equally sized blocks and blurring or averaging the pixels of each block to achieve privacy. However, for moving objects, this method has several drawbacks. If the mosaic blocks are too large, the occluded object becomes too clearly visible, and the blurring effect is insufficient. Conversely, if the mosaic blocks are too small, the computational cost is too high, and the target area becomes too blurred, negatively impacting the overall visual quality of the image.

[0004] Therefore, there is an urgent need to provide a scheme for dynamically determining the size of unit blocks, such as mosaic blocks, during interference processing. Summary of the Invention

[0005] This application provides a terminal device, image processing method, and program product for image processing, which solves the problem of low accuracy in determining the size of mosaics for moving objects, and realizes the dynamic determination of the size of unit blocks using depth, taking into account both computational overhead and interference processing effect.

[0006] In a first aspect, embodiments of this application provide a terminal device for image processing, including a memory and a processor; the memory stores computer execution instructions, and the processor executes the computer execution instructions to achieve the following steps: determining the size parameters of a unit block based on the depth information of the target region of the image; and performing image perturbation processing on the target region using multiple unit blocks with the size parameters.

[0007] In one possible implementation, the processor is further configured to: acquire depth distribution data of the target region; and determine depth information of the target region based on the depth distribution data.

[0008] In one possible implementation, obtaining depth distribution data of the target region includes: obtaining the target region of the same detected target in multiple frames of images; performing depth estimation on the multiple frames of images to obtain depth data of each frame of images; and extracting the depth distribution data of the target region in each frame of images from the depth data of each frame of images.

[0009] In one possible implementation, depth estimation is performed on multiple frames of images to obtain depth data for each frame, including: performing random enhancement processing on each frame of the multiple frames; performing depth estimation on each enhanced frame to obtain initial depth data for each frame; and calibrating the initial depth data for each frame to obtain the depth data for each frame.

[0010] In one possible implementation, calibrating the initial depth data of each frame image to obtain the depth data of each frame image includes: dividing multiple frames into multiple batches; each batch contains multiple overlapping frames and multiple new frames, the overlapping frames include a reference frame, which is the frame where the detected target is first detected; based on the initial depth data of the reference frame, performing depth calibration on each frame image in the same batch to obtain the intra-batch calibration depth of the image; and based on the intra-batch calibration depth of the overlapping frames, performing depth smoothing processing on the new frames in adjacent batches to obtain the depth data of the new frames.

[0011] In one possible implementation, determining the depth information of the target region based on depth distribution data includes:

[0012] Based on the connectivity and / or depth distribution data range of the target region, the target regions of multiple frames of images are classified to obtain the first target region and the second target region; based on the depth distribution data of each second target region, the initial depth information of the second target region is determined, and the initial depth information of multiple second target regions is low-pass filtered to obtain the depth information of each second target region; based on the interpolation result of the depth information of the second target regions in adjacent frames, the depth information of the first target region is obtained.

[0013] In one possible implementation, determining the size parameters of a unit block based on depth information includes: comparing the depth information of the target region in the current frame with that in the previous frame; and determining the size parameters of the unit block in the current frame based on the comparison result and the preset interval where the depth information of the target region in the previous frame is located or the size parameters of the unit block in the previous frame.

[0014] In one possible implementation, a mapping relationship exists between a preset interval and the size parameters of a unit block; based on the comparison result and the preset interval containing the depth information of the target region in the previous frame or the size parameters of the unit block in the previous frame, the size parameters of the unit block in the current frame are determined, including: based on the comparison result of the difference between the depth information of the target region in the current frame and the previous frame and a preset threshold, determining that the preset interval corresponding to the current frame is the previous interval, the next interval, or remains unchanged; and based on the preset interval corresponding to the current frame and the mapping relationship, determining the size parameters of the unit block in the current frame.

[0015] In one possible implementation, the unit block is rectangular, with dimensions including width and height. Multiple unit blocks of these dimensions are used to perturb the target region, including: determining the minimum bounding rectangle of the target region; adjusting the corner points of the minimum bounding rectangle so that its width and height are C and R times the width and height of the unit block, respectively; where C and R are both positive integers; cropping the portion corresponding to the adjusted minimum bounding rectangle from the image or its associated perturbation image as a local image; generating a target image containing C×R unit blocks based on the local image, and using the target image to cover or replace the corresponding region of the image.

[0016] In one possible implementation, generating a target image composed of C×R unit blocks based on a local image includes: downsampling the local image to a C×R size; and upsampling the downsampled image back to its original size to obtain the target image.

[0017] In one possible implementation, the processor is further configured to: perform a morphological dilation operation on the target region to obtain a dilated region; determine a mixing coefficient for a pixel based on the distance between a pixel in the transition region and an interference region; the interference region is the region corresponding to the adjusted minimum bounding rectangle, and the transition region is the difference between the dilated region and the interference region; and, based on the mixing coefficient, blend the image and the target image to obtain a transition image, and replace the transition region of the image with the transition image, or use the transition image to cover the transition region of the image.

[0018] Secondly, embodiments of this application provide an image processing method, including: determining the size parameters of a unit block based on the depth information of a target region of an image; and performing image perturbation processing on the target region using multiple unit blocks with the size parameters.

[0019] In one possible implementation, the method further includes: acquiring the target region of the same detection target in multiple frames of images; performing depth estimation on the multiple frames of images to obtain depth data of each frame of images; extracting depth distribution data of the target region in each frame of images from the depth data of each frame of images; and determining the depth information of the target region based on the depth distribution data of the target region.

[0020] In one possible implementation, determining the size parameters of a unit block based on the depth information of the target region of the image includes: comparing the depth information of the target region in the current frame with that in the previous frame; and determining the size parameters of the unit block in the current frame based on the comparison result and the preset interval where the depth information of the target region in the previous frame is located or the size parameters of the unit block in the previous frame.

[0021] In one possible implementation, the unit block is rectangular, with dimensions including width and height. Multiple unit blocks of these dimensions are used to perturb the target region, including: determining the minimum bounding rectangle of the target region; adjusting the corner points of the minimum bounding rectangle so that its width and height are C and R times the width and height of the unit block, respectively; where C and R are both positive integers; cropping the portion corresponding to the adjusted minimum bounding rectangle from the image or its associated perturbation image as a local image; generating a target image containing C×R unit blocks based on the local image, and using the target image to cover or replace the corresponding region of the image.

[0022] The method provided in this application can be executed locally on the terminal or in conjunction with cloud / edge computing, and is applicable to real-time privacy protection scenarios such as security monitoring, live conference streaming, and AR / VR.

[0023] Thirdly, embodiments of this application provide an image processing apparatus, including: a size adaptive module, used to determine the size parameters of a unit block based on depth information of an image target region; and a perturbation processing module, used to perform image perturbation processing on the target region using multiple unit blocks with the size parameters.

[0024] Fourthly, embodiments of this application provide a display device for displaying multiple frames of images after image perturbation processing; at least two of the multiple frames have different depth information of the target region where the detected target is located; and at least two frames use different unit block size parameters when performing image perturbation processing.

[0025] In some possible implementations, the display device is specifically used to display an image after image perturbation processing sent by the terminal device provided in the first aspect of this application, or an image processed based on the method provided in the second aspect of this application.

[0026] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the image processing method provided by the second aspect and / or various possible implementations of the second aspect.

[0027] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the image processing method provided by the second aspect and / or various possible implementations of the second aspect.

[0028] The terminal device, image processing method, and program product for image processing provided in this application dynamically configure the size of the unit blocks based on the depth of the target region in the image, thereby using unit blocks of corresponding sizes to cover the target region and achieve visual interference of the target region. For near-distance targets, larger unit blocks are automatically matched for coverage, reducing the number of unit blocks while ensuring visual interference effect, saving the computational load of image perturbation processing, and improving processing efficiency. At the same time, it avoids covering distant targets with a large number of small, dense unit blocks, which would affect the overall visual effect of the image. For distant targets, smaller, densely arranged unit blocks are used to form coverage, refining the interference granularity and improving the interference effect on target details, avoiding the exposure of the target's outline or key features due to excessively large unit blocks. This method improves interference effectiveness, computational efficiency, and visual coordination by adaptively adjusting the unit block size with depth, and can better meet the needs of target visual interference in dynamic scenes. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0030] Figure 1 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 1 ;

[0031] Figure 2 A schematic diagram illustrating the results of multi-frame image perturbation processing provided in an embodiment of this application;

[0032] Figure 3 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 2 ;

[0033] Figure 4 This is a schematic diagram illustrating the image batching results provided in real time for this application;

[0034] Figure 5 A schematic diagram of the transition region and interference region provided in the embodiments of this application;

[0035] Figure 6 A schematic diagram illustrating the image processing results provided in an embodiment of this application;

[0036] Figure 7 A schematic diagram of the target area detection results provided in an embodiment of this application;

[0037] Figure 8 A schematic diagram of the target region tracking framework provided in an embodiment of this application;

[0038] Figure 9A schematic diagram illustrating the target area tracking results provided in an embodiment of this application;

[0039] Figure 10 A schematic diagram of various scene types of images provided in the embodiments of this application;

[0040] Figure 11 A schematic diagram of the structure of the first detector provided in an embodiment of this application;

[0041] Figure 12 For this application Figure 11 A schematic diagram of the decoder structure of the generator in the illustrated embodiment;

[0042] Figure 13 For this application Figure 11 A schematic diagram of the discriminator in the illustrated embodiment;

[0043] Figure 14 A schematic diagram illustrating a second tracker data processing procedure provided in an embodiment of this application;

[0044] Figure 15 A flowchart illustrating the target region tracking method provided in this application embodiment. Figure 2 ;

[0045] Figure 16 This is a schematic diagram of the structure of a terminal device for image processing provided in this application.

[0046] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0047] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0048] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0049] The image processing method provided in this application can be executed locally on the terminal or in collaboration with cloud / edge computing, and is suitable for security monitoring, live conference streaming, AR / VR or other scenarios that require privacy protection.

[0050] Figure 1 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 1 This method can be executed by any electronic device with corresponding data processing capabilities, such as terminal devices for image processing, display devices, etc. Figure 1 As shown, the image processing method includes:

[0051] Step S101: Determine the size parameters of the unit block based on the depth information of the target region in the image.

[0052] The image can be a two-dimensional or three-dimensional image, such as a naked-eye stereo image. The target region is the area in the image that needs to be perturbed. It can be a human body part, such as a face or private parts, or an area that needs to be protected, such as a license plate or the adhesive area on a delivery label. The depth information of the target region is used to characterize the depth of the target region. It can be a single depth value or data representing the depth distribution.

[0053] A unit block is the basic unit in image perturbation processing; it is an independent block-shaped unit that constitutes the visual interference effect in the target area. During image perturbation processing, multiple arranged unit blocks cover the target area, and visual interference in the target area is achieved by filling the unit blocks with preset patterns or colors, such as colors extracted from the original image.

[0054] The dimensional parameters of a unit block include, but are not limited to: side length, radius, width, and height.

[0055] The shape of the unit block should be one that can cover the entire plane. For example, the unit block can be a square, a rectangle, a hexagon, or other regular shapes, or an irregular shape, such as a cloud shape.

[0056] By utilizing segmentation networks or detectors, and tracking networks or trackers, it is possible to detect and track the target region of the same target across multiple frames of images, thus obtaining the target region in each frame. The segmentation network or detector is used to detect the mask of the target region in a single frame of image, while the tracking network or tracker is used to track the spatial trajectory of the target region as it changes over time, thereby achieving the association of target regions of the same target across multiple frames of images.

[0057] Specifically, object detection can be performed on the image first to obtain the target region. After obtaining the target region, the depth distribution data of the target region can be extracted from the image's depth data.

[0058] For example, the depth data of an image can be obtained from the depth map associated with the image, or it can be obtained using depth estimation based on a deep learning algorithm.

[0059] When the image is a three-dimensional image, the depth distribution data of the target area can be directly extracted from the three-dimensional coordinates of the pixels within the target area of ​​the image.

[0060] Optionally, the image processing method further includes: acquiring depth distribution data of the target region of the image; and determining depth information of the target region based on the depth distribution data.

[0061] The depth distribution data of the target area can be statistically analyzed to obtain the depth information of the target area. For example, the depth information of the target area can be the median or mean of the depth distribution data of the target area, or the data that appears most frequently in the depth distribution data of the target area.

[0062] There is a mapping relationship between the size parameters of the unit block and the depth information of the target region. Based on the changes in the depth information of the target region in multiple frames of images, the interval of the depth information of the target region in each frame of images is dynamically determined, thereby realizing the dynamic adjustment of the unit block size parameters.

[0063] The depth information of the target region is used to characterize the overall depth level of the target region. The size parameter of the cell block can be negatively correlated with the depth information of the target region, that is, the smaller the depth information of the target region, the larger the size parameter of the cell block.

[0064] In some embodiments, the size parameters of the unit block can be determined based on the interval where the depth information is located. The size parameters of the unit block can be pre-mapped for various intervals where the depth information is located to obtain a mapping relationship. Based on this mapping relationship and the interval where the depth information of the target region of the image is located, the size parameters of the unit block used to cover the target region can be determined.

[0065] In other embodiments, a conversion relationship between depth information and size parameters can be pre-designed, and the depth information of the target area can be substituted into the conversion relationship to obtain the size parameters of the unit block.

[0066] The depth information or depth distribution data of the target region can be output by the tracking network or tracker during target region tracking. When performing target region tracking, a depth-based tracker, such as a subsequent third tracker, can be used to output the depth of the tracked target region. Based on this, the depth of the target region output by the third tracker can be directly obtained as the depth distribution data or depth information of the target region when the third tracker successfully tracks the target region.

[0067] Step S102: Using multiple unit blocks with the stated size parameters, perform image perturbation processing on the target region.

[0068] Multiple unit blocks with corresponding size parameters can be displayed in the target area, and the patterns or fill colors displayed in the multiple unit blocks can be used to cover, replace, or occlude the target area.

[0069] When using multiple unit blocks for image perturbation, these unit blocks need to be arranged in a suitable manner to cover the entire target area according to the outline and shape of the target area.

[0070] For example, when the target area is a detected face area in an image, and the unit block is determined to be a square with a side length of 5 pixels based on the depth information, the square unit blocks with a side length of 5 pixels will be arranged in a grid pattern to closely cover the face area. Each unit block can be filled with a preset single color, such as the average value of the corresponding part of the pixel value in the original image. Through this combination of dense, uniformly sized unit blocks, the original features of the face are occluded, achieving visual interference.

[0071] When performing image perturbation processing, the color or pattern used to fill the unit block can be extracted from the original image, can be a default color or pattern, or can be extracted from other images that match the image and are used for image perturbation. For example, the other image can be a template image provided by the system. Taking the detection target as a face as an example, the template image can be an image of another face or an image of a cartoon character.

[0072] Taking the processing of multiple consecutive frames of images as an example, target detection and tracking can be used to identify the same target, such as the same face, in the target region of each of the multiple frames. Then, based on the depth information of the target region of each frame, the size parameters of the unit block used to cover the target region of that frame are determined, thereby realizing the image perturbation processing of multiple frames of images through dynamically sized unit blocks.

[0073] For example, Figure 2 This is a schematic diagram illustrating the multi-frame image perturbation processing results provided in an embodiment of this application. In a video segment, there is a license plate that needs to be obscured, and the vehicle is in motion, causing the depth of the license plate to decrease. From frame i to frame j (j>i), the vehicle's depth gradually decreases (i.e., the distance between the vehicle and the viewpoint gradually decreases). Figure 2 As shown, in the i-th frame of the video, the target area corresponding to the license plate is region 21, which has a large depth information, meaning that the license plate is far away from the observer. Therefore, the size of the license plate in the image is also small, and the corresponding unit block size parameter is also small, such as 5 pixels. In the j-th frame, the target area corresponding to the license plate is region 22, which has a small depth information. The size of the license plate in the image is also large, and the corresponding unit block size parameter is also large, such as 10 pixels.

[0074] For multi-image scenarios such as image sequences or video frames, the image processing method provided in this embodiment can be used to continuously perturb the tracked target region based on the target detection / segmentation and tracking results of each frame in the image sequence or video. For each frame image in which the target region is tracked, the size parameters of the unit block are dynamically determined through the depth information of the target region in the image, and multiple unit blocks with corresponding size parameters are used to perturb the image of that frame.

[0075] When a target region is not detected in a frame, or the depth information of the target region fails to be acquired, the perturbation of that frame can be performed based on the depth information of the target region in the previous frame that detected the same target, or the result of image perturbation processing.

[0076] Specifically, for the current frame where target region tracking fails, the target region of the nearest preceding frame that detected the same target can be directly determined as the target region of the current frame, or the nearest preceding frame where target region tracking was successful. Alternatively, based on the optical flow fields of the preceding and current frames, the target region detected or successfully tracked in the preceding frame can be adjusted, and the adjusted target region can be used as the target region of the current frame. The distance between the target region of the preceding frame and the image edge can be limited to at least a set distance. If the depth information of the target region in the current frame fails to be obtained, the depth information of the target region in the current frame can be obtained by interpolating the depth information of the target regions in adjacent frames.

[0077] The image processing method provided in this embodiment dynamically configures the size of the cell blocks based on the depth of the target region in the image, thereby using cell blocks of corresponding sizes to cover the target region and achieve visual interference. For near-range targets, larger cell blocks are automatically matched for coverage, reducing the number of cell blocks while ensuring visual interference effect, saving computational load in image perturbation processing, and improving processing efficiency. At the same time, it avoids covering distant targets with a large number of small, dense cell blocks, which would affect the overall visual effect of the image. For distant targets, smaller, densely arranged cell blocks are used to form coverage, refining the interference granularity and improving the interference effect on target details, avoiding the exposure of the target's outline or key features due to excessively large cell blocks. This method improves interference effectiveness, computational efficiency, and visual coordination by adaptively adjusting the cell block size with depth, and can better meet the needs of target visual interference in dynamic scenes.

[0078] Figure 3 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 2 This embodiment is... Figure 1 Further refinement of the illustrated embodiment, such as Figure 3As shown, the image processing method includes:

[0079] Step S301: Obtain the target region of the same detection target in multiple frames of images.

[0080] Specifically, target detection and tracking are performed on multiple consecutive frames of images in a video or image sequence to obtain the target region of the same detected target in the multiple frames.

[0081] Multiple consecutive frames of images can be acquired. This step can determine the target region of the same detection in multiple frames of images based on the tracking results provided by the target region tracking method in subsequent embodiments. For the target region detected for the first time, a tracker can be established to determine the tracking region of the target region in subsequent frames, thereby obtaining the target regions of the same detection target in multiple frames of images.

[0082] Step S302: Perform depth estimation on the multi-frame images to obtain the depth data of each frame in the multi-frame images.

[0083] For multiple frames of images that detect the same target, depth estimation is performed frame by frame based on a depth estimation algorithm, or depth estimation and calibration are performed to obtain the depth data of each frame.

[0084] Any algorithm can be used for image depth estimation, such as depth estimation algorithms based on deep learning models.

[0085] For example, image depth estimation can be performed based on algorithms such as DepthAnything (Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data) or Unidepth (Universal Monocular Metric DepthEstimation).

[0086] DepthAnything is a monocular depth estimation model that improves robustness through large-scale unlabeled data and data augmentation. It also incorporates an auxiliary supervision mechanism that allows the model to inherit rich semantic prior information from the pre-trained encoder, resulting in high depth estimation accuracy. Unidepth is a general monocular depth estimation model capable of reconstructing 3D scenes from a single image without relying on additional information such as camera parameters. UniDepth implements a self-cueing camera module that predicts dense camera representations to modulate depth features. It also utilizes a pseudo-spherical output representation to decouple the camera and depth representations and introduces a geometric invariance loss to enhance the invariance of camera-cued depth features, resulting in high estimation accuracy and robustness.

[0087] Before depth estimation and calibration, the image can be preprocessed, such as by noise reduction or at least one random augmentation. Depth estimation is then performed on the preprocessed image, and depth estimation from multiple frames is used to perform image depth calibration, resulting in the image's depth data.

[0088] Optionally, the step of performing depth estimation on the multiple frames of images to obtain depth data for each frame includes: performing random enhancement processing on each frame of the multiple frames of images; performing depth estimation on each enhanced frame to obtain initial depth data for each frame; and calibrating the initial depth data for each frame to obtain depth data for each frame.

[0089] Random enhancement can add random noise, randomly adjust color values, contrast, etc., or perform random geometric transformations on the image, such as random rotation, random cropping, random scaling, random flipping, etc.

[0090] The initial depth of the image can be calibrated, either by smoothing the depth of adjacent frames or by dividing the images into batches, first calibrating the depth of the same batch of images, and then calibrating the depth between adjacent batches.

[0091] When batching, there can be at least one overlapping frame between adjacent batches, so that depth calibration between adjacent batches can be achieved by using the depth of the overlapping frame in different batches.

[0092] Optionally, calibrating the initial depth data of each frame image to obtain the depth data of each frame image includes: dividing the multiple frames into multiple batches; each batch contains multiple overlapping frames and multiple newly added frames, the overlapping frames include a reference frame, the reference frame being the frame where the target is first detected; based on the initial depth data of the reference frame, performing depth calibration on each frame image in the same batch to obtain the intra-batch calibration depth of the image; and based on the intra-batch calibration depth of the overlapping frames, performing depth smoothing processing on the newly added frames in adjacent batches to obtain the depth data of the newly added frames.

[0093] For example, multiple frames of images can be batched using an overlapping sliding window to obtain multiple batches, while a reference frame is added to each batch.

[0094] New frames in each batch can be obtained through a non-overlapping sliding window. The overlapping frames in the next batch include the reference frame and at least one frame selected from the new frames in the previous batch.

[0095] When performing target detection on an image sequence or video, the first frame of the target detected is the reference frame for that target.

[0096] The number of overlapping frames in a batch is less than the number of new frames. For example, the number of new frames can be N times the number of overlapping frames, where N is an integer greater than 1. For instance, the number of overlapping frames could be 4, and the number of new frames could be 8.

[0097] In one embodiment, overlapping frames in the same batch can be composed of several non-contiguous frames from multiple images. That is, overlapping frames in the same batch are distributed at intervals, and the intervals between different overlapping frames can be the same or different.

[0098] For example, Figure 4 This is a schematic diagram of the image batching results provided in real time according to this application, such as... Figure 4 As shown, for multiple frames of images that detect the same target ( Figure 4 Taking 36 frames as an example, the images are numbered according to their relative frame numbers to obtain images from frame 0 to frame 35. Figure 4 Taking each batch as an example, which includes 4 overlapping frames and 8 newly added frames, such as Figure 4 As shown, when performing image batching, a sliding window with a step size of 8 can be used to obtain the newly added frames in each batch. For example, frames 4-11 of the first batch, frames 12-19 of the second batch, frames 20-27 of the third batch, and frames 28-35 of the fourth batch. The newly added frames in each batch are consecutive frames that do not overlap. For the overlapping frames in each batch, in addition to the reference frame (frame 0), they can also include the second frame, the third-to-last frame, and the last frame of the newly added frames in the previous batch. Figure 4 The frames within each batch are represented by gray fill.

[0099] Figure 4 The batching method shown is only an example. The number of overlapping frames and new frames, as well as the selection method, can also be other methods, and this application does not limit them.

[0100] After dividing multiple frames of images into batches to obtain multiple batch images, depth data for each frame of image is obtained through intra-batch depth calibration and inter-batch depth calibration.

[0101] The primary purpose of intra-batch depth calibration is to standardize the depth scale of images within a batch. This can be achieved by using the range of initial depth data from reference frames within the batch as a benchmark. Specifically, the initial depth data of the current batch's reference frames is compared with the initial depth data of reference frames from previous batches to obtain the scale calibration parameters for the current batch, such as scaling factors and offsets. These parameters are then used to perform intra-batch depth calibration on the images within the current batch, yielding the intra-batch calibrated depth for each image within the current batch.

[0102] For a single frame within the current batch, its batch calibration depth can be: initial depth data × scaling factor + offset.

[0103] Inter-batch depth calibration is mainly used to achieve depth smoothing between adjacent batches. It mainly utilizes the intra-batch calibration depth of overlapping frames to achieve depth smoothing of newly added frames. Specifically, for the i-th batch (where i is an integer greater than 1), its adjacent batches include the (i-1)-th batch and the (i+1)-th batch. The overlapping frames between the i-th and (i-1)-th batches, as well as the overlapping frames between the i-th and (i+1)-th batches, can be used to perform secondary depth calibration on the newly added frames of the i-th batch. Specifically, by using the intra-batch calibration depth of the overlapping frames, continuous difference processing and depth smoothing are performed on the newly added frames, thereby solving the inter-batch scale drift problem caused by the sliding window.

[0104] During inter-batch depth calibration, the overlapping frames used are specifically the last frame of the previous batch and the last frame of the current batch.

[0105] by Figure 4 Taking the batching method shown as an example, for the second batch, the overlapping frames are frames 0, 5, 9, and 11, and the newly added frames are frames 12-19. During inter-batch depth calibration, based on the intra-batch calibration depth of frame 11 in the first batch and frame 19 in the third batch, continuous interpolation and depth smoothing are performed on the newly added frames 12-19 in the second batch to obtain the depth data of frames 12-19 in the second batch. This process is repeated to obtain the depth data of the newly added frames in each batch. Since the union of the newly added frames in all batches covers all frames that need to be depth estimated, the depth data of all frame images is obtained.

[0106] For the first batch, inter-batch calibration can be performed using only the overlapping frames from the next batch.

[0107] By segmenting images into batches and performing intra-batch and inter-batch depth calibration, the depth scale was unified, ensuring data consistency between frames and batches, improving the accuracy and stability of depth estimation for long sequence images, and providing reliable data support for the subsequent determination of unit block size.

[0108] Step S303: Extract the depth distribution data of the target region in each frame image from the depth data of that frame image.

[0109] After obtaining the depth data of each frame of the image, the depth distribution data of the target region is extracted from the depth data of the frame of the image based on the mask of the target region in that frame of the image.

[0110] Step S304: Determine the depth information of the target area based on the depth distribution data.

[0111] For example, the median of the depth distribution data of the target area can be determined as the depth information of the target area.

[0112] To improve the accuracy of depth information determination, anomaly detection can be performed on the target regions of each frame to filter out abnormal target regions. After determining the depth information of the target regions to be retained, interpolation can be used to obtain the depth information of the abnormal target regions.

[0113] Optionally, determining the depth information of the target region based on the depth distribution data includes: classifying the target regions of the multi-frame images based on the connectivity of the target regions and / or the value range of the depth distribution data to obtain a first target region and a second target region; determining the initial depth information of the second target region based on the depth distribution data of each second target region, and performing low-pass filtering on the initial depth information of multiple second target regions to obtain the depth information of each second target region; and obtaining the depth information of the first target region based on the interpolation result of the depth information of the second target regions in adjacent frames.

[0114] The first target region is an abnormal target region, and the estimated depth information is less accurate. The first target region can be a disconnected target region or a target region with an abnormal depth distribution range.

[0115] First, connectivity detection is performed on the target regions in each frame to identify disconnected target regions as the first target regions. Next, depth range detection is performed on connected target regions to identify target regions with excessively large depth range variations as the first target regions. These two steps achieve the detection of abnormal target regions, and the remaining target regions are designated as the second target regions.

[0116] For target regions that are not identified as the first target region after connectivity detection and depth range detection, the ratio of the size to the depth of these target regions can be determined. Based on this ratio, these target regions are further filtered to identify the target regions with larger ratio deviations as the first target regions, and the remaining regions as the second target regions.

[0117] For each second target region, the initial depth value of the second target region is first determined based on the depth distribution data of the second target region. In order to avoid drastic changes in the depth value, the initial depth value of each second target region is low-pass filtered, such as Gaussian filtering, mean filtering, median filtering, etc., to remove depth jumps, noise, etc., and obtain the depth value of each second target region.

[0118] After determining the depth value of each second target region, for the target region sequence composed of the first target region and the second target region (sorted according to the order of the corresponding frames), the depth value of each first target region can be obtained by interpolating the depth values ​​of adjacent second target regions in the sequence by distance.

[0119] Before interpolation, the sequence can be segmented. Specifically, if there are two frames in the sequence with a breakpoint number greater than a preset frame count threshold, the sequence is segmented into multiple segments, using the latter or former frame of those two frames as the breakpoint. Within each segment, the depth value of the first target region within that segment is obtained by interpolating the depth values ​​of adjacent second target regions by distance. Here, the breakpoint number is the number of frames between two frames detecting the same target in the original video or image sequence.

[0120] For example, the preset frame rate threshold can be 10, 15, 20, 30, 35 or other values.

[0121] After obtaining the depth information, such as the depth value, of the target region in each frame, the size parameters of the cell block can be obtained by mapping based on the interval where the depth information is located.

[0122] To avoid frequent changes in the unit block size parameters between adjacent frames, which would cause the perturbation processing part of the image to scroll too obviously, the size parameters of the unit block used in the perturbation processing of each frame can be determined by the following steps, such as width and height.

[0123] Step S305: Compare the depth information of the target region in the current frame with that in the previous frame.

[0124] Step S306: Based on the comparison results and the preset interval where the depth information of the target region in the previous frame is located or the size parameters of the unit block in the previous frame, determine the size parameters of the unit block in the current frame.

[0125] The depth information is divided into multiple preset intervals based on at least one threshold. Each preset interval is mapped to the size parameter of a cell block, meaning there is a mapping relationship between the preset interval and the size parameter of the cell block.

[0126] Based on the comparison results and the size parameters of the previous frame's unit block, it can be determined whether to continue maintaining the size parameters of the previous frame's unit block, that is, whether to determine whether the size parameters of the current frame's unit block remain unchanged.

[0127] The threshold for dividing depth intervals can be determined based on the detected target. The detected target has an upper limit on its motion speed. Based on this upper limit and the acquisition frequency of the image acquisition device, the upper limit of the target's displacement during the acquisition of adjacent frames is determined. Based on this upper limit of displacement, a threshold is determined such that the interval between adjacent depth intervals is at least this upper limit of displacement.

[0128] For intermediate frames (excluding the reference frame of the same detection target) that detect the same target, when determining the cell block size parameters used for the intermediate frame, it is necessary to compare the depth information of the target region of the intermediate frame (i.e., the current frame) with that of the previous frame. That is, compare the depth information of the target region of the current frame with that of the target region of the previous frame. Based on the comparison result and the preset interval (or the size parameters of the mapped cell block) where the target region of the previous frame is located, determine the size parameters of the cell block of the current frame.

[0129] If the difference in depth information between the target region of the current frame and the previous frame is within the allowable range, the size parameters of the unit block of the previous frame are maintained; if it exceeds the allowable range, the size parameters of the unit block of the previous frame are increased by one level (the depth information of the target region of the current frame is greater than the depth information of the target region of the previous frame) or decreased by one level (the depth information of the target region of the current frame is less than the depth information of the target region of the previous frame) to obtain the size parameters of the unit block of the current frame.

[0130] Optionally, the preset interval and the size parameter of the unit block have a mapping relationship; determining the size parameter of the current frame unit block based on the comparison result and the preset interval where the depth information of the target region of the previous frame is located or the size parameter of the unit block of the previous frame includes: determining the preset interval corresponding to the current frame as the previous interval, the next interval, or unchanged based on the comparison result of the difference between the depth information of the target region of the current frame and the previous frame and a preset threshold; determining the size parameter of the current frame unit block according to the preset interval corresponding to the current frame and the mapping relationship.

[0131] The comparison results are categorized into three types: rising, falling, and holding scenarios. In both rising and falling scenarios, the difference between the depth information of the target region in the current frame and the previous frame exceeds a preset threshold, causing a change in the size parameter of the current frame's unit block. In the rising scenario, the depth information of the target region in the current frame is greater than that in the previous frame, and the size parameter of the current frame's unit block is: the size parameter of the previous frame's unit block + "1". Here, "1" indicates that the size parameter of the current frame's unit block is the size parameter of the unit block corresponding to the next preset interval of the preset interval in the previous frame, i.e., the next size parameter. In the falling scenario, the depth information of the target region in the current frame is less than that in the previous frame, and the size parameter of the current frame's unit block is: the size parameter of the previous frame's unit block - "1", i.e., the previous size parameter of the previous frame's unit block.

[0132] For example, taking a preset threshold of 100 as an example, the size parameters of each preset interval and its corresponding unit block are as follows: [0-100) and the first size parameter, [100-200) and the second size parameter, [200-300) and the third size parameter, etc. Assuming that the unit block size parameter of the previous frame is the second size parameter, when the depth information of the target area of ​​the current frame is greater than 205 (usually less than 300), the size parameter of the unit block of the current frame is determined to be the third size parameter; when the depth information of the target area of ​​the current frame is less than 95, the size parameter of the unit block of the current frame is determined to be the first size parameter.

[0133] To further stabilize the size parameters of image units across multiple consecutive frames, the size parameters of the unit blocks are only changed after the differences all exceed a preset threshold and a sufficient number of frames have been reached. In other words, it is necessary to count the number of frames that satisfy the rising scene condition, and only after the number of frames exceeds a set number will the size parameters of the current frame's unit block be adjusted to the next or previous size parameter of the original size parameter.

[0134] Step S307: Determine the minimum bounding rectangle of the target region.

[0135] Specifically, for each frame of the image where a detected target is located, the minimum bounding rectangle (BBox) of the region where the detected target is located (i.e., the target region) in that frame is determined. This minimum bounding rectangle can be determined by the non-zero regions (i.e., regions with values ​​of 1) in the mask of the target region. The coordinates of the corner points of the minimum bounding rectangle are then obtained, specifically the coordinates of the top-left and bottom-right corners, denoted as (x1, y1) and (x2, y2) respectively.

[0136] Step S308: Adjust the corner points of the minimum bounding rectangle so that the width and height of the adjusted minimum bounding rectangle are C and R times the width and height of the unit block, respectively.

[0137] Where C and R are both positive integers.

[0138] Specifically, for each frame of image where a target is detected, the corner coordinates (x1, y1) and (x2, y2) of the minimum bounding rectangle of the target region of that frame of image are adjusted based on the size parameters of the unit block, so that the width and height of the adjusted region are integer multiples of the unit block, that is, the adjusted BBox can be filled by an integer number of unit blocks.

[0139] When adjusting, the coordinates of the top left corner can be rounded down, and the coordinates of the top right corner can also be rounded down.

[0140] For example, taking a square unit block with a side length of 10 pixels, and (x1,y1) and (x2,y2) as (103,52) and (197,108) respectively, the coordinates of the upper left and upper right corners of the adjusted BBox can be (100,50) and (200,110).

[0141] Step S309: From the image or its associated perturbation image, extract the portion corresponding to the adjusted minimum bounding rectangle as a local image.

[0142] Using the adjusted BBox, the image of the target region detected from the original image, or other images such as perturbation images associated with the original image, is used to extract the corresponding region and obtain a local image.

[0143] The perturbation image associated with the original image can be another image with a similar style to the original image, used for perturbation processing of the target region of the original image. Perturbation images can be pre-associated with different detection targets, thus determining the associated perturbation image based on the detected targets in the original image. Alternatively, the similarity between multiple pre-stored perturbation images and the detected targets in the original image can be calculated, and the perturbation image with a similarity within a set range can be used as the perturbation image associated with the original image.

[0144] Step S310: Based on the local image, generate a target image containing C×R unit blocks, and use the target image to cover or replace the corresponding area of ​​the image.

[0145] After capturing a local image, the local image is processed into a pixel-style target image containing C×R unit blocks. This target image is then used to cover or replace the corresponding area of ​​the original image (the adjusted BBox), thereby achieving occlusion or blurring of the detected target and preventing the exposure of the target's details.

[0146] Any method can be used to process a local image into a target image containing C×R unit blocks. The target image is then generated using a method similar to mosaicking. Specifically, the average pixel value of the region corresponding to each unit block in the original image or associated perturbation image is used as the fill color of that unit block to obtain the target image.

[0147] Optionally, generating a target image composed of C×R unit blocks based on the local image includes: downsampling the local image to a C×R size; upsampling the downsampled image back to its original size to obtain the target image.

[0148] For example, taking a local image with a size of 100×100 and C and R of 10, the local image is first downsampled to 10×10 to obtain a low-resolution version of the local image; the low-resolution version of the local image is then enlarged back to its original size, i.e., 100×100, to obtain a pixel-style or mosaic-style target image.

[0149] By downsampling and then upsampling, the original image is reduced in size and then enlarged, efficiently generating a pixel-style target image. This preserves the visual style of the original image while improving the efficiency of perturbation processing.

[0150] In this embodiment, for the perturbation processing of the same target in an image sequence or video frame, target detection and tracking are used to track the same detected target, obtaining the target region of the detected target in each frame image. Depth distribution data of the target region is obtained through depth estimation, and depth values ​​representing the overall depth level of the target region are obtained using the depth distribution data. By simplifying the depth distribution data into depth values, the determination of the unit block size parameter is simplified and the determination efficiency is improved. When determining the unit block size based on the depth value, in order to avoid frequent changes in the unit block size between adjacent frames, the size of the unit block in the next frame is determined based on the comparison results of the depths of adjacent frames. This improves the continuity of the unit blocks and avoids visual flickering of the unit blocks due to small depth fluctuations caused by depth estimation errors, thereby improving the stability and visual smoothness of target region processing in consecutive frames. By adjusting the target region to an integer multiple of the unit block size, it is ensured that the generated multiple unit blocks completely cover the area, avoiding edge truncation, improving the regularity and accuracy of visual interference processing, while simplifying the generation logic and enhancing operational efficiency and effect stability.

[0151] Optionally, the image processing method further includes: performing a morphological dilation operation on the target region to obtain a dilated region; determining a mixing coefficient for the pixels based on the distance between pixels in the transition region and the interference region; the interference region being the region corresponding to the adjusted minimum bounding rectangle, and the transition region being the difference between the dilated region and the interference region; mixing the image and the target image based on the mixing coefficient to obtain a transition image, and using the transition image to cover or replace the transition region of the image.

[0152] Morphological dilation is used to expand the range of a target region. The specific process is as follows: Select a structuring element of a specific shape (such as a rectangle, circle, etc.) and use it to traverse every pixel of the target region; for each pixel, take the maximum value of all pixels within the coverage area of ​​the structuring element (grayscale image) or perform a logical "OR" operation (binary image) as the new value of that pixel; finally, the boundary of the target region expands outward, fills the small gaps within the region, increases the area of ​​the region, strengthens the connectivity of the region, and obtains the expanded region.

[0153] For each pixel in the transition region within the expanded region, a blending coefficient is calculated based on the pixel's distance from the adjusted bounding box (BBox). This blending coefficient is then used to blend the pixel value in the original image with the pixel value of the corresponding pixel (e.g., the nearest pixel) in the adjusted BBox, resulting in the pixel value in the transition image. This process iterates through all pixels in the transition region to obtain a transition image, which is then used to cover or replace the transition region of the original image. Alternatively, the transition image can be stitched together with the target region to obtain a stitched image, which is then used to cover or replace the corresponding region of the original image, thus perturbing the original image.

[0154] The smaller the distance between the pixel and the adjusted BBox, the greater the proportion of the pixel in the target image during the blending process, i.e., the larger the blending coefficient, and the closer the pixel value is to the pixel in the target image.

[0155] For a pixel in the transition region, its pixel value in the transition image is: mixing coefficient × p1 + (1 - mixing coefficient) × p2, where p1 is the pixel value of the nearest pixel in the target image and p2 is the pixel value of the pixel in the original image.

[0156] For example, Figure 5 This is a schematic diagram of the transition region and interference region provided in the embodiments of this application, as shown below. Figure 5 As shown, the target is a face. In one frame of the image, the adjusted bounding box corresponding to the target area of ​​the face is region 510. Through morphological dilation, region 510 is expanded to region 520. The portion of region 520 excluding region 510 is the transition region. Figure 5 It can be seen that in this transition region, the image used for perturbation processing in region 510 and the original image are mixed, so that the perturbation processing transitions from "0" (no perturbation processing, i.e., the original image style) to "1", which improves the visual coherence of the image.

[0157] Figure 6 This is a schematic diagram of the image processing results provided in the embodiments of this application. Figure 6 Taking the detection target as a face as an example, such as Figure 6 As shown, the region where the face is located in the image or multiple frames of the image is obtained, namely region 60. Then, a unit block of a certain size is used as a mosaic block to perform mosaic processing on region 60, i.e., mosaicking. Alternatively, a unit block of a certain size is used to replace the face in region 60 with other pixel-style faces, such as cartoon characters.

[0158] Figure 6 The processing method shown is only an example. Other processing methods can also be used to blur the target area, and this application does not limit this to them.

[0159] This embodiment also provides a target region tracking method, including: determining the scene type of each frame in a series of consecutive frames; selecting a target detector from a set of preset detectors based on the scene type; performing target region detection on each frame using the corresponding target detector; and tracking the target region in subsequent frames based on the target region detected in any frame.

[0160] Among them, consecutive frames can be consecutive frames in a video or a sequence of images. The scene type is determined by one or more of the following: the overall features of the image, the features of the target region in the image, and the relationship between the target region and its surrounding regions.

[0161] Specifically, the scene type of an image can be determined based on factors such as image resolution, image illumination intensity, image background complexity, distance between the target region and the viewpoint, movement speed of the target region, whether the target region is occluded, whether the target region crosses an object, the type of the target region, and the texture similarity between the target region and its surrounding regions.

[0162] Scene types can be classified based on the distance between the target region in the image and the viewpoint, such as into distant scenes and close scenes; scene types can also be classified based on whether the target region is occluded, such as into occluded scenes and unoccluded scenes; scene types can also be classified based on the type of the target region, such as into human body parts, license plates, etc., or based on whether the target region has weak texture edges, such as into weak texture edge scenes and strong texture edge scenes. Alternatively, multi-dimensional features from the aforementioned features can be used to comprehensively classify scenes, achieving more types of scene classification.

[0163] First, target regions in the image can be identified. Then, based on the features of the extracted target regions and their surrounding areas, the scene type of the image can be determined. Any algorithm can be used to identify target regions. The target regions identified in this step are only used to determine the scene type and are different from the target regions detected by the subsequent target detector. For example, in this step, a default detector can be directly used to detect target regions in the image. This default detector can be one of several preset detectors or a non-preset detector.

[0164] Optionally, the target region is the region where the target part of the target object is located; determining the scene type of each frame in a series of consecutive frames includes: for each frame, identifying the target part of the target object in that frame; and determining the scene type of that frame based on at least one of the following: the distance between the target part and the viewpoint, whether the target part is occluded, and the degree of difference between the texture features of the edge of the target part and the surrounding texture features. The viewpoint is used to characterize the position of the observer or reference camera (also known as a virtual camera).

[0165] The target area refers to the region of the target object to be detected. It can be a local area with a specific function or an area unsuitable for display. The surrounding texture features are the texture features of the region surrounding the target area.

[0166] The target object can be a person, animal, car, express delivery package, etc. The target location can be the tattoo area, a specific organ, license plate, shipping label sticker area, etc.

[0167] To determine the scene type of an image, target objects within the image are first identified. Based on these identified objects, target parts of those objects are then identified. The distance between the target part and the viewpoint (the origin of the camera coordinate system) is calculated based on the coordinates of the target part, such as the coordinates of the four corners of its bounding box. Alternatively, the distance can be represented by the number of pixels contained in the target part. The percentage of pixels in the target part that are occluded is then counted. If this percentage is greater than a preset percentage, the target part is determined to be occluded. Alternatively, if occluded pixels exist in a preset area of ​​the target part, the target part is also determined to be occluded. Texture features at the edges of the target part are extracted. Edges can be areas less than or equal to 5 pixels from the bounding box. These texture features are compared with the texture features of the surrounding areas of the target part to determine the degree of difference.

[0168] The scene type of the image, i.e. the detection scene of the target area, is determined by comparing the determined distance and degree of difference from the viewpoint with the preset threshold, as well as the result of whether the target part is occluded.

[0169] Specifically, if the target part in the image is occluded, or the distance between the target part and the viewpoint is greater than a preset distance, or the difference between the texture features of the edge of the target part and the surrounding texture features is less than a preset degree, then the scene type of the image is determined to be the first type or the target scene type; otherwise, it is determined to be the second type.

[0170] By utilizing multi-dimensional features for scene segmentation, the accuracy of scene segmentation is improved, and the difficulty of detecting target regions in images is effectively identified, providing a reliable foundation for matching detectors.

[0171] In some embodiments, a pre-trained model can be used to identify the scene type of an image. Optionally, determining the scene type of each frame in a series of consecutive frames includes: for each frame, inputting the frame into a pre-trained scene classification model to obtain the scene type of the frame; wherein the scene classification model is pre-trained based on training samples of multiple scene types.

[0172] Training samples can be labeled manually to indicate the scene type of the image.

[0173] Scene classification models are used to classify images by scene type and determine the scene type of the input image.

[0174] For each frame of a video or image sequence, the frame is input into a pre-trained scene classification model, which extracts and analyzes the features of the image to identify the scene type.

[0175] Specifically, scene classification models can determine the scene type of an image based on features extracted from the image, such as light features, occlusion features of the target area, distance between the target area and the viewpoint, image perspective, and pose of the object containing the target area in the image.

[0176] Scene types can include two types, such as the first type and the second type, which are the aforementioned target scene types. Under the first type, the target area is relatively blurry and difficult to detect, while under the second type, the target area is obvious and easy to detect.

[0177] The scene classification model can be trained in advance using training samples of various scene types, enabling the model to recognize multiple scene types.

[0178] In some embodiments, the training samples for the target scene type among multiple scene types include at least one of the following: a first sample where the target region is occluded, a second sample where the distance between the target region and the viewpoint is greater than a preset distance, and a third sample where the difference between the texture features of the edge of the target region and the surrounding texture features is less than a preset degree. The viewpoint is used to characterize the position of the observer or reference camera.

[0179] Detecting target regions in images with specific target scene types is quite challenging, and conventional detection methods are insufficient for accurate detection. Therefore, it is necessary to use detectors designed specifically for these target scene types.

[0180] The target area is occluded, making part of the target area invisible, which increases the difficulty of target area detection and makes it easy to miss detection. The scene type of the corresponding image is determined as the target scene type.

[0181] For example, the preset distance can be 10 meters, 20 meters, 50 meters or other longer distances.

[0182] When the distance between the target area and the viewpoint is greater than a preset distance, the image of the target area in the image is smaller and darker overall, making the image easily blurry and difficult to identify, thus making detection more difficult. The scene type of the corresponding image is determined as the target scene type.

[0183] The surrounding texture features are the texture features of the area surrounding the target region. If the difference between the texture features of the target region edge and the surrounding texture features is less than a preset threshold, it indicates that the target region has weak texture edges, making it difficult to accurately segment the target region. The scene type of the corresponding image is then determined as the target scene type.

[0184] For example, scene classification models can employ Transformer models, such as the Swin-Transformer model. The Swin-Transformer model uses a hierarchical network structure to progressively extract feature maps of different resolutions from low to high, corresponding to scene information at different scales. This enables the extraction of both close-range details and global features. Through sliding window self-attention mechanisms and shifted windows, it balances and enhances the correlation between close-range details and global features. Furthermore, through the hierarchical structure and sliding window, it simultaneously extracts multi-dimensional features such as distance, occlusion state, and lighting conditions of the target region. Based on the extracted features, scene classification is achieved, resulting in the scene label of the image, i.e., the scene type.

[0185] By training the model with a large number of training samples collected for various scene types, the model can automatically learn the representation of multi-dimensional features in images of various scene types, thereby achieving scene classification and improving the accuracy of scene classification.

[0186] There are at least two preset detectors, and each detector can correspond to one or more scene types. The target detector can be selected based on the correspondence between detectors and scene types.

[0187] The detector can be implemented using a segmentation network or a detection network. It can detect the target region in the image by detection or instance segmentation and output a mask of the target region.

[0188] Specifically, for each frame of an image in a video or image sequence, the target detector corresponding to the image is determined from a set of multiple pre-defined detectors based on the scene type of the image.

[0189] Multiple preset detectors are pre-designed for each scene type to detect target regions in images of various scene types.

[0190] The preset detectors can be single-stage detectors, two-stage detectors, end-to-end detectors, etc. Single-stage detectors can be models like the YOLO (You Only Look Once) series or SSD (Single Shot MultiBox Detector). Two-stage detectors can be based on single-stage detectors by adding SAM (SegmentAnything Model) series models, such as SAM2, or models like Faster R-CNN (Faster Region-based Convolutional Neural Network) or Mask R-CNN (MaskRegion-based Convolutional Neural Network). End-to-end detectors can be models like CenterNet (Center-based Object Detection Network) or DETR (Detection Transformer).

[0191] For example, a single-stage detector can be a YOLOv8 model, or a model consisting of YOLOv8 and SAM2.

[0192] For example, the model composed of YOLOv8 and SAM2 can be the detector corresponding to the first type mentioned above.

[0193] Each frame of a series of images is input into the target detector corresponding to its scene type, and the target detector performs target region detection on the input images.

[0194] If multiple frames of images correspond to the same target detector, the multiple frames of images can be input into the target detector in frame order or other order to perform target region detection frame by frame.

[0195] This step yields the single-frame detection results for each image frame, which can be the bounding box corresponding to the detected target region. The single-frame detection results may also include a segmentation mask and its identifier (e.g., Identity, ID), where the segmentation mask is used to segment the detected target region.

[0196] Taking a series of consecutive frames containing two scene types (e.g., the first type and the second type) as an example, for the first type of image, the target region is detected by the first detector, and for the second type of image, the target region is detected by the second detector.

[0197] For example, Figure 7 This is a schematic diagram of the target area detection results provided in an embodiment of this application. Figure 7 Taking the detection target as a face as an example, Figure 7 As shown, the selected target detector performs face detection on the input image to obtain the region where the face is located, i.e., region 700.

[0198] Each time the target region is tracked, a tracking result is obtained. The tracking result indicates whether the target region was successfully tracked, and the tracked region is the tracked area.

[0199] Considering the spatiotemporal correlation between adjacent frames in a video or image sequence, in order to improve the accuracy of target region detection, this step introduces a target tracking mechanism. Using the single-frame detection result output in the previous step, the target region is tracked on multiple consecutive frames in the video or image sequence to obtain the tracking results of each intermediate frame. The intermediate frames are the frames other than the first frame in the multiple consecutive frames.

[0200] Target region tracking is specifically used to strengthen the correlation of target regions of the same detected target across frames. By establishing a continuous representation of the target region in the time dimension, it addresses the problems of missed detections, false detections, and ID jumps that easily occur in single-frame detection. Based on the motion trajectory, position, and features of the target region in the previous frame, the position of the target region in the current frame is predicted. This prediction is then corrected by combining the single-frame detection result of the current frame (i.e., the target region detected in the current frame), resulting in the tracking region and its tracking score. The tracking score represents the probability that the tracking region in the current frame successfully tracks the target region in the previous frame.

[0201] Specifically, a tracking network can be used to track target regions in multiple frames of images to obtain the spatial trajectory of the target region of the same detected target over time.

[0202] For the same target region (corresponding to the same detection target) in different frames, such as the license plate of the same vehicle in different frames, cross-frame tracking of the target region can be achieved by associating the detection target with the Mask ID and bounding box of different frames.

[0203] Optionally, the consecutive multi-frame images are consecutive multi-frame images in a three-dimensional video; based on the target region detected in any frame image, the tracking of the target region is performed on the surface of the three-dimensional object in the subsequent frame image, including: based on the target region detected in any frame image, the tracking of the target region is performed on the surface of the three-dimensional object in the subsequent frame image.

[0204] 3D video is a stereoscopic visual content with depth perception constructed using digital technology. Its core lies in utilizing optical principles, image algorithms, or device interaction to allow viewers to experience a spatial sense beyond the two-dimensional plane. 3D video can be generated using technologies such as AR (Augmented Reality), VR (Virtual Reality), and glasses-free 3D display. Before presenting a 3D video, the method provided in this application's embodiments is required to track the 3D video target area, thereby enabling subsequent processing such as occlusion, replacement, encryption, and deformation of the target area content.

[0205] For 3D video, when tracking a target, it is necessary to use the single-frame detection results and tracking results of each previous frame to track the target region on the surface of the 3D object in the current frame and obtain the tracking result.

[0206] Specifically, in single-frame detection, based on the target detector, target recognition and localization are performed on the surface of the 3D object contained in each frame image, and the target region in each frame image is detected. Based on the detection results of the target region in each frame image obtained in the previous step, the motion trajectory and changes of the target region on the surface of the 3D object are tracked between different frames in consecutive frames. In this step, by analyzing the feature similarity and positional changes of the target region between consecutive frames, the target regions in different frames are associated, thereby achieving target tracking.

[0207] Target region tracking on the surface of a three-dimensional object can be performed using the tracking method provided in the aforementioned embodiments, and will not be repeated here.

[0208] The aforementioned method can be implemented using the designed target region tracking framework. Figure 8 A schematic diagram of the target region tracking framework provided in the embodiments of this application, as shown below. Figure 8 As shown, the target area tracking framework includes a scene classification model, multiple preset detectors, such as detector 1 to detector j, where j is an integer greater than 1, and a tracking module.

[0209] Scene labels are determined for consecutive frames of images using a scene classification model. A detector matching the scene labels is then used to detect target regions in the images. For each frame's detected target region, a tracking module tracks it. During tracking, if a target region existed in the previous frame, it tracks that region, predicts its position in the current frame, and obtains the tracking region. It then checks if a target region in the current frame matches the tracking region. If so, tracking is successful, and the tracking result is output. If not, a new tracker is initialized based on the target region detected in the current frame to obtain tracking results for the target region detected in the current frame in subsequent frames.

[0210] For example, Figure 9 This is a schematic diagram of the target area tracking results provided in an embodiment of this application. Figure 9 Taking the tracking of two individuals, Target 1 and Target 2, as an example, Figure 9 As shown, in the three consecutive frames, target 1 is detected for the first time in the first frame, and a tracker for target 1 is established. Target 1 is tracked in the subsequent frames, namely the second and third frames, using the tracker. In the second frame, target 1 is tracked by the tracker, and at the same time, a new target, target 2, is detected by the selected target detector, and a tracker for target 2 is established. In the third frame, target 1 is not tracked, but target 2 is tracked. Figure 9 The target area is indicated by a box.

[0211] After obtaining the tracking results, the tracking results and single-frame detection results can be output together, or only the tracking results can be output. The tracking results may include the detection results of the target region in the first frame image, so that the image processing device can perform image processing on the continuous multi-frame images based on the tracking results, or the tracking results and single-frame detection results, such as target region occlusion, target region content replacement, target region content deformation, target region content encryption, etc.

[0212] Specifically, for a current frame where target region tracking fails, the target region of the nearest preceding frame that detected the same target, or the nearest preceding frame where target region tracking was successful, can be directly determined as the target region of the current frame. Alternatively, based on the optical flow fields of the preceding and current frames, the target region detected or successfully tracked in the preceding frame can be adjusted, and the adjusted target region can be used as the target region or tracking region of the current frame. The distance between the target region of the preceding frame and the image edge can be limited to at least a set distance.

[0213] The target region tracking method provided in this embodiment, for target detection in video or image sequences, dynamically matches a suitable detector (i.e., a target detector) from multiple provided detectors based on the scene type of each frame. This allows for the use of the corresponding target detector for each frame to perform target region detection, improving the accuracy of target region detection in a single frame, achieving precise target region localization, and reducing the probability of missed and false detections due to scene differences. Furthermore, by combining this with a target tracking mechanism, the method can fully exploit the spatiotemporal correlation between frames in a video or image sequence. Even in complex scenarios such as rapid target movement or partial occlusion, it can establish stable target associations based on historical frame detection and tracking results, ensuring the continuity of the target trajectory and the consistency of the detection results, further improving the accuracy and reliability of target detection.

[0214] Optionally, the multiple detectors include a first detector corresponding to the target scene type, the first detector being used to detect target regions of the image of the target scene type; the first detector includes a generator and a discriminator, the generator including an encoder and a decoder; the encoder is used to extract multi-scale encoded features of the input image, the encoder including multiple improved multi-scale modules, the improved multi-scale modules including multi-scale modules connected to an adaptation network, the adaptation network including a normalization layer, a first fully connected layer, a second fully connected layer, an activation layer and a residual connection layer connected in sequence; the parameters of the multi-scale modules are frozen during the training of the first detector; the encoder adopts a U-shaped network structure (U-NET) to decode the multi-scale encoded features to obtain a prediction mask; the connected regions of the prediction mask are concatenated with the input image and then input into the discriminator to obtain the confidence that the prediction mask is a target region, and the determination of whether to remove the prediction mask is based on the confidence.

[0215] The first detector corresponds to the target scene type, which is an image where the target region features are not obvious and it is difficult to detect, such as when the target region is occluded, the target region is far from the viewpoint, or the target region has weak texture edges.

[0216] When the target area is occluded or invisible, or the target area is blurry or small, or the target area is difficult to segment due to weak texture edges, the accuracy of existing detectors is poor and there are cases of missed detection. In order to solve this problem, this embodiment provides a detector for such image target areas, namely the first detector.

[0217] Figure 10 This is a schematic diagram of various scene types provided in the embodiments of this application. The scene types are divided according to whether there is occlusion, distance (distance between the target area and the viewpoint), and whether the texture is obvious (or whether it has a faint texture edge). Among them, the scene type of the occluded, distant, or textured images is the target scene type, and the rest are classified as other scene types. Figure 10 As shown, in the first set of images, the detected targets are all human faces. In the image on the right, the face is obscured by trees, so it is classified as a target scene type. In the second set of images, the detected targets are license plates. The distance between the vehicles and the viewpoint varies, so the images that are farther away are classified as target scene types. In the third set of images, the surface texture of the detected targets differs from the surrounding texture. The texture on the right is less obvious, so it is classified as a target scene type.

[0218] Figure 11 A schematic diagram of the structure of the first detector provided in the embodiments of this application is shown below. Figure 11As shown, the first detector includes a generator and a discriminator. The generator generates a predicted mask for the image and includes an encoder and a decoder. The encoder extracts and encodes features from the input image, converting the image information into a feature vector. The decoder decodes the feature vector and outputs a predicted mask. The discriminator takes the result of concatenating the predicted mask and the original image as input and determines whether the region indicated by the predicted mask is the target region, i.e., it performs a true / false judgment on whether the predicted mask represents the target region.

[0219] See also Figure 11 To adapt to target region detection in complex scene types, its encoder includes multiple improved multi-scale blocks. Figure 11 Taking four improved multi-scale modules as an example, more or fewer improved multi-scale modules can also be used, and this application does not limit this. The improved multi-scale module is based on the multi-scale module provided by the SAM2 model, with an adapter network connected on top of it. The adapter network includes a normalization layer, a first fully connected layer, a second fully connected layer, an activation layer, and a residual connection layer connected in sequence. The activation layer can be GELU (Gaussian Error Linear Unit).

[0220] The encoder of the generator of the first detector can be obtained by adjusting the multi-scale module in SAM2 to the aforementioned improved multi-scale module. In this way, the advantages of the SAM2 large model in global context deep semantic understanding can be retained, while adapting to target region detection of target scene type images.

[0221] When training the encoder of the generator, the parameters of the multi-scale modules can be frozen, so that only the network parameters of the adapted network added during improvement are trained.

[0222] The encoder of the generator can use a U-shaped network to decode and integrate the outputs of the various improved multi-scale modules included in the decoder to obtain the generator's output, i.e., the prediction mask.

[0223] The encoder may include multiple modules, each containing a combination of multiple convolutional layers, batch normalization layers, and activation layers.

[0224] Optional, Figure 12 For this application Figure 11 The schematic diagram of the decoder structure of the generator in the illustrated embodiment is as follows: Figure 12As shown, the decoder includes: a decoding module connected to each improved multi-scale module; the decoding module connected to the first improved multi-scale module includes a first module; and the decoding modules connected to the other improved multi-scale modules include the first module and a second module. The first module includes a first convolutional layer, a first batch normalization layer, a second convolutional layer, a second batch normalization layer, a residual connection layer, and an activation layer connected in sequence. The second module is used to upsample the output of the connected first module and then perform channel concatenation with the output of the other connected second modules. The concatenated features are then processed sequentially through the following layers to become the output of the second module: a third batch normalization layer, an activation layer, a third convolutional layer, a fourth batch normalization layer, an activation layer, a fourth convolutional layer, and a residual connection layer.

[0225] The decoder uses U-net results, which can fuse features at different levels through skip connections, thereby effectively preserving detailed information and improving the accuracy of mask segmentation. At the same time, the symmetric structure of U-net can efficiently utilize computing resources and perform well even with small sample data, making it suitable for scenarios with relatively limited data.

[0226] See also Figure 12 The last second module is followed by multiple convolutional layers.

[0227] Figure 12 Taking the decoder connecting four improved multi-scale modules as an example, the output of each improved multi-scale module first passes through a convolutional layer to obtain feature maps of different scales but with the same number of channels. The output of the first convolutional layer is upsampled to enlarge the scale, and then element-wise added (summed) with the feature maps of other scales to achieve feature fusion at different scales. After convolution, the feature maps of the four scales enter the first module for further feature extraction. After processing by the first module, except for the feature map corresponding to the upsampling operation, the feature maps of the other scales sequentially enter the second module for further feature processing, gradually restoring the spatial resolution and integrating semantic information to obtain the final output feature map. This feature map therefore passes through multiple convolutional layers, with the number of channels gradually changing, to obtain the final output prediction mask.

[0228] The predicted mask output by the generator is concatenated with the image input to the generator and then input into the discriminator. The discriminator judges and filters the predicted mask output by the generator to obtain the target region and its confidence level.

[0229] Taking an RGB (Red-Green-Blue) image as an example, the RGB image input to the generator is concatenated with the predicted mask output by the generator to obtain a 4-channel RGBA image. This RGBA image is then input into a discriminator, which extracts and transforms image features step by step to obtain the target region and its confidence level in the RGBA image. Target regions with low confidence levels can be filtered out.

[0230] Figure 13 For this application Figure 11 The schematic diagram of the discriminator in the illustrated embodiment is as follows: Figure 13 As shown, the discriminator includes convolutional layers and multiple residual modules ( Figure 13 Taking M as an example (M is an integer greater than 1), the activation layer and the pooling layer.

[0231] like Figure 13 As shown, the residual module includes two branches: the main path branch on the left and the shortcut branch on the right. The main path branch includes an instance normalization layer, an activation layer, a convolutional layer, an average pooling layer, and a downsampling layer. The shortcut branch includes a convolutional layer, an average pooling layer, and a downsampling layer. The shortcut branch and the main path branch are connected element-wise through the output layer to achieve residual connection, thus obtaining the output of the residual module.

[0232] Connectivity analysis is performed on the predicted mask output by the generator to obtain multiple candidate regions. For each candidate region, it is superimposed with the original image (the image input to the generator) to obtain an RGBA image that is input to the discriminator. After passing through the convolutional layer of the discriminator, the number of channels is changed, and multiple residual modules are stacked to increase the number of channels. Then, it passes through a combination of multiple convolutional layers and activation layers in sequence. Figure 13 (Taking two combinations as an example), adjust the size of the feature map.

[0233] In addition to the target scene type, multiple scene types also include a second type. The detector corresponding to the second type is called the second detector. That is, in addition to the first detector, there is also a second detector. The second detector is used to detect the target region in the second type of image. The image type other than the target scene type can be denoted as the second type.

[0234] The second detector can be any detector used for target region detection, such as the YOLO model.

[0235] Optionally, the plurality of detectors further includes a second detector for detecting target regions in images other than the target scene type; the second detector includes an object detection module and an instance segmentation module; the object detection module is used to detect the position of the target in the input image and obtain the instance detection box of the target; the instance segmentation module is used to perform instance segmentation on the input image using the instance detection box as a prompt to obtain the detection result of the target region of the input image.

[0236] The specific objects to be detected can be parts of the human body, license plates, etc.

[0237] For example, the object detection module can be a YOLO series model, such as the latest version of the YOLOv8 model. The instance segmentation module can be a SAM2 model.

[0238] Specifically, the raw image is input into the object detection module, such as the YOLOv8 model, to detect the location of the target in the image and obtain the instance detection box of the target. The object detection module includes a cue encoder, an image encoder, and a mask decoder. The information with the instance detection box (i.e., the cue word) is input into the cue encoder to obtain the cue code; the raw image is processed by the image encoder to extract multi-dimensional features; the features output by the image encoder and the cue code are input together into the mask decoder. The mask decoder generates an instance segmentation mask based on the input information, thereby achieving instance segmentation of the specific object, i.e., the target, and obtaining the target region.

[0239] By using instance detection boxes as prompts, the segmentation module is guided to focus on the target area, reducing background interference and improving the accuracy of target area detection.

[0240] Optionally, the step of tracking the target region detected in any frame image and then tracking the target region in subsequent frames includes: for any preceding frame image in which the target region is detected, tracking the target region in subsequent frames based on multiple trackers and the target region detected in the preceding frame image, thereby obtaining the tracking score and tracking region of the subsequent frames; the subsequent frames are later than the preceding frames in time. The number of trackers can be two, three, or other numbers.

[0241] Each tracker can be used separately to track the target region in subsequent frames based on the detection results of the target region in the previous frame image, thus obtaining the tracking score and tracking region of the subsequent frames.

[0242] Multiple trackers can include mask trackers based on the SAM2 model, as well as trackers based on Kalman filters, trackers based on optical flow, and so on.

[0243] The mask tracker based on the SAM2 model utilizes the segmentation capability of the SAM2 model, using the predicted mask of the target region in the previous frame as a cue word, and then achieving target region localization and tracking in subsequent frames through mask matching.

[0244] The Kalman filter-based tracker uses the position of the target region detected in the previous frame to construct a target region motion model. Based on the target region motion model, it predicts the position of the target region in subsequent frames to obtain the predicted region. By matching the predicted region with the target region detected in subsequent frames, the tracking region is determined and the tracking score is calculated.

[0245] The optical flow-based tracker calculates the motion vectors (i.e., optical flow fields) of pixels in adjacent frames to track the displacement of the target region in subsequent frames, thus obtaining the predicted region.

[0246] By outputting the tracking region and its tracking score, it is easier to determine whether to merge the tracking region and the corresponding target region in subsequent processing.

[0247] The final tracking result for subsequent frames can be determined based on the tracking results of multiple trackers. This can be achieved by determining the tracking result of the tracker with the highest tracking score as the final tracking result for subsequent frames, or by weighting the tracking regions of multiple trackers based on their tracking scores, and then using the weighted result to obtain the final tracking result for subsequent frames. Alternatively, one tracker can be used for initial tracking, and the results can be used to perform fine-tuning with SAM2, followed by verification of the fine-tuning results using other trackers.

[0248] By using multiple trackers and fusing the results from these trackers, the accuracy and robustness of target area tracking are improved.

[0249] Optionally, based on multiple trackers and the target region detected in the preceding frame image, the target region is tracked in subsequent frame images to obtain the tracking score and tracking region of the subsequent frame. This includes: based on the target region detected in the preceding frame image, the multiple trackers are used to sequentially track the target region in the subsequent frame image; based on the tracking result of the first tracker that successfully tracks the target region, the tracking score and tracking region of the subsequent frame are determined; the tracking score of the tracker that successfully tracks the target region is greater than a preset score; or, based on the target region detected in the preceding frame image, the multiple trackers are used to track the target region in the subsequent frame image respectively to obtain the tracking results of each tracker, wherein the tracking results include a tracking score and a tracking result; based on the tracking result of the target tracker, the tracking score and tracking region of the subsequent frame are determined; the target tracker is the tracker with the highest tracking score that successfully tracks the target region. Taking a maximum tracking score of 1 point as an example, the preset score can be 0.5, 0.6, or other values.

[0250] To improve efficiency, the target area can be tracked by each tracker in a certain order. Once a tracker has successfully tracked the target area, there is no need to use subsequent trackers. The tracking result of the successfully tracked tracker can be directly output.

[0251] To ensure tracking quality, each tracker can be used to track the target area separately, and the tracking results of each tracker can be obtained. Then, the tracker with the highest tracking score among the trackers that successfully tracked the target area can be selected as the target tracker, and the tracking result of the target tracker can be output.

[0252] Optionally, the plurality of trackers includes at least two of a first tracker, a second tracker, and a third tracker. The first tracker includes an instance segmentation module that uses the target region detected in a preceding frame as a cue word. The second tracker is configured to: extract target points from the target region of a reference frame; determine matching points between the target point features of the current frame and the reference frame; propagate the extracted target points to the current frame based on the optical flow fields calculated by the current frame and the target preceding frame of the current frame, obtaining candidate points for the current frame; filter and weight the candidate points of the same target point to obtain optical flow tracking points for the target point; determine the tracking point corresponding to the target point in the current frame based on the matching points and optical flow tracking points of the same target point; determine the tracking region of the target region in the current frame based on the tracking points corresponding to each extracted target point in the current frame, and determine the tracking score of the tracking region based on the proportion of filtered candidate points; wherein the reference frame is the first frame in which the target region is detected, and the target preceding frame is the preceding frame of the current frame sampled after the reference frame. The third tracker includes a Kalman filter with the depth, image coordinates, and velocity of the target region as state variables.

[0253] For example, the instance segmentation module can be a SAM2 model.

[0254] When using multiple trackers to perform cross-frame tracking in sequence, the order of the first to third trackers is: first tracker, second tracker, third tracker.

[0255] The instance segmentation module includes an image encoder, a memory attention layer, a cue encoder, a mask encoder, and an output layer. The image encoder extracts and encodes features from the input image, converting raw image data into an abstract feature representation. The memory attention layer processes the features extracted by the image encoder using attention branches, along with memory information derived from previously processed images. This memory information allows subsequent processing to focus on key parts of the image, while also enhancing feature representations. The cue encoder uses the target region detected in the previous frame as a cue word, encoding the cue word to provide guidance for the subsequent mask decoder. The mask decoder combines the features output from the memory attention layer and the cue word encoding to generate a mask of the target in the image, determining the specific region of the target, obtaining the tracking region and its tracking score, and outputting it through the output layer. The module also includes a memory bank and a memory encoder. The memory encoder further encodes the image's feature representation, mask, or other relevant information to obtain memory information. The memory bank stores the memory information processed by the memory encoder for later retrieval.

[0256] The second tracker is a tracker that combines feature matching and optical flow. It achieves continuous tracking of the target region through tracking point sampling, multi-frame optical flow and feature fusion, and morphological post-processing.

[0257] A reference frame (also called a keyframe) is the frame in which the target region is first detected; that is, the target region was not detected in the frame preceding the reference frame. If the same target exists in multiple frames, the target regions detected in those multiple frames can be considered as one instance region, corresponding to one instance ID. If, during tracking in the current frame, there is a target region that does not match any of the obtained instance regions, a new instance region is added for that target region. This target region is the first target region detected in the current frame, and the current frame is a reference frame.

[0258] Figure 14 This is a schematic diagram of a second tracker data processing procedure provided in an embodiment of this application, as shown below. Figure 14 As shown, the second tracker achieves cross-frame tracking of the target region mainly through the following process:

[0259] From the new instance region detected in the reference frame, such as frame N, multiple target points are sparsely sampled and selected using contour uniform segmentation. For each target point, based on feature matching, the matching point of the target point in the current frame, such as frame N+i, is determined. The target point and the matching point are two points that are feature matched. False matching can also be filtered out using methods such as RANSAC. Then, multiple neighboring points are found around the target point in the reference frame. Based on the matching points of these neighboring points in the current frame, linear interpolation is performed according to distance weights. The position of the matching point of the target point in the current frame is then fine-tuned based on the interpolation result.

[0260] Calculate the optical flow field based on the (N+i)th frame and each frame in the preceding frame. Figure 14 Taking frames N+i-16 to N+i-1 as an example, where i is an integer greater than or equal to 16; by calculating the optical flow field, the target point is predicted to propagate to the corresponding point in the current frame under the optical flow field, thus obtaining candidate points of the target point; the candidate points of the same target point are filtered and weighted to obtain the optical flow tracking point of the target point; the filtering specifically filters out flying points, that is, points with excessive positional deviations, and the weights used in the weighting can be determined based on the frame number; the matching points and optical flow tracking points of the same target point are weighted to obtain the tracking point corresponding to the target point in the current frame, and the weights can be default values, for example, the weight of the optical flow tracking point is 0.6 and the weight of the matching point is 0.4; after obtaining the tracking points corresponding to multiple target points in the current frame, the position of the tracking points can be adjusted using TPS (Thin Plate Spline) to obtain the initial tracking area; since there are holes or disconnected areas in the initial tracking area, morphological operations, such as opening and closing operations, can be used to fill the disconnected areas or holes to obtain the complete tracking area. The score of the tracking region can be determined based on the number of filtered flying points, or based on the number of filtered flying points and the proportion of the tracking region with holes or disconnected areas.

[0261] The third tracker primarily predicts the real-time 3D spatial coordinates of the target region through depth estimation and Kalman filtering, then back-projects them onto the image to obtain the tracking region and its tracking score. The third tracker mainly consists of three modules: a depth estimation and calibration module, a Kalman filtering module, and a post-processing module. The depth estimation and calibration module uses AI (Artificial Intelligence) algorithms to estimate the depth of the target region detected in each frame of the image, and adjusts the target region depth using adjacent frames and reference frames to obtain a more accurate depth. The Kalman filtering module is responsible for predicting the position and velocity of the tracked target region in space, and updates the filter system based on the tracking region obtained in each frame, thereby achieving continuous optimization. The post-processing module combines the position of the tracking region predicted by the Kalman filter module in the next frame with the tracking region output by the first tracker and / or the second tracker in the next frame to obtain the final tracking region in the next frame, which is then used to update the coefficients of the Kalman filter. Essentially, it uses the depth of the tracking region predicted by the Kalman filter to remove outliers from the tracking region output by the first tracker and / or the second tracker, thus obtaining a more accurate tracking result.

[0262] Depth estimation can be performed using any easy algorithm, such as DepthAnything or Unidepth.

[0263] During depth calibration, a sliding window can be used to first perform batch calibration (within the same sliding window), and then use overlapping frames between adjacent batches to achieve inter-batch calibration, thereby improving the smoothness of depth changes at matching points in adjacent frames.

[0264] KF (Kalman Filter) initialization: The Kalman filter can be initialized with the center point, aspect ratio, height and depth of the target area, and velocity as states. The observation data of the target area can be provided by the first tracker and the second tracker. By calculating the intersection-union ratio of the tracking area of ​​the previous frame and the tracking area of ​​the current frame, and judging whether it meets the set condition, if it does, the KF covariance update is triggered and the stable frame number is incremented by 1. If the number is less than 0, the number of stable frames is reset (set to 0). After the number of stable frames reaches a certain value, such as 30 frames, the position of the target area in the next frame is predicted by KF. The intersection-union ratio (IU) of the predicted target area and the detected target area in the next frame is calculated. Combined with the estimated depth change, it is determined whether there is a match. If there is a match, KF state update is performed to correct the covariance, the number of stable frames is continued to accumulate (+1), and the number of lost frames is cleared. If there is no match (the IU is not up to standard or the depth change is too large), the number of lost frames is increased by 1. If the number of lost frames exceeds a certain number of frames, such as 10 frames, it is considered that the corresponding target has been lost, the number of stable frames is reset, and the initialization step is returned to be re-executed.

[0265] After obtaining the tracking results from multiple trackers, it is first determined whether the target region in the current frame has been successfully tracked relative to the previous frame. If so, the detection results of the tracked region and the target region that has been successfully tracked in the current frame are fused to obtain the latest position and shape of the target region in the current frame, which is the final tracking region.

[0266] Successful tracking can be determined by the intersection-union ratio (IUU) and feature similarity between the target region in the previous frame and the tracking region of the target region in the current frame.

[0267] For the same target region (same instance region) in the final tracking region across multiple frames, a weighted intersection-union matching method can be used to ensure the continuity of target region tracking. The weights can be determined based on the frame sequence interval; the larger the interval, the smaller the weight.

[0268] If the target region tracking fails in the next frame with respect to the preceding frame, the target region detected in the next frame is used as a new instance region to track the new instance region from subsequent frames. That is, the next frame is used as a new reference frame for cross-frame tracking.

[0269] Figure 15 A flowchart illustrating the target region tracking method provided in this application embodiment. Figure 2 ,like Figure 15 As shown, in this embodiment, the target area tracking method specifically includes the following steps:

[0270] Step S701: Obtain the next frame image.

[0271] Step S702: Perform scene detection and classification, that is, determine the scene type of the next frame image.

[0272] Step S703: Use the detector corresponding to the scene type to detect the target area.

[0273] Step S704: Determine whether there is an instance region in the previous frame, that is, whether there is a target region or tracking region that needs to be tracked in the previous frame; if yes, proceed to step S705; if no, proceed to step S707.

[0274] Step S705: Track the instance region of the previous frame to obtain the old instance tracking region of the next frame, that is, the region corresponding to the instance region of the previous frame in the next frame, which is also the tracking region of the next frame.

[0275] Step S706: Determine the target regions in the target regions detected in the next frame that are other than the target regions that match the old instance tracking regions as the instance regions of the next frame.

[0276] Step S707: Determine the target region to be detected in the next frame as the instance region of the next frame.

[0277] Step S708: Determine whether the old instance tracking region in the next frame has been successfully tracked; if yes, proceed to step S709; if no, proceed to step S710.

[0278] Step S709: Merge the old instance tracking region and its matching target region to obtain the old instance tracking region in the next frame and output it.

[0279] Step S710: The target region matched by the old instance tracking region is used as the instance region of the next frame.

[0280] Step S711: Determine if there is a next frame; if yes, return to step S701; otherwise, end.

[0281] For the first frame in a series of consecutive images, only target region detection is required, and the detected target region is directly determined as the instance region of the first frame.

[0282] Corresponding to the target region tracking method provided in the foregoing embodiments, this application also provides a target region tracking device, which includes: a scene classification module for determining the scene type of each frame in a series of consecutive frames; a detector determination module for selecting a target detector from a plurality of preset detectors based on the scene type; a target region detection module for performing target region detection on each frame using the corresponding target detector; and a target tracking module for tracking the target region in subsequent frames based on the target region detected in any frame.

[0283] In one possible implementation, the scene classification module is specifically used to: for each frame of image, input the frame of image into a pre-trained scene classification model to obtain the scene type of the frame of image; wherein, the scene classification model is pre-trained based on training samples of multiple scene types; the training samples of the target scene type among the multiple scene types include at least one of the following: a first sample in which the target area is occluded, a second sample in which the distance between the target area and the viewpoint is greater than a preset distance, and a third sample in which the difference between the texture features of the edge of the target area and the surrounding texture features is less than a preset degree; the viewpoint is used to characterize the position of the observer or reference camera.

[0284] In one possible implementation, multiple detectors include a first detector corresponding to the target scene type, used to detect target regions of the image of the target scene type; the first detector includes a generator and a discriminator, the generator including an encoder and a decoder; the encoder is used to extract multi-scale encoded features of the input image, the encoder including multiple improved multi-scale modules, the improved multi-scale modules including multi-scale modules connected to an adaptation network, the adaptation network including a normalization layer, a first fully connected layer, a second fully connected layer, an activation layer and a residual connection layer connected in sequence; the parameters of the multi-scale modules are frozen during the training of the first detector; the encoder adopts a U-shaped network structure to decode the multi-scale encoded features to obtain a prediction mask; the connected regions of the prediction mask are concatenated with the input image and then input into the discriminator to obtain the confidence that the prediction mask is the target region, and the determination of whether to remove the prediction mask is based on the confidence.

[0285] In one possible implementation, the decoder includes: a decoding module connected to each of the improved multi-scale modules; the decoding module connected to the first improved multi-scale module includes a first module; the decoding modules connected to the other improved multi-scale modules include a first module and a second module; the first module includes a first convolutional layer, a first batch normalization layer, a second convolutional layer, a second batch normalization layer, a residual connection layer, and an activation layer connected in sequence; the second module is used to upsample the output of the connected first module and then perform channel concatenation with the output of the other connected second modules, and the concatenated features are processed sequentially through the following layers to become the output of the second module: a third batch normalization layer, an activation layer, a third convolutional layer, a fourth batch normalization layer, an activation layer, a fourth convolutional layer, and a residual connection layer.

[0286] In one possible implementation, the multiple detectors further include a second detector for detecting target regions in images other than target scene types; the second detector includes an object detection module and an instance segmentation module; the object detection module is used to detect the position of the target in the input image and obtain the instance detection box of the target; the instance segmentation module is used to perform instance segmentation on the input image using the instance detection box as a prompt word to obtain the detection result of the target region of the input image.

[0287] In one possible implementation, the target region is the region where the target part of the target object is located; the scene classification module is specifically used to: for each frame of image, identify the target part of the target object in the frame of image; determine the scene type of the frame of image based on at least one of the following: the distance between the target part and the viewpoint, whether the target part is occluded, and the degree of difference between the texture features of the edge of the target part and the surrounding texture features; the viewpoint is used to characterize the position of the observer or reference camera.

[0288] In one possible implementation, the target tracking module is specifically configured to: for any preceding frame image in which a target region is detected, track the target region in subsequent frame images based on multiple trackers and the target region detected in the preceding frame image, and obtain the tracking score and tracking region of the subsequent frame; the subsequent frame is later than the preceding frame in time. In another possible implementation, the target tracking module is specifically configured to: based on the target region detected in the preceding frame image, use the multiple trackers to sequentially track the target region in subsequent frame images, and determine the tracking score and tracking region of the subsequent frame based on the tracking result of the first tracker that successfully tracks the target region; the tracking score of the tracker that successfully tracks the target region is greater than a preset score; or, based on the target region detected in the preceding frame image, use the multiple trackers to track the target region in subsequent frame images respectively, and obtain the tracking results of each tracker, wherein the tracking results include a tracking score and a tracking result; determine the tracking score and tracking region of the subsequent frame based on the tracking results of the target tracker; the target tracker is the tracker with the highest tracking score that successfully tracks the target region. In one possible implementation, the multiple trackers include at least two of a first tracker, a second tracker, and a third tracker; the first tracker includes an instance segmentation module that uses the target region detected in the preceding frame as a cue word; the second tracker is used to: extract target points from the target region of a reference frame; determine matching points where the target point features of the current frame and the reference frame match; propagate the extracted target points to the current frame based on the optical flow field calculated by the current frame and the target preceding frame of the current frame, to obtain candidate points of the current frame; filter and weight the candidate points of the same target point to obtain optical flow tracking points of the target point; determine the tracking point corresponding to the target point in the current frame based on the matching point and the optical flow tracking point of the same target point; determine the tracking region of the target region in the current frame based on the tracking points corresponding to each extracted target point in the current frame, and determine the tracking score of the tracking region based on the proportion of the filtered candidate points; wherein, the reference frame is the first frame in which the target region is detected, and the target preceding frame is the preceding frame of the current frame sampled after the reference frame; the third tracker includes a Kalman filter with the depth, image coordinates, and velocity of the target region as variables of state.

[0289] In one possible implementation, the consecutive multi-frame images are consecutive multi-frame images in a three-dimensional video; the target tracking module is specifically used to: perform target region tracking on the surface of a three-dimensional object in subsequent frame images based on the target region detected in any frame image.

[0290] The target area tracking device provided in this embodiment can execute the target area tracking method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0291] Corresponding to the image processing method provided in the foregoing embodiments, this application also provides an image processing apparatus, including: a size adaptive module, used to determine the size parameters of a unit block based on the depth information of the target region of an image; and a perturbation processing module, used to perform image perturbation processing on the target region using multiple unit blocks with the size parameters.

[0292] In one possible implementation, the image processing apparatus further includes a depth distribution acquisition module for acquiring depth distribution data of a target region; and a depth information determination module for determining depth information of the target region based on the depth distribution data.

[0293] In one possible implementation, the depth distribution acquisition module includes: a target region detection unit for acquiring the target region of the same target in multiple frames of images; a depth estimation unit for performing depth estimation on the multiple frames of images to obtain depth data for each frame of images; and a depth distribution extraction unit for extracting depth distribution data of the target region in each frame of images from the depth data of each frame of images.

[0294] In one possible implementation, the depth estimation unit is specifically used for: performing random enhancement processing on each frame of the multi-frame image; performing depth estimation on each enhanced frame of the image to obtain initial depth data for each frame of the image; and calibrating the initial depth data of each frame of the image to obtain depth data for each frame of the image.

[0295] In one possible implementation, the depth estimation unit is specifically configured to: perform random enhancement processing on each frame of the multi-frame images; perform depth estimation on each enhanced frame to obtain initial depth data for each frame; divide the multi-frame images into multiple batches; each batch contains multiple overlapping frames and multiple new frames, the overlapping frames containing reference frames, the reference frames being the frames where the detected target is first detected; perform depth calibration on each frame of the same batch based on the initial depth data of the reference frames to obtain the intra-batch calibration depth of the images; and perform depth smoothing processing on the new frames of adjacent batches based on the intra-batch calibration depth of the overlapping frames to obtain the depth data of the new frames.

[0296] In one possible implementation, the depth information determination module is specifically used for: classifying target regions of multiple frames of images based on the connectivity and / or the value range of depth distribution data of the target region to obtain a first target region and a second target region; determining the initial depth information of the second target region based on the depth distribution data of each second target region, and performing low-pass filtering on the initial depth information of multiple second target regions to obtain the depth information of each second target region; and obtaining the depth information of the first target region based on the interpolation result of the depth information of the second target regions in adjacent frames.

[0297] In one possible implementation, the size adaptive module is specifically used to: compare the depth information of the target region in the current frame with that in the previous frame; and determine the size parameters of the current frame unit block based on the comparison result and the preset interval where the depth information of the target region in the previous frame is located or the size parameters of the unit block in the previous frame.

[0298] In one possible implementation, there is a mapping relationship between the preset interval and the size parameters of the unit block; the size adaptive module is specifically used to: compare the depth information of the target region of the current frame and the previous frame; based on the comparison result of the difference between the depth information of the target region of the current frame and the previous frame and the preset threshold, determine whether the preset interval corresponding to the current frame is the previous interval, the next interval, or remains unchanged; and determine the size parameters of the unit block of the current frame according to the preset interval corresponding to the current frame and the mapping relationship.

[0299] In one possible implementation, the unit block is rectangular, with dimensions including width and height; the perturbation processing module includes: a rectangle adjustment unit, used to determine the minimum bounding rectangle of the target area and adjust the corner points of the minimum bounding rectangle so that the width and height of the adjusted minimum bounding rectangle are C and R times the width and height of the unit block, respectively; C and R are both positive integers; an image cropping unit, used to crop the portion corresponding to the adjusted minimum bounding rectangle from the image or its associated perturbation image as a local image; and a perturbation processing unit, used to generate a target image containing C×R unit blocks based on the local image, and use the target image to cover or replace the corresponding area of ​​the image.

[0300] In one possible implementation, the perturbation processing unit is specifically configured to: downsample a local image to a C×R size; upsample the downsampled image back to its original size to obtain a target image; and use the target image to cover or replace the corresponding region of the image.

[0301] In one possible implementation, the target region tracking device further includes a dilation module, configured to: perform morphological dilation on the target region to obtain a dilated region; determine a mixing coefficient for a pixel based on the distance between a pixel in the transition region and an interference region; the interference region is the region corresponding to the adjusted minimum bounding rectangle, and the transition region is the difference between the dilated region and the interference region; and, based on the mixing coefficient, blend the image and the target image to obtain a transition image, and replace the transition region of the image with the transition image, or use the transition image to cover the transition region of the image.

[0302] The target area tracking device provided in this embodiment can execute the target area tracking method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0303] This application also provides an electronic device, including a memory and a processor. The memory stores computer execution instructions, and the processor executes the computer instructions to implement the methods provided in any of the foregoing embodiments of this application, such as image processing methods and / or target area tracking methods.

[0304] Figure 16 This is a schematic diagram of the structure of a terminal device for image processing provided in this application. Figure 16 As shown, the terminal device 1600 for image processing provided in this embodiment includes a processor 1601 and a memory 1602.

[0305] In the specific implementation process, the memory 1602 stores computer execution instructions, and the processor 1601 executes the computer execution instructions stored in the memory 1602, so that the processor 1601 executes the above-mentioned image processing method.

[0306] The processor 1601 executes computer execution instructions to perform the following steps: determining the size parameters of the unit blocks based on the depth information of the target region of the image; and performing image perturbation processing on the target region using multiple unit blocks with the size parameters.

[0307] The specific implementation process of processor 1601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0308] Optionally, the terminal device 1600 for image processing also includes a communication component. The processor 1601, memory 1602, and communication component are connected via a bus.

[0309] This application also provides a display device for displaying multiple frames of images after image perturbation processing; in at least two of the multiple frames, the depth information of the target region where the detected target is located is different; and the size parameters of the unit blocks used in the image perturbation processing of the at least two frames are different.

[0310] For example, the display device can be a head-mounted display, a glasses-free stereoscopic display, smart glasses, etc.

[0311] The multi-frame images after image perturbation processing displayed on the display device can be sent to the display device by the terminal device 1600 for image processing, or can be obtained by the processor built into the display device itself executing the image processing method provided in any of the foregoing embodiments.

[0312] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0313] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0314] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0315] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0316] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0317] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0318] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0319] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0320] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0321] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0322] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0323] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0324] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A terminal device for image processing, characterized in that, Including memory and processor; The memory stores computer-executable instructions, and the processor executes the computer-executable instructions to perform the following steps: For a series of consecutive frames in a 3D video, the scene type of each frame is determined, and a detector that matches the scene type is used to detect the target region of each frame. Based on the target region detected in any frame image, the target region tracking is performed on the surface of the three-dimensional object in subsequent frame images to obtain the target region of the same detected target in the consecutive multiple frame images; Depth estimation is performed on the consecutive multiple frames of images to obtain the depth data of each frame; The depth distribution data of the target region in each frame of the image is extracted from the depth data of that frame. Based on the depth distribution data, the depth information of the target area is determined; Based on the depth information of the target region, the size parameters of the unit block are determined; The target region is perturbed using multiple unit blocks with the stated size parameters.

2. The device according to claim 1, characterized in that, The step of performing depth estimation on the consecutive multiple frames of images to obtain depth data for each frame includes: Random enhancement processing is performed on each frame of the continuous multi-frame images; Depth estimation is performed on each enhanced frame to obtain the initial depth data for each frame. The initial depth data of each frame is calibrated to obtain the depth data of each frame.

3. The device according to claim 2, characterized in that, The calibration of the initial depth data for each frame of image to obtain the depth data for each frame includes: The multi-frame images are divided into multiple batches; each batch contains multiple overlapping frames and multiple newly added frames, the overlapping frames include reference frames, and the reference frames are the frames in which the detected target is first detected; Based on the initial depth data of the reference frame, depth calibration is performed on each frame of the same batch of images to obtain the intra-batch calibration depth of the images. Based on the intra-batch calibration depth of overlapping frames, depth smoothing is performed on newly added frames in adjacent batches to obtain the depth data of the newly added frames.

4. The device according to claim 1, characterized in that, Determining the depth information of the target region based on the depth distribution data includes: Based on the range of values ​​of connectivity and / or depth distribution data of the target region, the target regions of the multi-frame images are classified to obtain a first target region and a second target region. Based on the depth distribution data of each second target region, the initial depth information of the second target region is determined, and the initial depth information of multiple second target regions is low-pass filtered to obtain the depth information of each second target region. The depth information of the first target region is obtained by interpolation based on the depth information of the second target region in adjacent frames.

5. The device according to claim 1, characterized in that, Determining the size parameters of the unit block based on the depth information of the target region includes: Compare the depth information of the target region in the current frame with that in the previous frame; Based on the comparison results and the preset interval where the depth information of the target region in the previous frame is located, or the size parameters of the unit block in the previous frame, the size parameters of the unit block in the current frame are determined.

6. The device according to claim 5, characterized in that, The preset interval and the size parameters of the unit block have a mapping relationship; determining the size parameters of the current frame unit block based on the comparison result and the preset interval or the size parameters of the previous frame unit block where the depth information of the target region of the previous frame is located includes: Based on the comparison between the difference in target region depth information between the current frame and the previous frame and the preset threshold, the preset interval corresponding to the current frame is determined to be the previous interval, the next interval, or remain unchanged from the preset interval corresponding to the previous frame. The size parameters of the current frame unit block are determined based on the preset interval corresponding to the current frame and the mapping relationship.

7. The device according to any one of claims 1-6, characterized in that, The unit block is rectangular, and the size parameters include width and height; the multiple unit blocks using the size parameters perform image perturbation processing on the target region, including: Determine the minimum bounding rectangle of the target region; Adjust the corner points of the minimum bounding rectangle so that the width and height of the adjusted minimum bounding rectangle are C and R times the width and height of the unit block, respectively; C and R are both positive integers. From the image or its associated perturbation image, the portion corresponding to the adjusted minimum bounding rectangle is extracted as a local image; Based on the local image, a target image containing C×R unit blocks is generated, and the target image is used to cover or replace the corresponding area of ​​the image.

8. The device according to claim 7, characterized in that, The step of generating a target image composed of C×R unit blocks based on the local image includes: The local image is downsampled to C×R size; The downsampled image is upsampled back to its original size to obtain the target image.

9. The device according to claim 7, characterized in that, The processor is also configured to: A morphological dilation operation is performed on the target region to obtain the dilated region; The mixing coefficient of a pixel is determined based on the distance between the pixel in the transition region and the interference region; the interference region is the region corresponding to the adjusted minimum bounding rectangle, and the transition region is the difference between the dilated region and the interference region. Based on the mixing coefficient, the image and the target image are mixed to obtain a transition image, and the transition image is used to cover or replace the transition region of the image.

10. An image processing method, characterized in that, include: For a series of consecutive frames in a 3D video, the scene type of each frame is determined, and a detector that matches the scene type is used to detect the target region of each frame. Based on the target region detected in any frame image, the target region tracking is performed on the surface of the three-dimensional object in subsequent frame images to obtain the target region of the same detected target in the consecutive multiple frame images; Depth estimation is performed on the consecutive multiple frames of images to obtain the depth data of each frame; The depth distribution data of the target region in each frame of the image is extracted from the depth data of that frame. Based on the depth distribution data, the depth information of the target area is determined; Based on the depth information of the target region, the size parameters of the unit block are determined; The target region is perturbed using multiple unit blocks with the stated size parameters.

11. The method according to claim 10, characterized in that, Determining the size parameters of the unit block based on the depth information of the target region includes: Compare the depth information of the target region in the current frame with that in the previous frame; Based on the comparison results and the preset interval where the depth information of the target region in the previous frame is located, or the size parameters of the unit block in the previous frame, the size parameters of the unit block in the current frame are determined.

12. The method according to claim 10 or 11, characterized in that, The unit block is rectangular, and the size parameters include width and height; the multiple unit blocks using the size parameters perform image perturbation processing on the target region, including: Determine the minimum bounding rectangle of the target region; Adjust the corner points of the minimum bounding rectangle so that the width and height of the adjusted minimum bounding rectangle are C and R times the width and height of the unit block, respectively; C and R are both positive integers. From the image or its associated perturbation image, the portion corresponding to the adjusted minimum bounding rectangle is extracted as a local image; Based on the local image, a target image containing C×R unit blocks is generated, and the target image is used to cover or replace the corresponding area of ​​the image.

13. A display device, characterized in that, The display device is used to perform the method provided by any one of claims 10-12 to realize image perturbation processing of consecutive multi-frame images in a three-dimensional video, and to display the consecutive multi-frame images after image perturbation processing.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 10-12.

Images