Foreground target bullet screen anti-shielding method based on artificial intelligence

Abstract

The invention discloses a foreground target bullet screen anti-shielding method based on artificial intelligence, and the method comprises the following steps: S10, carrying out the sampling of an original video at a preset frequency, and obtaining a video stream image; s20, processing the video stream by using a shot switching detection algorithm, and judging whether shot switching occurs in the current video frame or not; s30, processing the current image by using a face detector to obtain target face information of the current image; s40, processing the current image by using an instance divider to obtain target instance segmentation information of the current image; s50, a semantic segmentation device is used for processing the current image, target human body semantic segmentation information of the current image is obtained, the human body semantic segmentation information is a one-dimensional feature map with the same width and height as the original image, and the value on each pixel represents the probability that the pixel is a human body; s60, processing the current image by using a depth estimator to obtain depth estimation information of the current image; and S70, sending the information to a post-processing module for processing.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an artificial intelligence-based foreground target barrage anti-occlusion method. Background Art

[0002] The barrage technology originated from Japanese video websites. It allows users to post rolling subtitle comments while the video is playing. These comments will appear on the video screen in the form of barrage. This technology was soon introduced to China, and many video platforms further promoted the development of barrage culture. Barrage not only enables real-time interaction between the audience and the work, but also facilitates the audience's secondary creation of the work on the screen, becoming an indispensable and important part of the current film and television pop culture.

[0003] However, barrage technology also has some flaws. For example, the large number of barrages may affect the user's viewing experience, especially when there are too many barrages, they may cover the screen and affect the viewing of video content.

[0004] The large number of barrages will block the screen, resulting in a decrease in user experience. The purpose of inventing anti-blocking barrage technology is to solve the occlusion problem caused by too many barrages during video viewing, especially when the barrage blocks important visual content in the video, such as people's faces, specific targets or key information, which will affect the user's viewing experience. By developing intelligent barrage anti-blocking technology, the barrage can accurately "bypass" the core area of ​​the video, thereby achieving the dual advantages of immersive viewing and barrage interaction, and improving user viewing satisfaction and participation.

[0005] Anti-blocking bullet screen technology is an application of mask technology. Mask technology refers to creating a transparent "mask" that can cover a specific area of ​​the video screen. In this way, the bullet screen can be controlled to be displayed only in the area outside the mask, thereby avoiding blocking important video content.

[0006] Existing anti-occlusion barrage technology often has the following problems: (1) In the process of automatic detection and mask generation, misrecognition may occur, resulting in areas that should not be occluded being incorrectly occluded, or areas that should be occluded not being processed correctly; (2) There is no intelligent judgment on the detected targets, and many targets that do not need to be processed are also occluded, resulting in a poor barrage viewing experience; (3) The automatic detection and mask generation technology based on a single frame has no temporal semantic coherence, and it is easy to cause the barrage to flicker in the case of incomplete detection. Summary of the invention

[0007] In view of the above problems, the present invention provides an artificial intelligence-based foreground target barrage anti-occlusion method, which ensures that the target mask is tracked from multiple dimensions since it appears by combining depth of field information, semantic information, and instance information, thereby ensuring the continuity of the mask, preventing the frequent appearance of barrage, and improving the user's barrage viewing experience.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] A method for preventing foreground target barrage from being blocked based on artificial intelligence comprises the following steps:

[0010] S10, sampling the original video at a preset frequency to obtain a video stream image; the original video is a video that needs to be protected from foreground human body bullet screen;

[0011] S20, using a shot switching detection algorithm to process the video stream to determine whether a shot switching occurs in the current video frame; if a shot switching occurs in the current frame, resetting the post-processing module cache; the post-processing module cache includes target tracking information, depth information of the main target in the previous frame, and whether there is a mask output in the previous frame;

[0012] S30, using a face detector to process the current image to obtain target face information of the current image;

[0013] S40, using an instance segmenter to process the current image to obtain instance segmentation information of the current image target;

[0014] S50, using a semantic segmenter to process the current image to obtain semantic segmentation information of a human body target in the current image, wherein the human body semantic segmentation information is a one-dimensional feature map with the same width and height as the original image, and the value of each pixel represents the probability that the pixel is a human body;

[0015] S60, using a depth estimator to process the current image to obtain depth estimation information of the current image, where the depth estimation information is a one-dimensional feature map with the same width and height as the original image, and the value of each pixel represents the depth information of the object at the pixel;

[0016] S70, sending the current image target face information, the current image target instance segmentation information, the current image target human body semantic segmentation information and the current image depth estimation information to a post-processing module for processing, wherein the post-processing module includes a foreground mask selection algorithm and a temporal mask compensation algorithm, wherein the foreground mask selection algorithm is used to screen out the instances to be output in the current picture, construct mask information based on the information of the current frame, and obtain the foreground mask of the current frame; and the temporal mask compensation algorithm is used to track the instances that have been output in the previous frame, and obtain the temporal compensation mask;

[0017] S80, taking a union of the current frame foreground mask and the timing compensation mask to obtain a final output mask of the current frame, and outputting it to the outside as the current frame result;

[0018] S90, saving the instance information used by the foreground mask selection algorithm and the temporal mask compensation algorithm of the current frame for processing by the foreground mask selection algorithm and the temporal mask compensation algorithm of the next frame image;

[0019] S100, performing the above processing from S20 to S90 on the next frame of image until the video stream image processing is completed.

[0020] In a possible implementation manner, the use of the shot switching detection algorithm to process the video stream in S20 specifically includes: S201, scaling the image to a certain size; S202, performing a difference with the image retained in the previous frame, counting the number of differences greater than a preset image change threshold, obtaining the number of pixel changes, and counting the number of differences less than a preset image unchanged threshold, obtaining the number of pixels unchanged; if the number of pixel changes is greater than the preset threshold and the number of pixels unchanged is less than the preset threshold, it is determined that a shot change has occurred in the frame; S203, converting the image to a hue-saturation-brightness HSV color space, extracting the histogram information of the hue-saturation component, and comparing it with the histogram information of the hue-saturation component of the previous frame to obtain Histogram correlation of hue and saturation components. At the same time, the image is converted to grayscale color space, grayscale histogram information is extracted, and compared with the grayscale histogram information of the previous frame to obtain the histogram correlation of the grayscale histogram. If the histogram correlation of the hue and saturation components is greater than the preset threshold and the histogram correlation of the grayscale histogram is less than the preset threshold, it is determined that the frame has a lens change; S204, update the cache information, save the image after frame scaling, the histogram of the hue and saturation components, and the grayscale histogram information, and use it as a basis for judging the next frame; S205, output the lens switching information of the current video frame. When the output result is true, it indicates that the current frame has a lens switch. When the output result is false, it indicates that the current frame has not a lens switch.

[0021] In a possible implementation, the face detector in S30 uses the nano architecture of the YOLOv8 model, and the specific working process includes:

[0022] S301, preprocess the current video stream image, scale the video stream image to a picture with a width of 640 pixels and a height of 384 pixels; convert the storage channel format of the picture from [height, width, color channel] to [batch, color channel, height, width]; normalize each pixel value and divide each pixel by 255;

[0023] S302, after the preprocessed image is sent to the model, the feature extraction is first performed through the backbone network to generate multiple feature maps of different scales. The above feature maps capture the information of the image at different levels; the above feature maps are sent to the neck network, and the neck network further processes and fuses the feature maps output by the backbone network. The feature maps processed by the neck network are sent to the prediction head, which is responsible for generating detection results based on the feature maps. The detection result is a matrix, in which each row represents a detected object, and each column represents an attribute of the object, in order: the horizontal coordinate of the center point of the bounding box, the vertical coordinate of the center point of the bounding box, the width of the bounding box, the height of the bounding box, the confidence of the yolo detection, and the probability of the face category;

[0024] S303, the neck network of yolov8 has feature map outputs of multiple sizes, each feature map is equipped with a prediction head, a prediction matrix is ​​generated, and the output matrices of all prediction heads are spliced ​​column by column to generate a new prediction matrix. Each row represents a detection result, and each column represents an attribute of the detection result, in order: the horizontal coordinate of the center point of the bounding box, the vertical coordinate of the center point of the bounding box, the width of the bounding box, the height of the bounding box, the confidence of the yolo detection, and the probability of the face category; the confidence of the yolo detection is multiplied by the probability of the face category to obtain the face confidence. The detection result contains the bounding box information and the face confidence information;

[0025] S304, filtering the detection results of the face detected by yolo in the obtained matrix whose confidence level is lower than a preset threshold of 0.5, and performing a non-maximum suppression operation on the filtered detection results to filter out overlapping bounding boxes;

[0026] S305, convert the coordinates of the bounding box of the final detection result from the normalized coordinates back to the coordinates of the original video stream image, multiply the width and height of the bounding box by the aspect ratio of the original image, and multiply the horizontal and vertical coordinates of the center point by the aspect ratio of the original image accordingly; output the detection result after the coordinate conversion.

[0027] In a possible implementation manner, the model used by the instance segmenter in S40 is SOLOv2, and processing the current image using the instance segmenter includes:

[0028] S401, pre-processing the current video stream image, scaling the video stream image to an image with a width of 640 pixels and a height of 360 pixels; converting the video into RGB format; and normalizing the video;

[0029] S402, inputting the preprocessed image into the model to obtain a feature map, inputting the preprocessed image into the backbone network for feature extraction, and then obtaining feature maps of different scales through the feature pyramid network, upsampling or downsampling the feature maps of different scales to match a unified resolution, and further performing feature fusion through a convolutional layer;

[0030] S403, dividing the obtained feature map into a grid of 9 rows and 16 columns, each grid is responsible for predicting targets within a certain range, and for each grid, a mask kernel and mask kernel parameters are learned from the feature map through a series of convolution operations;

[0031] S404, for each grid unit, convolve the obtained mask kernel with the mask feature to generate an instance mask; for each grid, the prediction head generates an instance confidence score for each grid, the score indicates whether the grid contains the target instance, and the instance mask and the instance confidence score together constitute the output of the grid;

[0032] S405, merging the outputs of all grids, and filtering the grid outputs with instance confidence scores less than 0.5;

[0033] S406, obtaining a mask feature by performing operations including fusion and upsampling on the feature map output by the feature pyramid network, wherein the mask feature includes information on the position and shape of the target;

[0034] S407, performing a convolution operation on all mask kernels and mask features to generate an instance mask of a grid output;

[0035] S408, filtering the output instance mask using matrix non-maximum suppression processing;

[0036] S409, merge the instances filtered by matrix non-maximum suppression processing and output them, and the output matrix shape is (n,h,w,2), where n represents the number of output instances, h and w are the width and height of the original image, and n instances output n binary mask images with the same width and height as the original image. Each mask image represents the information of an instance, and for each pixel therein, 0 represents the background and 1 represents the foreground of the current instance.

[0037] In a possible implementation manner, the model used by the semantic segmenter in S50 is a PID neural network, and the specific working process includes:

[0038] S501, preprocessing the current video stream image, scaling the video stream image to a picture with a width of 640 pixels and a height of 384 pixels; converting the video into RGB format; and normalizing the video;

[0039] S502, firstly passing the preprocessed image through the backbone network to obtain deep features, and capture local features and context information in the image;

[0040] S503, extract fusion features, send deep features to P branch, I branch, and D branch respectively, and obtain features of each branch, where P branch is responsible for parsing and retaining detailed information in high-resolution feature maps; I branch is responsible for aggregating local and global context information to parse long-term dependencies between pixels, using an encoder-decoder structure, where the encoder gradually reduces the spatial resolution to capture context, and the decoder gradually restores the resolution to locate details; D branch focuses on capturing high-frequency details, including edge and texture information, through a series of convolutional layers and dilated convolutions;

[0041] S504, post-processing to obtain a semantic segmentation map, which is a semantic segmentation map of the same size as the input image, in which each pixel is assigned a category label, and the feature map output by the boundary attention guided fusion BAG module is upsampled to match the resolution of the original input image; the upsampled feature map will be sent to the semantic head to obtain a semantic feature map, which includes 1x1 convolution and softmax operations; 1x1 convolution is used for channel number adjustment and feature fusion, and the softmax operation assigns a category probability to each pixel; for each pixel in the semantic feature map, select the category with the highest human probability, and when the semantic human category probability is greater than the threshold of 0.5, it is used as the human semantic output, and the others are all backgrounds, generating a semantic segmentation map; output the semantic segmentation map, and its output matrix shape is (1, h, w, 2), indicating that a 2-value matrix with the same width and height as the original image is generated, 0 represents the pixel as the background, and 1 represents the pixel as the foreground.

[0042] In a possible implementation manner, the model used by the depth estimator in S60 is DepthAnything, and the specific working process includes:

[0043] S601, preprocessing the current video stream image, scaling the video stream image to an image with a width of 448 pixels and a height of 224 pixels; converting the video into RGB format; and normalizing the video;

[0044] S602, sending the preprocessed image to the encoder of the model for encoding. The feature extraction encoder is the vits encoder of DINOv2, which will extract features from the preprocessed image;

[0045] S603, sending the obtained code to the DPT decoder for decoding, mapping the feature back to the depth map, and obtaining a depth feature map.

[0046] S603, upsampling the depth feature map to the same size as the original image to obtain a depth map, where the value of each pixel in the depth map represents the depth of the location, and the smaller the value, the farther the location is, and the larger the value, the closer the location is.

[0047] In a possible implementation manner, the foreground mask selection algorithm process in S70 specifically includes:

[0048] S7011, based on the area, width and height information, ignore the instances whose area, width and height are less than the preset threshold; S7012, count the number of instances containing ignore marks, if it exceeds the preset number, and call the post-processing module cache to query whether there is foreground information in the previous frame mask, if the previous frame mask has no foreground information output, then the frame outputs a blank mask without foreground information; S7013, match the face information with the instance information, and mark the instances with face information as containing faces; S7014, if there is no face information in the current frame, the frame does not output the mask; S7015, exclude the instances containing ignore marks, select the instances with face information, score them based on their face position information, area information, and aspect ratio information, and select the best face; the instance corresponding to the best face is the best instance, and the center of gravity depth of field of this instance is the best depth of field of the current frame; S7016, the best depth of field of the current frame is not When it is empty, read the optimal depth of field of the previous frame. If the optimal depth of field of the current frame and the optimal depth of field of the previous frame are greater than the dynamic threshold, the value of the dynamic threshold is 0.2*the optimal depth of field of the previous frame, then traverse all current instance depths, and select the one closest to the optimal depth of field of the previous frame as the optimal depth of field of the current frame; S7017, exclude instances containing ignore marks, traverse instances with face information, if their depth of field and the optimal depth of field of the current frame are less than the dynamic optimal threshold, the value of the dynamic threshold is 0.2*the optimal depth of field of the current frame, then the instance is considered to be an important instance of the current picture and is marked as needing to be output; S7018, create a blank mask image with the same width and height as the original image, the mask image is a binary image, 0 represents the background, and 1 represents the foreground; all instance masks that need to be output are superimposed on the mask image, that is, the value of the corresponding position of the position with 1 in the instance mask on the mask image is set to 1, and the foreground mask of the current frame is obtained.

[0049] In a possible implementation manner, the specific process of selecting the optimal face in S7015 includes: traversing all face results of the current frame, and calculating the position score, area score, and width-height score of each face respectively, the position score calculation method is: assuming that the coordinates of the center point of the face are cx, cy, the width of the current frame image is imgw, and the height is imgh, the position score calculation formula is score1=1ifabs(0.5*imgw-cx) / imgw<0.3else 1-abs(0.5*imgw-cx) / imgw; the area score calculation method is: traversing all faces, taking the maximum area as max_face_area, for each face, its area is face_area, and the area score calculation formula is score2=face_area / max_area; the width-height score calculation method is: assuming that the long side of the face is l, the short side is s, and the area score calculation formula is score3=s / l; the sum of the three scores of the position score, area score and width-height score of the face is the highest, which is the optimal face.

[0050] In a possible implementation, the timing mask compensation algorithm process in S70 includes: S7021, reading the output instance information of the previous frame, reading the instance information of the current frame, matching the two based on the center of gravity and the bounding box, if the current frame instance can match the output instance of the previous frame, then output the instance mask, and mark the matched previous frame instance as matched; S7022, for the unmatched previous frame output instance, take the intersection of it and the semantic segmentation and depth of field information results of the current frame, if the mask area after the intersection is larger than the previous frame instance area * a preset threshold, then take the intersection result as a new instance to be output; S7023, establish a blank mask image with the same width and height as the original image, superimpose all instance masks to be output on the mask image, and obtain a timing compensation mask.

[0051] In a possible implementation, the matching calculation method in S7021 is: first calculate the centers of the two instances respectively, the calculation method is that for the instance, take all the points that are foreground, calculate the average of x and y, that is, the center of gravity; if the center of gravity position is not classified as foreground in the instance, then all foreground points are arranged in order from small to large in the horizontal coordinate and from small to large in the vertical coordinate into a one-dimensional vector, and the middle point is taken as the center of gravity; calculate the Euclidean distance between the centers of gravity of the two instances; if the distance is less than 0.25 times the height of the previous frame instance and less than 0.25 times the width of the previous frame instance, the two instances are considered to match.

[0052] The present invention has the following beneficial effects:

[0053] (1) A new shot-cut detection method is designed, which has high sensitivity to general shot cuts and maintains high performance to avoid misjudgment of fast-moving images, laying the foundation for subsequent timing processing.

[0054] (2) A deep learning model is used to obtain instance segmentation information, depth of field information, and face information in the picture, determine the main target of the current frame from multiple dimensions, and generate a foreground mask, focusing only on the main target so that the main target is not blocked by the barrage. At the same time, the barrage will still cover the secondary target to ensure the barrage viewing experience.

[0055] (3) Use a deep learning model to obtain instance segmentation information, depth of field information, semantic segmentation information in the picture, and lens switching information provided by the lens detector, to ensure that the corresponding mask is continuously output from multiple dimensions after the main target is established. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 The present invention is a flowchart of the steps of a method for preventing foreground target barrage from being blocked based on artificial intelligence according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] See also Figure 1 , which is a flowchart of a method for preventing foreground target bullet screen from being blocked based on artificial intelligence according to an embodiment of the present invention, comprising the following steps:

[0059] S10, sampling the original video at a preset frequency to obtain a video stream image; wherein the original video is a video that needs to be protected from foreground human body bullet screen;

[0060] S20, using a shot switching detection algorithm to process the video stream to determine whether a shot switching occurs in the current video frame; if a shot switching occurs in the current frame, resetting the post-processing module cache; wherein the post-processing module cache includes target tracking information, depth information of the main target of the previous frame, and whether there is a mask output in the previous frame;

[0061] S30, using a face detector to process the current image to obtain target face information of the current image;

[0062] S40, using an instance segmenter to process the current image to obtain instance segmentation information of the current image target;

[0063] S50, using a semantic segmenter to process the current image to obtain semantic segmentation information of a human body target in the current image, wherein the human body semantic segmentation information is a one-dimensional feature map with the same width and height as the original image, and the value of each pixel represents the probability that the pixel is a human body;

[0064] S60, using a depth estimator to process the current image to obtain depth estimation information of the current image, wherein the depth estimation information is a one-dimensional feature map with the same width and height as the original image, and the value of each pixel represents the depth information of the object at the pixel;

[0065] S70, sending the current image target face information, the current image target instance segmentation information, the current image target human body semantic segmentation information and the current image depth estimation information to a post-processing module for processing, wherein the post-processing module includes a foreground mask selection algorithm and a timing mask compensation algorithm, wherein the foreground mask selection algorithm is used to screen out instances that need to be output in the current picture, construct mask information based on information of the current frame, and obtain a foreground mask of the current frame; the timing mask compensation algorithm is used to track instances that have been output in the previous frame, to prevent the lack of key position masks due to jitter of the model results, to improve the robustness of the overall algorithm, and to obtain a timing compensation mask;

[0066] S80, taking a union of the current frame foreground mask and the timing compensation mask to obtain a final output mask of the current frame, and outputting it to the outside as the current frame result;

[0067] S90, saving the instance information used by the foreground mask selection algorithm and the temporal mask compensation algorithm of the current frame for processing by the foreground mask selection algorithm and the temporal mask compensation algorithm of the next frame image;

[0068] S100, performing the above processing from S20 to S90 on the next frame of image until the video stream image processing is completed.

[0069] In the actual processing, the processes of S30, S40, S50, and S60 are not in order and can be processed simultaneously. The lens detector of S20 can not only have high sensitivity to general lens switching, but also maintain high performance for fast-moving images without misjudgment, laying the foundation for subsequent sequential processing. S40 uses the instance segmentation result for bullet screen mask processing, which is the basis for the mask to distinguish the main target and shield the secondary target. Based on the foreground mask selection algorithm in S30, S40, S60, and S80, by combining face information, depth of field information, and instance information, the main target can be effectively found from the current screen, and only the main target can be masked, so that users can watch the main actors and participate in the bullet screen interaction when watching again. Based on the temporal mask compensation algorithm in S40, S50, S60, and S70, by combining depth of field information, semantic information, and instance information, it ensures that the target mask has been tracked from the moment it appears from multiple dimensions, ensuring the continuity of the mask, preventing the frequent occurrence of bullet screens, and improving the user's bullet screen viewing experience.

[0070] In a specific application example, the use of the lens switching detection algorithm to process the video stream in S20 specifically includes: S201, scaling the image to a certain size; S202, making a difference with the retained image of the previous frame, counting the number of differences greater than a preset image change threshold, obtaining the number of pixel changes, and counting the number of differences less than a preset image unchanged threshold, obtaining the number of pixels unchanged, if the number of pixel changes is greater than the preset threshold and the number of pixels unchanged is less than the preset threshold, it is determined that a lens change occurs in the frame; S203, converting the image to the hue-saturation-brightness HSV color space, extracting the histogram information of the hue-saturation component, and comparing it with the histogram information of the hue-saturation component of the previous frame to obtain the hue At the same time, the image is converted into a grayscale color space, the grayscale histogram information is extracted, and compared with the grayscale histogram information of the previous frame to obtain the histogram correlation of the grayscale histogram. If the histogram correlation of the hue-saturation component is greater than a preset threshold and the histogram correlation of the grayscale histogram is less than the preset threshold, it is determined that a lens change occurs in the frame; S204, updating the cache information, saving the image after frame scaling, the histogram of the hue-saturation component, and the grayscale histogram information, which are used as a basis for judging the next frame; S205, outputting the lens switching information of the current video frame. When the output result is true, it indicates that a lens switch occurs in the current frame. When the output result is false, it indicates that a lens switch does not occur in the current frame.

[0071] In a specific application example, the face detector in S30 uses the nano architecture of the YOLOv8 model. YOLOv8 belongs to the YOLO series of algorithms. On the basis of inheriting the advantages of the YOLO series, it adopts an anchor-free design to improve the detection accuracy; introduces new loss functions, such as Distribution Focal Loss, to better balance positive and negative samples and improve the learning efficiency of the model during training; introduces new or improved feature fusion modules (such as BiFPN and AFPN), and uses technologies such as SPD-Conv to enhance the detection capabilities of small targets and multi-scale targets; adopts a variety of attention mechanisms to enhance the feature extraction capabilities of the model, and innovates on the convolution layer, such as using deformable convolution and dynamic convolution. The structure of YOLOv8 consists of three parts: (1) Backbone: used to extract image features, using C2f modules and SPPF modules, reducing image size and enhancing features through convolution layers. (2) Neck: located between the backbone and the head, it fuses features of different scales through SPPF, PAA and PAN modules to enhance feature expression. (3) Head network: It is the decision-making part of the model and is responsible for generating the final detection results. The feature map is generated based on the neck network to generate the detection results, including the location of the bounding box, the confidence and the category probability. The nano architecture indicates that the depth and width of the standard YOLOv8 model have been reduced to adapt to resource-constrained environments while maintaining the effectiveness and efficiency of the model. In the actual configuration, the depth factor is 0.33 and the width factor is 0.25. The number of layers of the nano architecture YOLOv8 model is 33% of the standard version, and the number of channels per layer is 25%. The model input is a video image, and the output is the location information of the face and the confidence of the face. Its specific working process includes:

[0072] S301, preprocess the current video stream image, scale the video stream image to a picture with a width of 640 pixels and a height of 384 pixels; convert the storage channel format of the picture from [height, width, color channel] to [batch, color channel, height, width]; normalize each pixel value, and the specific normalization operation is to divide each pixel by 255;

[0073] S302, after the preprocessed image is sent to the model, the feature extraction is first performed through the backbone network to generate multiple feature maps of different scales. The above feature maps capture the information of the image at different levels; the above feature maps are sent to the neck network, and the neck network further processes and fuses the feature maps output by the backbone network. The feature maps processed by the neck network are sent to the prediction head, which is responsible for generating detection results based on the feature maps. The detection result is a matrix, in which each row represents a detected object, and each column represents an attribute of the object, in order: the horizontal coordinate of the center point of the bounding box, the vertical coordinate of the center point of the bounding box, the width of the bounding box, the height of the bounding box, the confidence of the yolo detection, and the probability of the face category;

[0074] S303, the neck network of yolov8 has feature map outputs of multiple sizes, each feature map is equipped with a prediction head, a prediction matrix is ​​generated, and the output matrices of all prediction heads are spliced ​​column by column to generate a new prediction matrix. Each row represents a detection result, and each column represents an attribute of the detection result, in order: the horizontal coordinate of the center point of the bounding box, the vertical coordinate of the center point of the bounding box, the width of the bounding box, the height of the bounding box, the confidence of the yolo detection, and the probability of the face category; the confidence of the yolo detection is multiplied by the probability of the face category to obtain the face confidence. The detection result contains the bounding box information and the face confidence information;

[0075] S304, filtering the detection results of the face detected by yolo in the obtained matrix whose confidence level is lower than a preset threshold of 0.5, and performing a non-maximum suppression operation on the filtered detection results to filter out overlapping bounding boxes;

[0076] S305, convert the coordinates of the bounding box of the final detection result from the normalized coordinates back to the coordinates of the original video stream image, multiply the width and height of the bounding box by the aspect ratio of the original image, and multiply the horizontal and vertical coordinates of the center point by the aspect ratio of the original image accordingly; output the detection result after the coordinate conversion.

[0077] The specific process of non-maximum suppression is as follows: sort all detection results in descending order according to confidence to form a candidate list; select the detection result with the highest confidence, add it to the final detection result, and remove it from the candidate list; calculate the intersection over union (IoU) value of the bounding box of the detection result and the bounding box of all other detection results in the candidate list; if the IoU value exceeds the preset threshold of 0.5, remove the corresponding detection result from the candidate list; repeat the above process until all detection results are processed and there are no detection results in the candidate list.

[0078] The specific calculation method of the detection box intersection over union (IoU) is: the detection box intersection over union of bounding box A and bounding box B = (intersection area) / (area of ​​A + area of ​​B – intersection area). For a bounding box, the coordinates of its upper left corner are (x1, y1)(x1, y1), and the coordinates of its lower right corner are (x2, y2)(x2, y2). Its area A can be calculated by the following formula: A = (x2-x1+1) × (y2-y1+1). For two bounding boxes, their intersection area interAreainterArea can be calculated by the following steps: Calculate the intersection coordinates of the two bounding boxes on the x-axis and y-axis: xx1 = max(x11, x12), yy1 = max(y11, y12), xx2 = min(x21, x22), yy2 = min(y21, y22); Calculate the width and height of the intersection and ensure that they are non-negative: w = max(0, xx2-xx1+1) h = max(0, yy2-yy1+1); Calculate the intersection area: interArea = w × h. Among them, min() is the minimum operation and max() is the maximum operation. The calculation formula of the detection box intersection over union ratio is IoU = (interArea) / (Area1 + Area2 - interArea).

[0079] In a specific application example, the model used by the instance segmenter in S40 is SOLOv2. SOLOv2 performs fast and accurate instance segmentation tasks directly at the pixel level in the form of a fully convolutional network. Its main structure is: (1) Feature extraction: Use a pre-trained network such as ResNet to extract image features, and then process it through a feature pyramid network (FPN) to obtain a multi-scale fused feature map; (2) Dynamic instance segmentation: Mask kernel branch: Parallel to the semantic category branch, predict the convolution kernel weights, taking into account spatial location information, mask feature branch: Learn a unified high-resolution mask feature, which is achieved by fusing the feature maps of each layer of FPN; (3) Prediction head: Contains convolutional layers, processes the feature maps output by FPN, and outputs category and mask predictions (4) Matrix optimized non-maximum suppression NMS: Optimizes the non-maximum suppression process to improve segmentation efficiency and accuracy. During training, the classification loss and mask loss are trained at the same time to ensure that the model accurately predicts the target category and position. The model takes a video image as input and outputs the human instance information of the current image, including (1) semantic category: the model predicts an output for each grid cell in the image, indicating whether the grid cell contains the center of the human body; (2) instance mask: the model predicts an accurate pixel-level mask for each human instance, which can be distinguished even if the human bodies overlap; (4) segmentation instance confidence: the model predicts the probability of the category for the instance. Using the instance segmenter to process the current image includes:

[0080] S401, preprocess the current video stream image, scale the video stream image to a picture with a width of 640 pixels and a height of 360 pixels; convert the video into RGB format; normalize the video; the normalization parameter mean is [123.675, 116.28, 103.53], the standard deviation is [58.395, 57.12, 57.375], and the mean is first subtracted from each color channel and then divided by the standard deviation, that is, the red channel value is subtracted from 123.675 and then divided by 58.395, the green channel value is subtracted from 116.28 and then divided by 57.12, and the blue channel value is subtracted from 103.53 and then divided by 57.375;

[0081] S402, inputting the preprocessed image into the model to obtain a feature map, inputting the preprocessed image into the backbone network for feature extraction, and then obtaining feature maps of different scales through the feature pyramid network, upsampling or downsampling the feature maps of different scales to match a unified resolution, and further performing feature fusion through a convolutional layer;

[0082] S403, dividing the obtained feature map into a grid of 9 rows and 16 columns, each grid is responsible for predicting targets within a certain range, and for each grid, a mask kernel and mask kernel parameters are learned from the feature map through a series of convolution operations;

[0083] S404, for each grid unit, convolve the obtained mask kernel with the mask feature to generate an instance mask; for each grid, the prediction head generates an instance confidence score for each grid, the score indicates whether the grid contains the target instance, and the instance mask and the instance confidence score together constitute the output of the grid;

[0084] S405, merging the outputs of all grids, and filtering the grid outputs with instance confidence scores less than 0.5;

[0085] S406, obtaining a mask feature by performing operations including fusion and upsampling on the feature map output by the feature pyramid network, wherein the mask feature includes information on the position and shape of the target;

[0086] S407, performing a convolution operation on all mask kernels and mask features to generate an instance mask of a grid output;

[0087] S408, filtering the output instance mask using matrix non-maximum suppression processing;

[0088] S409, merge the instances filtered by matrix non-maximum suppression processing and output them, and the output matrix shape is (n,h,w,2), where n represents the number of output instances, h and w are the width and height of the original image, and n instances output n binary mask images with the same width and height as the original image. Each mask image represents the information of an instance, and for each pixel therein, 0 represents the background and 1 represents the foreground of the current instance.

[0089] The specific process of filtering the output instance mask using matrix non-maximum suppression processing in S408 includes:

[0090] (1) Sort the prediction results from high to low according to the confidence score, and take the top 50 predictions for subsequent operations, and discard the rest; if the original results are less than 50, take all of them.

[0091] (2) For all the prediction results obtained, calculate the intersection over union (IoU) between the masks of all the prediction results and generate an N×N IoU matrix. Mask intersection over union (IoU) is an indicator to measure the degree of overlap between two masks, which is defined as the intersection area of ​​the two masks divided by their union area. The specific calculation process is as follows: for mask a and mask b, flatten their H*W mask matrix into a vector of [1, H*W] to obtain mask vector A and mask vector B; calculate the dot product of A and B to obtain the intersection area of ​​the two; calculate the total number of activated pixels of the two masks minus their intersection to obtain the union area of ​​the two; the mask intersection over union of the mask pair = intersection area / union area.

[0092] In actual calculation, the matrix operations based on the above algorithm are performed in parallel on the GPU. The specific operation is to obtain N binary masks, each of which is of size H×W. Each mask can be flattened into a vector of H×W dimensions. Therefore, all masks can be represented as an N×(H×W) matrix M, where each row represents a mask. By calculating the self-dot product of the matrix M, an N×N intersection matrix I is obtained, where I ij Represents the intersection between mask i and mask j. By calculating the sum of the elements of each row of matrix M, we get an N-dimensional vector S, where S i Represents the total number of pixels of mask i. Construct an N×N diagonal matrix D, where the elements on the diagonal are the elements of S, that is, D ii =S i Next, we can calculate D+D T -I to get the union matrix U. The IOU matrix is ​​equal to the intersection matrix I divided by the union matrix U.

[0093] (3) For each prediction result, calculate a decay factor, which indicates the degree to which the mask is suppressed by other masks. Multiply the decay factor by the confidence score of the prediction result to update the confidence score of the prediction result. The calculation method is: for each prediction result, calculate its decay value with other prediction results, the calculation method is decay = 1-iou ij , take the minimum value of all attenuation values ​​as the attenuation factor.

[0094] (4) Sort the updated confidence scores of all prediction results from high to low, traverse the sorted confidence score list, and for each mask: if the confidence score of the prediction result is lower than the confidence threshold of 0.5, discard the prediction result; if the IoU between the prediction result and all other retained prediction results is greater than the overlap threshold of 0.7, discard the prediction result; if the prediction result is not discarded, add it to the list of retained prediction results. After the traversal is completed, output the list of retained prediction results.

[0095] In a specific application example, the semantic segmentation model used by the semantic segmentation in S50 is the PID neural network PIDNet. PIDNet is a real-time semantic segmentation network. It introduces an auxiliary differential branch to solve the overshoot problem that may occur in the traditional two-branch network when fusing high-resolution details and low-resolution context information. The architecture of PIDNet contains three complementary branches: the proportion (P) branch: responsible for parsing and saving the detail information in the high-resolution feature map; the integral (I) branch: responsible for aggregating context information locally and globally to resolve the long-distance dependencies between pixels; the differential (D) branch: responsible for extracting high-frequency features to predict and emphasize the edge areas in the image. PIDNet also uses a pixel attention guided fusion module (Pag) and a boundary attention guided fusion module (Bag). These modules use the edge features extracted from the D branch to guide the fusion of the P branch and the I branch, thereby enhancing the features of the edge and boundary areas while maintaining the detail information. The model input is a video image and the output is the semantic segmentation information of the human body. This information is a one-dimensional feature map with the same width and height as the original image. The value of each pixel represents whether the pixel is the background or the foreground. The specific working process includes:

[0096] S501, preprocess the current video stream image, scale the video stream image to a picture with a width of 640 pixels and a height of 384 pixels; convert the video into RGB format; normalize the video; the normalization parameter mean is [123.675, 116.28, 103.53], the standard deviation is [58.395, 57.12, 57.375], and the mean is first subtracted from each color channel and then divided by the standard deviation, that is, the red channel value is subtracted from 123.675 and then divided by 58.395, the green channel value is subtracted from 116.28 and then divided by 57.12, and the blue channel value is subtracted from 103.53 and then divided by 57.375;

[0097] S502, firstly passing the preprocessed image through the backbone network to obtain deep features, and capture local features and context information in the image;

[0098] S503, extract fusion features, send deep features to P branch, I branch, and D branch respectively, and obtain features of each branch, where P branch is responsible for parsing and retaining detailed information in high-resolution feature maps; I branch is responsible for aggregating local and global context information to parse long-term dependencies between pixels, using an encoder-decoder structure, where the encoder gradually reduces the spatial resolution to capture context, and the decoder gradually restores the resolution to locate details; D branch focuses on capturing high-frequency details, including edge and texture information, through a series of convolutional layers and dilated convolutions;

[0099] S504, post-processing to obtain a semantic segmentation map, which is a semantic segmentation map of the same size as the input image, in which each pixel is assigned a category label, and the feature map output by the boundary attention guided fusion BAG module is upsampled to match the resolution of the original input image; the upsampled feature map will be sent to the semantic head to obtain a semantic feature map, which includes 1x1 convolution and softmax operations; 1x1 convolution is used for channel number adjustment and feature fusion, and the softmax operation assigns a category probability to each pixel; for each pixel in the semantic feature map, select the category with the highest human probability, and when the semantic human category probability is greater than the threshold of 0.5, it is used as the human semantic output, and the others are all backgrounds, generating a semantic segmentation map; output the semantic segmentation map, and its output matrix shape is (1, h, w, 2), indicating that a 2-value matrix with the same width and height as the original image is generated, 0 represents the pixel as the background, and 1 represents the pixel as the foreground.

[0100] Further, in S503, the parallel aggregation pyramid pooling module PAPPM is located in the I branch, and the context information of different scales is quickly aggregated through the parallel structure to improve the computational efficiency. The input is the feature map of the I branch, and the output is the feature map after multi-scale context aggregation. Multi-scale features are extracted through pooling and convolution operations of different scales, and then fused through upsampling and element-level addition to construct a global scene prior. The process is as follows: the input feature map is firstly subjected to pooling operations of different scales to obtain context information of different scales; each feature map is subjected to ReLU activation function and 1x1 convolution operation to reduce the number of channels and further extract features; each pooled feature map is restored to the same size as the original input feature map by upsampling; the fused feature map is processed through a compressed convolution layer to reduce the number of channels and integrate information; the compressed fused feature map is added to the original input feature map to obtain the final output feature map.

[0101] The pixel attention guided fusion PAG module is located in the P branch. It selectively learns useful semantic features from the I branch through the attention mechanism to avoid being overwhelmed by the contextual information of the I branch. The input is the feature map of the P branch and the I branch, and the output is the fused feature map, which combines the detail information of the P branch and the contextual information of the I branch. The features are selectively fused by calculating the pixel attention weights between the proportional P branch and the I branch feature maps. The process is as follows: its input is the feature map of the two branches, and the output is the feature map that integrates the details and contextual information. The specific working process is as follows: the P branch and I branch features are input, and they are processed by 1x1 convolution layers respectively to reduce the number of channels and extract key features. The obtained feature maps are subjected to pixel-level dot product operations, and the attention weight σ is generated by the Sigmoid function, σ=Sigmoid(f p (v p )·f i (v i )), where f p f i Represents a combination of convolution, batch normalization, and ReLU activation functions; the feature map of the I branch is weighted using the calculated attention weights, and then element-wise added to the feature map of the P branch to achieve feature fusion. pag =σ·v i +(1-σ)·v p .

[0102] The boundary attention guided fusion BAG module is located in the fusion stage of the network. It combines the outputs of the P branch, I branch, and D branch, uses the boundary attention weight to guide the fusion of the P branch and the I branch, and strengthens the model's ability to recognize boundaries and small objects. The input is the feature map of the P branch, I branch, and D branch, and the output is the fused feature map, in which the contribution of the P branch and the I branch is dynamically adjusted according to the boundary features of the D branch. The process is: guide the fusion of the proportional P branch and the I branch feature map by calculating the Sigmoid output of the D branch feature map. Its input is the feature map of the three branches, and the output is a feature map that integrates details and context information. The output of the D branch v d To calculate the boundary attention weight σ, the Sigmoid function is used to implement σ = Sigmoid (v d ). Dynamically fuse the features of the P branch and the I branch. The fusion operation is as follows: f out represents the combination of convolution, batch normalization, and ReLU activation functions, Represents element-wise multiplication.

[0103] In a specific application example, the model used by the depth estimator in S60 is Depth Anything. DepthAnything is an advanced monocular depth estimation model. The Depth Anything model uses VisionTransformer (ViT) as the backbone network. ViT is a deep learning model based on the Transformer architecture, which divides the image into a series of patches and converts each patch into a vector representation as an input sequence. These vectors are processed by a multi-layer Transformer encoder, which includes a self-attention mechanism and a feedforward neural network layer to capture contextual dependencies in the image. In ViT, the input image is first divided into non-overlapping patches of fixed size, and then these patches are linearly mapped to a new dimensional space to form a serialized input. In addition, a special category marker is added to the beginning of the sequence, and a position embedding is added to each patch vector to preserve the spatial information in the original image. ViT processes the input sequence through a self-attention mechanism and a feedforward network layer to extract the features of the image. After feature extraction, the model fuses feature maps of different scales through a feature fusion module to retain more details and context information. Finally, the model outputs the final depth map through the depth prediction module. The model inputs the video image and outputs the depth map, predicting the depth value of each pixel in the input image. The specific working process includes:

[0104] S601, preprocess the current video stream image, scale the video stream image to an image with a width of 448 pixels and a height of 224 pixels; convert the video into RGB format; normalize the video; the normalization parameter mean is [123.675, 116.28, 103.53], the standard deviation is [58.395, 57.12, 57.375], and the mean is first subtracted from each color channel and then divided by the standard deviation, that is, the red channel value is subtracted from 123.675 and then divided by 58.395, the green channel value is subtracted from 116.28 and then divided by 57.12, and the blue channel value is subtracted from 103.53 and then divided by 57.375;

[0105] S602, sending the preprocessed image to the encoder of the model for encoding. The feature extraction encoder is the vits encoder of DINOv2, which will extract features from the preprocessed image;

[0106] S603, sending the obtained code to the DPT decoder for decoding, mapping the feature back to the depth map, and obtaining a depth feature map.

[0107] S603, upsampling the depth feature map to the same size as the original image to obtain a depth map, where the value of each pixel in the depth map represents the depth of the location, and the smaller the value, the farther the location is, and the larger the value, the closer the location is.

[0108] In a specific application example, the foreground mask selection algorithm process in S70 specifically includes:

[0109] S7011, based on the area, width and height information, ignore the instances whose area, width and height are less than the preset threshold. In practice, the area of ​​the instance should be greater than 0.01 times the area of ​​the video stream image, the height should be greater than 0.25 times the height of the video stream image, and the width should be greater than 0.1 times the width of the video stream image. The parameters here can be modified according to actual needs; S7012, count the number of instances containing ignore marks. If it exceeds the preset number, and the post-processing module cache is called to query whether there is foreground information in the previous frame mask, if the previous frame mask has no foreground information output, then the frame outputs a blank mask without foreground information; S7013, match the face information with the instance information, and mark the instances with face information as containing faces; S7014, if there is no face information in the current frame, the frame does not output the mask; S7015, exclude the instances containing ignore marks, select the instances with face information, score them based on their face position information, area information, and aspect ratio information, and select the best face; the best face corresponds to the instance The optimal instance is the instance with the center depth of field of the instance as the optimal depth of field of the current frame; S7016, when the optimal depth of field of the current frame is not empty, read the optimal depth of field of the previous frame. If the optimal depth of field of the current frame and the optimal depth of field of the previous frame are greater than the dynamic threshold, the value of the dynamic threshold is 0.2*the optimal depth of field of the previous frame, then traverse all the current instance depths of field, and select the one closest to the optimal depth of field of the previous frame as the optimal depth of field of the current frame; S7017, exclude the instance containing the ignore mark, traverse the instance with face information, and if its depth If the depth and the optimal depth of field of the current frame are less than the dynamic optimal threshold, and the value of the dynamic threshold is 0.2*the optimal depth of field of the current frame, the instance is considered to be an important instance of the current picture and is marked as needing to be output; S7018, establish a blank mask image with the same width and height as the original image, and the mask image is a binary image, 0 represents the background, and 1 represents the foreground; superimpose all instance masks that need to be output on the mask image, that is, set the value of the corresponding position of the position with a value of 1 in the instance mask on the mask image to 1, and obtain the foreground mask of the current frame.

[0110] Furthermore, the specific process of selecting the optimal face in S7015 includes: traversing all face results of the current frame, and calculating the position score, area score, and width-height score of each face respectively. The position score is calculated as follows: assuming that the coordinates of the center point of the face are cx, cy, the width of the current frame image is imgw, and the height is imgh. The position score calculation formula is score1=1ifabs(0.5*imgw-cx) / imgw<0.3else 1-abs(0.5*imgw-cx) / imgw; the area score calculation method is: traversing all faces, taking the maximum area as max_face_area, for each face, its area is face_area, and the area score calculation formula is score2=face_area / max_area; the width-height score calculation method is: assuming that the long side of the face is l, and the short side is s, and the area score calculation formula is score3=s / l; the sum of the three scores of the position score, area score and width-height score of the face is the highest, and the optimal face is.

[0111] In a specific embodiment, the timing mask compensation algorithm process in S70 includes: S7021, reading the output instance information of the previous frame, reading the instance information of the current frame, matching the two based on the center of gravity and the bounding box, if the current frame instance can match the output instance of the previous frame, then output the instance mask, and mark the matched previous frame instance as matched; S7022, for the unmatched previous frame output instance, take the intersection of it and the semantic segmentation and depth of field information results of the current frame, if the mask area after the intersection is larger than the previous frame instance area * the preset threshold, then take the intersection result as a new instance to be output; S7023, establish a blank mask image with the same width and height as the original image, superimpose all instance masks to be output on the mask image, and obtain a timing compensation mask.

[0112] Furthermore, the matching calculation method in S7021 is as follows: first, the centers of the two instances are calculated respectively. For each instance, all the points that are foreground are taken, and the average of x and y is calculated, which is the center of gravity. If the center of gravity position is not classified as foreground in the instance, all the foreground points are arranged into a one-dimensional vector in the order of the horizontal coordinate from small to large and the vertical coordinate from small to large, and the middle point is taken as the center of gravity; the Euclidean distance between the centers of gravity of the two instances is calculated. If the distance is less than 0.25 times the height of the previous frame instance and less than 0.25 times the width of the previous frame instance, the two instances are considered to match.

[0113] The above-set artificial intelligence-based foreground target barrage anti-occlusion method designs a new lens switching detection method, which is highly sensitive to general lens switching and maintains high performance for fast-moving images without misjudgment, laying the foundation for subsequent timing processing. By using a deep learning model to obtain instance segmentation information, depth of field information, and face information in the picture, the main target of the current frame is determined from multiple dimensions, and a foreground mask is generated, so that only the main target can be focused on, so that the main target is not obscured by the barrage. At the same time, the barrage will still cover the secondary target to ensure the barrage viewing experience. The deep learning model is used to obtain instance segmentation information, depth of field information, semantic segmentation information in the picture, and lens switching information provided by the lens detector, ensuring from multiple dimensions that when the main target is established, the corresponding mask is continuously output.

[0114] It should be understood that the exemplary embodiments described herein are illustrative rather than restrictive. Although one or more embodiments of the present invention have been described in conjunction with the accompanying drawings, it should be understood by those skilled in the art that various changes in form and detail may be made without departing from the spirit and scope of the present invention as defined by the appended claims.

Claims

1. A method for preventing foreground target barrage from being blocked based on artificial intelligence, characterized in that: The following steps are involved: S10, sampling the original video at a preset frequency to obtain a video stream image; The original video is a video that needs to be protected from foreground human body bullet screen; S20, using a shot switching detection algorithm to process the video stream to determine whether a shot switching occurs in the current video frame; If a shot switch occurs in the current frame, a post-processing module cache is reset; the post-processing module cache includes target tracking information, depth information of the main target in the previous frame, and whether there is a mask output in the previous frame; S30, using a face detector to process the current image to obtain target face information of the current image; S40, using an instance segmenter to process the current image to obtain instance segmentation information of the current image target; S50, using a semantic segmenter to process the current image to obtain semantic segmentation information of a human body target in the current image, wherein the human body semantic segmentation information is a one-dimensional feature map with the same width and height as the original image, and the value of each pixel represents the probability that the pixel is a human body; S60, using a depth estimator to process the current image to obtain depth estimation information of the current image, where the depth estimation information is a one-dimensional feature map with the same width and height as the original image, and the value of each pixel represents the depth information of the object at the pixel; S70, sending the current image target face information, the current image target instance segmentation information, the current image target human body semantic segmentation information and the current image depth estimation information to a post-processing module for processing, wherein the post-processing module includes a foreground mask selection algorithm and a temporal mask compensation algorithm, wherein the foreground mask selection algorithm is used to screen out the instances to be output in the current picture, construct mask information based on the information of the current frame, and obtain the foreground mask of the current frame; and the temporal mask compensation algorithm is used to track the instances that have been output in the previous frame, and obtain the temporal compensation mask; S80, taking a union of the current frame foreground mask and the timing compensation mask to obtain a final output mask of the current frame, and outputting it to the outside as the current frame result; S90, saving the instance information used by the foreground mask selection algorithm and the temporal mask compensation algorithm of the current frame for processing by the foreground mask selection algorithm and the temporal mask compensation algorithm of the next frame image; S100, performing the above processing from S20 to S90 on the next frame of image until the video stream image processing is completed.

2. The method for preventing foreground target barrage from being blocked based on artificial intelligence according to claim 1, characterized in that: The use of the lens switching detection algorithm to process the video stream described in S20 specifically includes: S201, scaling the image to a certain size; S202, making a difference with the retained image of the previous frame, counting the number of differences greater than a preset image change threshold, obtaining the number of pixel changes, and counting the number of differences less than a preset image unchanged threshold, obtaining the number of pixels unchanged; if the number of pixel changes is greater than the preset threshold and the number of pixels unchanged is less than the preset threshold, it is determined that a lens change has occurred in the frame; S203, converting the image to the hue-saturation-brightness HSV color space, extracting the histogram information of the hue-saturation component, and comparing it with the histogram information of the hue-saturation component of the previous frame to obtain the hue-saturation component. At the same time, the image is converted into a grayscale color space, the grayscale histogram information is extracted, and compared with the grayscale histogram information of the previous frame to obtain the histogram correlation of the grayscale histogram. If the histogram correlation of the hue-saturation component is greater than a preset threshold and the histogram correlation of the grayscale histogram is less than the preset threshold, it is determined that a lens change occurs in the frame; S204, updating the cache information, saving the image after frame scaling, the histogram of the hue-saturation component, and the grayscale histogram information, which are used as a basis for judging the next frame; S205, outputting the lens switching information of the current video frame. When the output result is true, it indicates that a lens switch occurs in the current frame. When the output result is false, it indicates that a lens switch does not occur in the current frame.

3. The method for preventing foreground target barrage from being blocked based on artificial intelligence according to claim 1, characterized in that: The face detector in S30 uses the nano architecture of the YOLOv8 model. The specific working process includes: S301, preprocess the current video stream image, scale the video stream image to a picture with a width of 640 pixels and a height of 384 pixels; convert the storage channel format of the picture from [height, width, color channel] to [batch, color channel, height, width]; normalize each pixel value and divide each pixel by 255; S302, after the preprocessed image is sent to the model, the feature extraction is first performed through the backbone network to generate multiple feature maps of different scales. The above feature maps capture the information of the image at different levels; the above feature maps are sent to the neck network, and the neck network further processes and fuses the feature maps output by the backbone network. The feature maps processed by the neck network are sent to the prediction head, which is responsible for generating detection results based on the feature maps. The detection result is a matrix, in which each row represents a detected object, and each column represents an attribute of the object, in order: the horizontal coordinate of the center point of the bounding box, the vertical coordinate of the center point of the bounding box, the width of the bounding box, the height of the bounding box, the confidence of the yolo detection, and the probability of the face category; S303, the neck network of yolov8 has feature map outputs of multiple sizes, each feature map is equipped with a prediction head, a prediction matrix is ​​generated, and the output matrices of all prediction heads are spliced ​​column by column to generate a new prediction matrix. Each row represents a detection result, and each column represents an attribute of the detection result, in order: the horizontal coordinate of the center point of the bounding box, the vertical coordinate of the center point of the bounding box, the width of the bounding box, the height of the bounding box, the confidence of the yolo detection, and the probability of the face category; the confidence of the yolo detection is multiplied by the probability of the face category to obtain the face confidence. The detection result contains the bounding box information and the face confidence information; S304, filtering the detection results of the face detected by yolo in the obtained matrix whose confidence level is lower than a preset threshold of 0.5, and performing a non-maximum suppression operation on the filtered detection results to filter out overlapping bounding boxes; S305, convert the coordinates of the bounding box of the final detection result from the normalized coordinates back to the coordinates of the original video stream image, multiply the width and height of the bounding box by the aspect ratio of the original image, and multiply the horizontal and vertical coordinates of the center point by the aspect ratio of the original image accordingly; output the detection result after the coordinate conversion.

4. The method for preventing foreground target barrage from being blocked based on artificial intelligence according to claim 1, characterized in that: The model used by the instance segmenter in S40 is SOLOv2, and processing the current image using the instance segmenter includes: S401, pre-processing the current video stream image, scaling the video stream image to an image with a width of 640 pixels and a height of 360 pixels; converting the video into RGB format; and normalizing the video; S402, inputting the preprocessed image into the model to obtain a feature map, inputting the preprocessed image into the backbone network for feature extraction, and then obtaining feature maps of different scales through the feature pyramid network, upsampling or downsampling the feature maps of different scales to match a unified resolution, and further performing feature fusion through a convolutional layer; S403, dividing the obtained feature map into a grid of 9 rows and 16 columns, each grid is responsible for predicting targets within a certain range, and for each grid, a mask kernel and mask kernel parameters are learned from the feature map through a series of convolution operations; S404, for each grid unit, convolve the obtained mask kernel with the mask feature to generate an instance mask; for each grid, the prediction head generates an instance confidence score for each grid, the score indicates whether the grid contains the target instance, and the instance mask and the instance confidence score together constitute the output of the grid; S405, merging the outputs of all grids, and filtering the grid outputs with instance confidence scores less than 0.5; S406, obtaining a mask feature by performing operations including fusion and upsampling on the feature map output by the feature pyramid network, wherein the mask feature includes information on the position and shape of the target; S407, performing a convolution operation on all mask kernels and mask features to generate an instance mask of a grid output; S408, filtering the output instance mask using matrix non-maximum suppression processing; S409, merge the instances filtered by matrix non-maximum suppression processing and output them, and the output matrix shape is (n,h,w,2), where n represents the number of output instances, h and w are the width and height of the original image, and n instances output n binary mask images with the same width and height as the original image. Each mask image represents the information of an instance, and for each pixel therein, 0 represents the background and 1 represents the foreground of the current instance.

5. The method for preventing foreground target barrage from being blocked based on artificial intelligence according to claim 1, characterized in that: The model used by the semantic segmentation in S50 is a PID neural network, and the specific working process includes: S501, preprocessing the current video stream image, scaling the video stream image to a picture with a width of 640 pixels and a height of 384 pixels; converting the video into RGB format; and normalizing the video; S502, firstly passing the preprocessed image through the backbone network to obtain deep features, and capture local features and context information in the image; S503, extract fusion features, send deep features to P branch, I branch, and D branch respectively, and obtain features of each branch, where P branch is responsible for parsing and retaining detailed information in high-resolution feature maps; I branch is responsible for aggregating local and global context information to parse long-term dependencies between pixels, using an encoder-decoder structure, where the encoder gradually reduces the spatial resolution to capture context, and the decoder gradually restores the resolution to locate details; D branch focuses on capturing high-frequency details, including edge and texture information, through a series of convolutional layers and dilated convolutions; S504, post-processing to obtain a semantic segmentation map, which is a semantic segmentation map of the same size as the input image, in which each pixel is assigned a category label, and the feature map output by the boundary attention guided fusion BAG module is upsampled to match the resolution of the original input image; the upsampled feature map will be sent to the semantic head to obtain a semantic feature map, which includes 1x1 convolution and softmax operations; 1x1 convolution is used for channel number adjustment and feature fusion, and the softmax operation assigns a category probability to each pixel; for each pixel in the semantic feature map, select the category with the highest human probability, and when the semantic human category probability is greater than the threshold of 0.5, it is used as the human semantic output, and the others are all backgrounds, generating a semantic segmentation map; output the semantic segmentation map, and its output matrix shape is (1, h, w, 2), indicating that a 2-value matrix with the same width and height as the original image is generated, 0 represents the pixel as the background, and 1 represents the pixel as the foreground.

6. The method for preventing foreground target barrage from being blocked based on artificial intelligence according to claim 1, characterized in that: The model used by the depth estimator in S60 is Depth Anything, and the specific working process includes: S601, preprocessing the current video stream image, scaling the video stream image to an image with a width of 448 pixels and a height of 224 pixels; converting the video into RGB format; and normalizing the video; S602, sending the preprocessed image to the encoder of the model for encoding. The feature extraction encoder is the vits encoder of DINOv2, which will extract features from the preprocessed image; S603, sending the obtained code to the DPT decoder for decoding, mapping the feature back to the depth map, and obtaining a depth feature map. S603, upsampling the depth feature map to the same size as the original image to obtain a depth map, where the value of each pixel in the depth map represents the depth of the location, and the smaller the value, the farther the location is, and the larger the value, the closer the location is.

7. The method for preventing foreground target barrage from being blocked based on artificial intelligence according to claim 1, characterized in that: The foreground mask selection algorithm process in S70 specifically includes: S7011, based on the area, width and height information, ignore the instances whose area, width and height are less than the preset threshold; S7012, count the number of instances containing ignore marks, if it exceeds the preset number, and call the post-processing module cache to query whether there is foreground information in the previous frame mask, if the previous frame mask has no foreground information output, then the frame outputs a blank mask without foreground information; S7013, match the face information with the instance information, and mark the instances with face information as containing faces; S7014, if there is no face information in the current frame, the frame does not output the mask; S7015, exclude the instances containing ignore marks, select the instances with face information, score them based on their face position information, area information, and aspect ratio information, and select the best face; the instance corresponding to the best face is the best instance, and the center of gravity depth of field of this instance is the best depth of field of the current frame; S7016, the best depth of field of the current frame is not When it is empty, read the optimal depth of field of the previous frame. If the optimal depth of field of the current frame and the optimal depth of field of the previous frame are greater than the dynamic threshold, the value of the dynamic threshold is 0.2*the optimal depth of field of the previous frame, then traverse all current instance depths, and select the one closest to the optimal depth of field of the previous frame as the optimal depth of field of the current frame; S7017, exclude instances containing ignore marks, traverse instances with face information, if their depth of field and the optimal depth of field of the current frame are less than the dynamic optimal threshold, the value of the dynamic threshold is 0.2*the optimal depth of field of the current frame, then the instance is considered to be an important instance of the current picture and is marked as needing to be output; S7018, create a blank mask image with the same width and height as the original image, the mask image is a binary image, 0 represents the background, and 1 represents the foreground; all instance masks that need to be output are superimposed on the mask image, that is, the value of the corresponding position of the position with 1 in the instance mask on the mask image is set to 1, and the foreground mask of the current frame is obtained.

8. The method for preventing foreground target barrage from being blocked based on artificial intelligence according to claim 7, characterized in that: The specific process of selecting the optimal face in S7015 includes: traversing all face results of the current frame, and calculating the position score, area score, and width-height score of each face respectively. The position score is calculated as follows: assuming that the coordinates of the center point of the face are cx, cy, the width of the current frame image is imgw, and the height is imgh. The position score calculation formula is score1=1ifabs(0.5*imgw-cx) / imgw<0.3else 1-abs(0.5*imgw-cx) / imgw; the area score calculation method is: traversing all faces, taking the maximum area as max_face_area, and the area of ​​each face is face_area, and the area score calculation formula is score2=face_area / max_area; the width-height score calculation method is: assuming that the long side of the face is l, and the short side is s, and the area score calculation formula is score3=s / l; the sum of the three scores of the position score, area score and width-height score of the face is the highest, and the optimal face is obtained.

9. The method for preventing foreground target barrage from being blocked based on artificial intelligence according to claim 1, characterized in that: The temporal mask compensation algorithm process in S70 includes: S7021, reading the output instance information of the previous frame, reading the instance information of the current frame, matching the two based on the center of gravity and the bounding box, if the current frame instance can match the output instance of the previous frame, then output the instance mask, and mark the matched previous frame instance as matched; S7022, for the unmatched previous frame output instance, take the intersection of it and the semantic segmentation and depth of field information results of the current frame, if the mask area after the intersection is larger than the area of ​​the previous frame instance * the preset threshold, then take the intersection result as a new instance to be output; S7023, establish a blank mask image with the same width and height as the original image, superimpose all instance masks to be output on the mask image, and obtain a temporal compensation mask.

10. The artificial intelligence-based foreground target barrage anti-blocking method according to claim 9, characterized in that: The matching calculation method in S7021 is as follows: first calculate the centers of the two instances respectively. The calculation method is that for the instance, take all the points that are foregrounds, calculate the average of x and y, which is the center of gravity. If the center of gravity position is not classified as foreground in the instance, arrange all the foreground points in the order of the horizontal coordinate from small to large and the vertical coordinate from small to large into a one-dimensional vector, and take the middle point as the center of gravity; calculate the Euclidean distance between the centers of gravity of the two instances; if the distance is less than 0.25 times the height of the previous frame instance and less than 0.25 times the width of the previous frame instance, the two instances are considered to match.

Images