Prospective target barrage anti-occlusion method based on artificial intelligence

Abstract

The application discloses a foreground target barrage anti-shielding method based on artificial intelligence, and comprises the following steps: S10, sampling original video at a preset frequency to obtain video stream images; S20, processing the video stream by using a shot change detection algorithm to determine whether the current video frame has shot change; S30, processing the current image by using a face detector to obtain current image target face information; S40, processing the current image by using an instance segmenter to obtain current image target instance segmentation information; S50, processing the current image by using a semantic segmenter to obtain current image target human body semantic segmentation information, wherein the human body semantic segmentation information is a one-dimensional feature map with the same width and height as the original image, and the value on each pixel represents the probability that the pixel is a human body; S60, processing the current image by using a depth estimator to obtain current image depth estimation information; and S70, sending the above information to a post-processing module for processing.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to an artificial intelligence-based method for preventing foreground target bullet screen occlusion. Background Technology

[0002] Bullet comments, or danmaku, originated on Japanese video websites. They allow users to post scrolling subtitles and comments while videos are playing, which then appear as bullet comments on the screen. This technology was quickly introduced to China, where numerous video platforms further propelled the development of bullet comment culture. Bullet comments not only enable real-time interaction between viewers and the content but also foster secondary creation by viewers on screen, becoming an indispensable part of contemporary film and television culture.

[0003] However, bullet screen technology also has some drawbacks. For example, the large number of bullet screens may affect the user's viewing experience, especially when there are too many bullet screens, which may cover the screen and affect the viewing of the video content.

[0004] The excessive appearance of bullet comments can obscure the screen, leading to a decline in the user's viewing experience. The invention of anti-obscuring bullet comment technology aims to solve this problem during video viewing, especially when bullet comments obscure important visual content such as faces, specific targets, or key information, thus affecting the user's viewing experience. By developing intelligent anti-obscuring bullet comment technology, bullet comments can accurately "bypass" the core areas of the video, achieving the dual advantages of immersive viewing and bullet comment interaction, thereby improving user satisfaction and engagement.

[0005] Anti-obstruction bullet screen technology is an application of masking technology. Masking technology refers to creating a transparent "mask" that can cover a specific area of ​​the video screen. In this way, bullet screen comments can be controlled to only be displayed in the area outside the mask, thus avoiding obscuring important video content.

[0006] Existing anti-obstruction bullet screen technologies often have the following problems: (1) In the process of automatic detection and mask generation, misidentification may occur, resulting in areas that should not be obscured being obscured incorrectly, or areas that should be obscured not being properly processed; (2) There is no intelligent judgment for detected targets, and many targets that do not need to be processed are also obscured, resulting in a worse bullet screen viewing experience; (3) The automatic detection and mask generation technology based on a single frame lacks sequential semantic continuity, and in the case of incomplete detection, it is easy to cause the bullet screen to flicker. Summary of the Invention

[0007] In view of the above-mentioned problems, the present invention provides an artificial intelligence-based method for preventing occlusion of foreground target bullet comments. By combining depth information, semantic information, and instance information, the method ensures that the target mask is continuously tracked from the moment it appears, thus guaranteeing the continuity of the mask and preventing the frequent occurrence of bullet comments that appear and disappear, thereby improving the user's bullet comment viewing experience.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] An artificial intelligence-based method for preventing foreground target occlusion in bullet comments includes the following steps:

[0010] S10, sample the original video at a preset frequency to obtain a video stream image; the original video is the video that needs to be protected against foreground human body barrage occlusion.

[0011] S20, the video stream is processed using a camera transition detection algorithm to determine whether a camera transition has occurred in the current video frame; if a camera transition has occurred in the current frame, the post-processing module cache is reset; the post-processing module cache includes target tracking information, the depth information of the main target in the previous frame, and whether there was a mask output in the previous frame;

[0012] S30, use a face detector to process the current image to obtain the target face information of the current image;

[0013] S40, use the instance segmenter to process the current image to obtain the target instance segmentation information of the current image;

[0014] S50, use a semantic segmenter to process the current image to obtain the semantic segmentation information of the target human body in the current image. The semantic segmentation information of the human body is a one-dimensional feature map with the same width and height as the original image. The value of each pixel represents the probability that the pixel is a human body.

[0015] S60, use a depth estimator to process the current image to obtain the current image depth estimation information. The depth estimation information is a one-dimensional feature map with the same width and height as the original image. The value of each pixel represents the depth information of the object in that pixel.

[0016] S70, the current image target face information, current image target instance segmentation information, current image target human semantic segmentation information, and current image depth estimation information are sent to the post-processing module for processing. The post-processing module includes a foreground mask selection algorithm and a temporal mask compensation algorithm. The foreground mask selection algorithm is used to filter out the instances that need to be output in the current frame, construct mask information based on the information of the current frame, and obtain the foreground mask of the current frame. The temporal mask compensation algorithm is used to track the instances that have been output in the previous frame and obtain the temporal compensation mask.

[0017] S80: Take the union of the foreground mask and the timing compensation mask of the current frame to obtain the final output mask of the current frame, and output it to the outside as the result of the current frame.

[0018] S90, save the instance information used by the foreground mask selection algorithm and the temporal mask compensation algorithm in the current frame, and use it for the foreground mask selection algorithm and the temporal mask compensation algorithm processing of the next frame image;

[0019] S100, the next frame image is processed as described in S20 to S90 above, until the video stream image processing is completed.

[0020] In one possible implementation, the processing of the video stream using the lens switching detection algorithm in S20 specifically includes: S201, scaling the image to a certain size; S202, calculating the difference between the image and the previous frame, counting the number of pixels whose difference is greater than a preset image change threshold to obtain the number of changed pixels, and counting the number of pixels whose difference is less than a preset image unchanged threshold to obtain the number of unchanged pixels. If the number of changed pixels is greater than the preset threshold and the number of unchanged pixels is less than the preset threshold, then it is determined that a lens switch has occurred in this frame; S203, converting the image to the hue / saturation / luminance HSV color space, extracting the histogram information of the hue / saturation components, comparing it with the histogram information of the hue / saturation components of the previous frame to obtain the histogram correlation of the hue / saturation components, and simultaneously converting the image to the grayscale color space to extract the grayscale histogram... The histogram information is compared with the grayscale histogram information of the previous frame to obtain the histogram correlation of the grayscale histogram. If the histogram correlation of the hue saturation component is greater than a preset threshold and the histogram correlation of the grayscale histogram is less than a preset threshold, it is determined that a shot change has occurred in the frame. S204, update the cache information and save the scaled image of the current frame, the histogram of the hue saturation component, and the grayscale histogram information as the basis for judging the next frame. S205, output the shot change information of the current video frame. When the output result is true, it indicates that a shot change has occurred in the current frame. When the output result is false, it indicates that a shot change has not occurred in the current frame.

[0021] In one possible implementation, the face detector in S30 uses a nano architecture based on the YOLOv8 model, and its specific operation includes:

[0022] S301, preprocess the current video stream image, scale the video stream image to an image with a width of 640 pixels and a height of 384 pixels; convert the image storage channel format from [height, width, color channel] to [batch, color channel, height, width]; normalize each pixel value, and divide each pixel by 255;

[0023] S302: After the preprocessed image is fed into the model, it first performs feature extraction through the backbone network, generating multiple feature maps at different scales. These feature maps capture information from different levels of the image. The feature maps are then fed into the neck network, which further processes and fuses the feature maps output by the backbone network. The feature maps processed by the neck network are then fed into the prediction head, which is responsible for generating detection results based on the feature maps. The detection results are a matrix, where each row represents a detected object and each column represents an attribute of that object, in the following order: x-coordinate of the bounding box center point, y-coordinate of the bounding box center point, width of the bounding box, height of the bounding box, confidence score of YOLO detection, and probability of face category.

[0024] The S303 and YOLOv8 neck network outputs feature maps of multiple sizes. Each feature map is paired with a prediction head, generating a prediction matrix. The output matrices of all prediction heads are concatenated column-wise to generate a new prediction matrix. Each row represents a detection result, and each column represents an attribute of that detection result, in the following order: x-coordinate of the bounding box center point, y-coordinate of the bounding box center point, width of the bounding box, height of the bounding box, YOLO detection confidence score, and probability of face class. The face confidence score is obtained by multiplying the YOLO detection confidence score by the probability of face class. The detection result includes bounding box information and face confidence score information.

[0025] S304, filter out the face detection results in the obtained matrix whose face confidence is lower than the preset threshold of 0.5 by YOLO detection, and perform non-maximum suppression operation on the filtered detection results to filter out overlapping bounding boxes;

[0026] S305 transforms the coordinates of the bounding box of the final detection result from normalized coordinates back to the coordinates of the original video stream image, multiplies the width and height of the bounding box by the aspect ratio of the original image, and also multiplies the horizontal and vertical coordinates of the center point by the aspect ratio of the original image accordingly; and outputs the detection result after coordinate transformation.

[0027] In one possible implementation, the instance segmenter in S40 uses the SOLOv2 model, and processing the current image using the instance segmenter includes:

[0028] S401, preprocess the current video stream image, scale the video stream image to a width of 640 pixels and a height of 360 pixels; convert the video to RGB format; normalize the video;

[0029] S402 inputs the preprocessed image into the model to obtain feature maps, inputs the preprocessed image into the backbone network for feature extraction, then obtains feature maps of different scales through the feature pyramid network, performs upsampling or downsampling operations on the feature maps of different scales to match a uniform resolution, and performs further feature fusion through convolutional layers.

[0030] S403 divides the obtained feature map into a grid of 9 rows and 16 columns. Each grid is responsible for predicting targets within a certain range. For each grid, a mask kernel and mask kernel parameters are learned from the feature map through a series of convolution operations.

[0031] S404: For each grid cell, the obtained mask kernel and mask features are convolved to generate an instance mask; for each grid, the prediction head generates an instance confidence score for each grid, the score indicating whether the grid contains a target instance, and the instance mask and instance confidence score together constitute the grid output;

[0032] S405 merges the outputs of all grids and filters out grid outputs with an instance confidence score less than 0.5;

[0033] S406, A mask feature is obtained by performing operations including fusion and upsampling on the feature map output by the feature pyramid network. The mask feature includes information about the position and shape of the target.

[0034] S407, performs a convolution operation on all mask kernels and mask features to generate an instance mask for grid output;

[0035] S408 filters the output instance mask using matrix non-maximum suppression.

[0036] S409 merges the instances filtered by matrix non-maximum suppression and outputs them. The output matrix has the shape (n,h,w,2), where n represents the number of output instances, h and w are the width and height of the original image, and n instances output n binary masks with the same width and height as the original image. Each mask represents the information of an instance. For each pixel, 0 represents the background and 1 represents the foreground of the current instance.

[0037] In one possible implementation, the semantic segmenter in S50 uses a PID neural network model, and its specific operation includes:

[0038] S501: Preprocess the current video stream image by scaling it to a width of 640 pixels and a height of 384 pixels; convert the video to RGB format; and normalize the video.

[0039] S502 first passes the preprocessed image through the backbone network to obtain deep features, capturing local features and contextual information in the image;

[0040] S503 extracts fused features by feeding deep features into the P, I, and D branches to obtain features for each branch. The P branch is responsible for parsing and preserving detailed information in the high-resolution feature map; the I branch is responsible for aggregating local and global contextual information to resolve long-term dependencies between pixels, using an encoder-decoder structure where the encoder progressively reduces spatial resolution to capture context, and the decoder progressively restores resolution to locate details; the D branch focuses on capturing high-frequency details, including edge and texture information, through a series of convolutional layers and dilated convolutions.

[0041] S504, post-processing yields a semantic segmentation map, which is the same size as the input image. Each pixel is assigned a class label. The feature map output by the boundary attention-guided fusion BAG module is upsampled to match the resolution of the original input image. The upsampled feature map is then fed into the semantic head to obtain a semantic feature map, which includes 1x1 convolution and softmax operations. The 1x1 convolution is used for adjusting the number of channels and feature fusion, while the softmax operation assigns a class probability to each pixel. For each pixel in the semantic feature map, the class with the highest human probability is selected, and when the semantic human class probability is greater than a threshold of 0.5, it is used as the human semantic output. All others are considered background, generating a semantic segmentation map. The semantic segmentation map is output, and its output matrix shape is (1, h, w, 2), indicating that a binary matrix with the same width and height as the original image has been generated. 0 represents the pixel as background, and 1 represents the pixel as foreground.

[0042] In one possible implementation, the depth estimator in S60 uses the model DepthAnything, and its specific operation includes:

[0043] S601, preprocess the current video stream image, scale the video stream image to a width of 448 pixels and a height of 224 pixels; convert the video to RGB format; normalize the video;

[0044] S602, the preprocessed image is fed into the model's encoder for encoding. The encoder for feature extraction is the DINOv2 vits encoder, which will extract features from the preprocessed image.

[0045] S603, the obtained encoding is sent to the DPT decoder for decoding, and the features are mapped back to the depth map to obtain the depth feature map;

[0046] S604, upsample the depth feature map to the same size as the original image to obtain a depth map. The value of each pixel on the depth map represents the depth of that location. The smaller the value, the farther the location is, and the larger the value, the closer the location is.

[0047] In one possible implementation, the foreground mask selection algorithm process in S70 specifically includes:

[0048] S7011: Based on area and width / height information, instances with areas, widths, and heights smaller than preset thresholds are marked as ignored. S7012: The number of instances containing the ignored marker is counted. If the number exceeds a preset number, the post-processing module cache is retrieved to check if foreground information exists in the previous frame's mask. If the previous frame's mask did not output foreground information, a blank mask without foreground information is output for this frame. S7013: Face information is matched with instance information, and instances containing face information are marked as containing a face. S7014: If the current frame does not contain face information, no mask is output for this frame. S7015: Instances containing the ignored marker are excluded, and instances containing face information are selected. Based on the face's position, area, and aspect ratio information, a score is given to select the optimal face. The instance corresponding to the optimal face is the optimal instance, and the centroid depth of this instance is the optimal depth of field for the current frame. S7016: If the optimal depth of field for the current frame is not empty, the optimal depth of field for the previous frame is read. If the optimal depth of field of the current frame is greater than the optimal depth of field of the previous frame (the dynamic threshold is calculated as 0.2 * the optimal depth of field of the previous frame), then all instance depths are traversed, and the one closest to the optimal depth of field of the previous frame is selected as the optimal depth of field of the current frame. In step S7017, instances containing ignore markers are excluded, and instances with facial information are traversed. If their depth of field is less than the optimal depth of field of the current frame (the dynamic threshold is calculated as 0.2 * the optimal depth of field of the current frame), then the instance is considered an important instance in the current frame and marked as needing to be output. In step S7018, a blank mask image with the same width and height as the original image is created. This mask image is a binary image, where 0 represents the background and 1 represents the foreground. All instance masks that need to be output are superimposed on the mask image, that is, the value of the position where 1 is in the instance mask is set to 1 at the corresponding position on the mask image, thus obtaining the foreground mask of the current frame.

[0049] In one possible implementation, the specific process of selecting the optimal face in S7015 includes: traversing all face results in the current frame, and calculating the position score, area score, and width-to-height ratio of each face. The position score is calculated as follows: let the coordinates of the face center point be cx, cy, the width of the current frame image be imgw, and the height be imgh. The position score calculation formula is score1 = 1 ifabs(0.5 *imgw–cx) / imgw < 0.3 else 1 - abs(0.5 *imgw–cx) / imgw; The area score is calculated as follows: traversing all faces, taking the maximum area as max_face_area, and for each face, its area is face_area. The area score calculation formula is score2 = face_area / max_face_area; The width-to-height ratio is calculated as follows: let the long side of the face be l, and the short side be s. The width-to-height ratio calculation formula is score3 = s / l; the face with the highest sum of the position score, area score, and width-to-height ratio is the optimal face.

[0050] In one possible implementation, the temporal mask compensation algorithm in S70 includes: S7021, reading the output instance information of the previous frame, reading the instance information of the current frame, matching the two based on the centroid and bounding box, if the current frame instance can match the output instance of the previous frame, then outputting the instance mask, and marking the matched previous frame instance as matched; S7022, for the unmatched output instance of the previous frame, taking the intersection of it with the semantic segmentation and depth information results of the current frame, if the mask area after taking the intersection is greater than the area of ​​the previous frame instance * a preset threshold, then taking the intersection result as a new instance to be prepared for output; S7023, establishing a blank mask image with the same width and height as the original image, and superimposing all the instance masks that need to be output onto the mask image to obtain the temporal compensation mask.

[0051] In one possible implementation, the matching calculation method in S7021 is as follows: First, calculate the center of each of the two instances. The calculation method is as follows: for each instance, take all points that are foreground and calculate the mean of x and y, which is the centroid. If the centroid position is not classified as foreground in the instance, then arrange all foreground points in order of ascending horizontal coordinate and ascending vertical coordinate into a one-dimensional vector, and take the middle point as the centroid. Calculate the Euclidean distance between the centroids of the two instances. If the distance is less than 0.25 times the height and less than 0.25 times the width of the previous frame instance, then the two instances are considered to be matched.

[0052] The present invention has the following beneficial effects:

[0053] (1) A new lens switching detection method was designed, which is highly sensitive to general lens switching and maintains high performance to avoid misjudgment of fast-moving scenes, laying the foundation for subsequent timing processing.

[0054] (2) Using a deep learning model, the instance segmentation information, depth information, and face information in the picture are obtained. The main target of the current frame is determined from multiple dimensions, and a foreground mask is generated. Only the main target is focused on, so that the main target is not obscured by the barrage. At the same time, the barrage will still cover the secondary target, ensuring the barrage viewing experience.

[0055] (3) Use deep learning models to obtain instance segmentation information, depth information, semantic segmentation information and lens switching information provided by the lens detector in the picture, and ensure that the corresponding mask is continuously output after the main target is established from multiple dimensions. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating the steps of an artificial intelligence-based method for preventing foreground target barrage obstruction, according to an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] See Figure 1 The diagram shows a flowchart of a method for preventing foreground target occlusion based on artificial intelligence according to an embodiment of the present invention, including the following steps:

[0059] S10, sample the original video at a preset frequency to obtain a video stream image; where the original video is the video for which foreground human body bullet screen anti-occlusion is required;

[0060] S20: The video stream is processed using a camera transition detection algorithm to determine whether a camera transition has occurred in the current video frame. If a camera transition has occurred in the current frame, the post-processing module cache is reset. The post-processing module cache includes target tracking information, the depth information of the main target in the previous frame, and whether there was a mask output in the previous frame.

[0061] S30, use a face detector to process the current image to obtain the target face information of the current image;

[0062] S40, use the instance segmenter to process the current image to obtain the target instance segmentation information of the current image;

[0063] S50, use a semantic segmenter to process the current image to obtain the semantic segmentation information of the target human body in the current image. The human body semantic segmentation information is a one-dimensional feature map with the same width and height as the original image. The value of each pixel represents the probability that the pixel is a human body.

[0064] S60, use a depth estimator to process the current image to obtain the current image depth estimation information, where the depth estimation information is a one-dimensional feature map with the same width and height as the original image, and the value of each pixel represents the depth information of the object in that pixel;

[0065] S70, the current image target face information, current image target instance segmentation information, current image target human semantic segmentation information, and current image depth estimation information are sent to the post-processing module for processing. The post-processing module includes a foreground mask selection algorithm and a temporal mask compensation algorithm. The foreground mask selection algorithm is used to filter out the instances that need to be output in the current frame, construct mask information based on the information of the current frame, and obtain the foreground mask of the current frame. The temporal mask compensation algorithm is used to track the instances that have been output in the previous frame, prevent the lack of key position masks due to model result jitter, improve the overall algorithm robustness, and obtain the temporal compensated mask.

[0066] S80: Take the union of the foreground mask and the timing compensation mask of the current frame to obtain the final output mask of the current frame, and output it to the outside as the result of the current frame.

[0067] S90, save the instance information used by the foreground mask selection algorithm and the temporal mask compensation algorithm in the current frame, and use it for the foreground mask selection algorithm and the temporal mask compensation algorithm processing of the next frame image;

[0068] S100, the next frame image is processed as described in S20 to S90 above, until the video stream image processing is completed.

[0069] In actual processing, processes S30, S40, S50, and S60 can be performed simultaneously without regard to their order. The camera detector in S20 is highly sensitive to general camera transitions while maintaining high performance against fast-moving scenes, laying the foundation for subsequent temporal processing. S40 uses instance segmentation results for bullet screen masking, which is the basis for masking the distinction between primary and secondary targets. Based on the foreground mask selection algorithms in S30, S40, S60, and S80, by combining facial information, depth information, and instance information, the main target in the current frame can be effectively identified, and only the main target is masked, allowing users to both enjoy the main characters and participate in bullet screen interaction. Based on the temporal mask compensation algorithms in S40, S50, S60, and S70, by combining depth information, semantic information, and instance information, the target mask is continuously tracked from its appearance, ensuring mask continuity and preventing frequent intermittent bullet screen occurrences, thus improving the user's bullet screen viewing experience.

[0070] In a specific application example, the video stream processing using the shot switching detection algorithm in S20 specifically includes: S201, scaling the image to a certain size; S202, calculating the difference between the image and the previous frame, counting the number of pixels whose difference is greater than a preset image change threshold to obtain the number of changed pixels, and counting the number of pixels whose difference is less than a preset unchanged image threshold to obtain the number of unchanged pixels. If the number of changed pixels is greater than the preset threshold and the number of unchanged pixels is less than the preset threshold, then a shot switching is determined to have occurred in that frame; S203, converting the image to the HSV color space, extracting the histogram information of the hue / saturation components, and comparing it with the histogram information of the hue / saturation components of the previous frame to obtain the hue / saturation component. The histogram correlation of the saturation component is calculated. Simultaneously, the image is converted to grayscale color space, and grayscale histogram information is extracted and compared with the grayscale histogram information of the previous frame to obtain the histogram correlation of the grayscale histograms. If the histogram correlation of the hue and saturation components is greater than a preset threshold and the histogram correlation of the grayscale histogram is less than a preset threshold, then it is determined that a shot change has occurred in this frame. S204: Update the cache information, saving the scaled image of the current frame, the histogram of the hue and saturation components, and the grayscale histogram information, to be used as the basis for judging the next frame. S205: Output the shot change information of the current video frame. When the output result is true, it indicates that a shot change has occurred in the current frame; when the output result is false, it indicates that a shot change has not occurred in the current frame.

[0071] In a specific application example, the face detector in S30 uses the nano architecture of the YOLOv8 model. YOLOv8 belongs to the YOLO series of algorithms. On the basis of inheriting the advantages of the YOLO series, it adopts an anchorless design to improve the detection accuracy; introduces new loss functions, such as Distribution Focal Loss, to better balance positive and negative samples and improve the learning efficiency of the model during training; introduces new or improved feature fusion modules (such as BiFPN and AFPN), and uses SPD-Conv and other technologies to enhance the detection ability of small targets and multi-scale targets; adopts a variety of attention mechanisms to enhance the feature extraction ability of the model, and innovates on the convolutional layer, such as using deformable convolution and dynamic convolution. The structure of YOLOv8 consists of three parts: (1) Backbone: used to extract image features, using C2f module and SPPF module, reducing image size and enhancing features through convolutional layers. (2) Neck: located between the backbone and the head, using SPPF, PAA and PAN modules to fuse features of different scales and enhance feature expression. (3) Head Network: This is the decision-making part of the model, responsible for generating the final detection results. It generates feature maps based on the neck network, producing detection results including bounding box positions, confidence scores, and class probabilities. The nano architecture represents a reduction in both depth and width compared to the standard YOLOv8 model to adapt to resource-constrained environments while maintaining model effectiveness and efficiency. In practical configurations, the depth factor is 0.33, the width factor is 0.25, and the nano architecture YOLOv8 model has 33% of the number of layers as the standard version, with 25% of the channels per layer. The model takes video images as input and outputs face location information and face confidence scores. Its specific working process includes:

[0072] S301, preprocess the current video stream image, scale the video stream image to an image with a width of 640 pixels and a height of 384 pixels; convert the image storage channel format from [height, width, color channel] to [batch, color channel, height, width]; normalize each pixel value, the specific normalization operation is to divide each pixel by 255;

[0073] S302: After the preprocessed image is fed into the model, it first performs feature extraction through the backbone network, generating multiple feature maps at different scales. These feature maps capture information from different levels of the image. The feature maps are then fed into the neck network, which further processes and fuses the feature maps output by the backbone network. The feature maps processed by the neck network are then fed into the prediction head, which is responsible for generating detection results based on the feature maps. The detection results are a matrix, where each row represents a detected object and each column represents an attribute of that object, in the following order: x-coordinate of the bounding box center point, y-coordinate of the bounding box center point, width of the bounding box, height of the bounding box, confidence score of YOLO detection, and probability of face category.

[0074] The S303 and YOLOv8 neck network outputs feature maps of multiple sizes. Each feature map is paired with a prediction head, generating a prediction matrix. The output matrices of all prediction heads are concatenated column-wise to generate a new prediction matrix. Each row represents a detection result, and each column represents an attribute of that detection result, in the following order: x-coordinate of the bounding box center point, y-coordinate of the bounding box center point, width of the bounding box, height of the bounding box, YOLO detection confidence score, and probability of face class. The face confidence score is obtained by multiplying the YOLO detection confidence score by the probability of face class. The detection result includes bounding box information and face confidence score information.

[0075] S304, filter out the face detection results in the obtained matrix whose face confidence is lower than the preset threshold of 0.5 by YOLO detection, and perform non-maximum suppression operation on the filtered detection results to filter out overlapping bounding boxes;

[0076] S305 transforms the coordinates of the bounding box of the final detection result from normalized coordinates back to the coordinates of the original video stream image, multiplies the width and height of the bounding box by the aspect ratio of the original image, and also multiplies the horizontal and vertical coordinates of the center point by the aspect ratio of the original image accordingly; and outputs the detection result after coordinate transformation.

[0077] The specific process of nonmaximum suppression is as follows: sort all detection results in descending order of confidence to form a candidate list; select the detection result with the highest confidence, add it to the final detection result, and remove it from the candidate list; calculate the intersection-union ratio (IoU) of the bounding box of the detection result with the bounding boxes of all other detection results in the candidate list; if the IoU value exceeds a preset threshold of 0.5, remove the corresponding detection result from the candidate list; repeat the above process until all detection results have been processed and there are no detection results in the candidate list.

[0078] The specific calculation method for the intersection-over-unity (IoU) of bounding boxes is as follows: the IoU of bounding boxes A and B is equal to (intersection area) / (area of ​​A + area of ​​B – intersection area). For a bounding box with its top-left corner coordinates (x1, y1) and bottom-right corner coordinates (x2, y2), its area A can be calculated using the following formula: A = (x2 − x1 + 1) × (y2 − y1 + 1). For two bounding boxes, their intersection area, interArea, can be calculated using the following steps: Calculate the intersection coordinates of the two bounding boxes on the x and y axes: xx1 = max(x11, x12), yy1 = max(y11, y12), xx2 = min(x21, x22), yy2 = min(y21, y22); Calculate the width and height of the intersection, ensuring they are non-negative: w = max(0, xx2 − xx1 + 1) h = max(0, yy2 − yy1 + 1); Calculate the intersection area: interArea = w × h. Here, min() is the minimum value operation, and max() is the maximum value operation. The cross-union ratio (IoU) of the detection boxes is calculated as IoU = (interArea) / (Area1 + Area2 - interArea).

[0079] In a specific application example, the instance segmenter in S40 uses the SOLOv2 model. SOLOv2 performs fast and accurate instance segmentation at the pixel level directly in the form of a fully convolutional network. Its main structure is as follows: (1) Feature extraction: Image features are extracted using a pre-trained network such as ResNet, and then processed by the Feature Pyramid Network (FPN) to obtain a multi-scale fused feature map; (2) Dynamic instance segmentation: Mask kernel branch: Parallel to the semantic category branch, predicts the convolution kernel weights, taking into account spatial location information; Mask feature branch: Learns a unified high-resolution mask feature, which is achieved by fusing the feature maps of each layer of FPN; (3) Prediction head: Contains convolutional layers, processes the feature maps output by FPN, and outputs the category and mask prediction; (4) Matrix optimization non-maximum suppression (NMS): Optimizes the non-maximum suppression process to improve segmentation efficiency and accuracy. During training, the loss is the same as the training classification loss and mask loss to ensure that the model accurately predicts the target category and location. The model takes a video image as input and outputs human instance information of the current image, including (1) semantic category: the model predicts an output for each grid cell in the image, indicating whether the grid cell contains the center of the human body; (2) instance mask: the model predicts a precise pixel-level mask for each human instance, which can distinguish even if the human bodies overlap; (4) segmentation instance confidence: the model predicts the probability of the category for the instance. Processing the current image using the instance segmenter includes:

[0080] S401: Preprocess the current video stream image by scaling it to a width of 640 pixels and a height of 360 pixels; convert the video to RGB format; normalize the video; the normalization parameters have a mean of [123.675, 116.28, 103.53] and a standard deviation of [58.395, 57.12, 57.375]. Subtract the mean from each color channel and then divide by the standard deviation. That is, the red channel value will be subtracted by 123.675 and then divided by 58.395, the green channel value will be subtracted by 116.28 and then divided by 57.12, and the blue channel value will be subtracted by 103.53 and then divided by 57.375.

[0081] S402 inputs the preprocessed image into the model to obtain feature maps, inputs the preprocessed image into the backbone network for feature extraction, then obtains feature maps of different scales through the feature pyramid network, performs upsampling or downsampling operations on the feature maps of different scales to match a uniform resolution, and performs further feature fusion through convolutional layers.

[0082] S403 divides the obtained feature map into a grid of 9 rows and 16 columns. Each grid is responsible for predicting targets within a certain range. For each grid, a mask kernel and mask kernel parameters are learned from the feature map through a series of convolution operations.

[0083] S404: For each grid cell, the obtained mask kernel and mask features are convolved to generate an instance mask; for each grid, the prediction head generates an instance confidence score for each grid, the score indicating whether the grid contains a target instance, and the instance mask and instance confidence score together constitute the grid output;

[0084] S405 merges the outputs of all grids and filters out grid outputs with an instance confidence score less than 0.5;

[0085] S406, A mask feature is obtained by performing operations including fusion and upsampling on the feature map output by the feature pyramid network. The mask feature includes information about the position and shape of the target.

[0086] S407, performs a convolution operation on all mask kernels and mask features to generate an instance mask for grid output;

[0087] S408 filters the output instance mask using matrix non-maximum suppression.

[0088] S409 merges the instances filtered by matrix non-maximum suppression and outputs them. The output matrix has the shape (n,h,w,2), where n represents the number of output instances, h and w are the width and height of the original image, and n instances output n binary masks with the same width and height as the original image. Each mask represents the information of an instance. For each pixel, 0 represents the background and 1 represents the foreground of the current instance.

[0089] The specific process of filtering the output instance mask using matrix non-maximum suppression in S408 includes:

[0090] (1) Sort the prediction results from high to low confidence scores and take the 50 highest predictions for subsequent operations, and discard the rest; if there are fewer than 50 original results, take all of them.

[0091] (2) For all the predicted results, calculate the intersection-union ratio (IoU) between the masks of all the predicted results to generate an N×N IoU matrix. The mask intersection-union ratio (IoU) is an indicator that measures the degree of overlap between two masks, defined as the area of ​​the intersection of two masks divided by the area of ​​their union. The specific calculation process is as follows: For masks a and b, flatten their H*W mask matrices into a vector of [1, H*W] to obtain mask vector A and mask vector B; calculate the intersection area of ​​A and B by calculating the dot product of A and B; calculate the union area of ​​the two masks by subtracting their intersection from the total number of activated pixels of the two masks; the mask intersection-union ratio of this mask pair = intersection area / union area.

[0092] In actual computation, matrix operations based on the above algorithm are executed in parallel on the GPU. Specifically, the N binary masks, each of size H×W, are flattened into an H×W dimensional vector. Therefore, all masks can be represented as an N×(H×W) matrix M, where each row represents a mask. An N×N intersection matrix I is obtained by calculating the self-dot product of matrix M, where I... ij Let S represent the intersection between masks i and j. By summing the elements of each row of matrix M, we obtain an N-dimensional vector S, where S... i Let represent the total number of pixels in mask i. Construct an N×N diagonal matrix D, where the elements on the diagonal are the elements of S, i.e., D ∈ S. ii =S i Next, we can calculate D+D. T -I is used to obtain the union matrix U. The IOU matrix is ​​equal to the intersection matrix I divided by the union matrix U.

[0093] (3) For each prediction result, calculate a decay factor, which represents the degree to which the mask is suppressed by other masks. Update the confidence score of the prediction result by multiplying the decay factor by the confidence score of the prediction result. The calculation method is as follows: For each prediction result, calculate its decay value with other prediction results, calculated as decay = 1–iou ij The minimum value among all attenuation values ​​is taken as the attenuation factor.

[0094] (4) Sort all the updated confidence scores of the prediction results from high to low. Iterate through the sorted confidence score list. For each mask: if the confidence score of the prediction result is lower than the confidence threshold of 0.5, discard the prediction result. If the IoU between the prediction result and all other retained prediction results is greater than the overlap threshold of 0.7, discard the prediction result. If the prediction result is not discarded, add it to the list of retained prediction results. Output the list of retained prediction results after the iteration is complete.

[0095] In a specific application example, the semantic segmenter in the S50 uses the PID neural network PIDNet. PIDNet is a real-time semantic segmentation network that addresses the overshoot problem that may occur in traditional two-branch networks when fusing high-resolution details and low-resolution contextual information by introducing an auxiliary differential branch. The PIDNet architecture comprises three complementary branches: a proportional (P) branch responsible for parsing and preserving detailed information in the high-resolution feature map; an integral (I) branch responsible for aggregating contextual information locally and globally to resolve long-distance dependencies between pixels; and a differential (D) branch responsible for extracting high-frequency features to predict and emphasize edge regions in the image. PIDNet also employs a pixel attention-guided fusion module (Pag) and a boundary attention-guided fusion module (Bag). These modules utilize edge features extracted from the D branch to guide the fusion of the P and I branches, thereby enhancing the features of edges and boundary regions while preserving detailed information. The model takes a video image as input and outputs human semantic segmentation information. This information is a one-dimensional feature map with the same width and height as the original image, where the value of each pixel represents whether the pixel is background or foreground. The specific working process includes:

[0096] S501: Preprocess the current video stream image by scaling it to a width of 640 pixels and a height of 384 pixels; convert the video to RGB format; normalize the video; the normalization parameters have a mean of [123.675, 116.28, 103.53] and a standard deviation of [58.395, 57.12, 57.375]. Subtract the mean from each color channel and then divide by the standard deviation. That is, the red channel value will be subtracted by 123.675 and then divided by 58.395, the green channel value will be subtracted by 116.28 and then divided by 57.12, and the blue channel value will be subtracted by 103.53 and then divided by 57.375.

[0097] S502 first passes the preprocessed image through the backbone network to obtain deep features, capturing local features and contextual information in the image;

[0098] S503 extracts fused features by feeding deep features into the P, I, and D branches to obtain features for each branch. The P branch is responsible for parsing and preserving detailed information in the high-resolution feature map; the I branch is responsible for aggregating local and global contextual information to resolve long-term dependencies between pixels, using an encoder-decoder structure where the encoder progressively reduces spatial resolution to capture context, and the decoder progressively restores resolution to locate details; the D branch focuses on capturing high-frequency details, including edge and texture information, through a series of convolutional layers and dilated convolutions.

[0099] S504, post-processing yields a semantic segmentation map, which is the same size as the input image. Each pixel is assigned a class label. The feature map output by the boundary attention-guided fusion BAG module is upsampled to match the resolution of the original input image. The upsampled feature map is then fed into the semantic head to obtain a semantic feature map, which includes 1x1 convolution and softmax operations. The 1x1 convolution is used for adjusting the number of channels and feature fusion, while the softmax operation assigns a class probability to each pixel. For each pixel in the semantic feature map, the class with the highest human probability is selected, and when the semantic human class probability is greater than a threshold of 0.5, it is used as the human semantic output. All others are considered background, generating a semantic segmentation map. The semantic segmentation map is output, and its output matrix shape is (1, h, w, 2), indicating that a binary matrix with the same width and height as the original image has been generated. 0 represents the pixel as background, and 1 represents the pixel as foreground.

[0100] Furthermore, in S503, the parallel aggregation pyramid pooling module PAPPM is located in the I branch. It rapidly aggregates contextual information at different scales through a parallel structure, improving computational efficiency. The input is the feature map of the I branch, and the output is the feature map after multi-scale context aggregation. Multi-scale features are extracted through pooling and convolution operations at different scales, and then fused through upsampling and element-wise addition to construct a global scene prior. The process is as follows: its input is the deep feature map extracted by the backbone network, and the output is a feature map fused with multi-scale contextual information. The process is as follows: the input feature map first obtains contextual information at different scales through pooling operations at different scales; each feature map is processed by the ReLU activation function and a 1x1 convolution operation to reduce the number of channels and further extract features; each pooled feature map is upsampled to restore its size to the same as the original input feature map size; the fused feature map is processed through a compression convolutional layer to reduce the number of channels and integrate information; the compressed fused feature map is added to the original input feature map to obtain the final output feature map.

[0101] The Pixel Attention-Guided Fusion (PAG) module is located in the P branch. It selectively learns useful semantic features from the I branch through an attention mechanism, avoiding being overwhelmed by the contextual information of the I branch. The input is the feature maps of the P and I branches, and the output is a fused feature map, which combines the detailed information from the P branch and the contextual information from the I branch. Features are selectively fused by calculating pixel attention weights between the P and I branch feature maps. The process is as follows: the input is the feature maps of both branches, and the output is a feature map that fuses detailed and contextual information. Specifically, the P and I branch features are input and processed through 1x1 convolutional layers to reduce the number of channels and extract key features. The resulting feature maps undergo pixel-level dot product operations, and attention weights σ are generated using the Sigmoid function, where σ = Sigmoid(f p (v p )·f i (v i )), where f p f i This represents a combination of convolution, batch normalization, and ReLU activation; the calculated attention weights are used to weight the feature maps of branch I, and then element-wise added to the feature maps of branch P to achieve feature fusion. (Out) pag =σ·v i +(1-σ)·v p .

[0102] The Boundary Attention-Guided Fusion (BAG) module is located in the network's fusion stage. It combines the outputs of the P, I, and D branches, using boundary attention weights to guide the fusion of the P and I branches, enhancing the model's ability to recognize boundaries and small objects. The input consists of feature maps from the P, I, and D branches, and the output is the fused feature map. The contributions of the P and I branches are dynamically adjusted based on the boundary features of the D branch. The process is as follows: the sigmoid output of the D branch feature map guides the fusion of the P and I branch feature maps. Its input consists of feature maps from the three branches, and the output is a feature map that fuses details and contextual information. The output v of the D branch... d To calculate the boundary attention weights σ, we use the Sigmoid function to achieve σ = Sigmoid(v d Dynamically merge the features of P branches and I branches. The merging operation is as follows: Out bag =f out ((1-σ) v i +σ v p ), f out This represents a combination of convolution, batch normalization, and the ReLU activation function. This indicates element-wise multiplication.

[0103] In a specific application example, the depth estimator described in S60 uses the Depth Anything model. Depth Anything is an advanced monocular depth estimation model that employs VisionTransformer (ViT) as its backbone network. ViT is a deep learning model based on the Transformer architecture that segments an image into a series of patches and converts each patch into a vector representation as an input sequence. These vectors are processed by a multi-layer Transformer encoder, including self-attention mechanisms and feedforward neural network layers to capture contextual dependencies in the image. In ViT, the input image is first segmented into fixed-size non-overlapping patches, which are then linearly mapped to a new dimensional space to form a sequential input. Furthermore, a special category label is added to the beginning of the sequence, and positional embeddings are added to each patch vector to preserve spatial information from the original image. ViT processes the input sequence through self-attention mechanisms and feedforward network layers to extract image features. After feature extraction, the model fuses feature maps of different scales through a feature fusion module to retain more details and contextual information. Finally, the model outputs a final depth map through a depth prediction module. The model takes a video image as input and outputs a depth map, predicting the depth value of each pixel in the input image. The specific workflow includes:

[0104] S601: Preprocess the current video stream image by scaling it to a width of 448 pixels and a height of 224 pixels; convert the video to RGB format; normalize the video; the normalization parameters have a mean of [123.675, 116.28, 103.53] and a standard deviation of [58.395, 57.12, 57.375]. Subtract the mean from each color channel and then divide by the standard deviation. That is, the red channel value will be subtracted by 123.675 and then divided by 58.395, the green channel value will be subtracted by 116.28 and then divided by 57.12, and the blue channel value will be subtracted by 103.53 and then divided by 57.375.

[0105] S602, the preprocessed image is fed into the model's encoder for encoding. The encoder for feature extraction is the DINOv2 vits encoder, which will extract features from the preprocessed image.

[0106] S603, the obtained encoding is sent to the DPT decoder for decoding, and the features are mapped back to the depth map to obtain the depth feature map;

[0107] S604, upsample the depth feature map to the same size as the original image to obtain a depth map. The value of each pixel on the depth map represents the depth of that location. The smaller the value, the farther the location is, and the larger the value, the closer the location is.

[0108] In a specific application example, the foreground mask selection algorithm process in S70 includes:

[0109] S7011: Based on area, width, and height information, instances with areas, widths, and heights less than preset thresholds are marked as ignored. In practice, the instance area should be greater than 0.01 times the video stream image area, the height should be greater than 0.25 times the video stream image height, and the width should be greater than 0.1 times the video stream image width. These parameters can be modified according to actual needs. S7012: Count the number of instances containing the ignore marker. If the number exceeds the preset number, and the post-processing module cache is retrieved to check if the previous frame mask has foreground information, if the previous frame mask did not output foreground information, then a blank mask without foreground information is output for this frame. S7013: Match face information with instance information, and mark instances with face information as containing faces. S7014: If the current frame does not have face information, then no mask is output for this frame. S7015: Exclude instances containing the ignore marker and select instances with face information. Based on the face's location, area, and aspect ratio information, a score is assigned to select the optimal face. The instance corresponding to the optimal face is designated as the optimal instance, and its centroid depth of field is the optimal depth of field for the current frame. In S7016, if the optimal depth of field for the current frame is not empty, the optimal depth of field for the previous frame is read. If the optimal depth of field for the current frame is greater than or equal to the optimal depth of field for the previous frame, a dynamic threshold is set to 0.2 * the optimal depth of field for the previous frame. Then, all instance depths are iterated, and the one closest to the optimal depth of field for the previous frame is selected as the optimal depth of field for the current frame. In S7017, instances containing ignore markers are excluded. Instances with face information are iterated. If their depth of field is less than or equal to the optimal depth of field for the current frame, a dynamic threshold is set to 0.2 * the optimal depth of field for the current frame. This instance is considered an important instance in the current frame and is marked for output. In S7018, a blank mask image with the same width and height as the original image is created. This mask image is a binary image, where 0 represents the background. 1 represents the foreground; all instance masks that need to be output are superimposed on the mask image, that is, the value of the position where 1 is in the instance mask is set to 1 at the corresponding position on the mask image, thus obtaining the foreground mask of the current frame.

[0110] Furthermore, the specific process for selecting the optimal face in S7015 includes: traversing all face results in the current frame, and calculating the position score, area score, and width-to-height ratio for each face. The position score is calculated as follows: Let the center point coordinates of the face be cx, cy, the width of the current frame image be imgw, and the height be imgh. The position score calculation formula is score1 = 1 if abs(0.5 *imgw–cx) / imgw < 0.3 else 1 - abs(0.5 *imgw–cx) / imgw; The area score is calculated as follows: traversing all faces, taking the maximum area as max_face_area, and for each face, its area is face_area. The area score calculation formula is score2 = face_area / max_face_area; The width-to-height ratio is calculated as follows: Let the long side of the face be l, and the short side be s. The width-to-height ratio calculation formula is score3 = s / l; the face with the highest sum of the position score, area score, and width-to-height ratio is the optimal face.

[0111] In a specific application example, the temporal mask compensation algorithm in S70 includes the following steps: S7021, read the output instance information of the previous frame, read the instance information of the current frame, and match the two based on the centroid and bounding box. If the current frame instance can match the output instance of the previous frame, output the instance mask and mark the matched previous frame instance as matched; S7022, for the unmatched output instance of the previous frame, take the intersection of it with the semantic segmentation and depth information results of the current frame. If the mask area after taking the intersection is greater than the area of ​​the previous frame instance * a preset threshold, take the intersection result as a new instance to be prepared for output; S7023, create a blank mask image with the same width and height as the original image, and overlay all the instance masks that need to be output onto the mask image to obtain the temporal compensation mask.

[0112] Furthermore, the matching calculation method in S7021 is as follows: First, the center of each instance is calculated separately. The calculation method is as follows: For each instance, all points that are foreground are taken, and the mean of x and y is calculated, which is the centroid. If the centroid position is not classified as foreground in the instance, all foreground points are arranged into a one-dimensional vector in ascending order of horizontal coordinate and ascending vertical coordinate, and the midpoint of the vector is taken as the centroid. The Euclidean distance between the centroids of the two instances is then calculated. If the distance is less than 0.25 times the height and less than 0.25 times the width of the previous frame instance, then the two instances are considered to match.

[0113] The above-described AI-based foreground target anti-occlusion method for bullet comments employs a novel shot transition detection approach. This method exhibits high sensitivity to general shot transitions while maintaining high performance against fast-moving scenes, preventing false detections and laying the foundation for subsequent temporal processing. By utilizing a deep learning model to obtain instance segmentation, depth, and facial information from the image, the main target of the current frame is determined from multiple dimensions, and a foreground mask is generated. This allows focus to be placed solely on the main target, preventing it from being obscured by bullet comments, while ensuring that secondary targets are still covered by bullet comments, maintaining a good viewing experience. The use of deep learning models to obtain instance segmentation, depth, and semantic segmentation information, along with shot transition information from the shot detector, ensures that once the main target is identified, the corresponding mask is continuously output.

[0114] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A method for preventing foreground target occlusion in bullet screens based on artificial intelligence, characterized in that, Includes the following steps: S10, sample the original video at a preset frequency to obtain a video stream image; The original video is the one that needs to be protected from occlusion by foreground human figures in the bullet screen. S20: The video stream is processed using a camera transition detection algorithm to determine whether a camera transition has occurred in the current video frame. If a camera switch occurs in the current frame, the post-processing module cache is reset; the post-processing module cache includes target tracking information, the depth information of the main target in the previous frame, and whether there was a mask output in the previous frame; S30, use a face detector to process the current image to obtain the target face information of the current image; S40, use the instance segmenter to process the current image to obtain the target instance segmentation information of the current image; S50, use a semantic segmenter to process the current image to obtain the semantic segmentation information of the target human body in the current image. The semantic segmentation information of the human body is a one-dimensional feature map with the same width and height as the original image. The value of each pixel represents the probability that the pixel is a human body. S60, use a depth estimator to process the current image to obtain the current image depth estimation information. The depth estimation information is a one-dimensional feature map with the same width and height as the original image. The value of each pixel represents the depth information of the object in that pixel. S70, the current image target face information, current image target instance segmentation information, current image target human semantic segmentation information, and current image depth estimation information are sent to the post-processing module for processing. The post-processing module includes a foreground mask selection algorithm and a temporal mask compensation algorithm. The foreground mask selection algorithm is used to filter out the instances that need to be output in the current frame, construct mask information based on the information of the current frame, and obtain the foreground mask of the current frame. The temporal mask compensation algorithm is used to track the instances that have been output in the previous frame and obtain the temporal compensation mask. S80: Take the union of the foreground mask and the timing compensation mask of the current frame to obtain the final output mask of the current frame, and output it to the outside as the result of the current frame. S90, save the instance information used by the foreground mask selection algorithm and the temporal mask compensation algorithm in the current frame, and use it for the foreground mask selection algorithm and the temporal mask compensation algorithm processing of the next frame image; S100, the next frame image is processed as described in S20 to S90 above, until the video stream image processing is completed.

2. The artificial intelligence-based method for preventing foreground target occlusion in bullet screens as described in claim 1, characterized in that, The video stream processing using the shot switching detection algorithm described in S20 specifically includes: S201, scaling the image to a certain size; S202, calculating the difference between the image and the previous frame, counting the number of pixels whose difference is greater than a preset image change threshold to obtain the number of changed pixels, and counting the number of pixels whose difference is less than a preset unchanged image threshold to obtain the number of unchanged pixels. If the number of changed pixels is greater than the preset threshold and the number of unchanged pixels is less than the preset threshold, then a shot switching is determined to have occurred in that frame; S203, converting the image to the HSV color space, extracting the histogram information of the hue / saturation components, comparing it with the histogram information of the hue / saturation components of the previous frame to obtain the histogram correlation of the hue / saturation components, and simultaneously converting the image to the grayscale color space to extract the grayscale histogram... The histogram information is compared with the grayscale histogram information of the previous frame to obtain the histogram correlation of the grayscale histogram. If the histogram correlation of the hue saturation component is greater than a preset threshold and the histogram correlation of the grayscale histogram is less than a preset threshold, it is determined that a shot change has occurred in the frame. S204, update the cache information and save the scaled image of the current frame, the histogram of the hue saturation component, and the grayscale histogram information as the basis for judging the next frame. S205, output the shot change information of the current video frame. When the output result is true, it indicates that a shot change has occurred in the current frame. When the output result is false, it indicates that a shot change has not occurred in the current frame.

3. The artificial intelligence-based method for preventing foreground target occlusion in bullet screens as described in claim 1, characterized in that, The face detector in S30 uses the YOLOv8 model with a nano architecture. Its specific operation includes: S301, preprocess the current video stream image, scale the video stream image to an image with a width of 640 pixels and a height of 384 pixels; convert the image storage channel format from [height, width, color channel] to [batch, color channel, height, width]; normalize each pixel value, and divide each pixel by 255; S302: After the preprocessed image is fed into the model, it first performs feature extraction through the backbone network, generating multiple feature maps at different scales. These feature maps capture information from different levels of the image. The feature maps are then fed into the neck network, which further processes and fuses the feature maps output by the backbone network. The feature maps processed by the neck network are then fed into the prediction head, which is responsible for generating detection results based on the feature maps. The detection results are a matrix, where each row represents a detected object and each column represents an attribute of that object, in the following order: x-coordinate of the bounding box center point, y-coordinate of the bounding box center point, width of the bounding box, height of the bounding box, confidence score of YOLO detection, and probability of face category. The S303 and YOLOv8 neck network outputs feature maps of multiple sizes. Each feature map is paired with a prediction head, generating a prediction matrix. The output matrices of all prediction heads are concatenated column-wise to generate a new prediction matrix. Each row represents a detection result, and each column represents an attribute of that detection result, in the following order: x-coordinate of the bounding box center point, y-coordinate of the bounding box center point, width of the bounding box, height of the bounding box, YOLO detection confidence score, and probability of face class. The face confidence score is obtained by multiplying the YOLO detection confidence score by the probability of face class. The detection result includes bounding box information and face confidence score information. S304, filter out the face detection results in the obtained matrix whose face confidence is lower than the preset threshold of 0.5 by YOLO detection, and perform non-maximum suppression operation on the filtered detection results to filter out overlapping bounding boxes; S305 transforms the coordinates of the bounding box of the final detection result from normalized coordinates back to the coordinates of the original video stream image, multiplies the width and height of the bounding box by the aspect ratio of the original image, and also multiplies the horizontal and vertical coordinates of the center point by the aspect ratio of the original image accordingly; and outputs the detection result after coordinate transformation.

4. The artificial intelligence-based method for preventing foreground target occlusion in bullet screens as described in claim 1, characterized in that, The instance segmenter in S40 uses the SOLOv2 model, and the processing of the current image using the instance segmenter includes: S401, preprocess the current video stream image, scale the video stream image to a width of 640 pixels and a height of 360 pixels; convert the video to RGB format; normalize the video; S402 inputs the preprocessed image into the model to obtain feature maps, inputs the preprocessed image into the backbone network for feature extraction, then obtains feature maps of different scales through the feature pyramid network, performs upsampling or downsampling operations on the feature maps of different scales to match a uniform resolution, and performs further feature fusion through convolutional layers. S403 divides the obtained feature map into a grid of 9 rows and 16 columns. Each grid is responsible for predicting targets within a certain range. For each grid, a mask kernel and mask kernel parameters are learned from the feature map through a series of convolution operations. S404: For each grid cell, the obtained mask kernel and mask features are convolved to generate an instance mask; for each grid, the prediction head generates an instance confidence score for each grid, the score indicating whether the grid contains a target instance, and the instance mask and instance confidence score together constitute the grid output; S405 merges the outputs of all grids and filters out grid outputs with an instance confidence score less than 0.5; S406, A mask feature is obtained by performing operations including fusion and upsampling on the feature map output by the feature pyramid network. The mask feature includes information about the position and shape of the target. S407, performs a convolution operation on all mask kernels and mask features to generate an instance mask for grid output; S408 filters the output instance mask using matrix non-maximum suppression. S409 merges the instances filtered by matrix non-maximum suppression and outputs them. The output matrix has the shape (n,h,w,2), where n represents the number of output instances, h and w are the width and height of the original image, and n instances output n binary masks with the same width and height as the original image. Each mask represents the information of an instance. For each pixel, 0 represents the background and 1 represents the foreground of the current instance.

5. The artificial intelligence-based method for preventing foreground target occlusion in bullet screens as described in claim 1, characterized in that, The semantic segmenter in S50 uses a PID neural network model, and its specific working process includes: S501: Preprocess the current video stream image by scaling it to a width of 640 pixels and a height of 384 pixels; convert the video to RGB format; and normalize the video. S502 first passes the preprocessed image through the backbone network to obtain deep features, capturing local features and contextual information in the image; S503 extracts fused features by feeding deep features into the P, I, and D branches to obtain features for each branch. The P branch is responsible for parsing and preserving detailed information in the high-resolution feature map; the I branch is responsible for aggregating local and global contextual information to resolve long-term dependencies between pixels, using an encoder-decoder structure where the encoder progressively reduces spatial resolution to capture context, and the decoder progressively restores resolution to locate details; the D branch focuses on capturing high-frequency details, including edge and texture information, through a series of convolutional layers and dilated convolutions. S504, post-processing yields a semantic segmentation map, which is the same size as the input image. Each pixel is assigned a class label. The feature map output by the boundary attention-guided fusion BAG module is upsampled to match the resolution of the original input image. The upsampled feature map is then fed into the semantic head to obtain a semantic feature map, which includes 1x1 convolution and softmax operations. The 1x1 convolution is used for adjusting the number of channels and feature fusion, while the softmax operation assigns a class probability to each pixel. For each pixel in the semantic feature map, the class with the highest human probability is selected, and when the semantic human class probability is greater than a threshold of 0.5, it is used as the human semantic output. All others are considered background, generating a semantic segmentation map. The semantic segmentation map is output, and its output matrix shape is (1, h, w, 2), indicating that a binary matrix with the same width and height as the original image has been generated. 0 represents the pixel as background, and 1 represents the pixel as foreground.

6. The artificial intelligence-based method for preventing foreground target occlusion in bullet screens as described in claim 1, characterized in that, The depth estimator in S60 uses the model Depth Anything, and its specific working process includes: S601, preprocess the current video stream image, scale the video stream image to a width of 448 pixels and a height of 224 pixels; convert the video to RGB format; normalize the video; S602, the preprocessed image is fed into the model's encoder for encoding. The encoder for feature extraction is the DINOv2 vits encoder, which will extract features from the preprocessed image. S603, the obtained encoding is sent to the DPT decoder for decoding, and the features are mapped back to the depth map to obtain the depth feature map; S604, upsample the depth feature map to the same size as the original image to obtain a depth map. The value of each pixel on the depth map represents the depth of that location. The smaller the value, the farther the location is, and the larger the value, the closer the location is.

7. The artificial intelligence-based method for preventing foreground target occlusion in bullet screens as described in claim 1, characterized in that, The foreground mask selection algorithm process in S70 specifically includes: S7011: Based on area and width / height information, instances with areas, widths, and heights smaller than preset thresholds are marked as ignored. S7012: The number of instances containing the ignored marker is counted. If the number exceeds a preset number, the post-processing module cache is retrieved to check if foreground information exists in the previous frame's mask. If the previous frame's mask did not output foreground information, a blank mask without foreground information is output for this frame. S7013: Face information is matched with instance information, and instances containing face information are marked as containing a face. S7014: If the current frame does not contain face information, no mask is output for this frame. S7015: Instances containing the ignored marker are excluded, and instances containing face information are selected. Based on the face's position, area, and aspect ratio information, a score is given to select the optimal face. The instance corresponding to the optimal face is the optimal instance, and the centroid depth of this instance is the optimal depth of field for the current frame. S7016: If the optimal depth of field for the current frame is not empty, the optimal depth of field for the previous frame is read. If the optimal depth of field of the current frame is greater than the optimal depth of field of the previous frame (the dynamic threshold is calculated as 0.2 * the optimal depth of field of the previous frame), then all instance depths are traversed, and the one closest to the optimal depth of field of the previous frame is selected as the optimal depth of field of the current frame. In step S7017, instances containing ignore markers are excluded, and instances with facial information are traversed. If their depth of field is less than the optimal depth of field of the current frame (the dynamic threshold is calculated as 0.2 * the optimal depth of field of the current frame), then the instance is considered an important instance in the current frame and marked as needing to be output. In step S7018, a blank mask image with the same width and height as the original image is created. This mask image is a binary image, where 0 represents the background and 1 represents the foreground. All instance masks that need to be output are superimposed on the mask image, that is, the value of the position where 1 is in the instance mask is set to 1 at the corresponding position on the mask image, thus obtaining the foreground mask of the current frame.

8. The artificial intelligence-based method for preventing foreground target occlusion in bullet screens as described in claim 7, characterized in that, The specific process for selecting the optimal face in S7015 includes: traversing all face results in the current frame, and calculating the position score, area score, and width-to-height ratio for each face. The position score is calculated as follows: Let the coordinates of the face center point be cx, cy, the width of the current frame image be imgw, and the height be imgh. The position score calculation formula is score1 = 1 if abs(0.5 *imgw–cx) / imgw < 0.3 else 1 - abs(0.5 *imgw–cx) / imgw; The area score is calculated as follows: traversing all faces, taking the maximum area as max_face_area, and for each face, its area is face_area. The area score calculation formula is score2 = face_area / max_face_area; The width-to-height ratio is calculated as follows: Let the long side of the face be l, and the short side be s. The width-to-height ratio calculation formula is score3 = s / l; the face with the highest sum of the position score, area score, and width-to-height ratio is the optimal face.

9. The artificial intelligence-based method for preventing foreground target occlusion in bullet screens as described in claim 1, characterized in that, The temporal mask compensation algorithm in S70 includes the following steps: S7021, reading the output instance information of the previous frame, reading the instance information of the current frame, matching the two based on the centroid and bounding box, if the current frame instance can match the output instance of the previous frame, then outputting the instance mask, and marking the matched previous frame instance as matched; S7022, for the unmatched output instance of the previous frame, taking the intersection of it with the semantic segmentation and depth information results of the current frame, if the mask area after taking the intersection is greater than the area of ​​the previous frame instance * a preset threshold, then taking the intersection result as a new instance to be prepared for output; S7023, establishing a blank mask image with the same width and height as the original image, and superimposing all the instance masks that need to be output onto the mask image to obtain the temporal compensation mask.

10. The artificial intelligence-based method for preventing foreground target occlusion in bullet screens as described in claim 9, characterized in that, The matching calculation method in S7021 is as follows: First, calculate the center of each of the two instances. The calculation method is as follows: For each instance, take all points that are foreground and calculate the mean of x and y, which is the centroid. If the centroid position is not classified as foreground in the instance, then arrange all foreground points in order of ascending horizontal coordinate and ascending vertical coordinate into a one-dimensional vector, and take the middle point as the centroid. Calculate the Euclidean distance between the centroids of the two instances. If the distance is less than 0.25 times the height and less than 0.25 times the width of the previous frame instance, then the two instances are considered to be matched.

Images