Live broadcasting room content display method and device

Abstract

The invention discloses a live broadcast room content display method and device, and the method comprises the steps: obtaining the bullet screen information of a slide speech of a live broadcast scene, extracting the text information and score data of user evaluation from the bullet screen information, carrying out the normalization processing of the text information and score data, constructing an emotion evaluation model corresponding to the bullet screen information, and carrying out the emotion evaluation of the emotion evaluation model. The method comprises the following steps: acquiring image information of a live scene, identifying and analyzing the image information by adopting a mistakenly-intercepted slide identification algorithm to obtain a slide detection model, and constructing a lightweight code rate saving model according to an emotion evaluation model and the slide detection model, the live broadcast scene is subjected to feature low-rank reconstruction based on the lightweight code rate saving model so as to complete live broadcast room content display, the problem of intelligent screenshot of character close-up and the problem of intelligent screenshot of slides can be solved, the code rate is saved, the image quality of video live broadcast is enhanced, and the user experience is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video live broadcast, and particularly relates to a method and device for displaying content in a live broadcast room. Background Art

[0002] At present, the emergence of online live broadcast has driven the rapid development of the live broadcast industry, making live broadcast not only an important pillar of the cultural and entertainment industry, but also gaining a certain position in the fields of business, academics, etc. Live broadcasts such as business press conference live broadcasts, academic seminar live broadcasts or open class live broadcasts, which have characteristics such as serious content, long duration, a large amount of content, and lack of post-production, are conference-type live broadcasts. Image intelligent analysis for live broadcast scenarios will mainly target conference live broadcasts. Taking an academic seminar live broadcast as an example, a complete live broadcast conference may include a large number of people such as several hosts, leaders, speakers, etc., and several speech slides with a total of hundreds of pages, with a total duration of several hours or even several days. For viewers who cannot watch the conference-type live broadcast completely, it is relatively difficult to quickly grasp the live broadcast content. In order to take care of such viewers, the form of picture and text live broadcast has emerged. The picture and text live broadcast is carried out in the form of pictures and text, and the real-time dynamics are displayed in the form of pictures and text.

[0003] However, as a form of online live broadcast, the picture and text live broadcast can describe the main content of the live broadcast in text in the absence of video images, and at the same time, select pictures such as close-ups of the host's speech, group photos of guests, etc. to enhance the viewer's intuitive feeling of the live broadcast scene. At present, the production of picture and text live broadcast mainly relies on manual screenshotting and matching with text. However, in the face of long-duration and high-density live broadcast sources, using manual methods to capture live broadcast images not only requires additional labor costs, but also manual operations are prone to errors, and the efficiency is difficult to keep up with the speed of the live broadcast, and the screenshot effect is also difficult to be standardized and guaranteed. Therefore, there is an urgent need to provide a method and device for displaying content in a live broadcast room to solve the above-mentioned existing technical problems. Summary of the Invention

[0004] In view of this, the present invention provides a method and device for displaying content in a live broadcast room that improve the intelligent and accurate screenshotting of person close-ups and slides and the smoothness of the playback picture quality, and specifically adopts the following technical solutions to achieve.

[0005] In a first aspect, the present invention provides a method for displaying content in a live broadcast room, including the following steps:

[0006] Obtain the bullet screen information of the slide speech in the live broadcast scene, and extract the text information and scoring data evaluated by the user from the bullet screen information, where the text information includes negative words and adverbs of degree;

[0007] Perform normalization processing on the text information and scoring data, and construct an emotional evaluation model corresponding to the bullet screen information;

[0008] Obtain the image information of the live broadcast scene, and use the mis-cut slide recognition algorithm to identify and analyze the image information to obtain the slide detection model. Among them, the image information includes face images and slide images, and the mis-cut slide recognition algorithm includes ghost mis-cut recognition based on the transparency matrix and splicing mis-cut recognition based on window matching;

[0009] Construct a lightweight bitrate saving model according to the sentiment evaluation model and the slide detection model, and perform feature low-rank reconstruction on the live broadcast scene based on the lightweight bitrate saving model to complete the display of the content in the live broadcast room.

[0010] As an optimization of the above technical solution, obtain the bullet screen information of the slide speech in the live broadcast scene, and extract the text information and score data of the user evaluation from the bullet screen information, including:

[0011] Classify the sentiment words in the obtained text information into positive and negative. Among them, positive is set to value 1, and negative is set to -1. The sentiment value of each word in the text information is w j , and the negation word and its number before the obtained sentiment word s j are represented by -1 and k respectively, and calculate the influence degree of the negation word on the sentiment:

[0012] degree(s j ) = deg * -1 k (1)

[0013] Among them, in the calculation process, the adverb of degree before the sentiment word is set to w j , and the adverb of degree is represented by the already marked adverb of degree. The sentiment calculation expression of the influencing factor of the adverb of degree is:

[0014] degree(s j ) = degree(w j ) * (-1) k (2)

[0015] According to the analysis of the multi-sentiment word operation degree(ρ i ) expression of the text information of the live video is:

[0016]

[0017] As an optimization of the above technical solution, perform normalization processing on the text information and score data, and construct a sentiment evaluation model corresponding to the bullet screen information, including:

[0018]

[0019] Formula (3) is used to calculate the sentiment score of the comment statement, and expand the sentiment score of the user's bullet screen information during the live broadcast to the same range as the score:

[0020]

[0021] Among them, p max , p min represent the maximum and minimum sentiment score values in the text information calculated according to the model, i represents the i-th comment, and j represents the j-th sentiment word in the comment;

[0022] Based on the sentiment information in the user's bullet screen, strengthen the comprehensive analysis to obtain the final score, and the corresponding calculation expression is:

[0023] Score(i) = w1 * score i + w2 * degree(p i ) (5)

[0024] Among them, Score(i) represents the final score of the i-th live broadcast room, score i is the score given by the user to the live broadcast room, w1 is the weight value of the live broadcast room score, degree(p i ) is the sentiment score of the user's bullet screen comment, and w2 is the weight value of the user's bullet screen score.

[0025] As an optimization of the above technical solution, obtain the image information of the live broadcast scene, and use the mis - intercepted slide recognition algorithm to identify and analyze the image information to obtain the slide detection model, including:

[0026] Use the structural similarity SSIM as an index to measure the similarity of two images, and measure the image similarity from three quantities: brightness l, contrast c, and structure s. The expressions of the three quantities corresponding to SSIM are:

[0027]

[0028] Among them, x, y are two images to be compared, μ is the average pixel value, σ 2 is the covariance of x and y, and SSIM is calculated according to the pixel value range:

[0029] SSIM(x,y) = [l(x,y) α ·c(x,y) β ·s(x,y) γ (7)

[0030] Let α, β, γ be 1, and we can get:

[0031]

[0032] If pixels are regarded as three - dimensional points, the Euclidean distance between corresponding pixel points in two images can reflect the difference degree of pixels, and the average value of the distances between all pixel points reflects the difference degree of the images. The picture difference degree diff of the images is as follows:

[0033]

[0034] where x and y are images of the same size, n is the total number of pixels in x or y, r x 、g x 、b x are the red, green, and blue channel images of image x respectively, r y 、g y 、b y are the red, green, and blue channel images of image y respectively. Convert SSIM into a form describing the picture difference: ssim = 100(1 - SSIM).

[0035] As an optimization of the above - mentioned technical solution, for two images x and y whose difference degree is calculated by 3×3 segmentation, each image is horizontally and vertically divided into nine sub - blocks. The content expression of the sub - blocks is:

[0036] B x (i,j)=x[ih:(i + 1)h,jω:(j + 1)ω] (10)

[0037] where (i,j) represents the coordinates of the sub - block, h and ω are the height and width of the image, and x[a:b,c:d] represents the sub - image in the range of the a - th to b - th rows and the c - th to d - th columns of image x. Calculate the picture difference degree for each pair of corresponding sub - blocks in x and y to obtain the picture difference degree matrix of the two images:

[0038] D(x,y,3,3)=[diff(B x (i,j),B y (i,j))],i,j∈[1,3] (11)

[0039] Analyze the situation of matrix D when analyzing the picture change:

[0040] Case 1: When neither the slide content nor the picture - in - picture content changes, all elements in matrix D are relatively small;

[0041] Case 2: When the slide content does not change while the picture - in - picture content changes, the difference degree of the position of the picture - in - picture corresponding to matrix element D 3,3 becomes larger;

[0042] Case 3: When the slide content changes while the picture - in - picture content does not change, several elements in the matrix except D 3,3 become larger;

[0043] Case 4: When the content of the slide and the content of the picture-in-picture change, at least two elements in matrix D are relatively large. The indicators corresponding to matrix D are:

[0044]

[0045] where n is the number of matrix elements, μ(i, j) and σ 2 (i, j) represent the average value and variance of the remaining matrix elements after excluding element D i,j ;

[0046] After calculating the exclusion variance for all elements in the matrix, the element with the smallest exclusion variance is the element with the highest degree of dispersion. The corresponding exclusion mean is the expression of the picture difference degree after being corrected by picture segmentation:

[0047]

[0048] As an optimization of the above technical solution, the ghosting and misclipping recognition based on the transparency matrix includes:

[0049] It is preset that there are three pictures involved in the ghosting and misclipping problems, namely the old slide page X, the new slide page Y, and the misclipped picture E generated by overlapping. The overlapping method of the images satisfies:

[0050] E = θX + (1 - θ)Y (14)

[0051] where θ is the transparency, and its value range is (0, 1). The process of gradually changing from 0 to 1 is the picture gradual change process; when the misclipped picture is generated, the corresponding θ is unknown, but by transforming formula (14), we can get:

[0052]

[0053] where A is a matrix with the same shape as X and Y, and each element represents the transparency of the corresponding position pixel. Substituting formula (14) into formula (15), we get:

[0054]

[0055] By comparing the variance of all elements in matrix A in formula (15) with the preset threshold t, it is judged whether a picture is composed of the superposition of the other two pictures. The expression for judging whether the image is overlapped is:

[0056]

[0057] As an optimization of the above technical solution, the splicing misclipping recognition based on window matching includes:

[0058] There are three pictures involved in the preset splicing and truncation problem, namely the old slide page X, the new slide page Y, and the truncated picture E formed by splicing. The part from X in the truncated picture E is E X , and the part from Y is E Y , set a sliding window W on X X , each time the window slides, look for whether there is an area in E that is the same as the window content;

[0059] If a matching area W is found in E E , through W X and W E coordinates calculate the moving direction and distance when the picture X flips the page, use an image filtering algorithm based on the picture difference degree, and use W X as the filtering matrix to move pixel by pixel on E and calculate the picture difference degree. The corresponding expression is:

[0060] DIFF(i,j)=diff(W x ,E[i,i+h,j:j+ω]) (18)

[0061] Among them, DIFF is the filtering result. (i,j) is the pixel coordinate, h and ω are the height and width of the sliding window. E[a:b,c:d] represents the sub-picture in the range of the a~bth row and the c~dth column of the image E. Each pixel in DIFF represents the picture difference degree between W x at the current position and E. The position where the minimum value in DIFF is located is the most matching value of W x ;

[0062] Before calculating the formula (18), it is necessary to judge the content of W x , and judge by calculating the pixel variance of W x :

[0063]

[0064] As the optimization of the above technical solution, compare the new screenshot with the previous several screenshots to calculate the difference degree to judge whether it is a repeated screenshot, including:

[0065] Step 1: Obtain a key frame A from the video source, calculate the picture difference degree diff with the previous key frame B, and set the previous key frame as A for the next round of calculation;

[0066] Step 2: If it is in the restricted state currently, enter Step 3, otherwise enter Step 4;

[0067] Step 3: If the diff values of num_active consecutive occurrences are all less than the activation threshold t_active, it indicates that a continuous static image appears in the picture. At this time, enter the activation state and return to Step 2; otherwise, return to Step 1.

[0068] Step 4: If the diff is greater than the screenshot threshold t_shot, save the current key frame image to a file to complete a screenshot; otherwise, return to Step 1.

[0069] Step 5: Execute the misintercepted slide correction mechanism for the just-captured picture, including identifying overlapping misinterceptions, splicing misinterceptions, and duplicate misinterceptions. If it is determined to be a misintercepted picture, delete it.

[0070] Step 6: If the diff values of num_inactive consecutive occurrences are all greater than t_shot, it indicates that the picture is changing rapidly continuously. At this time, num_inactive non-slide contents are misintercepted. After deleting all of them, enter the idle state and return to Step 1; otherwise, directly return to Step 1.

[0071] As an optimization of the above technical solution, a lightweight bitrate saving model is constructed according to the emotion evaluation model and the slide detection model, including:

[0072] Adopt the lightweight bitrate saving model LRBSN based on feature low-rank reconstruction to guide the video encoder to output a video stream with a lower bitrate. The LRBSN model uses information distillation and channel splitting to achieve lightweight and high-performance inference. Among them, the LRBSN model includes a pixel processing module and a splitting conversion module. The working process of the LRBSN model includes:

[0073] The first step: The pixel processing module downsamples the image to reduce the scale of the image, and the scale is converted into the number of channels. The pixel processing module rearranges the feature map of H×r×W×r×C into the dimension of H×W×C×r 2 where r is the reduction factor, and the expressions for the changes in the width, height, and number of channels of the feature map are:

[0074]

[0075] The second step: Input the feature map downsampled by the pixel processing module into a 3×3 convolutional layer for preliminary feature extraction. The channels multiplied in the first step will be fused, and a ReLU activation function is connected after convolution.

[0076] The third step: Input the features into the splitting conversion module for deep feature extraction based on information distillation and channel separation;

[0077] The fourth step: Add the output features of the splitting conversion module to its input, and add them to the output features of the second step;

[0078] Step 5: Feed the added features into a 3×3 convolutional layer for feature extraction again;

[0079] Step 6: Input the output features of Step 5 into the pixel processing module for upsampling to restore the feature map to the input scale.

[0080] In a second aspect, the present invention also provides a device for displaying live broadcast content, including:

[0081] A data acquisition unit, configured to acquire the bullet screen information of the slide presentation in the live broadcast scene, and extract the text information and score data of the user evaluation from the bullet screen information, where the text information includes negative words and adverbs of degree;

[0082] A model construction unit, configured to perform normalization processing on the text information and score data, and construct a sentiment evaluation model corresponding to the bullet screen information;

[0083] An information analysis unit, configured to acquire the image information of the live broadcast scene, and perform recognition and analysis on the image information by using a mis - cropped slide recognition algorithm to obtain a slide detection model, where the image information includes face images and slide images, and the mis - cropped slide recognition algorithm includes ghost mis - cropped recognition based on a transparency matrix and stitching mis - cropped recognition based on window matching;

[0084] A video processing unit, configured to construct a lightweight bitrate saving model according to the sentiment evaluation model and the slide detection model, and perform feature low - rank reconstruction on the live broadcast scene based on the lightweight bitrate saving model to complete the display of the live broadcast content.

[0085] The present invention provides a method and device for displaying live broadcast content. By acquiring the bullet screen information of the slide presentation in the live broadcast scene, extracting the text information and score data of the user evaluation from the bullet screen information, performing normalization processing on the text information and score data, constructing a sentiment evaluation model corresponding to the bullet screen information, acquiring the image information of the live broadcast scene, performing recognition and analysis on the image information by using a mis - cropped slide recognition algorithm to obtain a slide detection model, constructing a lightweight bitrate saving model according to the sentiment evaluation model and the slide detection model, and performing feature low - rank reconstruction on the live broadcast scene based on the lightweight bitrate saving model to complete the display of the live broadcast content, it can solve the problems of intelligent screenshot of character close - ups and intelligent screenshot of slides, save the bitrate, enhance the video quality of the live broadcast, and improve the user experience. Description of the Drawings

[0086] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0087] Figure 1 It is a flowchart of the method for displaying the content of the live broadcast room of the present invention;

[0088] Figure 2 It is a structural block diagram of the device for displaying the content of the live broadcast room of the present invention. Specific embodiments

[0089] The following details the embodiments of the present invention. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0090] Refer to Figure 1 , the present invention provides a method for displaying the content of a live broadcast room, including the following steps:

[0091] S1: Obtain the barrage information of the slide speech in the live broadcast scene, and extract the text information and scoring data of the user evaluation from the barrage information. Among them, the text information includes negative words and degree adverbs;

[0092] S2: Perform normalization processing on the text information and scoring data, and construct an emotional evaluation model corresponding to the barrage information;

[0093] S3: Obtain the image information of the live broadcast scene, and use the mis - cut slide recognition algorithm to perform recognition and analysis on the image information to obtain a slide detection model. Among them, the image information includes face images and slide images, and the mis - cut slide recognition algorithm includes ghost mis - cut recognition based on the transparency matrix and splicing mis - cut recognition based on window matching;

[0094] S4: Construct a lightweight bitrate saving model according to the emotional evaluation model and the slide detection model, and perform feature low - rank reconstruction on the live broadcast scene based on the lightweight bitrate saving model to complete the display of the content of the live broadcast room.

[0095] In this embodiment, relying solely on sentiment quantification for the text of user evaluations in bullet comments is insufficient because it may contain words with various other attributes, which can have a significant impact on the user's sentiment polarity. Therefore, negative words and adverbs of degree are considered when quantifying the sentiment of user evaluations. In live broadcast scenarios such as academic seminars and business press conferences, keynote speeches often appear, and there are usually accompanying slides. As a dynamic combination of text and pictures, the slides contain rich information and are a powerful supplement to the keynote speech. The content and format of the slides vary widely and can include various texts, word arts, pictures, and dynamic effects, etc. The content layout also varies according to the author's preference. Therefore, it is difficult to determine whether a picture is a slide from its content or format. Obtain the bullet comment information of the slide presentation in the live broadcast scenario, and extract the text information and scoring data of user evaluations from the bullet comment information, including: classify the sentiment words in the obtained text information into positive and negative, where positive is set to value 1 and negative is set to -1, and the sentiment value of each word in the text information is w j , the obtained sentiment word s j , the negative word and its number before the obtained sentiment word s are represented by -1 and k respectively, and calculate the influence degree of the negative word on the sentiment:

[0096] degree(s j ) = deg * -1 k (1)

[0097] Among them, in the calculation process, the adverb of degree before the sentiment word is set as w j , the adverb of degree is represented by the pre-labeled adverb of degree, and the sentiment calculation expression of the influencing factor of the adverb of degree is:

[0098] degree(s j ) = degree(w j ) * (-1) k (2)

[0099] According to the analysis of the text information of the live video, the multi-sentiment word operation degree(ρ i ) expression is:

[0100]

[0101] As an optimization of the above technical solution, normalize the text information and scoring data, and construct a sentiment evaluation model corresponding to the bullet comment information, including:

[0102] The normalization expression is

[0103] Formula (3) is used to calculate the sentiment score of the comment statement, and expand the sentiment score of the bullet comment information of the user during the live broadcast process to the same range as the score:

[0104]

[0105] Among them, p max and p min represent the maximum and minimum sentiment score values in the text information calculated according to the model, i represents the i-th comment, and j represents the j-th sentiment word in the comment;

[0106] Based on the sentiment information in the user's bullet screen, through enhanced comprehensive analysis, the final score is obtained, and the corresponding calculation expression is:

[0107] Score(i) = w1 * score i + w2 * degree(p i ) (5)

[0108] Among them, Score(i) represents the final score of the i-th live broadcast room, score i is the score given by the user to the live broadcast room, w1 is the weight value of the live broadcast room score, and degree(p i ) is the sentiment score of the user's bullet screen comment, and w2 is the weight value of the user's bullet screen score.

[0109] It should be noted that to obtain the image information of the live broadcast scene, the misintercepted slide recognition algorithm is used to identify and analyze the image information to obtain the slide detection model, including:

[0110] The structural similarity SSIM is used as an index to measure the similarity of two images. The similarity of the images is measured from three quantities: brightness l, contrast c, and structure s. The expressions for the three quantities corresponding to SSIM are:

[0111]

[0112] Among them, x and y are two images to be compared, μ is the mean of pixel values, and σ 2 is the covariance of x and y. SSIM is calculated according to the pixel value range:

[0113] SSIM(x, y) = [l(x, y) α · c(x, y) β · s(x, y) γ (7)

[0114] Let α, β, and γ be 1, and we get:

[0115]

[0116] If pixels are regarded as three-dimensional points, the Euclidean distance between corresponding pixel points in two images can reflect the difference degree of pixels, and the average value of the distances between all pixel points reflects the difference degree of the images. The picture difference degree diff of the images is as follows:

[0117]

[0118] where x and y are images of the same size, n is the total number of pixels in x or y, and r x 、g x 、b x are the red, green, and blue channel images of image x respectively, and r y 、g y 、b y are the red, green, and blue channel images of image y respectively. Convert SSIM to a form describing the picture difference: ssim = 100(1 - SSIM).

[0119] The picture difference degree is an index used to measure the difference degree between two pictures, and the picture similarity can also be used to describe this degree. From different perspectives such as the color distribution, texture structure, or pixel value difference of the images, the difference degree of the images can be analyzed. Considering from the perspective of color distribution, the color histogram of the images can be statistically calculated and compared as the difference degree. The slide detection algorithm based on the picture difference degree is as follows: when diff is greater than the threshold, the slide has definitely changed and can be screenshot; when diff is less than the threshold, if the overall diff of the picture is also less than the threshold, it means the slide has not changed and there is no need to screenshot; otherwise, a judgment needs to be made based on the position of the maximum value in the difference degree matrix. By obtaining the bullet screen information of the slide presentation in the live broadcast scene, extracting the text information and scoring data of the user evaluation from the bullet screen information, normalizing the text information and scoring data, constructing an emotional evaluation model corresponding to the bullet screen information, obtaining the image information of the live broadcast scene, using the mis - intercepted slide recognition algorithm to identify and analyze the image information to obtain the slide detection model, constructing a lightweight bitrate - saving model based on the emotional evaluation model and the slide detection model, and performing feature low - rank reconstruction on the live broadcast scene based on the lightweight bitrate - saving model to complete the content display of the live broadcast room, the problems of intelligent screenshot of character close - ups and intelligent screenshot of slides can be solved, the bitrate is saved, the picture quality of the video live broadcast is enhanced, and the user experience is improved.

[0120] Optionally, for two images x and y whose difference degree is calculated by 3×3 segmentation, each image is horizontally and vertically divided into nine sub - blocks, and the content expression of the sub - block is:

[0121] B x (i,j) = x[ih:(i + 1)h,jω:(j + 1)ω] (10)

[0122] Among them, (i, j) represents the coordinates of the sub-block, h and ω are the height and width of the image, and x[a:b, c:d] represents the sub-image in the range of the a-th to b-th rows and the c-th to d-th columns of the image x. The picture difference matrix of the two images is obtained by calculating the picture difference degree for each pair of corresponding sub-blocks in x and y:

[0123] D(x, y, 3, 3) = [diff(B x (i, j), B y (i, j))], i, j ∈ [1, 3] (11)

[0124] Analyze the situation of matrix D when the picture changes:

[0125] Case 1: When neither the slide content nor the picture-in-picture content changes, all elements in matrix D are relatively small;

[0126] Case 2: When the slide content does not change while the picture-in-picture content changes, the difference degree of the corresponding picture-in-picture position in matrix element D 3,3 becomes larger;

[0127] Case 3: When the slide content changes while the picture-in-picture content does not change, several elements in the matrix except D 3,3 will become larger;

[0128] Case 4: When both the slide content and the picture-in-picture content change, at least two elements in matrix D are relatively large. The index corresponding to matrix D is:

[0129]

[0130] Among them, n is the number of matrix elements, μ(i, j), σ 2 (i, j) represents the average value and variance of the remaining matrix elements after excluding element D i,j ;

[0131] After calculating the excluded variance for all elements in the matrix, the element with the smallest excluded variance is the element with the highest degree of dispersion. The corresponding excluded mean value is the expression of the picture difference degree after picture segmentation correction:

[0132]

[0133] In this embodiment, the ghosting misinterception recognition based on the transparency matrix includes:

[0134] It is preset that there are three pictures involved in the ghosting misinterception problem, namely the old slide page X, the new slide page Y, and the misintercepted picture E generated by overlapping. The overlapping method of the images satisfies:

[0135] E = θX + (1 - θ)Y (14)

[0136] Among them, θ is the transparency, and its value range is (0, 1). The process of gradually changing from 0 to 1 is the process of the picture fading. When the mis - truncated picture is generated, the corresponding θ cannot be known, but by transforming formula (14), we can get:

[0137]

[0138] Among them, A is a matrix with the same shape as X and Y, and each element represents the transparency of the pixel at the corresponding position. Substituting formula (14) into formula (15), we get:

[0139]

[0140] By calculating the variance of all elements of matrix A in formula (15) and comparing it with the preset threshold t, it is determined whether a picture is composed of the superposition of two other pictures. The expression for determining whether an image is overlapped is:

[0141]

[0142] When actually calculating formula (16), the situation of division by zero needs to be noted. When a non - zero number is divided by zero, the result should be set to ±255 according to the sign; when zero is divided by zero, since its result can be any value and has no reference significance, it should be excluded and not participate in the calculation of the variance. When E is an overlapping image, the variance of A is extremely small; otherwise, it is not. Based on this, it can be determined whether an image is an overlapping image.

[0143] It should be noted that the recognition of splicing mis - truncation based on window matching includes:

[0144] There are three pictures involved in the preset splicing mis - truncation problem, namely the old slide page X, the new slide page Y, and the mis - truncated picture E formed by splicing. The part of E from X is E X , and the part from Y is E Y , set a sliding window W on X X , and each time the window slides, look for whether there is an area in E with the same content as the window content;

[0145] If a matching area W E is found in E, through W X and W E coordinates, calculate the moving direction and distance of the picture X when turning the page. Use the image filtering algorithm based on the picture difference degree, and use W X as the filtering matrix to move pixel - by - pixel on E and calculate the difference degree of the picture. The corresponding expression is:

[0146] DIFF(i,j) = diff(W x ,E[i,i + h,j:j+ω]) (18)

[0147] Among them, DIFF is the filtering result. (i, j) is the pixel coordinate, and h and ω are the height and width of the sliding window. E[a:b, c:d] represents the sub-image in the range of the a-th to b-th rows and the c-th to d-th columns of the image E. Each pixel in DIFF represents the x degree of difference between the current position and the screen of E. The position where the minimum value in DIFF is located is x the most matching value;

[0148] Before calculating the formula (18), it is necessary to judge the content of x W. By calculating the pixel variance of x W, the judgment is made:

[0149]

[0150] When it is greater than the preset variance threshold, it can be considered that the content of x W has a certain complexity and can be used for screen matching; when no area in E is found to be the same as the content of x W, a non-matching problem will occur. At this time, the minimum value in DIFF is also relatively large, and the matching result is meaningless. By setting a threshold for the minimum value of DIFF, such matching window pairs can be excluded to solve the non-matching problem. When there are similar or even identical non-corresponding parts in X and Y, a mis-matching problem will occur. When turning the page, this part of X in E moves out of the screen and the similar part of Y is moved into the screen. At this time, x W will match the similar part in Y, and the calculated page-turning direction and distance are incorrect, which is obviously not the expected matching result. Picture-in-picture will also cause mis-matching situations, that is, the picture-in-picture in X matches the picture-in-picture in E. The key to truly solving the mis-matching problem with the sliding window is to obtain multiple pairs of matching windows by setting appropriate sliding ranges and step sizes. Each pair of windows can calculate a displacement vector. If the screen E is indeed displaced from X, most of these displacement vectors should be the same.

[0151] Group all the mutually equal displacement vectors obtained as a group, and use the number of vectors in the group as the histogram height. By statistically analyzing all displacement vectors, a displacement vector histogram can be obtained. When E is a non-stitched image, it has almost no matching part with X. When the number of final matching windows is small (the ratio to the total number of sliding windows is less than the preset threshold t w ), it can be considered that E is a non-stitched image. In actual applications, for higher requirements on the weak team determination accuracy, the displacement vector between (Y, E) can be calculated and compared with the displacement vector of (X, E) to make a determination.

[0152] Optionally, compare the new screenshot with the previous several screenshots to calculate the difference degree to determine whether it is a duplicate screenshot, including:

[0153] Step 1: Obtain a key frame A from the video source, calculate the frame difference diff with the previous key frame B, and set the previous key frame as A for the next round of calculation;

[0154] Step 2: If it is in the restricted state currently, go to Step 3; otherwise, go to Step 4;

[0155] Step 3: If num_active consecutive diff values are all less than the activation threshold t_active, it indicates that there is a continuous static image in the frame. At this time, enter the activation state and return to Step 2; otherwise, return to Step 1;

[0156] Step 4: If diff is greater than the screenshot threshold t_shot, save the current key frame image to a file to complete a screenshot; otherwise, return to Step 1;

[0157] Step 5: Execute the mis - screenshot slide correction mechanism for the just - captured image, including identifying overlapping mis - screenshots, stitching mis - screenshots, and duplicate mis - screenshots. If it is determined to be a mis - screenshot image, delete it;

[0158] Step 6: If num_inactive consecutive diff values are all greater than t_shot, it indicates that the frame is changing rapidly continuously. At this time, num_inactive non - slide content images are mis - captured. After deleting all of them, enter the idle state and return to Step 1; otherwise, directly return to Step 1.

[0159] Among them, a lightweight bitrate saving model is constructed based on the sentiment evaluation model and the slide detection model, including:

[0160] Adopt the lightweight bitrate saving model LRBSN based on feature low - rank reconstruction to guide the video encoder to output a video stream with a lower bitrate. The LRBSN model uses information distillation and channel splitting to achieve lightweight and high - performance inference. Among them, the LRBSN model includes a pixel processing module and a splitting conversion module. The working process of the LRBSN model includes:

[0161] The first step: The pixel processing module downsamples the image to reduce the scale of the image. The scale is converted to the number of channels. The pixel processing module rearranges the feature map of H×r×W×r×C into the dimension of H×W×C×r 2 where r is the reduction factor. The expressions for the changes in the width, height, and number of channels of the feature map are:

[0162]

[0163] Step 2: Input the feature map after downsampling by the pixel processing module into a 3×3 convolutional layer for preliminary feature extraction. The number of channels in Step 1 will be fused, and after convolution, a ReLU activation function is connected.

[0164] Step 3: Input the features into the splitting and transformation module for deep feature extraction based on information distillation and channel separation.

[0165] Step 4: Add the output features of the splitting and transformation module to its input, and then add the result to the output features of Step 2.

[0166] Step 5: Feed the added features into a 3×3 convolutional layer for feature extraction again.

[0167] Step 6: Input the output features of Step 5 into the pixel processing module for upsampling to restore the feature map to the input scale.

[0168] Specifically, the misinterception of the slide detection algorithm based on the picture difference mainly occurs during the page-turning process of the slides. Common page-turning processes of the slides include the fade-in type and the displacement type. During the fade-in type of page-turning, the picture gradually changes from the old slide to the new slide. At this time, the picture is actually the overlap of the old and new slides. The slide detection algorithm based on the picture difference will detect this overlapping picture as a new slide, which is called the ghosting misinterception. During the displacement type of page-turning, the old slide moves out of the picture in a certain direction, and at the same time, the new slide moves into the picture from outside the picture. At this time, the picture is actually the splicing of a part of the old and new slides.

[0169] Refer to Figure 2 , the present invention also provides a live broadcast room content display device, including:

[0170] A data acquisition unit, configured to acquire the barrage information of the slide presentation in the live broadcast scene, and extract the text information and score data of the user evaluation from the barrage information, wherein the text information includes negative words and adverbs of degree;

[0171] A model construction unit, configured to perform normalization processing on the text information and score data, and construct a sentiment evaluation model corresponding to the barrage information;

[0172] An information analysis unit, configured to acquire the image information of the live broadcast scene, and use the misintercepted slide recognition algorithm to perform recognition and analysis on the image information to obtain a slide detection model, wherein the image information includes face images and slide images, and the misintercepted slide recognition algorithm includes ghosting misintercept recognition based on the transparency matrix and splicing misintercept recognition based on window matching;

[0173] A video processing unit is configured to construct a lightweight bitrate saving model based on an emotion evaluation model and a slide detection model, and perform feature low-rank reconstruction on a live broadcast scenario based on the lightweight bitrate saving model to complete the display of the content in the live broadcast room.

[0174] In the actual application scenario of the video live broadcast system in this embodiment, limited by expensive network bandwidth and cost control, users are sensitive to the video bitrate. At the same time, the network speed of users often cannot bear high-bitrate videos. As the super-resolution multiple exponent of the video bitrate increases, the network bandwidth cost also increases accordingly. Therefore, in the application of super-resolution in the video live broadcast system, the saving of bitrate cost is also a factor that needs to be considered. The rank of a matrix is the number of linearly independent column (or row) vectors at most. The rank of an image indicates the richness of the information contained in the image. The more information the image contains, the higher the rank; conversely, the rank is smaller. Noise information generated through such means as poor lighting environment, quantization of image spatial encoding and decoding, and quantization of video temporal encoding and decoding will increase the rank of the image. Therefore, the smaller the rank of the image, the less noise information in the image, thus improving the image quality and viewing effect of the video live broadcast.

[0175] In all the examples shown and described here, any specific value should be construed as merely exemplary, not as a limitation. Therefore, other examples of the exemplary embodiments may have different values.

[0176] It should be noted that similar reference numerals and letters denote similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0177] The above-described embodiments merely represent several implementation manners of the present invention. Their descriptions are relatively specific and detailed, but should not be construed as limiting the scope of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention.

Claims

1. A method for displaying content in a live broadcast room, characterized in that, The steps include: Obtain the bullet screen information of the slide presentation in the live broadcast scenario, and extract the text information and rating data of user evaluations from the bullet screen information. Among them, the text information includes negative words and adverbs of degree; Perform normalization processing on the text information and rating data, and construct an emotional evaluation model corresponding to the bullet screen information; Obtain the image information of the live broadcast scenario, and use the mis-cut slide recognition algorithm to perform recognition and analysis on the image information to obtain a slide detection model. Among them, the image information includes face images and slide images, and the mis-cut slide recognition algorithm includes ghost mis-cut recognition based on the transparency matrix and splicing mis-cut recognition based on window matching; Construct a lightweight bitrate saving model according to the emotional evaluation model and the slide detection model, and perform feature low-rank reconstruction on the live broadcast scenario based on the lightweight bitrate saving model to complete the display of the live broadcast room content.

2. The method for displaying the content of the live broadcast room according to claim 1, wherein Obtain the bullet screen information of the slide presentation in the live broadcast scenario, and extract the text information and rating data of user evaluations from the bullet screen information, including: Classify the sentiment words in the obtained text information into positive and negative. Among them, the positive is set to value 1, and the negative is set to -1. The sentiment value of each word in the text information is w j , the obtained sentiment word s j The negation words before the sentiment word s and their number are represented by -1 and k respectively, and calculate the influence degree of the negation words on the sentiment: degree(s j ) = deg * -1 k (1) Among them, the intensifier before the sentiment word is set as w during the calculation process j , the sentiment adverb is represented by the intensifier that has been marked, and the sentiment calculation expression of the influencing factor of the intensifier is: degree(s j ) = degree(w j ) * (-1) k (2) Analysis of multi-emotional word operation degree (ρ i ) based on the text information of the live video, the expression is as follows:

3. The live broadcast room content display method according to claim 2, wherein Perform normalization processing on the text information and rating data, and construct an emotional evaluation model corresponding to the bullet screen information, including: The normalized expression is Formula (3) is used to calculate the emotional score of the comment statement, and the emotional score of the bullet screen information of the user during the live broadcast process is extended to the same range as the rating: where p max and p min represent the maximum and minimum sentiment score values in the text information calculated according to the model, i represents the i-th comment, and j represents the j-th sentiment word in the comment; Based on the comprehensive analysis of the emotional information in the user's bullet screen, the final score is obtained, and the corresponding calculation expression is: Score(i) = w1 * score i + w2 * degree(p i )(5) Among them, Score(i) represents the final score of the i-th live broadcast room, score i is the score given by the user to the live broadcast room, w1 is the weight value of the live broadcast room score, degree(p i ) is the emotional score of the user's bullet screen comments, and w2 is the weight value of the user's bullet screen score.

4. The live broadcast room content display method according to claim 1, characterized in that, Obtain the image information of the live broadcast scenario, and use the mis-cut slide recognition algorithm to perform recognition and analysis on the image information to obtain a slide detection model, including: Use the structural similarity SSIM as an index to measure the similarity of two images, and measure the image similarity from three quantities: brightness l, contrast c, and structure s. The expressions of the three quantities corresponding to SSIM are: Among them, x and y are two images to be compared, μ is the mean pixel value, and σ 2 is the covariance between x and y. The SSIM is calculated according to the pixel value range: SSIM(x,y) = [l(x,y) α ·c(x,y) β ·s(x,y) γ (7) Let α, β, and γ be 1 to obtain: If the pixels are regarded as a three-dimensional point, the Euclidean distance between the corresponding pixel points of the two images can reflect the difference degree of the pixels, and the average value of the distances between all pixel points reflects the difference degree of the images. The picture difference degree diff of the image is: Among them, x and y are images of the same size, n is the total number of pixels of x or y, r x , g x , b x are respectively the red, green, and blue channel images of image x, r y , g y , b y are respectively the red, green, and blue channel images of image y. Convert SSIM into a form describing the picture difference: ssim = 100(1 - SSIM).

5. The live broadcast room content display method according to claim 4, wherein, It also includes For two images x and y whose difference degree is calculated by 3×3 segmentation, each image is horizontally and vertically segmented into nine sub-blocks, and the content expression of the sub-blocks is: B x (i,j) = x[ih:(i + 1)h,jω:(j + 1)ω](10) Among them, (i,j) represents the coordinates of the sub-block, h and ω are the height and width of the image, and x[a:b,c:d] represents the sub-image in the range of the a~bth row and the c~dth column of the image x. Calculate the picture difference degree for each pair of corresponding sub-blocks in x and y to obtain the picture difference degree matrix of the two images: D(x,y,3,3) = [diff(B x (i,j), B y (i,j))], i, j ∈ [1, 3] (11) Analyze the situation of matrix D when the picture changes: Situation 1, when neither the slide content nor the picture-in-picture content changes, all elements in matrix D are small; Case 2: When the content of the slide remains unchanged while the content of the picture-in-picture changes, the difference degree of the corresponding picture-in-picture position becomes larger for matrix element D 3,3 The difference degree of the corresponding picture-in-picture position becomes larger; Case 3, when the content of the slide changes while the content of the picture-in-picture does not change, several elements in the matrix except D 3,3 will become larger; Situation 4, when the slide content and the picture-in-picture content change, at least two elements in matrix D are large, and the index corresponding to matrix D is: where n is the number of matrix elements, and μ(i, j) and σ 2 (i, j) represent the mean and variance of the remaining matrix elements after excluding element D i,j ; After calculating the exclusion variance for all elements in the matrix, the element with the smallest exclusion variance is the element with the highest degree of dispersion, and the corresponding exclusion mean is the expression of the picture difference degree after being corrected by picture segmentation:

6. The method for displaying live broadcast content according to claim 5, wherein The ghost mis-cut recognition based on the transparency matrix includes: There are three pictures involved in the ghosting misinterception problem, namely the old slide page X, the new slide page Y, and the misintercepted picture E generated by overlapping. The overlapping method of the images satisfies: E = θX+(1 - θ)Y(14) where θ is the transparency, and its value range is (0, 1). The process of gradually changing from 0 to 1 is the process of gradual change of the picture. When the misintercepted picture is generated, the corresponding θ cannot be known, but by transforming formula (14), we can get: where A is a matrix with the same shape as X and Y, and each element represents the transparency of the pixel at the corresponding position. Substituting formula (14) into formula (15), we get: By calculating the variance of all elements of matrix A in formula (15) and comparing it with the preset threshold t, it is judged whether a picture is composed of the superposition of the other two pictures. The expression for judging whether an image is overlapped is:

7. The method for displaying live broadcast content according to claim 5, wherein The splicing misinterception recognition based on window matching includes: There are three pictures involved in the preset splicing misinterception problem, namely the old slide page X, the new slide page Y, and the misintercepted picture E formed by splicing. The part from X in the misintercepted picture E is E X , and the part from Y is E Y , set a sliding window W on X X , and each time the window slides, look in area E to see if there is an area with the same content as the window content; If a matching area W is found in E E , through W X and W E calculate the moving direction and distance when the screen X flips. Use an image filtering algorithm based on the screen difference degree to move W X pixel by pixel on E as the filtering matrix and calculate the difference degree of the screen. The corresponding expression is: DIFF(i,j) = diff(W x , E[i, i + h, j:j + ω])(18) Among them, DIFF is the filtering result. (i, j) is the pixel coordinate, and h and ω are the height and width of the sliding window. E[a:b, c:d] represents the sub-image in the range of the a-th to b-th rows and the c-th to d-th columns of the image E. Each pixel in DIFF represents W x the degree of screen difference between the current position and E, and the position where the minimum value in DIFF is located is W x the most matching value; Before calculating the formula (18), it is necessary to judge the content of W x by calculating the pixel variance of W x for judgment:

8. The live broadcast room content display method according to claim 7, wherein Comparing the new screenshot with the previous several screenshots to calculate the difference degree to judge whether it is a repeated screenshot, including: Step 1: Obtain a key frame A from the video source, calculate the picture difference degree diff with the previous key frame B, and set the previous key frame as A for the next round of calculation; Step 2: If the current is in the restricted state, enter Step 3, otherwise enter Step 4; Step 3: If the diff values of num_active consecutive activation quantities are all less than the activation threshold t_active, it means that there is a continuous static picture in the picture. At this time, enter the activation state and return to Step 2, otherwise return to Step 1; Step 4: If diff is greater than the screenshot threshold t_shot, save the current key frame picture to the file to complete a screenshot, otherwise return to Step 1; Step 5: Execute the misintercepted slide correction mechanism for the just intercepted picture, including identifying overlapping misinterception, splicing misinterception and repeated misinterception. If it is determined to be a misintercepted picture, delete it; Step 6: If the diff values of num_inactive consecutive restricted quantities are all greater than t_shot, it means that the picture changes rapidly continuously. At this time, num_inactive non-slide contents are misintercepted. After deleting all of them, enter the idle state and return to Step 1, otherwise directly return to Step 1.

9. The method for displaying live broadcast content according to claim 1, wherein Construct a lightweight bitrate saving model according to the sentiment evaluation model and the slide detection model, including: Adopt the lightweight bitrate saving model LRBSN based on feature low-rank reconstruction to guide the video encoder to output a video stream with a lower bitrate. The LRBSN model uses information distillation and channel splitting to achieve lightweight and high-performance inference. Among them, the LRBSN model includes a pixel processing module and a splitting conversion module. The working process of the LRBSN model includes: Step 1: The pixel processing module downsamples the image to reduce the scale of the image. The scale is converted into the number of channels, and the pixel processing module rearranges the feature map of H×r×W×r×C into the dimension of H×W×C×r 2 , where r is the reduction factor, and the expressions for the changes in the width, height, and number of channels of the feature map are as follows: Step 2: Input the feature map downsampled by the pixel processing module into a 3×3 convolutional layer for preliminary feature extraction. The number of channels in Step 1 will be fused, and after convolution, a ReLU activation function is connected; Step 3: Input the features into the splitting conversion module for deep feature extraction based on information distillation and channel separation; Step 4: Add the output features of the splitting conversion module to its input, and add them to the output features of Step 2; Step 5: Feed the added features into a 3×3 convolutional layer for feature extraction again; Step 6: Input the output features of Step 5 into a pixel processing module for upsampling to restore the feature map to the input scale.

10. A live broadcast room content display device for the live broadcast room content display method according to any one of claims 1-9, characterized in that, Including: A data acquisition unit, configured to acquire the bullet screen information of the slide presentation in the live broadcast scenario, and extract the text information and scoring data of the user evaluation from the bullet screen information, where the text information includes negative words and adverbs of degree; A model construction unit, configured to perform normalization processing on the text information and scoring data, and construct a sentiment evaluation model corresponding to the bullet screen information; An information analysis unit, configured to acquire the image information of the live broadcast scenario, and perform recognition and analysis on the image information by using a mis-cut slide recognition algorithm to obtain a slide detection model, where the image information includes face images and slide images, and the mis-cut slide recognition algorithm includes ghost mis-cut recognition based on a transparency matrix and stitching mis-cut recognition based on window matching; A video processing unit, configured to construct a lightweight bitrate saving model according to the sentiment evaluation model and the slide detection model, and perform feature low-rank reconstruction on the live broadcast scenario based on the lightweight bitrate saving model to complete the display of the live broadcast room content.

Images