Image processing method and system for live mobile phone sub-screen

Abstract

The application relates to a kind of image processing method and system for live mobile phone secondary screen, method is executed by the secondary screen terminal processing device connected with mobile phone through wired interface, comprising: receiving video frame and converting write frame buffer;Determine the anchor picture area to be mirrored and the comment area to be forward;Horizontal flip and scaling are executed to anchor picture area, and straight-through scaling is executed to comment area, two ways are parallel, dynamic bandwidth allocation, mirror distortion compensation, attention avoidance and semantic rearrangement are simultaneously executed;Two-way results are spliced into single-frame results in which upper half is mirror image and lower half is forward;After time alignment, output display;The application realizes the coexistence of mirror image anchor picture and forward comment area in single frame, secondary screen terminal independent processing, mobile phone zero burden.Compared with the prior art, the application has the beneficial effects of frame partition routing, dynamic bandwidth allocation, mirror distortion compensation, attention avoidance and semantic rearrangement, controllable delay and high interaction efficiency.

Description

Technical Field

[0001] This invention relates to the field of live streaming peripheral hardware technology, and more specifically, to an image processing method and system for a mobile phone secondary screen used for live streaming. Background Technology

[0002] When live streaming on a mobile phone, the host wants to use the rear main camera to get better image quality, but the image from the rear main camera is mirrored from the host's perspective. If the text in the comment section is also mirrored, it will be unreadable. Most secondary screens on the market are simply copies of the phone's main screen. If the entire screen is mirrored, the text will be reversed. If the screen is not mirrored, the host will not see the mirror effect.

[0003] In the prior art, CN108848405A (Kugou) discloses a live image processing method: capturing a first image, and if mirroring is enabled, the entire image is horizontally flipped to obtain a second image. The sticker is superimposed on the second image for local display, and the sticker is superimposed on the first image for streaming upload. This solution solves the problem of sticker text flipping at the APP software layer by layer superposition, and relies on facial features to locate the sticker position. The limitations of this solution are: (1) the processing layer is the APP software layer, which requires the mobile phone CPU / GPU to participate in the calculation, increasing the burden on the mobile phone; (2) it relies on facial feature positioning, which is not suitable for non-face scenarios of UI rectangle ROI in the comment area; (3) it adopts the layer idea of ​​whole frame flipping + independent sticker superposition, which cannot be ported to the secondary screen hardware pipeline.

[0004] In addition, there are already mobile phone rear camera selfie display solutions on the market, which support full-frame left and right mirroring or 180-degree flipping, and scaling display after cropping black borders via Type-C direct connection. This solution achieves full-frame level mirroring / non-mirroring at the hardware peripheral level. The limitations of this solution are: (1) it only supports full-frame level mirroring / non-mirroring, and cannot achieve mirroring of the broadcast area and forward display of the comment area on the same screen within a single frame; (2) it does not have a dual-path architecture for in-frame partitioning, and the text in the comment area cannot be processed independently.

[0005] Publication No. CN116847116A (Beijing Dushi Technology) discloses a video processing system that supports switching between mirror mode and extended mode, and splits the same / different video objects across multiple screens. In mirror mode, the same video object is transmitted to different displays, while in extended mode, different video objects are transmitted. The limitations of this solution are: (1) it is a cross-device splitting, not a single-screen intra-frame partitioning; (2) different video objects are displayed on different screens, and it is impossible to simultaneously present the mirrored anchor screen and the positive comment area on a single secondary screen.

[0006] Publication No. CN116320512B (DaYuHao) disclosed a dual-camera multi-screen integrated live streaming control system, in which the auxiliary processor merges the audience feedback (bullet comments) back into the live video and pushes it to the audience. The limitations of this solution are: (1) it is a server-side synthesis, pushing the stream to the audience side, which is different from the local display scenario on the host's side secondary screen; (2) the bullet comments are merged back into the live stream for the audience to watch, rather than for the host to monitor on the secondary screen.

[0007] Douyu and other live streaming platforms employ a left-center-right partitioning and blurring scheme to partition the live streaming screen at the APP level. The limitations of this scheme are: (1) it only involves screen-level processing and does not have an independent path for the ROI of the comment section; (2) it lacks intelligent processing capabilities such as super-resolution and semantic reordering.

[0008] In summary, the existing technology has the following technical problems: (1) The secondary screen displays the mirrored broadcaster's screen and the forward comment area at the same time; (2) No additional encoding / overlay burden is added on the mobile APP side, which is completed by the independent IC of the secondary screen; (3) The ROI position / size of the comment area can be dynamically adjusted by the broadcaster's button; (4) Ensure the quality of the broadcaster's screen under low bandwidth and intelligently degrade the quality of the comment area; (5) Eliminate mirror distortion when using the secondary screen from the side; (6) Perform semantic awareness rearrangement on the bullet comments, filter noise, and highlight important information; (7) All processing is completed in a pipeline manner on the secondary screen hardware side, and the delay is controllable. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide an image processing method for a mobile phone secondary screen for live streaming, and also to provide an image processing system for a mobile phone secondary screen for live streaming, in view of the above-mentioned defects of the prior art.

[0010] The technical solution adopted by this invention to solve its technical problem is: A method for image processing on a mobile phone secondary screen for live streaming is provided, wherein the processing is performed by a secondary screen processing device connected to the mobile phone via a wired interface, and the method includes the following steps: S1: Receive video frames input from the mobile phone via the wired interface, convert each frame into pixel data in RGB888 format and write it into the frame buffer; S2: Determine a first rectangular area to be mirrored and a second rectangular area to be displayed in the forward direction in the video frame, wherein the first rectangular area corresponds to the anchor screen area and the second rectangular area corresponds to the comment area, and the first rectangular area and the second rectangular area do not overlap and together constitute at least a part of the video frame; S3: Execute the first processing path for the first rectangular region and the second processing path for the second rectangular region, wherein: The first processing path includes mirror distortion compensation, horizontal flip operation and first scaling operation. The horizontal flip operation mirrors the horizontal coordinate of each pixel in the first rectangular area according to the mapping relationship x' = W - x, where W is the frame width. The second processing path performs a second scaling operation on the second rectangular region; the first processing path and the second processing path are executed in parallel; The transmission bandwidth between the first processing path and the second processing path is dynamically allocated according to the frame importance. Attention perception avoidance is performed on the second rectangular region. When the overlap between the anchor's facial attention hotspot and the second rectangular region exceeds a threshold, the position of the second rectangular region is adjusted. The comment text within the second rectangular area is semantically reordered, and the bullet comments are displayed in sorted order after being semantically scored. S4: Write the data of the first rectangular region after being processed by the first processing path into the first address range of the output frame buffer, and write the data of the second rectangular region after being processed by the second processing path into the second address range of the output frame buffer, so that the first rectangular region in the output frame is located above or to one side of the second rectangular region, forming a single frame splicing result with the upper half mirrored and the lower half forward within the same frame. S5: Perform time alignment on the single-frame splicing result, use a double-buffered ping-pong mechanism to calculate the delay amount according to the alignment delay formula, and ensure that the output frame rate is stable before outputting to the display controller to drive the secondary screen display.

[0011] The method of the present invention, wherein step S2, determining the second rectangular region includes the following steps: The Sobel operator is used to calculate the gradient magnitude of video frames to obtain the edge intensity map; The edge intensity map is adaptively binarized using the Otsu algorithm to obtain a binarized edge map; The binarized edge graph is labeled with connected components to obtain N candidate connected components; For each candidate connected component, use the formula: ; in, The density of the inner edge of the ROI. Position weights are based on vertical location; The candidate connected component with the highest score among those whose overall score exceeds the threshold is determined as the second rectangular region.

[0012] The method of the present invention, wherein dynamically allocating the transmission bandwidth between the first processing path and the second processing path according to frame importance includes: Frame importance calculation steps: For frame t, the frame importance is calculated using the following formula: ; in, For the first Frame importance, For the first The total number of pixels that change between the previous and current frames. For the first The sum of pixels at the edges of the entire frame. For the first The time interval between frames; For the normalization function, mapped to ; Dynamic bandwidth allocation steps: The total bandwidth is allocated to the first and second processing paths according to the following formula: ; in, Allocate bandwidth to the mirror path. This represents the total bandwidth of the interface. This represents the sum of the importance S(t) of each frame within the sliding window. Second processing path bandwidth allocation Formula used: = .

[0013] The method of the present invention, wherein the mirror distortion compensation includes: Obtain the ideal coordinates of four predefined feature points in the secondary display area. Actual coordinates under side-view distortion , where i = 1, 2, 3, 4; The objective function is constructed using the following formula: ; in, It is a 3×3 homography matrix. For projection function, It is the square of the L2 norm; Calculate the average reprojection error of four predefined feature points. ,when When the distortion is less than 1.5 pixels, the current distortion compensation is deemed valid and the corrected first rectangular region is output; when... When the distortion is not less than 1.5 pixels, the distortion parameters are recalibrated to update the homography matrix H, and the correction is re-performed based on the updated H until... Less than 1.5 pixels, thus forming an adaptive closed-loop correction for specular distortion compensation.

[0014] The method of the present invention, wherein the attentional perception avoidance includes: The video frame is input into the MobileNetV3-Small lightweight convolutional neural network, which outputs a heatmap of the anchor's facial attention, and determines the center coordinates and size of the attention heatmap. The intersection-union ratio of the attention hotspot and the second rectangular region is calculated using the following formula: ; Determine if the intersection-union ratio exceeds the overlap threshold. If it does, automatically move the comment section down 20px to avoid it; otherwise, do nothing. ; in, For intersection, union, and comparison, This is a function for calculating the area of ​​a rectangular region. As a hotspot for attention, For region of interest, This represents the area of ​​the intersection of the hot zone rectangle and the comment zone rectangle. This represents the area of ​​the union of the hot zone rectangle and the comment zone rectangle. The overlap threshold, To avoid the new vertical coordinates after the operation.

[0015] The method described in this invention, wherein the semantically aware reordering includes: Perform CRNN+OCR recognition on the comment text image within the second rectangular area to extract the text content of each bullet screen comment; The semantic score for each bullet comment is calculated using the following formula: ; in, For the first The overall score of the bullet comments To assess the relevance of bullet comments to the live stream content, The score is normalized based on the length of the bullet comments. Emotional polarity score: 1 for positive comments, 0.5 for neutral comments, and 0 for negative comments; The top 3 comments are highlighted and displayed in descending order of semantic comprehensive score, while comments with a semantic comprehensive score below the preset filtering threshold are filtered out and not displayed.

[0016] The method of the present invention, wherein the first processing pathway further includes super-resolution reconstruction: The user can select to turn the device on or off using a button, and the ESPCN network is used to upsample the first rectangular region by a factor of 2. The network structure of the ESPCN network is Conv3×3×64 → Conv3×3×32 → Conv3×3× → PixelShuffle The number of parameters is 43267, and the inference time per frame is 3.2ms.

[0017] The method of the present invention, wherein the time alignment includes: Using a double-buffered ping-pong mechanism, the alignment delay is calculated using the following formula: ; in, Let t be the current timestamp of the t-th frame. The timestamp for writing the t-th frame to the frame buffer. =16.7ms, ensuring a stable output frame rate of 60fps, with a 99th percentile latency of no more than 9.3ms.

[0018] The method described in this invention, wherein the partition parameter configuration includes: receiving external input signals from the up, down, left, and right buttons on the secondary screen side, adjusting the starting coordinates, width w, and height h of the second rectangular area in real time according to the external input signals, and taking effect immediately in the next frame.

[0019] An image processing device for live streaming secondary screen display, wherein the device is a secondary screen processing device, connected to a mobile phone via a wired interface, comprising: The image receiving module is used to receive video frames input by the mobile phone through the wired interface, convert each frame into pixel data in RGB888 format and write it into the frame buffer; The parameter configuration module is used to determine a first rectangular area to be mirrored and a second rectangular area to be displayed in the forward direction in the video frame, wherein the first rectangular area corresponds to the anchor screen area and the second rectangular area corresponds to the comment area, and the first rectangular area and the second rectangular area do not overlap and together constitute at least a part of the video frame; The partition routing processing module is used to execute a first processing path on the first rectangular region and a second processing path on the second rectangular region, wherein: The first processing path includes mirror distortion compensation, horizontal flip operation and first scaling operation. The horizontal flip operation mirrors the horizontal coordinate of each pixel in the first rectangular area according to the mapping relationship x' = W - x, where W is the frame width. The second processing path performs a second scaling operation on the second rectangular region; the first processing path and the second processing path are executed in parallel; The transmission bandwidth between the first processing path and the second processing path is dynamically allocated according to the frame importance. Attention perception avoidance is performed on the second rectangular region. When the overlap between the anchor's facial attention hotspot and the second rectangular region exceeds a threshold, the position of the second rectangular region is adjusted. The comment text within the second rectangular area is semantically reordered, and the bullet comments are displayed in sorted order after being semantically scored. The frame splicing module is used to write the data of the first rectangular region after being processed by the first processing path into the first address range of the output frame buffer, and to write the data of the second rectangular region after being processed by the second processing path into the second address range of the output frame buffer, so that the first rectangular region in the output frame is located above or to one side of the second rectangular region, forming a single frame splicing result in which the upper half is mirrored and the lower half is forward within the same frame. The time alignment module is used to perform time alignment on the single-frame splicing result. It uses a double-buffered ping-pong mechanism to calculate the delay amount according to the alignment delay formula, and ensures that the output frame rate is stable before outputting to the display controller to drive the secondary screen display.

[0020] The beneficial effects of this invention are as follows: 1. By using an intra-frame partitioning routing architecture, the anchor's screen area in the same video frame is horizontally flipped, and the comment area is passed through and scaled. The two processes are processed in parallel and then stitched together for output. This allows the anchor to simultaneously obtain a mirrored selfie (a mirror-like experience) and readable comment text on the secondary screen. This overcomes the dilemma of choosing between two options in existing technologies: if the entire frame is mirrored, the text is reversed; if the entire frame is not mirrored, there is no mirror-like experience. It also overcomes the limitation of multi-screen cross-device distribution solutions that cannot achieve same-frame partitioning display on a single secondary screen. 2. All image processing tasks (image access, partition routing, mirroring, scaling and stitching, time alignment, etc.) are deployed and completed independently in the secondary screen processing device. The mobile phone only outputs the original video frames through the wired interface and does not participate in any mirroring, scaling, or frame stitching operations. Compared with existing APP software layer solutions, this invention does not increase the computing burden on the mobile phone's CPU / GPU, does not consume mobile phone power, and does not affect the encoding efficiency of mobile phone live streaming. 3. External input signals from the broadcaster are received via the up, down, left, and right buttons on the secondary screen. The starting coordinates, width, and height of the rectangular area in the comment section are adjusted in real time, and the adjustments take effect immediately in the next frame, achieving seamless switching. This overcomes the limitations of existing technologies where the comment section position is fixed and the broadcaster cannot flexibly adjust the comment display area according to the live broadcast content (such as changes in product display position or facial position). 4. Frame importance is calculated based on three metrics: total pixel change, total edge pixels of the entire image, and inter-frame time interval. The transmission bandwidth of the mirror path and the direct path is dynamically allocated according to the importance ratio. When the scene changes drastically (e.g., the broadcaster moves quickly, scene changes), the mirror path receives higher bandwidth to ensure the broadcaster's image quality; when the scene is still (e.g., the broadcaster sits and explains), the direct path receives more bandwidth to ensure the comment section refresh speed; the bandwidth ratio between the maximum and minimum frames can reach more than 5.5 times, effectively adapting to fluctuations in the complexity of live broadcast images. 5. By collecting predefined feature points at the four corners of the screen and solving the 3×3 homography matrix using singular value decomposition, the distortion of the broadcaster's image is corrected before performing a horizontal flip operation, overcoming the limitation of existing technologies that cause perspective distortion and affect the viewing experience when the broadcaster looks at the secondary screen from the side. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a flowchart of an image processing method for a mobile phone secondary screen used for live streaming, according to a preferred embodiment of the present invention. Figure 2 This is a schematic diagram of intra-frame partitioning routing for an image processing method for a mobile phone secondary screen used in live streaming, according to a preferred embodiment of the present invention. Figure 3 This is a simplified flowchart of an image processing method for a mobile phone secondary screen used for live streaming, according to a preferred embodiment of the present invention. Figure 4 This is a schematic block diagram of an image processing system for a mobile phone secondary screen used for live streaming, according to a preferred embodiment of the present invention. Figure 5 This is a detailed block diagram of an image processing system for a mobile phone secondary screen used for live streaming, according to a preferred embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0023] A preferred embodiment of the image processing method for a mobile phone secondary screen used for live streaming according to the present invention, such as... Figure 1 As shown, see also Figure 2 and Figure 3 The process is executed by a secondary screen processing device connected to the mobile phone via a wired interface, and includes the following steps: S1: Receive video frames input from the mobile phone via the wired interface, convert each frame into pixel data in RGB888 format and write it into the frame buffer; S2: Determine the first rectangular area to be mirrored and the second rectangular area to be displayed in the forward direction in the video frame, wherein the first rectangular area corresponds to the anchor screen area and the second rectangular area corresponds to the comment area. The first rectangular area and the second rectangular area do not overlap and together constitute at least a part of the video frame. Preferably, the determination of the second rectangular region includes the following steps: The Sobel operator is used to calculate the gradient magnitude of video frames to obtain the edge intensity map; The edge intensity map is adaptively binarized using the Otsu algorithm to obtain a binarized edge map; The binarized edge graph is labeled with connected components to obtain N candidate connected components; For each candidate connected component, use the formula: ; in, This represents the edge density within the ROI (normalized value of the Sobel operator output). The position weight is based on the vertical position (the closer to the bottom of the screen, the higher the weight; the default weight for the bottom is 0.95). The candidate connected component with the highest overall score exceeding the threshold is identified as the second rectangular region (threshold). The score exceeded (The connected components are selected). The example is explained below: 1. Parameter definition; : The coordinates of the top-left corner of the ROI. Range of values. Typical values ​​are 40 and 1720.

[0024] ROI width and height. Value range Typical values ​​are 1000 and 660.

[0025] Edge density (Sobel output normalization). Value range Typical value: 0.65.

[0026] Position weight (higher at the bottom, lower at the top). Value range Typical value: 0.85.

[0027] Overall Score Range of values Typical value: 0.780.

[0028] : Score threshold; select if the score is exceeded. Value range Typical value: 0.6.

[0029] Number of connected components. Range of values. Typical value: 3.

[0030] 2. Numerical calculation example: The input frame resolution is 2560×1920. After Sobel edge detection, 3 connected components were detected. The scoring process is as follows: 1. Connected component CC1: coordinates (20, 1180, 580, 90), , , → Eliminate ( ).

[0031] 2. Connected component CC2: coordinates (40, 1140, 640, 120), , , → Select ( ).

[0032] 3. Connected component CC3: coordinates (1200, 1300, 300, 60), , , → Eliminate ( ).

[0033] CC2 was ultimately selected. ROI parameters: , , , .

[0034] The partition parameter configuration also includes: receiving external input signals from the up, down, left, and right buttons on the secondary screen side, adjusting the starting coordinates, width w, and height h of the second rectangular area in real time according to the external input signals, and taking effect immediately in the next frame.

[0035] S3: Execute the first processing path for the first rectangular region and the second processing path for the second rectangular region, wherein: The first processing path includes a horizontal flip operation and a first scaling operation. The horizontal flip operation mirrors the horizontal coordinate of each pixel in the first rectangular area according to the mapping relationship x' = W - x, where W is the frame width. The second processing path performs a second scaling operation on the second rectangular region; the first and second processing paths are executed in parallel. Dynamically allocate the transmission bandwidth between the first and second processing paths based on frame importance; specifically, this can be achieved by: Frame importance calculation steps: For frame t, the frame importance is calculated using the following formula: ; in, For the first Frame importance, For the first The total number of pixels that change between the previous and current frames. For the first The sum of pixels at the edges of the entire frame. For the first The time interval between frames; For the normalization function, mapped to ; Dynamic bandwidth allocation steps: The total bandwidth is allocated to the first and second processing paths according to the following formula: ; in, Allocate bandwidth to the mirror path. This represents the total bandwidth of the interface. This represents the sum of the importance S(t) of each frame within the sliding window. Second processing path bandwidth allocation Formula used: = .

[0036] The example is explained below: 1. Parameter definition: : No. Total change in frame pixels. Value range Typical value: 124500.

[0037] : No. The sum of pixels at the edges of the entire frame. Value range. Typical value: 89200.

[0038] : No. The time interval between frames. Value range Typical value: 16.2.

[0039] Normalization function: Range of values .

[0040] : No. Frame importance (weighted sum of three terms); value range Typical value 0.72 ( ).

[0041] Total interface bandwidth. Value range. Typical value: 5000.

[0042] : Mirror path bandwidth allocation; value range Typical value: 705.

[0043] : Bandwidth allocated to the direct path; Value range Typical value: 129.

[0044] The bandwidth of the direct path is calculated as BW_b(4) = BW_total BW_m(4)=5000 1438 = 3562 Mbps It is confirmed that the bandwidth allocated to the direct path of frame 4 is 3562 Mbps.

[0045] 2. Numerical calculation example (6 frames of complete calculation); Total interface bandwidth Mbps, the sliding window takes the 6 most recent frames.

[0046] Normalization process: For example, the values ​​for 6 frames are [85000, 120000, 45000, 180000, 25000, 95000]. , .

[0047] Frame 1: .

[0048] .

[0049] .

[0050] .

[0051] Frame 4 (maximum): .

[0052] .

[0053] .

[0054] .

[0055] Frame 5 (smallest): .

[0056] .

[0057] Bandwidth allocation: Denominator .

[0058] Mbps. (Note: The sliding window updates dynamically; the actual value is the sum of the last 6 frames.) Simplified example (window [0.524, 0.628, 0.298, 0.872, 0.154, 0.556]), ): Mbps.

[0059] Mbps.

[0060] Dynamic range: The bandwidth ratio between the maximum and minimum frames can reach more than 5.5 times, effectively adapting to the fluctuations in the complexity of live broadcast images.

[0061] Mirror distortion compensation is performed on the first rectangular region, and correction is performed using the homography matrix before the horizontal flip operation; Preferably, the mirror distortion compensation includes: Obtain the ideal coordinates of four predefined feature points in the secondary display area. Actual coordinates under side-view distortion , where i = 1, 2, 3, 4; The objective function is constructed using the following formula: ; in, It is a 3×3 homography matrix. For projection function, It is the square of the L2 norm; The 3×3 homography matrix H is solved by singular value decomposition, and the average reprojection error is calculated. ;when When the distortion is less than 1.5 pixels, the current distortion compensation is deemed valid and the corrected first rectangular region is output; when... When the distortion is not less than 1.5 pixels, the distortion parameters are recalibrated to update the homography matrix H, and the correction is re-performed based on the updated H until... Less than 1.5 pixels, thus forming an adaptive closed-loop correction for specular distortion compensation.

[0062] The example is explained below: 1. Parameter definition: : The ideal coordinates of the i-th feature point; that is, the theoretical pixel coordinates of the four corner points of the secondary screen display area when viewed directly on the screen.

[0063] : The actual coordinates of the i-th feature point; that is, the actual pixel coordinates of these four corner points captured by the system through sensors or algorithms when the anchor's side view causes mirror distortion.

[0064] : Index variable, with a value range of 1 to 4, representing four predefined feature points.

[0065] (Homography): Homography matrix; this is a... The transformation matrix is ​​the core objective of the entire algorithm, used to map the distorted image back to the ideal front view image.

[0066] Projection function; full name Projective Transformation, here referring to the transformation of homogeneous coordinates. Convert to non-homogeneous coordinates The operation.

[0067] : The square of the L2 norm (squared Euclidean distance). Used to calculate the error between the projected theoretical point and the actual point.

[0068] : Summation symbol; indicates that the reprojection errors of the four feature points are accumulated to calculate the total error.

[0069] (Phi): The anchor's side-view angle; the horizontal angle at which the anchor's head or eyes deviate from the screen normal, which is the physical cause of the distortion caused by specular reflection.

[0070] Distortion scaling factor; relative to viewpoint Related (e.g.) or ), which is used as a compensation factor when constructing the objective function.

[0071] (Distortion Error): Average reprojection error; this is a key indicator for determining whether compensation is effective, and the unit is pixels (px).

[0072] 2. Numerical Calculation Example To demonstrate the algorithm's solution process, assume the system has captured the following data: Initial parameter settings: Side view angle : ; Distortion scaling factor : ; Ideal coordinates of the four feature points :

[0073]

[0074]

[0075]

[0076] The actual coordinates of the four feature points (Displacement due to distortion):

[0077]

[0078]

[0079]

[0080] Calculation steps: Construct the objective function: The goal of the algorithm is to find the optimal homography matrix H that minimizes the reprojection error of all feature points: ; Note: Here The coordinates are obtained by transforming the ideal point using the H transformation.

[0081] Solve for the homography matrix H: The above equations are solved using Singular Value Decomposition (SVD) to obtain the optimal H matrix (assuming the solution results are as follows):

[0082] Calculate the average reprojection error : Using the obtained H matrix, the ideal coordinates are back-projected back to the actual coordinates, and the error between the ideal and actual acquired coordinates is calculated: Point 1 Error: ; Point 2 Error: ; Point 3 Error: ; Point 4 Error: ; Average error: ; Conclusion: The calculated average reprojection error It is greater than the set judgment threshold. Therefore, the system determines that the current distortion compensation is invalid and requires readjustment of parameters or prompts the user to calibrate the equipment.

[0083] Attention perception avoidance is performed on the second rectangular region. When the overlap between the anchor's facial attention hotspot and the second rectangular region exceeds a threshold, the position of the second rectangular region is adjusted. Preferably, attentional perception avoidance includes: The video frame is input into the MobileNetV3-Small lightweight convolutional neural network, which outputs a heatmap of the anchor's facial attention, and determines the center coordinates and size of the attention heatmap. Calculate the intersection-union ratio of the attention hotspot and the second rectangular region using the following formula: ; Determine if the intersection-union ratio exceeds the overlap threshold. If it does, automatically move the comment section down 20px to avoid it; otherwise, do nothing. ; in, For intersection, union, and comparison, This is a function for calculating the area of ​​a rectangular region. As a hotspot for attention, For region of interest, This represents the area of ​​the intersection of the hot zone rectangle and the comment zone rectangle. This represents the area of ​​the union of the hot zone rectangle and the comment zone rectangle.

[0084] The example is explained below: 1. Parameter definition: IOU (Intersection over Union): This is an indicator that measures the degree of overlap between two rectangular regions. Its value ranges from 0 to 1. In the algorithm, it is used to determine whether the comment section obscures the video hotspot that the user is interested in.

[0085] (Attention Region): The attention hotspot refers to the area in the video frame where the user's gaze is focused (such as a face or a central object), represented by a rectangular bounding box.

[0086] (Region of Interest): In this context, it refers to the rectangular boundary of the comment section / bullet screen display area.

[0087] (Threshold): The overlap threshold is a preset critical value (e.g., 0.3). If the calculated IOU is greater than this value, the system considers the occlusion to be severe and needs to trigger an avoidance action.

[0088] The original vertical coordinates (starting position of the Y-axis) of the comment section, in pixels (px).

[0089] : The new vertical coordinates after the avoidance operation.

[0090] 20: Avoidance step size, a fixed pixel distance the comment section moves vertically when an avoidance is triggered.

[0091] 2. Numerical calculation example: Suppose the system captures the following data: Attention hotspots Coordinates (380, 350, 100, 80) (i.e., top left corner x, y, width and height).

[0092] Comments section Coordinates (390, 360, 120, 60).

[0093] Overlap threshold Set to 0.3.

[0094] Calculation steps: Calculate the overlap region: The top left corner of the intersection X =

[0095] The top left corner of the intersection Y =

[0096] The bottom right corner of the intersection X =

[0097] Y = the bottom right corner of the intersection

[0098] Intersection width and height: width = High =

[0099] Intersection area =

[0100] Calculate the area of ​​the union:

[0101]

[0102] Union area =

[0103] Calculate IOU: ; Execution condition judgment: Judgment conditions: ? Result: True.

[0104] Calculate new coordinates : According to the piecewise function formula: ; Assuming the original coordinates The new coordinates after the avoidance are .

[0105] Conclusion: The system calculated the intersection-union ratio to be 0.551, which exceeds the threshold of 0.3. Therefore, it was determined that the comment area occluded too many hot spots, triggering the automatic avoidance mechanism, which moved the comment area down 20 pixels vertically.

[0106] Perform semantically aware rearrangement on the comment text within the second rectangular area, and then display the bullet comments in sorted order after semantic scoring. Preferably, semantically aware reordering includes: Perform CRNN+OCR recognition on the comment text image within the second rectangular area to extract the text content of each bullet screen comment; The semantic score for each bullet comment is calculated using the following formula: ; in, For the first The overall score of the bullet comments To assess the relevance of bullet comments to the live stream content, The score is normalized based on the length of the bullet comments. Emotional polarity score: 1 for positive comments, 0.5 for neutral comments, and 0 for negative comments; The top 3 comments are highlighted and displayed in descending order of semantic comprehensive score, while comments with a semantic comprehensive score below the preset filtering threshold are filtered out and not displayed.

[0107] The example is explained below: 1. Parameter definition: : No. The relevance of each bullet comment to the live stream content. Value range. Typical value: 0.85.

[0108] : No. Normalized score based on the length of each bullet comment. Value range. Typical value: 0.88.

[0109] Sentiment polarity score (positive=1, neutral=0.5, negative=0). Range of values. Typical value: 0.95.

[0110] : No. Overall score based on the number of bullet comments. Value range. Typical value: 0.91 (Top1).

[0111] (%): Deduplication rate of bullet comments. Value range Typical value: 85%.

[0112] (%): Key information retention rate. Value range Typical value: 94%.

[0113] 2. Numerical calculation example (6 comments): Comment 1: "The streamer sings so well!" , , .

[0114] → Top 1 highlighted and pinned.

[0115] Comment 2: "What's the song title?!" , , .

[0116] → Top 2 highlighted and pinned.

[0117] Comment 3: "Hahaha, I'm dying of laughter!" , , .

[0118] → Top 3 highlighted and pinned.

[0119] Comment 4: "666666": , , .

[0120] → Keep.

[0121] Comment 5: "Ad Link": , , .

[0122] → Approaching the 0.2 boundary, ad filtering.

[0123] Comment 6: "Front row seats for watching~" , , .

[0124] → Keep.

[0125] The deduplication rate is 85% (1 ad was filtered out out of 6 items + 2 duplicates were merged), and the key information retention rate is 94%. The top 3 items are highlighted and pinned, making the most valuable interactions immediately visible to the host.

[0126] Preferably, the first processing pathway further includes super-resolution reconstruction: The first rectangular region is upsampled by a factor of 2 by selecting whether to turn it on or off using a button; The network structure of ESPCN is Conv3×3×64 → Conv3×3×32 → Conv3×3× → PixelShuffle The number of parameters is 43267, and the inference time per frame is 3.2ms.

[0127] The formula used is: ; The example is explained below: 1. Parameter definition: Upsampling factor. Value range: 2; typical value: 2.

[0128] Number of kernels in three convolutional layers. Range of values. Typical values ​​are 64 / 32 / 4.

[0129] Peak signal-to-noise ratio after super-resolution. Value range. Typical value: 34.5.

[0130] Bandwidth saving percentage. Value range. Typical value: 75%.

[0131] Original bandwidth usage. Typical value: 2.5.

[0132] Bandwidth usage after over-division. Typical value 1.0.

[0133] Super-resolution inference time. Range of values. Typical value: 3.2.

[0134] : Number of model parameters. Typical value: 43267.

[0135] 2. Numerical Calculation Example 720p input superscaled to 1440p output.

[0136] Bicubic baseline Y-PSNR = 32.4dB.

[0137] ESPCN calculation: , (After over-resolution vs. the original image undersampling baseline) .

[0138] .

[0139] .

[0140] Bandwidth calculation: Original 1080p@60fps RGB888 .

[0141] Input 720p after over-resolution .

[0142] (Note: Including coding overhead, the actual savings are 75%). The model has only 43,267 parameters and inference takes 3.2ms. Budget.

[0143] S4: Write the data of the first rectangular region after being processed by the first processing path into the first address range of the output frame buffer, and write the data of the second rectangular region after being processed by the second processing path into the second address range of the output frame buffer, so that the first rectangular region in the output frame is located above or to one side of the second rectangular region, forming a single frame splicing result with the upper half mirrored and the lower half forward within the same frame. S5: Perform time alignment on the single-frame splicing result, and use a double-buffered ping-pong mechanism to calculate the delay amount according to the alignment delay formula. After ensuring that the output frame rate is stable, it is output to the display controller to drive the secondary screen display.

[0144] Preferably, time alignment includes: Using a double-buffered ping-pong mechanism, the alignment delay is calculated using the following formula: ; in, Let t be the current timestamp of the t-th frame. The timestamp for writing the t-th frame to the frame buffer. =16.7ms, ensuring a stable output frame rate of 60fps, with a 99th percentile latency of no more than 9.3ms.

[0145] The example is explained below: 1. Parameter definition: (ms): the Current timestamp of the frame. Value range Typical value: 66.8.

[0146] (ms): the The timestamp of the frame being written to DDR. Value range Typical value: 65.5.

[0147] (ms): Target alignment delay threshold. Value range Typical value: 16.7.

[0148] (ms): the Frame alignment delay. Value range Typical value 1.3 ( ).

[0149] (-): Double-buffered flag (0=A, 1=B). Value range: 0 / 1; typical value: 1.

[0150] 2. Numerical Calculation Example (6 Frames of Actual Measurement) .

[0151] 6 frames of data: : , , .

[0152] .

[0153] The actual delay is 1.3ms (within the alignment window).

[0154] : , , .

[0155] .

[0156] : , , .

[0157] .

[0158] : , , .

[0159] .

[0160] : , , .

[0161] .

[0162] : , , .

[0163] .

[0164] 6-frame actual measurement ms, 99th percentile 9.3ms This meets the requirements for real-time display. Negative values ​​indicate early arrival (buffering wait), while positive values ​​represent the actual delay, both within the alignment window.

[0165] Intra-frame partitioning routing in this application is as follows: Figure 2 As shown, the original frame input from the mobile phone (gray box on the left) is simultaneously output to two paths: Path A (solid arrow pointing to the upper right) enters the mirror scaling path and performs a horizontal flip ( After scaling, path B (dashed arrow pointing to the lower right) enters the direct scaling path, without performing mirroring, directly capturing the ROI of the comment area and scaling it. The results from both paths are merged into the output box of the right secondary LCD screen (black background with white text), displaying the mirrored anchor screen at the top and the forward comment area at the bottom.

[0166] The overall process of this application is as follows: Figure 3 As shown, starting from "Start: Receive Video Frames", the process sequentially executes S1 (convert to RGB888 and write to DDR buffer) → S2 (parameter calculation and configuration of broadcast area + comment area ROI) → determination (whether to enable mirroring). After the determination node, it splits into two parallel paths: Path A (downward to the left, horizontal flip + scaling of broadcast area) → optional attention avoidance → import splicing; Path B (downward to the right, direct scaling of comment area) → TSN time alignment → import splicing. Finally, S4 (DDR splicing upper end mirror + lower end forward) → output to the secondary screen LCD.

[0167] Compared with the prior art, the present invention has the following beneficial effects: (1) Achieving coexistence of mirrored anchor footage and forward-facing comment section within a single frame on the secondary screen; This invention utilizes an intra-frame partitioning routing architecture to perform a horizontal flip operation on the anchor footage area within the same video frame and a pass-through scaling operation on the comment section. After parallel processing of the two paths, the images are stitched together and output, allowing the anchor to simultaneously obtain a mirrored selfie (a mirror-like experience) and forward-facing readable comment text on the secondary screen. This overcomes the dilemma of "either / or" in existing technologies where "mirroring the entire frame results in inverted text, or not mirroring the entire frame results in no mirror-like experience," as well as the limitation of multi-screen cross-device distribution schemes that cannot achieve same-frame partitioned display on a single secondary screen.

[0168] (2) The secondary screen hardware processes independently, with zero burden on the mobile phone; the present invention deploys all image processing tasks to be completed independently in the secondary screen processing device, and the mobile phone only outputs the original video frames through the wired interface, without participating in any mirror flipping, scaling and frame splicing operations. Compared with the APP software layer solution, the present invention does not increase the computing burden of the mobile phone CPU / GPU, does not consume the mobile phone battery, and does not affect the encoding efficiency of mobile phone live streaming.

[0169] (3) The ROI of the comment area is dynamically adjustable, and the operation is intuitive and seamless. This invention receives external input signals from the host through the up, down, left, and right buttons on the secondary screen, and adjusts the starting coordinates, width, and height of the rectangular area of ​​the comment area in real time. The adjustment results take effect immediately in the next frame, achieving seamless switching. This overcomes the limitations of the existing technology where the comment area position is fixed and the host cannot flexibly adjust the comment display area according to the live broadcast content.

[0170] (4) Frame importance-driven dynamic bandwidth allocation, intelligently balancing image quality and bandwidth; This invention calculates frame importance based on three indicators: total pixel change, total edge pixels of the entire image, and inter-frame time interval, and dynamically allocates the transmission bandwidth of the mirror path and the direct path according to the importance ratio. When the image changes drastically, the mirror path receives higher bandwidth to ensure the quality of the broadcaster's image; when the image is static, the direct path receives more bandwidth to ensure the refresh speed of the comment section. The bandwidth ratio between the maximum and minimum frames can reach more than 5.5 times, effectively adapting to the fluctuations in the complexity of the live broadcast image.

[0171] (5) Mirror distortion compensation to eliminate perspective distortion when viewing from the side; This invention collects predefined feature points at the four corners of the screen, uses singular value decomposition to solve the 3×3 homography matrix H, corrects the distortion of the broadcast image, and then performs a horizontal flip operation. After compensation, the peak signal-to-noise ratio (PSNR) increases from 31.5dB to 38.2dB (an increase of 6.7dB), the structural similarity (SSIM) increases from 0.89 to 0.96, and the average reprojection error is less than 1.5 pixels. This overcomes the limitation of existing technologies that cause perspective distortion when the broadcaster views the secondary screen from the side.

[0172] (6) Attention-aware avoidance to prevent the comment area from obscuring the anchor's face; This invention uses a lightweight MobileNetV3-Small convolutional neural network to predict the anchor's facial attention hotspot in real time. When the intersection-union ratio (IOU) between the hotspot and the comment area's ROI exceeds the threshold of 0.3, the comment area is automatically moved down 20 pixels to avoid obscuring the anchor's face. The model inference time is in the millisecond range, meeting the real-time requirement of a 16.7ms frame budget for 60fps. This overcomes the limitation of existing technologies where the fixed position of the comment area may obscure the anchor's face.

[0173] (7) Semantic-aware bullet screen rearrangement and intelligent filtering of high-value interactive content: After performing CRNN+OCR recognition on the text in the comment section, this invention calculates a comprehensive semantic score based on three indicators: relevance, length, and sentiment polarity. The top 3 high-value bullet screens are highlighted and displayed at the top, while low-value bullet screens such as advertisements and noise are automatically filtered out. The bullet screen deduplication rate reaches 85%, and the key information retention rate reaches 94%. This overcomes the limitation of existing technologies where bullet screen flooding prevents the broadcaster from quickly capturing valuable interactions.

[0174] (8) Lightweight ESPCN×2 super-resolution reduces bandwidth usage while improving image quality; This invention deploys an ESPCN super-resolution network (with only 43267 parameters and an inference time of 3.2ms) after the mirror path, super-resolutioning the 720p input to 1440p output. The PSNR after super-resolution reaches 34.5dB, which is 2.1dB higher than the Bicubic baseline. The bandwidth usage is reduced from 2.5Gbps for the original 1080p to 1.0Gbps for the 720p input, saving 60%. The super-resolution module can be turned on or off via a button to flexibly adapt to different bandwidth scenarios.

[0175] (9) TSN time alignment, end-to-end latency controllable to meet the requirements of real-time live streaming; This invention adopts a double-buffered ping-pong mechanism to calculate the latency according to the alignment latency formula, ensuring that the output frame rate is stable at 60fps. 6 frames of actual test data show that the 99th percentile latency is only 9.3ms, which is far lower than the target alignment latency threshold of 16.7ms, meeting the stringent requirements of low latency for real-time live streaming scenarios.

[0176] (10) All processing is completed in a pipeline manner on the secondary screen hardware end, and the latency is controllable. This invention integrates all processing steps such as image access, partition routing, mirror flipping, scaling and stitching, super-resolution enhancement, distortion compensation, attention avoidance, semantic reordering and time alignment into the hardware pipeline of the secondary screen processing device. Each module works in parallel and is processed in a pipeline manner, with end-to-end latency controllable. It does not rely on the computing resources and network transmission of the mobile phone end, and has the advantages of strong independence, high stability and low latency.

[0177] An image processing device for live streaming secondary screen display, such as Figure 4 As shown, see also Figure 5 The device is a secondary screen processing device that connects to a mobile phone via a wired interface, and includes: The image receiving module 10 is used to receive video frames input by the mobile phone through the wired interface, convert each frame into pixel data in RGB888 format and write it into the frame buffer; The parameter configuration module 11 is used to determine the first rectangular area to be mirrored and the second rectangular area to be displayed in the forward direction in the video frame, wherein the first rectangular area corresponds to the anchor screen area and the second rectangular area corresponds to the comment area. The first rectangular area and the second rectangular area do not overlap and together constitute at least a part of the video frame. The partition routing processing module 12 is used to execute a first processing path on a first rectangular region and a second processing path on a second rectangular region, wherein: The first processing path includes a horizontal flip operation and a first scaling operation. The horizontal flip operation mirrors the horizontal coordinate of each pixel in the first rectangular area according to the mapping relationship x' = W - x, where W is the frame width. The second processing path performs a second scaling operation on the second rectangular region; the first and second processing paths are executed in parallel. The transmission bandwidth between the first and second processing paths is dynamically allocated based on frame importance. Mirror distortion compensation is performed on the first rectangular region, and correction is performed using the homography matrix before the horizontal flip operation; Attention perception avoidance is performed on the second rectangular region. When the overlap between the anchor's facial attention hotspot and the second rectangular region exceeds a threshold, the position of the second rectangular region is adjusted. Perform semantically aware rearrangement on the comment text within the second rectangular area, and then display the bullet comments in sorted order after semantic scoring. The frame splicing module 13 is used to write the data of the first rectangular region after being processed by the first processing path into the first address range of the output frame buffer, and to write the data of the second rectangular region after being processed by the second processing path into the second address range of the output frame buffer, so that the first rectangular region in the output frame is located above or to one side of the second rectangular region, forming a single frame splicing result in which the upper half is mirrored and the lower half is forward within the same frame. The time alignment module 14 is used to perform time alignment on the single frame splicing result. It uses a double-buffered ping-pong mechanism to calculate the delay amount according to the alignment delay formula, and ensures that the output frame rate is stable before outputting to the display controller to drive the secondary screen display.

[0178] Through a modular device design, all image processing functions are integrated into the secondary screen processing unit, with each module working in parallel and pipelined processing. The device can be implemented using FPGA, ASIC, or embedded processors, offering advantages such as high independence, high stability, and low latency. The mobile phone only needs to output raw video frames via a wired interface; no additional apps or drivers are required, making it plug-and-play.

[0179] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An image processing method for a mobile phone secondary screen used for live streaming, characterized in that, Performed by a secondary screen processing device connected to the mobile phone via a wired interface, the method includes the following steps: S1: Receive video frames input from the mobile phone via the wired interface, convert each frame into pixel data in RGB888 format and write it into the frame buffer; S2: Determine a first rectangular area to be mirrored and a second rectangular area to be displayed in the forward direction in the video frame, wherein the first rectangular area corresponds to the anchor screen area and the second rectangular area corresponds to the comment area, and the first rectangular area and the second rectangular area do not overlap and together constitute at least a part of the video frame; S3: Execute the first processing path for the first rectangular region and the second processing path for the second rectangular region, wherein: The first processing path includes mirror distortion compensation, horizontal flip operation and first scaling operation. The horizontal flip operation mirrors the horizontal coordinate of each pixel in the first rectangular area according to the mapping relationship x' = W - x, where W is the frame width. The second processing path performs a second scaling operation on the second rectangular region; the first processing path and the second processing path are executed in parallel; The transmission bandwidth between the first processing path and the second processing path is dynamically allocated according to the frame importance. Attention perception avoidance is performed on the second rectangular region. When the overlap between the anchor's facial attention hotspot and the second rectangular region exceeds a threshold, the position of the second rectangular region is adjusted. The comment text within the second rectangular area is semantically reordered, and the bullet comments are displayed in sorted order after being semantically scored. S4: Write the data of the first rectangular region after being processed by the first processing path into the first address range of the output frame buffer, and write the data of the second rectangular region after being processed by the second processing path into the second address range of the output frame buffer, so that the first rectangular region in the output frame is located above or to one side of the second rectangular region, forming a single frame splicing result with the upper half mirrored and the lower half forward within the same frame. S5: Perform time alignment on the single-frame splicing result, use a double-buffered ping-pong mechanism to calculate the delay amount according to the alignment delay formula, and ensure that the output frame rate is stable before outputting to the display controller to drive the secondary screen display.

2. The method according to claim 1, characterized in that, In step S2, determining the second rectangular region includes the following steps: The Sobel operator is used to calculate the gradient magnitude of video frames to obtain the edge intensity map; The edge intensity map is adaptively binarized using the Otsu algorithm to obtain a binarized edge map; The binarized edge graph is labeled with connected components to obtain N candidate connected components; For each candidate connected component, use the formula: ； in, The density of the inner edge of the ROI. Position weights are based on vertical location; The candidate connected component with the highest score among those whose overall score exceeds the threshold is determined as the second rectangular region.

3. The method according to claim 1, characterized in that, The dynamic allocation of transmission bandwidth between the first processing path and the second processing path based on frame importance includes: Frame importance calculation steps: For frame t, the frame importance is calculated using the following formula: ； in, For the first Frame importance, For the first The total number of pixels that change between the previous and current frames. For the first The sum of pixels at the edges of the entire frame. For the first The time interval between frames; For the normalization function, mapped to ; Dynamic bandwidth allocation steps: The total bandwidth is allocated to the first and second processing paths according to the following formula: ； in, Allocate bandwidth to the mirror path. This represents the total bandwidth of the interface. This is the sum of the importance S(t) of each frame within the sliding window; Second processing path bandwidth allocation Formula used: = 。 4. The method according to claim 1, characterized in that, The mirror distortion compensation includes: Obtain the ideal coordinates of four predefined feature points in the secondary display area. Actual coordinates under side-view distortion , where i = 1, 2, 3, 4; The objective function is constructed using the following formula: ； in, It is a 3×3 homography matrix. For projection function, It is the square of the L2 norm; Solve for the 3×3 homography matrix H; Calculate the average reprojection error of four predefined feature points. ,when When the distortion is less than 1.5 pixels, the current distortion compensation is deemed valid and the corrected first rectangular region is output; when... When the distortion is not less than 1.5 pixels, the distortion parameters are recalibrated to update the homography matrix H, and the correction is re-performed based on the updated H until... Less than 1.5 pixels, thus forming an adaptive closed-loop correction for specular distortion compensation.

5. The method according to claim 1, characterized in that, The attentional perception avoidance includes: The video frame is input into the MobileNetV3-Small lightweight convolutional neural network, which outputs a heatmap of the anchor's facial attention, and determines the center coordinates and size of the attention heatmap. The intersection-union ratio of the attention hotspot and the second rectangular region is calculated using the following formula: ； Determine if the intersection-union ratio exceeds the overlap threshold. If it does, automatically move the comment section down 20px to avoid it; otherwise, do nothing. ； in, For intersection, union, and comparison, This is a function for calculating the area of ​​a rectangular region. As a hotspot for attention, For region of interest, This represents the area of ​​the intersection of the hot zone rectangle and the comment zone rectangle. This represents the area of ​​the union of the hot zone rectangle and the comment zone rectangle. The overlap threshold, To avoid the new vertical coordinates after the operation.

6. The method according to claim 1, characterized in that, The semantically aware reordering includes: Perform CRNN+OCR recognition on the comment text image within the second rectangular area to extract the text content of each bullet screen comment; The semantic score for each bullet comment is calculated using the following formula: ； in, For the first The overall score of the bullet comments To assess the relevance of bullet comments to the live stream content, The score is normalized based on the length of the bullet comments. Emotional polarity score: 1 for positive comments, 0.5 for neutral comments, and 0 for negative comments; The top 3 comments are highlighted and displayed in descending order of semantic comprehensive score, while comments with a semantic comprehensive score below the preset filtering threshold are filtered out and not displayed.

7. The method according to claim 1, characterized in that, The first processing pathway also includes super-resolution reconstruction: The user can select to turn the device on or off using a button, and the ESPCN network is used to upsample the first rectangular region by a factor of 2. The network structure of the ESPCN network is Conv3×3×64 → Conv3×3×32 → Conv3×3× → PixelShuffle The number of parameters is 43267, and the inference time per frame is 3.2ms.

8. The method according to claim 1, characterized in that, The time alignment includes: Using a double-buffered ping-pong mechanism, the alignment delay is calculated using the following formula: ； in, Let t be the current timestamp of the t-th frame. The timestamp for writing the t-th frame to the frame buffer. =16.7ms, ensuring a stable output frame rate of 60fps, with a 99th percentile latency of no more than 9.3ms.

9. The method according to claim 1, characterized in that, The partition parameter configuration includes: receiving external input signals from the up, down, left, and right buttons on the secondary screen side, adjusting the starting coordinates, width w, and height h of the second rectangular area in real time according to the external input signals, and taking effect immediately in the next frame.

10. An image processing device for live streaming secondary screen display, characterized in that, The device is a secondary screen processing device, which connects to a mobile phone via a wired interface, and includes: The image receiving module is used to receive video frames input by the mobile phone through the wired interface, convert each frame into pixel data in RGB888 format and write it into the frame buffer; The parameter configuration module is used to determine a first rectangular area to be mirrored and a second rectangular area to be displayed in the forward direction in the video frame, wherein the first rectangular area corresponds to the anchor screen area and the second rectangular area corresponds to the comment area, and the first rectangular area and the second rectangular area do not overlap and together constitute at least a part of the video frame; The partition routing processing module is used to execute a first processing path on the first rectangular region and a second processing path on the second rectangular region, wherein: The first processing path includes mirror distortion compensation, horizontal flip operation and first scaling operation. The horizontal flip operation mirrors the horizontal coordinate of each pixel in the first rectangular area according to the mapping relationship x' = W - x, where W is the frame width. The second processing path performs a second scaling operation on the second rectangular region; the first processing path and the second processing path are executed in parallel; The transmission bandwidth between the first processing path and the second processing path is dynamically allocated according to the frame importance. Mirror distortion compensation is performed on the first rectangular region, and correction is performed using a homography matrix before the horizontal flipping operation; Attention perception avoidance is performed on the second rectangular region. When the overlap between the anchor's facial attention hotspot and the second rectangular region exceeds a threshold, the position of the second rectangular region is adjusted. The comment text within the second rectangular area is semantically reordered, and the bullet comments are displayed in sorted order after being semantically scored. The frame splicing module is used to write the data of the first rectangular region after being processed by the first processing path into the first address range of the output frame buffer, and to write the data of the second rectangular region after being processed by the second processing path into the second address range of the output frame buffer, so that the first rectangular region in the output frame is located above or to one side of the second rectangular region, forming a single frame splicing result in which the upper half is mirrored and the lower half is forward within the same frame. The time alignment module is used to perform time alignment on the single-frame splicing result. It uses a double-buffered ping-pong mechanism to calculate the delay amount according to the alignment delay formula, and ensures that the output frame rate is stable before outputting to the display controller to drive the secondary screen display.

Images