Television image processing system and method based on artificial intelligence

Abstract

The invention discloses a television image processing system and method based on artificial intelligence, and relates to the technical field of television image processing, and the system comprises a screen projection transmission analysis module, a user watching analysis module, and an optimization strategy decision module. The projection screen transmission analysis module performs projection screen transmission feature labeling on each image frame pixel point of projection screen image historical data, trains a projection screen transmission image prediction neural network model, calculates feature deviation before and after projection screen transmission of to-be-transmitted image frame data, and performs region unit division on the to-be-transmitted image frame data; the user watching analysis module monitors eye data of all watching users, analyzes a user sight focus point and user watching direction information, determines an image optimization feature deviation threshold value of each region unit, and calculates to-be-optimized deviation of each region unit; and the optimization strategy decision-making module uses the trained image optimization strategy decision-making model to determine an image optimization strategy of the to-be-transmitted image frame data according to the to-be-optimized deviation of each area unit.

Description

Technical Field

[0001] The present invention relates to the technical field of television image processing, and specifically to a television image processing system and method based on artificial intelligence. Background Art

[0002] With the introduction of screen mirroring playback technology in various video websites, image optimization for television screen mirroring becomes particularly important in screen mirroring playback;

[0003] In the existing television screen mirroring playback, the following technical defects exist: Traditional compression algorithms (such as H.264 / HEVC) use global quantization parameters and lack dynamic perception of the image data transmission link state, resulting in different degrees of image distortion in different texture feature regions, leading to a rigid image region division method and being unable to effectively distinguish the distortion degrees of different image regions; When analyzing the local optimization strategy for the image after screen mirroring transmission, in the adaptive optimization method based on the user's vision for different image regions, there is a lack of reasonable analysis of the optimization strategies suitable for different image regions, resulting in a lack of balance in the distortion degrees of each image region in the final optimization effect and being unable to more accurately display the image features before screen mirroring image transmission;

[0004] Therefore, a television image processing system and method based on artificial intelligence are needed to solve the above technical defects. Summary of the Invention

[0005] The purpose of the present invention is to provide a television image processing system and method based on artificial intelligence to solve the problems raised in the prior art.

[0006] To achieve the above purpose, the present invention provides the following technical solutions:

[0007] A television image processing method based on artificial intelligence, the method includes the following steps:

[0008] Step S100: Obtain the historical data of the television screen mirroring image, perform screen mirroring transmission feature annotation on each image frame pixel point in the historical data of the screen mirroring image, and construct an image transmission feature data set;

[0009] Step S200: Use the image transmission feature data set to train a screen mirroring transmission image prediction neural network model, calculate the feature deviation of each pixel point before and after screen mirroring transmission of the image frame data to be transmitted, and then perform regional unit division according to the feature deviation;

[0010] Step S300: Construct a three-dimensional coordinate system, monitor the eye data of all viewing users, analyze the user's line of sight focus point and the user's gaze direction information, and then determine the image optimization feature deviation threshold of each regional unit and calculate the deviation to be optimized of each regional unit;

[0011] Step S400: Obtain the historical data of the optimized TV screen mirroring image, extract the image feature parameters, optimization strategy data, feature deviation between the optimized image frame data and the pre-screen mirroring image frame data, and optimization strategy resource consumption data, construct an image optimization feature dataset, and train an image optimization strategy decision model;

[0012] Step S500: According to the deviation to be optimized for each regional unit of the image frame data to be transmitted, use the image optimization strategy decision model to determine the image optimization strategy for the image frame data to be transmitted for TV screen mirroring.

[0013] In the above technical solution, the method for performing screen mirroring transmission feature annotation on each image frame pixel point in the historical data of the screen mirroring image in step S100 is as follows:

[0014] For any pixel point p in the image frame data, the screen mirroring transmission feature annotation is: p[(x_p,y_p),F_before(p),F_after(p),S_tran(p)]; where x_p is the number of pixel point p arranged from left to right in the image frame, y_p is the number of pixel point p arranged from top to bottom in the image frame, F_before(p) is the set of image feature parameters of pixel point p before transmission, F_after(p) is the set of image feature parameters of pixel point p after transmission, and S_tran(p) is the set of transmission link state parameters when the image frame data to which pixel point p belongs is screen mirroring transmitted;

[0015] Through pixel-level feature comparison, locate image transmission distortions such as color deviation and blocking effect caused by transmission, establish a quantitative relationship between link features and image distortion, and provide input dimensions for the prediction model.

[0016] In the above technical solution, step S200 includes the following steps:

[0017] Step S201: Use the image transmission feature dataset to train a screen mirroring transmission image prediction neural network model, and analyze the predicted data of the image feature parameters of each pixel point after the screen mirroring transmission of the image frame data to be transmitted according to the image transmission features of each pixel point of the image frame data to be transmitted;

[0018] Step S202: Calculate the deviation between the predicted data of each image feature parameter of each pixel point after the screen mirroring transmission of the image frame data to be transmitted and each image feature parameter before the screen mirroring transmission, and calculate the feature deviation of each pixel point of the image frame data to be transmitted before and after the screen mirroring transmission through weighted calculation;

[0019] Step S203: Set a feature deviation interval, and divide the pixel points with the same feature deviation within the same feature deviation interval and adjacent in the image frame data to be transmitted into the same regional unit;

[0020] For any regional unit, calculate the distances between the pixel points therein in the image, set a pixel distance threshold, and perform a threshold judgment on the two pixel points with the largest distance. If the distance is less than the pixel distance threshold, no further division is made. If the distance is greater than or equal to the pixel distance threshold, use the two pixel points as the clustering centers, and based on the distance between the pixel points as the clustering basis, further divide the regional unit into two regional units. Repeat the above operations until the distances between all pixel points in all regional units are less than the pixel distance threshold;

[0021] By constructing a neural network model to predict the image transmission distortion effect under different data transmission link states, quantitatively analyze the degree of image transmission distortion, and then classify and summarize the images in different regions, effectively avoiding the mixing of low-deviation pixel points and high-deviation pixel points into the same unit due to fixed division, and significantly improving the adaptability and rationality of local optimization of the screen-cast image.

[0022] In the above technical solution, the step S300 includes the following steps:

[0023] Step S301: Take the center point of the TV screen as the origin, the horizontal direction from left to right as the x-axis, the vertical direction from top to bottom as the y-axis, and the direction perpendicular to the screen as the z-axis to establish a three-dimensional coordinate system;

[0024] Step S302: Use a vision sensor to monitor and identify all user facial feature information, extract user line-of-sight information, and analyze the user line-of-sight focus point and user gaze direction information; wherein, the user line-of-sight focus point is the intersection direction of the user's binocular lines of sight, and the user gaze direction is the ray direction from the center point of the user's eyes to the user line-of-sight focus point;

[0025] Step S303: Use a clustering algorithm to calculate the spatial center point of the pixel points in each regional unit in the image frame data, and according to the formula: k_x = R_e × exp(-l), determine the focusing coefficient k_x of the user on any regional unit x; where R_e is the ratio of the distance from the center point of the user's eyes to the user line-of-sight focus point to the distance from the center point of the user's eyes to the intersection point of the user gaze direction and the plane where the TV screen is located, and l is the distance between the intersection point of the user gaze direction and the plane where the TV screen is located and the spatial center point of the regional unit x;

[0026] Step S303: Calculate the focusing coefficients of all users on each regional unit, and take the maximum value as the focusing coefficient of each regional unit. Set the image optimization feature deviation threshold th_x of any regional unit x as: th_x = (1 - α) × k_x × C_dev; where α is the image optimization adjustment coefficient, k_x is the focusing coefficient of any regional unit, and C_dev is the maximum deviation of the image optimization feature;

[0027] Step S304: Further, calculate the difference between the center of the feature deviation interval to which each regional unit belongs and the image optimization feature deviation threshold, and use it as the deviation to be optimized for each regional unit.

[0028] Break through the traditional two-dimensional plane analysis method based on users' visual perception in the technical solution, comprehensively analyze the azimuth relationship between the focus of users' line of sight, the direction of the line of sight and the center of the TV screen at the three-dimensional level, make the analysis of users' visual effects more refined, and further improve the practicality of the local optimization strategy of the screen mirroring image and the user experience by correlating and analyzing the users' visual effects and the optimization requirements of each area of the screen mirroring image.

[0029] In the above technical solution, the following contents are included in the step S400:

[0030] The input of the image optimization strategy decision model is the image feature parameters, the adopted optimization strategy, and the feature deviation between the image frame data after optimization using the current optimization strategy and the image frame data before screen mirroring. The output is the time-consuming data of the image optimization strategy; the resource consumption data of the optimization strategy is the time-consuming data of the image optimization strategy.

[0031] In the above technical solution, the following contents are included in the step S500:

[0032] Obtain the time interval between adjacent image frames to be transmitted, and set the image optimization time threshold after the screen mirroring transmission of the image frame data to be transmitted according to the time interval between adjacent image frames to be transmitted;

[0033] Use the image optimization strategy decision model, take the deviation to be optimized of each regional unit in the image frame data as the input, analyze the time-consuming information of optimizing each regional unit to meet the image optimization feature deviation threshold judgment by adopting different optimization strategies, and set the optimization strategy of the image frame data:

[0034] When the overall time consumption of each image unit in the image frame data for adopting each optimization strategy is greater than the image optimization time threshold, proportionally amplify the deviation to be optimized of each regional unit until there is an optimization strategy that satisfies that the overall time consumption of each image unit in the image frame data for adopting each optimization strategy is less than or equal to the image optimization time threshold, and use this optimization strategy as the final optimization strategy of the image frame data;

[0035] When there is an optimization strategy with an overall time consumption less than or equal to the image optimization time threshold among the optimization strategies adopted by each image unit in the image frame data, proportionally reduce the deviation to be optimized of each regional unit until there is only one optimization strategy that satisfies that the overall time consumption of each image unit in the image frame data for adopting this optimization strategy is less than or equal to the image optimization time threshold, and use this optimization strategy as the final optimization strategy of the image frame data;

[0036] By dynamically scaling the deviation to be optimized for the regional unit, screening and determining an optimization strategy that meets the requirements of the time interval between image frames, while ensuring that the screen-cast image meets the requirements of smooth display, it improves the consistency between the user's overall perception effect of the image and the original effect of the image, and effectively balances the overall optimization effect and the display effect of the screen-cast image.

[0037] An artificial intelligence-based television image processing system applying an artificial intelligence-based television image processing method in the above technical solution, the system includes: a screen-cast transmission analysis module, a user gaze analysis module, and an optimization strategy decision module;

[0038] The screen-cast transmission analysis module performs screen-cast transmission feature annotation on each pixel point of the historical data of the screen-cast image, trains a screen-cast transmission image prediction neural network model, calculates the feature deviation of each pixel point before and after the screen-cast transmission of the image frame data to be transmitted, and divides the image frame data to be transmitted into regional units; the user gaze analysis module monitors the eye data of all viewing users, analyzes the user's line-of-sight focus point and the user's gaze direction information, determines the image optimization feature deviation threshold of each regional unit, and calculates the deviation to be optimized for each regional unit; the optimization strategy decision module determines the image optimization strategy of the image frame data to be transmitted according to the deviation to be optimized for each regional unit, using the trained image optimization strategy decision model.

[0039] In the above technical solution, the screen-cast transmission analysis module includes: a transmission feature annotation unit, a feature deviation analysis unit, and a regional unit division unit;

[0040] The transmission feature annotation unit performs screen-cast transmission feature annotation on each pixel point of the historical data of the screen-cast image; the feature deviation analysis unit uses the screen-cast transmission image prediction neural network model to predict the image feature parameters after the screen-cast transmission of the image frame data to be transmitted, and calculates the feature deviation of each pixel point before and after the screen-cast transmission of the image frame data to be transmitted; the regional unit division unit divides the image frame data to be transmitted into regional units according to the feature deviation of each pixel point before and after the screen-cast transmission of the image frame data to be transmitted.

[0041] In the above technical solution, the user gaze analysis module includes: a user monitoring unit, an orientation analysis unit, and a deviation setting unit;

[0042] The user monitoring unit is used to monitor the eye data of all viewing users; the orientation analysis unit is used to construct a three-dimensional coordinate system, and analyze the orientation information of the user's line-of-sight focus point, the user's gaze direction and the intersection point of the plane where the television screen is located in the three-dimensional coordinate system; the deviation setting unit is used to determine the image optimization feature deviation threshold of each regional unit in the image frame data to be transmitted, and calculate the deviation to be optimized for each regional unit.

[0043] In the above technical solution, the optimization strategy decision module includes: a decision model construction unit and an optimization strategy decision unit;

[0044] The decision model construction unit trains an image optimization strategy decision model by constructing an image optimization feature data set; the optimization strategy decision unit adjusts the deviation to be optimized of each regional unit in the image frame data according to the overall time consumption of each optimization strategy for optimizing each regional unit in the image frame data, and then determines the final optimization strategy for image frame optimization.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] In the present invention, by analyzing the deviation synchronization mechanism of the multi-objective optimization strategy, multi-dimensional parameters such as image quality improvement, delay constraint, and resource consumption are collaboratively optimized based on the reinforcement learning framework, and the proportional scaling algorithm is used to forcibly meet the frame processing time threshold, solving the strategy conflict problem of the traditional rule engine, and significantly improving the fluency and image quality balance of high-dynamic scenes;

[0047] In the present invention, by constructing a three-dimensional visual perception dynamic optimization model, the binocular vision tracking technology is used to capture the user's gaze focus point in real time, combined with the depth proportional coefficient and the spatial projection algorithm, a three-dimensional space model is built for the fixation area, solving the positioning deviation of the traditional two-dimensional weight model under the tilted view angle, and dynamically adjusting the optimization intensity through the focusing coefficient to achieve reasonable on-demand allocation of image optimization resources;

[0048] In the present invention, the pixel-level transmission feature dynamic annotation technology is used to monitor and analyze the screen-cast image in real time, avoiding the lack of perception of local distortion by the traditional macro-block level compression algorithm. By fusing the transmission link state parameters and multi-dimensional image features, a pixel-level distortion traceability system is constructed, significantly improving the transmission defect positioning accuracy, and at the same time enhancing the adaptive ability of the system in a complex network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flowchart of a method for processing television images based on artificial intelligence according to the present invention;

[0050] Figure 2 is an organizational structure diagram of a system for processing television images based on artificial intelligence according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the protection scope of the present invention.

[0052] Example: Please refer to Figure 1 - Figure 2 , the present invention provides the following technical solutions:

[0053] As Figure 1 shown, the present invention provides a method for processing television image based on artificial intelligence, and the method includes the following steps:

[0054] Step S100: Obtain the historical data of the television screen mirroring image, perform screen mirroring transmission feature annotation on each image frame pixel point in the historical data of the screen mirroring image, and construct an image transmission feature data set;

[0055] Step S200: Use the image transmission feature data set to train a screen mirroring transmission image prediction neural network model, calculate the feature deviation of each pixel point before and after the screen mirroring transmission of the image frame data to be transmitted, and then perform regional unit division according to the feature deviation;

[0056] Step S300: Construct a three-dimensional coordinate system, monitor the eye data of all viewing users, analyze the user's line of sight focus point and the user's gaze direction information, and then determine the image optimization feature deviation threshold of each regional unit, and calculate the deviation to be optimized of each regional unit;

[0057] Step S400: Obtain the historical data of the television screen mirroring image optimization, extract image feature parameters, optimization strategy data, the feature deviation between the optimized image frame data and the image frame data before screen mirroring, and the resource consumption data of the optimization strategy, construct an image optimization feature data set, and train an image optimization strategy decision model;

[0058] Step S500: According to the deviation to be optimized of each regional unit of the image frame data to be transmitted, use the image optimization strategy decision model to determine the image optimization strategy of the television screen mirroring image frame data to be transmitted.

[0059] The method for performing screen mirroring transmission feature annotation on each image frame pixel point in the step S100 is as follows:

[0060] For any pixel point p in the image frame data, the screen mirroring transmission feature annotation is: p[(x_p, y_p), F_before(p), F_after(p), S_tran(p)]; where, x_p is the number of the pixel point p arranged from left to right in the image frame, y_p is the number of the pixel point p arranged from top to bottom in the image frame, F_before(p) is the set of image feature parameters of the pixel point p before transmission, F_after(p) is the set of image feature parameters of the pixel point p after transmission, and S_tran(p) is the set of transmission link state parameters when the image frame data to which the pixel point p belongs is screen mirroring transmitted;

[0061] In specific implementation, obtain the historical image data of TV screen mirroring, and extract the multi-dimensional features of each pixel point, including the multi-dimensional features of the pixel point before and after transmission, such as color space, local texture complexity, gradient intensity, etc.;

[0062] Since image encoding and compression are required during image screen mirroring transmission, which includes block processing of image data, and the attenuation degrees of pixel points in different blocks are also different, the azimuth data of pixel points in the image is introduced to participate in the analysis; in addition, due to different texture feature distortion degrees of the image during screen mirroring transmission under different transmission link states, it is necessary to synchronously analyze the link state parameters during image transmission.

[0063] The step S200 includes the following steps:

[0064] Step S201: Use the image transmission feature data set to train the screen mirroring transmission image prediction neural network model, and analyze the predicted data of the image feature parameters of each pixel point after screen mirroring transmission of the to-be-transmitted image frame data according to the image transmission features of each pixel point of the to-be-transmitted image frame data;

[0065] Step S202: Calculate the deviation between the predicted data of each image feature parameter of each pixel point after screen mirroring transmission of the to-be-transmitted image frame data and each image feature parameter before screen mirroring transmission, and calculate the feature deviation of each pixel point of the to-be-transmitted image frame data before and after screen mirroring transmission through weighted calculation;

[0066] Step S203: Set a feature deviation interval, and divide the pixel points with the same feature deviation in the same interval and adjacent in the to-be-transmitted image frame into the same region unit;

[0067] For any region unit, calculate the distance between each pixel point in the image, set a pixel distance threshold, and make a threshold judgment on the two pixel points with the largest distance. If the distance is less than the pixel distance threshold, no further division is made. If the distance is greater than or equal to the pixel distance threshold, use the two pixel points as the clustering centers and the distance between pixel points as the clustering basis to further divide the region unit into two region units, and repeat the above operation until the distance between each pixel point in all region units is less than the pixel distance threshold;

[0068] Construct a conditional generation neural network model, with the input being the image feature parameter data of the pixel point before transmission and the set of transmission link state parameters during screen mirroring transmission, and improve the prediction accuracy of the model for the distortion effect of complex image textures after screen mirroring transmission through an adversarial training strategy;

[0069] Further, according to the influence of the deviation of each image feature parameter on the image display effect, set the deviation weight of each image feature parameter, and comprehensively analyze the feature deviation of each pixel point through weighted calculation;

[0070] Among them, since the influence of the deviation of some image feature parameters on the image display effect is not linearly related, when setting the deviation weight of the image feature parameters, a dynamic weight method can be adopted to ensure a more accurate description of the influence of each image feature parameter on the overall image effect display at different deviation levels;

[0071] Furthermore, when performing local adaptive optimization of the image, considering that in the two-dimensional orientation of the image, the difference in the screen projection transmission and the image feature texture of pixel points with a relatively large distance is generally large. Therefore, for a regional unit, if there are pixel points with a large interval, the regional unit is further divided. After repeating multiple times, to ensure the refined segmentation of all regional units and avoid the loss of accuracy of the local image optimization strategy.

[0072] The step S300 includes the following steps:

[0073] Step S301: Taking the center point of the TV screen as the origin, the horizontal direction from left to right as the x-axis, the vertical direction from top to bottom as the y-axis, and the direction perpendicular to the screen as the z-axis, a three-dimensional coordinate system is established;

[0074] Step S302: Using a vision sensor to monitor and identify all user facial feature information, extracting user line-of-sight information, and analyzing the user line-of-sight focus point and the user's gaze direction information; among them, the user line-of-sight focus point is the intersection direction of the user's binocular line of sight, and the user's gaze direction is the ray direction from the center point of the user's eyes to the user line-of-sight focus point;

[0075] Step S303: Using a clustering algorithm to calculate the spatial center point of the pixel points in each regional unit in the image frame data, and according to the formula: k_x = R_e × exp(-l), determining the focus coefficient k_x of the user on any regional unit x; where, R_e is the ratio of the distance from the center point of the user's eyes to the user line-of-sight focus point to the distance from the center point of the user's eyes to the intersection point of the user's gaze direction and the plane where the TV screen is located, and l is the distance between the intersection point of the user's gaze direction and the plane where the TV screen is located and the spatial center point of the regional unit x;

[0076] Step S303: Calculating the focus coefficients of all users on each regional unit, and taking the maximum value as the focus coefficient of each regional unit, and setting the image optimization feature deviation threshold th_x of any regional unit x as: th_x = (1 - α) × k_x × C_dev; where, α is the image optimization adjustment coefficient, k_x is the focus coefficient of any regional unit, and C_dev is the maximum deviation of the image optimization feature;

[0077] Step S304: Then calculating the difference between the center of the feature deviation interval to which each regional unit belongs and the image optimization feature deviation threshold as the deviation to be optimized for each regional unit;

[0078] In specific implementation, a three-dimensional coordinate system is established with the TV screen as a reference. The user's facial features are captured by a multi-camera vision sensor, the positions of the pupils of both eyes and the line-of-sight directions are extracted, the three-dimensional coordinates of the intersection point (focus point) of the line-of-sight of both eyes are calculated, and a ray direction vector from the center of the user's both eyes to the focus point is generated.

[0079] For each regional unit, calculate the projection position of its spatial center point on the screen plane. Considering that in the general environment where a TV is located, the maximum viewing distance of a user is generally greater than the size of the environmental area, the ratio of the actual distance from the user's both eyes to the focus point to the theoretical maximum viewing distance is used to participate in the calculation of the focusing coefficient. The theoretical maximum viewing distance starts from the center of the user's binoculars and ends at the intersection point of the user's line of sight and the plane where the TV screen is located. If the intersection point of the user's line of sight and the plane where the TV screen is located exceeds the scope of the environment, such as interference from factors like wall obstruction, then the intersection point of the user's line of sight and the environmental boundary is used as the end point of the theoretical maximum viewing distance.

[0080] Meanwhile, when multiple users are watching the TV screen simultaneously, analyze the focusing coefficients of all users for each regional unit, and select the maximum value as the focusing coefficient of the regional unit to ensure that each user can obtain the best viewing experience when the screen mirroring image is optimized.

[0081] The following content is included in step S400:

[0082] The input of the image optimization strategy decision model is image feature parameters, the adopted optimization strategy, and the feature deviation between the image frame data after optimization using the current optimization strategy and the image frame data before screen mirroring. The output is the time-consuming data of the image optimization strategy; the resource consumption data of the optimization strategy is the time-consuming data of the image optimization strategy.

[0083] In specific implementation, the image optimization strategy decision model is constructed in the way of CNN (Convolutional Neural Network) + GNN (Graph Neural Network) + XGBoost (Extreme Gradient Boosting); structured relationship information is extracted by GNN (Graph Neural Network), local feature extraction is performed by CNN (Convolutional Neural Network), and final decision optimization is carried out by XGBoost to accurately predict the time consumed to optimize different image feature parameters to the image optimization feature deviation threshold using different optimization strategies.

[0084] The following content is included in step S500:

[0085] Obtain the time interval between consecutive frames of the image to be transmitted, and set the image optimization time threshold after the screen mirroring transmission of the image frame data to be transmitted according to the time interval between consecutive frames of the image to be transmitted.

[0086] Using the image optimization strategy decision model, taking the deviation to be optimized of each regional unit in the image frame data as the input, analyzing the time-consuming information of optimizing each regional unit to meet the image optimization feature deviation threshold judgment by adopting different optimization strategies, and setting the optimization strategy of the image frame data:

[0087] When the overall time consumption of each image unit in the image frame data adopting each optimization strategy is greater than the image optimization time threshold, the deviation to be optimized of each regional unit is enlarged proportionally until there is an optimization strategy that satisfies that the overall time consumption of each image unit in the image frame data adopting each optimization strategy is less than or equal to the image optimization time threshold, and this optimization strategy is used as the final optimization strategy of the image frame data;

[0088] When there is an optimization strategy with an overall time consumption less than or equal to the image optimization time threshold among the optimization strategies adopted by each image unit in the image frame data, the deviation to be optimized of each regional unit is reduced proportionally until there is and only one optimization strategy that satisfies that the overall time consumption of each image unit in the image frame data adopting this optimization strategy is less than or equal to the image optimization time threshold, and this optimization strategy is used as the final optimization strategy of the image frame data;

[0089] In specific implementation, considering that different optimization strategies have different optimization effects and optimization efficiencies on different image units, therefore, prediction is carried out through the image optimization strategy decision model, and then the optimization effects and optimization strategy time consumptions of each regional unit adopting different optimization strategies are analyzed;

[0090] When making an optimization strategy decision, it is assumed that the link transmission state is relatively good during the screen mirroring transmission of the image, the image distortion degree is relatively low, and the optimization strategy time loss is relatively short. At this time, the number of optimization strategies that satisfy the overall time consumption less than or equal to the image optimization time threshold is greater than 1. At this time, the requirements for the optimization effect of the strategy are improved, and the optimization strategy with the best optimization effect while satisfying the threshold judgment conditions is selected through a screening method;

[0091] It is assumed that when the image is being screen mirroring transmitted, due to the poor link transmission state, the image distortion degree is relatively high, and all optimization strategies cannot ensure that while meeting the image optimization feature deviation threshold, the overall time consumption is less than or equal to the image optimization time threshold. At this time, the image optimization requirements can be appropriately reduced to ensure the smoothness and coherence of the image display during screen mirroring.

[0092] As Figure 2 shown, the present invention also provides an artificial intelligence-based television image processing system, and the system includes: a screen mirroring transmission analysis module, a user gaze analysis module, and an optimization strategy decision module;

[0093] The screen mirroring transmission analysis module performs screen mirroring transmission feature annotation on each pixel point of the historical data of the screen mirroring images, trains a neural network model for predicting screen mirroring transmission images, calculates the feature deviation of each pixel point before and after screen mirroring transmission of the image frame data to be transmitted, and divides the image frame data to be transmitted into regional units; the user gaze analysis module monitors the eye data of all viewing users, analyzes the user's line of sight focus and user gaze direction information, determines the image optimization feature deviation threshold for each regional unit, and calculates the deviation to be optimized for each regional unit; the optimization strategy decision module determines the image optimization strategy for the image frame data to be transmitted according to the deviation to be optimized for each regional unit and using the trained image optimization strategy decision model.

[0094] The screen mirroring transmission analysis module includes: a transmission feature annotation unit, a feature deviation analysis unit, and a regional unit division unit;

[0095] The transmission feature annotation unit performs screen mirroring transmission feature annotation on each pixel point of the historical data of the screen mirroring images; the feature deviation analysis unit uses the neural network model for predicting screen mirroring transmission images to predict the image feature parameters after screen mirroring transmission of the image frame data to be transmitted, and calculates the feature deviation of each pixel point before and after screen mirroring transmission of the image frame data to be transmitted; the regional unit division unit divides the image frame data to be transmitted into regional units according to the feature deviation of each pixel point before and after screen mirroring transmission of the image frame data to be transmitted.

[0096] The user gaze analysis module includes: a user monitoring unit, an orientation analysis unit, and a deviation setting unit;

[0097] The user monitoring unit is used to monitor the eye data of all viewing users; the orientation analysis unit is used to construct a three-dimensional coordinate system and analyze the orientation information of the intersection point of the user's line of sight focus, the user's gaze direction and the plane where the TV screen is located in the three-dimensional coordinate system; the deviation setting unit is used to determine the image optimization feature deviation threshold for each regional unit in the image frame data to be transmitted and calculate the deviation to be optimized for each regional unit.

[0098] The optimization strategy decision module includes: a decision model construction unit and an optimization strategy decision unit;

[0099] The decision model construction unit trains the image optimization strategy decision model by constructing an image optimization feature data set; the optimization strategy decision unit adjusts the deviation to be optimized for each regional unit in the image frame data according to the overall time consumed by each optimization strategy to optimize each regional unit of the image frame data, and then determines the final optimization strategy for image frame optimization.

[0100] It is obvious to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, in all respects, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

Claims

1. An image processing method for a television based on artificial intelligence, characterized in that , The method includes the following steps: Step S100: Obtain the historical data of the TV screen mirroring images, perform screen mirroring transmission feature annotation on each image frame pixel point in the historical data of the screen mirroring images, and construct an image transmission feature data set; Step S200: Use the image transmission feature data set to train a screen mirroring transmission image prediction neural network model, calculate the feature deviation of each pixel point before and after the screen mirroring transmission of the image frame data to be transmitted, and then perform regional unit division according to the feature deviation; Step S300: Construct a three-dimensional coordinate system, monitor the eye data of all viewing users, analyze the user's line of sight focus point and the user's gaze direction information, determine the image optimization feature deviation threshold of each regional unit, and calculate the deviation to be optimized of each regional unit; Step S400: Obtain the historical data of the TV screen mirroring image optimization, extract the image feature parameters, optimization strategy data, feature deviation between the optimized image frame data and the image frame data before screen mirroring, and optimization strategy resource consumption data, construct an image optimization feature data set, and train an image optimization strategy decision model; Step S500: According to the deviation to be optimized of each regional unit of the image frame data to be transmitted, use the image optimization strategy decision model to determine the image optimization strategy of the TV screen mirroring image frame data to be transmitted.

2. The method for processing television image based on artificial intelligence according to claim 1, wherein The method for performing screen mirroring transmission feature annotation on each image frame pixel point in the historical data of the screen mirroring images in step S100 is as follows: For any pixel point p in the image frame data, the screen mirroring transmission feature annotation is: p[(x_p,y_p),F_before(p),F_after(p),S_tran(p)]; where, x_p is the serial number of pixel point p arranged from left to right in the image frame, y_p is the serial number of pixel point p arranged from top to bottom in the image frame, F_before(p) is the set of image feature parameters of pixel point p before transmission, F_after(p) is the set of image feature parameters of pixel point p after transmission, and S_tran(p) is the set of transmission link state parameters when the image frame data to which pixel point p belongs is screen mirroring transmitted.

3. The image processing method of a television based on artificial intelligence according to claim 1, wherein The following steps are included in step S200: Step S201: Use the image transmission feature data set to train a screen mirroring transmission image prediction neural network model, and analyze the predicted data of the image feature parameters of each pixel point after the screen mirroring transmission of the image frame data to be transmitted according to the image transmission features of each pixel point of the image frame data to be transmitted; Step S202: Calculate the deviation between the predicted data of each image feature parameter of each pixel point after the screen mirroring transmission of the image frame data to be transmitted and each image feature parameter before the screen mirroring transmission, and calculate the feature deviation of each pixel point before and after the screen mirroring transmission of the image frame data to be transmitted through weighted calculation; Step S203: Set a feature deviation interval, and divide the pixel points with the same feature deviation in the same feature deviation interval and adjacent in the image frame data to be transmitted into the same regional unit; For any region unit, calculate the distances between each pixel point in it in the image, set a pixel distance threshold, and perform a threshold judgment on the two pixel points with the largest distance. If the distance is less than the pixel distance threshold, no further division is made. If the distance is greater than or equal to the pixel distance threshold, take the two pixel points as the clustering centers, take the distance between the pixel points as the clustering basis, and further divide the region unit into two region units. Repeat the above operations until the distances between all pixel points in each region unit are less than the pixel distance threshold.

4. The image processing method of a television based on artificial intelligence according to claim 1, wherein The steps in step S300 include the following steps: Step S301: Taking the center point of the TV screen as the origin, the horizontal direction from left to right as the x-axis, the vertical direction from top to bottom as the y-axis, and the direction perpendicular to the screen as the z-axis, establish a three-dimensional coordinate system; Step S302: Use a vision sensor to monitor and identify all user facial feature information, extract user line-of-sight information, and analyze the user line-of-sight focus point and user gaze direction information; wherein, the user line-of-sight focus point is the intersection direction of the user's binocular lines of sight, and the user gaze direction is the ray direction from the center point of the user's binoculars to the user line-of-sight focus point; Step S303: Use a clustering algorithm to calculate the spatial center point of the pixel points in each region unit in the image frame data, and determine the focusing coefficient \(k_x\) of the user for any region unit \(x\) according to the formula: \(k_x = R_e\times\exp(-l)\); where \(R_e\) is the ratio of the distance from the center point of the user's binoculars to the user line-of-sight focus point to the distance from the center point of the user's binoculars to the intersection point of the user's gaze direction and the plane where the TV screen is located, and \(l\) is the distance between the intersection point of the user's gaze direction and the plane where the TV screen is located and the spatial center point of region unit \(x\); Step S303: Calculate the focusing coefficients of all users for each region unit, and take the maximum value as the focusing coefficient of each region unit. Set the image optimization feature deviation threshold \(th_x\) of any region unit \(x\) as: \(th_x=(1 - lpha)\times k_x\times C_dev\); where \(lpha\) is the image optimization adjustment coefficient, \(k_x\) is the focusing coefficient of any region unit, and \(C_dev\) is the maximum deviation of the image optimization feature; Step S304: Further calculate the difference between the center of the feature deviation interval to which each region unit belongs and the image optimization feature deviation threshold as the deviation to be optimized for each region unit.

5. A method for processing television image based on artificial intelligence according to claim 1, characterized in that, The content in step S400 includes the following: The input of the image optimization strategy decision model is image feature parameters, the optimization strategy adopted, and the feature deviation between the image frame data after optimization using the current optimization strategy and the image frame data before screen mirroring. The output is the time-consuming data of the image optimization strategy; the optimization strategy resource consumption data is the time-consuming data of the image optimization strategy.

6. The method for processing television image based on artificial intelligence according to claim 1, characterized in that The content in step S500 includes the following: Obtain the time interval between adjacent image frames to be transmitted, and set the image optimization time threshold after the screen mirroring transmission of the image frame data to be transmitted according to the time interval between adjacent image frames to be transmitted; Use the image optimization strategy decision model, take the deviation to be optimized of each region unit in the image frame data as the input, analyze the time-consuming information of optimizing each region unit to meet the image optimization feature deviation threshold judgment using different optimization strategies, and set the optimization strategy of the image frame data: When the overall time consumption of each image unit in the image frame data using each optimization strategy is greater than the image optimization time threshold, the to-be-optimized deviation of each region unit is enlarged proportionally until there is an optimization strategy that satisfies that the overall time consumption of each image unit in the image frame data using each optimization strategy is less than or equal to the image optimization time threshold, and this optimization strategy is used as the final optimization strategy for the image frame data; When there is an optimization strategy with an overall time consumption less than or equal to the image optimization time threshold among the optimization strategies used by each image unit in the image frame data, the to-be-optimized deviation of each region unit is reduced proportionally until there is exactly one optimization strategy that satisfies that the overall time consumption of each image unit in the image frame data using this optimization strategy is less than or equal to the image optimization time threshold, and this optimization strategy is used as the final optimization strategy for the image frame data.

7. An artificial intelligence-based television image processing system applying an artificial intelligence-based television image processing method according to any one of claims 1-6, characterized in that, The system includes: a screen mirroring transmission analysis module, a user gaze analysis module, and an optimization strategy decision module; The screen mirroring transmission analysis module performs screen mirroring transmission feature annotation on each pixel point of the historical screen mirroring image data, trains a screen mirroring transmission image prediction neural network model, calculates the feature deviation of each pixel point before and after the screen mirroring transmission of the to-be-transmitted image frame data, and divides the to-be-transmitted image frame data into region units; the user gaze analysis module monitors the eye data of all viewing users, analyzes the user's line-of-sight focus point and the user's gaze direction information, determines the image optimization feature deviation threshold of each region unit, and calculates the to-be-optimized deviation of each region unit; the optimization strategy decision module determines the image optimization strategy of the to-be-transmitted image frame data according to the to-be-optimized deviation of each region unit by using the trained image optimization strategy decision model.

8. An image processing system for a television based on artificial intelligence according to claim 7, wherein The screen mirroring transmission analysis module includes: a transmission feature annotation unit, a feature deviation analysis unit, and a region unit division unit; The transmission feature annotation unit performs screen mirroring transmission feature annotation on each pixel point of the historical screen mirroring image data; the feature deviation analysis unit uses the screen mirroring transmission image prediction neural network model to predict the image feature parameters after the screen mirroring transmission of the to-be-transmitted image frame data and calculates the feature deviation of each pixel point before and after the screen mirroring transmission of the to-be-transmitted image frame data; the region unit division unit divides the to-be-transmitted image frame data into region units according to the feature deviation of each pixel point before and after the screen mirroring transmission of the to-be-transmitted image frame data.

9. An image processing system for a television based on artificial intelligence according to claim 7, wherein, The user gaze analysis module includes: a user monitoring unit, an orientation analysis unit, and a deviation setting unit; The user monitoring unit is used to monitor the eye data of all viewing users; the orientation analysis unit is used to construct a three-dimensional coordinate system and analyze the orientation information of the user's line-of-sight focus point, the user's gaze direction, and the intersection point of the user's gaze direction and the plane where the TV screen is located in the three-dimensional coordinate system; the deviation setting unit is used to determine the image optimization feature deviation threshold of each region unit in the to-be-transmitted image frame data and calculate the to-be-optimized deviation of each region unit.

10. An image processing system for a television based on artificial intelligence according to claim 7, characterized in that, The optimization strategy decision module includes: a decision model construction unit and an optimization strategy decision unit; The decision model construction unit trains an image optimization strategy decision model by constructing an image optimization feature data set; the optimization strategy decision unit adjusts the deviation to be optimized for each regional unit in the image frame data according to the overall time consumed by each optimization strategy to optimize each regional unit in the image frame data, and then determines the final optimization strategy for image frame optimization.

Images