A television image processing system and method based on artificial intelligence

Abstract

The application discloses a television image processing system and method based on artificial intelligence, and relates to the technical field of television image processing.The system comprises a screen projection transmission analysis module, a user gaze analysis module and an optimization strategy decision module.The screen projection transmission analysis module performs screen projection transmission feature labeling on each image frame pixel point of historical screen projection image data, trains a screen projection transmission image prediction neural network model, calculates feature deviation before and after screen projection transmission of to-be-transmitted image frame data, and divides the to-be-transmitted image frame data into regional units.The user gaze analysis module monitors eye data of all watching users, analyzes user line-of-sight focus points and user gaze direction information, determines image optimization feature deviation thresholds of each regional unit, and calculates to-be-optimized deviations of each regional unit.The optimization strategy decision module determines an image optimization strategy of the to-be-transmitted image frame data by using the trained image optimization strategy decision model according to the to-be-optimized deviations of each regional unit.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of television image processing, and particularly relates to a television image processing system and method based on artificial intelligence. BACKGROUND

[0002] With the introduction of screen projection playing technology in various video websites, image optimization for television screen projection is particularly important in screen projection playing.

[0003] In existing television screen projection playing, the following technical defects exist: a traditional compression algorithm (such as H.264 / HEVC) uses a global quantization parameter, lacks dynamic perception of the state of an image data transmission link, and causes the distortion degree of images in different texture feature regions to be different, which leads to a rigid image region division method and the inability to effectively distinguish the distortion degree of different image regions; in analyzing a local optimization strategy for an image after screen projection transmission, a self-adaptive optimization method based on user vision for different image regions lacks reasonable analysis of suitable optimization strategies for different image regions, which leads to a lack of balance in the distortion degree of each image region in the final optimization effect and the inability to more accurately display image features before screen projection image transmission.

[0004] Therefore, a television image processing system and method based on artificial intelligence are needed to solve the above technical defects. SUMMARY

[0005] The present application aims to provide a television image processing system and method based on artificial intelligence to solve the problems in the prior art.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0007] A television image processing method based on artificial intelligence, the method comprising the following steps:

[0008] Step S100: acquiring television screen projection image historical data, performing screen projection transmission feature labeling on each image frame pixel point in the screen projection image historical data, and constructing an image transmission feature data set;

[0009] Step S200: training a screen projection transmission image prediction neural network model using the image transmission feature data set, and calculating the feature deviation of each pixel point before and after screen projection transmission of to-be-transmitted image frame data, and then performing region unit division according to the feature deviation;

[0010] Step S300: constructing a three-dimensional coordinate system, monitoring eye data of all watching users, analyzing user visual focus points and user gaze direction information, and then determining the image optimization feature deviation threshold of each region unit and calculating the to-be-optimized deviation of each region unit;

[0011] Step S400: Obtain the television screen projection image optimization history data, extract the image feature parameters, optimization strategy data, optimized image frame data and image frame data feature deviation before projection, and optimization strategy resource consumption data, construct an image optimization feature data set, and train an image optimization strategy decision model;

[0012] Step S500: According to the image frame data to be transmitted, the image optimization strategy decision model is used to determine the image optimization strategy of the television screen projection image frame data to be transmitted.

[0013] In the above technical solution, the step S100 is as follows:

[0014] For any pixel point p in the image frame data, the screen projection transmission feature labeling is: p[(x_p, y_p), F_before(p), F_after(p), S_tran(p)]; Wherein, x_p is the number of pixel point p arranged from left to right in the image frame, y_p is the number of pixel point p arranged from top to bottom in the image frame, F_before(p) is the image feature parameter set before pixel point p transmission, F_after(p) is the image feature parameter set after pixel point p transmission, S_tran(p) is the transmission link state parameter set of the image frame data to which the pixel point p belongs when the image frame data is projected and transmitted.

[0015] By comparing the pixel-level features, the image transmission distortion caused by transmission, such as color deviation and block effect, is located, and the quantitative relationship between link features and image distortion is established to provide input dimension for the prediction model.

[0016] In the above technical solution, the step S200 includes the following steps:

[0017] Step S201: Use the image transmission feature data set to train a screen projection transmission image prediction neural network model, and analyze the image transmission features of each pixel point of the image frame data to be transmitted according to the image transmission features of each pixel point of the image frame data to be transmitted, and predict the image feature parameters of each pixel point after the image frame data to be transmitted is projected and transmitted;

[0018] Step S202: Calculate the deviation between the image feature parameter prediction data of each pixel point of the image frame data to be transmitted after projection and transmission and the image feature parameters before projection and transmission, and calculate the feature deviation of each pixel point of the image frame data to be transmitted before and after projection and transmission by weighting;

[0019] Step S203: Set the feature deviation interval, and divide the pixel points with the same feature deviation interval and adjacent in the image frame data to be transmitted into the same area unit;

[0020] For any region unit, the distance between each pixel point in the image is calculated, a pixel distance threshold is set, and the two pixel points with the largest distance are subjected to threshold judgment, if the distance is less than the pixel distance threshold, no further division is made, if the distance is greater than or equal to the pixel distance threshold, the two pixel points are taken as clustering centers and the distance between the pixel points is taken as clustering basis, the region unit is further divided into two region units, and the above operation is repeated until the distance between each pixel point in all region units is less than the pixel distance threshold;

[0021] By constructing a neural network model, the image transmission distortion effect under different data transmission link states is predicted, the image transmission distortion degree is quantitatively analyzed, and the images in different regions are classified and summarized, so that the low deviation pixel points and high deviation pixel points caused by fixed division are effectively avoided, and the adaptability and rationality of local optimization of the projection image are significantly improved.

[0022] In the above technical solution, the step S300 comprises the following steps:

[0023] Step S301: A three-dimensional coordinate system is established with the center point of the TV screen as the origin, the horizontal left-to-right direction as the x-axis, the vertical top-to-bottom direction as the y-axis, and the vertical screen direction as the z-axis;

[0024] Step S302: All user facial feature information is monitored and identified using a visual sensor, user gaze information is extracted, and user gaze focus points and user gaze direction information are analyzed; wherein the user gaze focus point is the intersection direction of the user's two eye gaze lines, and the user gaze direction is the ray direction from the user's two eye center points to the user gaze focus point;

[0025] Step S303: A clustering algorithm is used to calculate the spatial center points of the pixel points in each region unit in the image frame data, and the focus coefficient k x of the user to any region unit x is determined according to the formula: k x = R e x exp(-l); wherein R e is the distance ratio of the user's two eye center points to the user's gaze focus point to the intersection distance of the user's two eye center points to the user's gaze direction and the plane of the TV screen, and l is the distance between the intersection of the user's two eye center points to the user's gaze direction and the plane of the TV screen and the spatial center point of the region unit x;

[0026] Step S303: The focus coefficients of all users to each region unit are calculated, and the maximum value is taken as the focus coefficient of each region unit, and the image optimization feature deviation threshold th x of any region unit x is set as: th x = (1-α) x k x x C dev; wherein α is an image optimization adjustment coefficient, k x is the focus coefficient of any region unit, and C dev is the maximum deviation of the image optimization feature;

[0027] Step S304: further calculate the difference between the feature deviation interval center to which each regional unit belongs and the image optimization feature deviation threshold value as the to-be-optimized deviation of each regional unit;

[0028] The traditional technical solution breaks through the two-dimensional plane analysis method based on user visual perception, comprehensively analyzes the orientation relationship among the user visual line focus, the visual line direction and the center of the TV screen in the three-dimensional level, makes the user visual effect analysis more refined, and further improves the practicality of the local optimization strategy of the projection image and the user experience by correlating the user visual effect and the optimization demand of each area of the projection image.

[0029] In the above technical solution, the step S400 includes the following contents:

[0030] The image optimization strategy decision model is inputted with the image feature parameters, the adopted optimization strategy, the feature deviation between the image frame data after optimization by the current optimization strategy and the image frame data before projection, and is outputted with the image optimization strategy time consumption data; the optimization strategy resource consumption data is the image optimization strategy time consumption data.

[0031] In the above technical solution, the step S500 includes the following contents:

[0032] The time interval between the to-be-transmitted image frames is acquired, and the image optimization time threshold value after the projection transmission of the to-be-transmitted image frame data is set according to the time interval between the to-be-transmitted image frames;

[0033] The image optimization strategy decision model is used, the to-be-optimized deviation of each regional unit in the image frame data is taken as the input, the time consumption information of using different optimization strategies to optimize each regional unit to meet the image optimization feature deviation threshold value is analyzed, and the optimization strategy of the image frame data is set:

[0034] When the overall time consumption of each image unit in the image frame data using each optimization strategy is greater than the image optimization time threshold value, the to-be-optimized deviation of each regional unit is enlarged in proportion, until there is an optimization strategy that meets the condition that the overall time consumption of each image unit in the image frame data using each optimization strategy is less than or equal to the image optimization time threshold value, and the optimization strategy is taken as the final optimization strategy of the image frame data;

[0035] When there is an optimization strategy in each optimization strategy that the overall time consumption of each image unit in the image frame data using the optimization strategy is less than or equal to the image optimization time threshold value, the to-be-optimized deviation of each regional unit is reduced in proportion, until there is only one optimization strategy that meets the condition that the overall time consumption of each image unit in the image frame data using the optimization strategy is less than or equal to the image optimization time threshold value, and the optimization strategy is taken as the final optimization strategy of the image frame data;

[0036] By dynamically scaling the to-be-optimized deviation of the regional unit, the optimization strategy meeting the image inter-frame time interval requirement is screened and determined, so that the screen projection image meets the requirement of smooth display, the consistency of the overall perception effect of the image and the original effect of the image is improved, and the balance of the overall optimization effect and the display effect of the effective screen projection image is effectively ensured.

[0037] An artificial intelligence-based television image processing system based on one of the artificial intelligence-based television image processing methods in the above technical solutions, the system comprising: a screen projection transmission analysis module, a user gaze analysis module, and an optimization strategy decision module.

[0038] The screen projection transmission analysis module performs screen projection transmission feature labeling on each image frame pixel point of the screen projection image historical data, trains a screen projection transmission image prediction neural network model, calculates the feature deviation of each pixel point before and after the screen projection transmission of the to-be-transmitted image frame data, and divides the to-be-transmitted image frame data into regional units; the user gaze analysis module monitors the eye data of all watching users, analyzes the user visual focus point and user gaze direction information, determines the image optimization feature deviation threshold of each regional unit, and calculates the to-be-optimized deviation of each regional unit; and the optimization strategy decision module determines the image optimization strategy of the to-be-transmitted image frame data using the trained image optimization strategy decision model according to the to-be-optimized deviation of each regional unit.

[0039] In the above technical solution, the screen projection transmission analysis module comprises a transmission feature labeling unit, a feature deviation analysis unit, and a regional unit division unit.

[0040] The transmission feature labeling unit performs screen projection transmission feature labeling on each image frame pixel point of the screen projection image historical data; the feature deviation analysis unit uses the screen projection transmission image prediction neural network model to predict the image feature parameters of the to-be-transmitted image frame data after the screen projection transmission, and calculates the feature deviation of each pixel point before and after the screen projection transmission of the to-be-transmitted image frame data; and the regional unit division unit divides the to-be-transmitted image frame data into regional units according to the feature deviation of each pixel point before and after the screen projection transmission of the to-be-transmitted image frame data.

[0041] In the above technical solution, the user gaze analysis module comprises a user monitoring unit, an orientation analysis unit, and a deviation setting unit.

[0042] The user monitoring unit is configured to monitor the eye data of all watching users; the orientation analysis unit is configured to construct a three-dimensional coordinate system, and analyze the orientation information of the intersection of the user visual focus point, the user gaze direction, and the plane where the television screen is located in the three-dimensional coordinate system; and the deviation setting unit is configured to determine the image optimization feature deviation threshold of each regional unit in the to-be-transmitted image frame data, and calculate the to-be-optimized deviation of each regional unit.

[0043] In the technical solution, the optimization strategy decision module comprises a decision model construction unit and an optimization strategy decision unit.

[0044] The decision model construction unit trains an image optimization strategy decision model by constructing an image optimization feature dataset.

[0045] Compared with the prior art, the present application has the following advantages:

[0046] In the present application, by analyzing the deviation synchronization mechanism of multi-objective optimization strategy, the multi-dimensional parameters such as picture quality improvement, time delay constraint and resource consumption are cooperatively optimized based on a reinforcement learning framework, the frame processing time threshold is forcibly met by using a proportional scaling algorithm, the strategy conflict problem of a traditional rule engine is solved, and the smoothness and picture quality balance of a high dynamic scene are significantly improved.

[0047] In the present application, by constructing a three-dimensional visual perception dynamic optimization model, the user's visual focus point is captured in real time by using binocular visual tracking technology, the gaze area is modeled in three-dimensional space by combining a depth scaling coefficient and a space projection algorithm, the positioning deviation of a traditional two-dimensional weight model under an inclined viewing angle is solved, and the optimization intensity is dynamically adjusted by using a focus coefficient, so that the image optimization resources are reasonably allocated on demand.

[0048] In the present application, pixel-level transmission feature dynamic labeling technology is used to monitor and analyze the projection image in real time, the perception deficiency of traditional macroblock-level compression algorithms to local distortion is avoided, a pixel-level distortion tracing system is constructed by fusing transmission link state parameters and multi-dimensional image features, the transmission defect positioning accuracy is significantly improved, and the adaptive ability of the system in a complex network environment is enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0049] Fig. 1 A flowchart of a television image processing method based on artificial intelligence is provided.

[0050] Fig. 2 A television image processing system organization structure diagram based on artificial intelligence is provided. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0052] Embodiment: please refer to Figs. 1-2 The application provides the following technical solutions:

[0053] As Fig. 1 The application provides a television image processing method based on artificial intelligence, which comprises the following steps:

[0054] Step S100: acquiring television screen projection image historical data, marking the screen projection transmission characteristics of each image frame pixel point in the screen projection image historical data, and constructing an image transmission characteristic data set;

[0055] Step S200: training a screen projection transmission image prediction neural network model using the image transmission characteristic data set, and calculating the characteristic deviation of each pixel point before and after the screen projection transmission of the to-be-transmitted image frame data, and then dividing the regions into units according to the characteristic deviation;

[0056] Step S300: constructing a three-dimensional coordinate system, monitoring the eye data of all watching users, analyzing the user visual focus point and user gaze direction information, and then determining the image optimization characteristic deviation threshold of each region unit and calculating the to-be-optimized deviation of each region unit;

[0057] Step S400: acquiring television screen projection image optimization historical data, extracting image characteristic parameters, optimization strategy data, the characteristic deviation of the image frame data after optimization and the image frame data before screen projection, optimization strategy resource consumption data, constructing an image optimization characteristic data set, and training an image optimization strategy decision model;

[0058] Step S500: determining the image optimization strategy of the television screen projection to-be-transmitted image frame data using the image optimization strategy decision model according to the to-be-optimized deviation of each region unit of the to-be-transmitted image frame data.

[0059] The method for marking the screen projection transmission characteristics of each image frame pixel point in the screen projection image historical data in the step S100 is as follows:

[0060] For any pixel point p in the image frame data, the screen projection transmission characteristic marking is: p[(x_p, y_p), F_before(p), F_after(p), S_tran(p)]; wherein x_p is the number of the pixel point p arranged from left to right in the image frame, y_p is the number of the pixel point p arranged from top to bottom in the image frame, F_before(p) is the image characteristic parameter set before the transmission of the pixel point p, F_after(p) is the image characteristic parameter set after the transmission of the pixel point p, and S_tran(p) is the transmission link state parameter set of the image frame data to which the pixel point p belongs when the image frame data is screen projected and transmitted;

[0061] In a specific implementation, the television screen projection history image data is acquired, and multi-dimensional features of each pixel point are extracted, including multi-dimensional features of the pixel points before and after transmission, including color space, local texture complexity, gradient intensity, etc.

[0062] However, since the image projection transmission needs to be encoded and compressed, etc., which includes block processing of image data, and the pixel points in different blocks also have different degrees of attenuation, the position data of the pixel points in the image is introduced for analysis. In addition, since the texture feature distortion degree of the image is different under different transmission link states during the projection transmission, the link state parameters during the image transmission need to be analyzed synchronously.

[0063] The step S200 includes the following steps:

[0064] Step S201: using the image transmission feature data set to train a projection transmission image prediction neural network model, and according to the image transmission feature of each pixel point of the to-be-transmitted image frame data, analyzing the image feature parameter prediction data of each pixel point after the projection transmission of the to-be-transmitted image frame data;

[0065] Step S202: calculating the deviation between the image feature parameter prediction data of each pixel point after the projection transmission of the to-be-transmitted image frame data and the image feature parameters before the projection transmission, and calculating the feature deviation of each pixel point before and after the projection transmission of the to-be-transmitted image frame data by weighting;

[0066] Step S203: setting a feature deviation interval, and dividing the pixel points with the same feature deviation in the same feature deviation interval and adjacent in the to-be-transmitted image frame into the same area unit;

[0067] For any area unit, the distance between each pixel point in the image is calculated, a pixel distance threshold is set, and a threshold judgment is made on the two pixel points with the largest distance. If the distance is less than the pixel distance threshold, no further division is made. If the distance is greater than or equal to the pixel distance threshold, the two pixel points are taken as clustering centers, and the distance between the pixel points is taken as the clustering basis, so that the area unit is further divided into two area units. The above operation is repeated until the distance between each pixel point in all area units is less than the pixel distance threshold;

[0068] A conditional generation neural network model is constructed, the input is the image feature parameter data before the transmission of the pixel points and the transmission link state parameter set during the projection transmission, and the prediction accuracy of the model on the distortion effect of the complex image texture after the projection transmission is improved through the adversarial training strategy;

[0069] Further, according to the influence of the deviation of each image feature parameter on the image display effect, the deviation weight of each image feature parameter is set, and the feature deviation of each pixel point is analyzed by weighting;

[0070] In the image feature parameter deviation weight setting process, since the influence of the deviation of part of the image feature parameters on the image display effect is not a linear correlation relationship, a dynamic weight method can be used to ensure more accurate description of the influence of each image feature parameter on the overall image effect display in different deviation levels.

[0071] Further, in the image local adaptive optimization process, considering that the difference between the pixel points with a large interval in the two-dimensional direction of the image and the image feature texture is generally large in the projection transmission, if there are pixel points with a large interval in a region unit, the region unit is further divided, and after repeated division, the refinement of all region units is ensured, and the precision loss of the image local optimization strategy is avoided.

[0072] The step S300 includes the following steps:

[0073] Step S301: Establish a three-dimensional coordinate system with the center point of the TV screen as the origin, the horizontal left-to-right direction as the x-axis, the vertical up-to-down direction as the y-axis, and the vertical screen direction as the z-axis.

[0074] Step S302: Use a visual sensor to monitor and identify all user facial feature information, extract user gaze information, and analyze user gaze focus points and user gaze direction information; wherein the user gaze focus point is the intersection direction of the user's two eye gaze directions, and the user gaze direction is the ray direction from the user's two eye center points to the user's gaze focus point.

[0075] Step S303: Use a clustering algorithm to calculate the spatial center points of the pixel points in each region unit in the image frame data, and determine the focus coefficient k x of the user on any region unit x according to the formula: k x = R e × exp (-l); wherein R e is the distance ratio of the user's two eye center points to the user's gaze focus point to the intersection distance of the user's two eye center points to the user's gaze direction and the plane of the TV screen, and l is the distance between the intersection of the user's two eye center points to the user's gaze direction and the plane of the TV screen and the spatial center point of the region unit x.

[0076] Step S303: Calculate the focus coefficients of all users on each region unit, and take the maximum value as the focus coefficient of each region unit. The image optimization feature deviation threshold th x of any region unit x is set as: th x = (1-α)×k x ×C dev; wherein α is an image optimization adjustment coefficient, k x is the focus coefficient of any region unit, and C dev is the maximum deviation of the image optimization feature.

[0077] Step S304: Further calculate the difference between the center of the feature deviation interval to which each region unit belongs and the image optimization feature deviation threshold, as the optimized deviation of each region unit.

[0078] In specific implementation, a body coordinate system is established with the TV screen as a reference, the user's facial features are captured by a multi-view vision sensor, the positions of the pupils of the two eyes and the directions of the lines of sight are extracted, the three-dimensional coordinates of the intersection points (focal points) of the lines of sight of the two eyes are calculated, and the direction vectors of the rays from the centers of the two eyes to the focal points are generated;

[0079] For each area unit, the projection position of the spatial center point thereof on the screen plane is calculated, and in consideration of the fact that the maximum viewing distance of a user in the environment where a general TV is located is generally greater than the size of the environment area, the proportion of the actual distance from the user's two eyes to the focal point to the theoretical maximum viewing distance is used to participate in the calculation of the focusing coefficient, wherein the theoretical maximum viewing distance takes the center of the user's two eyes as the starting point and takes the intersection point of the user's line of sight and the plane where the TV screen is located as the end point, and if the intersection point of the user's line of sight and the boundary of the environment is beyond the range of the environment, for example, due to the interference of factors such as a wall, the intersection point of the user's line of sight and the boundary of the environment is taken as the end point of the theoretical maximum viewing distance.

[0080] Meanwhile, in consideration of the fact that multiple users watch the TV screen at the same time, the focusing coefficients of all users to each area unit are analyzed, and the maximum value is selected as the focusing coefficient of the area unit, so as to ensure that each user can obtain the best viewing experience when the projection image is optimized.

[0081] The step S400 includes the following contents:

[0082] The image optimization strategy decision model is inputted with image feature parameters, an adopted optimization strategy, and the feature deviation between the image frame data after optimization by using the current optimization strategy and the image frame data before projection; and is outputted with image optimization strategy time consumption data; and the optimization strategy resource consumption data is the image optimization strategy time consumption data.

[0083] In specific implementation, the image optimization strategy decision model is constructed in a CNN (convolutional neural network) + GNN (graph neural network) + XGBoost (extreme gradient boosting) manner; the structured relationship information is extracted by GNN (graph neural network), the local feature is extracted by CNN (convolutional neural network), and the final decision optimization is performed by XGBoost, so as to accurately predict the time consumed for optimizing different image feature parameters to the image optimization feature deviation threshold by using different optimization strategies.

[0084] The step S500 includes the following contents:

[0085] The time interval between the to-be-transmitted image frames is acquired, and the image optimization time threshold after the projection transmission of the to-be-transmitted image frame data is set according to the time interval between the to-be-transmitted image frames;

[0086] The image optimization strategy decision model is used to take the to-be-optimized deviation of each regional unit in the image frame data as input, analyze the time consumption of each regional unit optimized by using different optimization strategies to meet the image optimization characteristic deviation threshold judgment, and set the optimization strategy of the image frame data:

[0087] When the overall time consumption of each image unit in the image frame data using each optimization strategy is greater than the image optimization time threshold, the to-be-optimized deviation of each regional unit is proportionally enlarged until there is an optimization strategy that meets the condition that the overall time consumption of each image unit in the image frame data using the optimization strategy is less than or equal to the image optimization time threshold, and the optimization strategy is taken as the final optimization strategy of the image frame data.

[0088] When there is an optimization strategy in each optimization strategy that meets the condition that the overall time consumption is less than or equal to the image optimization time threshold, the to-be-optimized deviation of each regional unit is proportionally reduced until there is only one optimization strategy that meets the condition that the overall time consumption of each image unit in the image frame data using the optimization strategy is less than or equal to the image optimization time threshold, and the optimization strategy is taken as the final optimization strategy of the image frame data.

[0089] In a specific implementation, different optimization strategies have different optimization effects and optimization efficiencies on different image units, so the image optimization strategy decision model is used to predict and analyze the optimization effect and optimization strategy time consumption of each regional unit using different optimization strategies.

[0090] When the optimization strategy is decided, it is assumed that the link transmission state is good when the image is transmitted by projection, the image distortion degree is low, and the optimization strategy time loss is short, so that the number of optimization strategies that meet the condition that the overall time consumption is less than or equal to the image optimization time threshold is greater than 1, and the requirement for the optimization effect of the strategy is improved, and the optimization strategy that meets the threshold judgment condition and has the best optimization effect is selected by selection and screening.

[0091] It is assumed that when the image is transmitted by projection, the link transmission state is poor, the image distortion degree is high, and all optimization strategies cannot ensure that the image optimization characteristic deviation threshold is met while the overall time consumption is less than or equal to the image optimization time threshold, so the image optimization requirement can be appropriately reduced to ensure the smoothness and coherence of the image display of the image projection.

[0092] As shown in Fig. 2 The application also provides a television image processing system based on artificial intelligence, which comprises a projection transmission analysis module, a user gaze analysis module, and an optimization strategy decision module.

[0093] The screen projection transmission analysis module labels the pixel points of each image frame of the screen projection image historical data with screen projection transmission features, trains a screen projection transmission image prediction neural network model, calculates the feature deviation of each pixel point before and after the screen projection transmission of the to-be-transmitted image frame data, and divides the to-be-transmitted image frame data into regional units; the user gaze analysis module monitors the eye data of all watching users, analyzes the user visual focus point and user gaze direction information, determines the image optimization feature deviation threshold of each regional unit, and calculates the to-be-optimized deviation of each regional unit; and the optimization strategy decision module determines the image optimization strategy of the to-be-transmitted image frame data according to the to-be-optimized deviation of each regional unit by using the trained image optimization strategy decision model.

[0094] The screen projection transmission analysis module comprises a transmission feature labeling unit, a feature deviation analysis unit, and a regional unit division unit.

[0095] The transmission feature labeling unit labels the pixel points of each image frame of the screen projection image historical data with screen projection transmission features; the feature deviation analysis unit uses a screen projection transmission image prediction neural network model to predict the image feature parameters of the to-be-transmitted image frame data after the screen projection transmission, and calculates the feature deviation of each pixel point before and after the screen projection transmission of the to-be-transmitted image frame data; and the regional unit division unit divides the to-be-transmitted image frame data into regional units according to the feature deviation of each pixel point before and after the screen projection transmission of the to-be-transmitted image frame data.

[0096] The user gaze analysis module comprises a user monitoring unit, an orientation analysis unit, and a deviation setting unit.

[0097] The user monitoring unit is configured to monitor the eye data of all watching users; the orientation analysis unit is configured to construct a three-dimensional coordinate system, and analyze the orientation information of the intersection of the user visual focus point, the user gaze direction, and the plane where the television screen is located in the three-dimensional coordinate system; and the deviation setting unit is configured to determine the image optimization feature deviation threshold of each regional unit in the to-be-transmitted image frame data, and calculate the to-be-optimized deviation of each regional unit.

[0098] The optimization strategy decision module comprises a decision model construction unit and an optimization strategy decision unit.

[0099] The decision model construction unit trains an image optimization strategy decision model by constructing an image optimization feature data set; and the optimization strategy decision unit adjusts the to-be-optimized deviation of each regional unit in the image frame data according to the overall time consumption of each optimization strategy for optimizing each regional unit of the image frame data, and then determines the final optimization strategy for the image frame optimization.

[0100] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims with respect to the figures of the patent document.

Claims

1. A television image processing method based on artificial intelligence, characterized in that... The method includes the following steps: Step S100: Obtain historical data of TV screen projection images, perform screen projection transmission feature annotation on each image frame pixel in the historical data of screen projection images, and construct an image transmission feature dataset; Step S200: Use the image transmission feature dataset to train the projection transmission image prediction neural network model, and calculate the feature deviation of each pixel before and after the projection transmission of the image frame data to be transmitted, and then divide the region units according to the feature deviation. Step S200 includes the following steps: Step S201: Use the image transmission feature dataset to train the projection transmission image prediction neural network model, and analyze the image feature parameter prediction data of each pixel after the projection transmission of the image frame data based on the image transmission features of each pixel. Step S202: Calculate the deviation between the predicted data of each image feature parameter after the projection transmission of each pixel of the image frame data to be transmitted and the image feature parameters before the projection transmission, and calculate the feature deviation of each pixel of the image frame data to be transmitted before and after the projection transmission by weighting. Step S203: Set the feature deviation interval, and divide the pixels with the same feature deviation interval and adjacent pixels in the image frame to be transmitted into the same region unit. For any region unit, calculate the distance between each pixel in the image, set a pixel distance threshold, and make a threshold judgment on the two pixels with the largest distance. If the distance is less than the pixel distance threshold, no further division is made. If the distance is greater than or equal to the pixel distance threshold, the region unit is further divided into two region units with the two pixels as the cluster center and the distance between pixels as the clustering basis. Repeat the above operation until the distance between each pixel in all region units is less than the pixel distance threshold. Step S300: Construct a three-dimensional coordinate system, monitor the eye data of all viewing users, analyze the user's gaze focus point and gaze direction information, determine the image optimization feature deviation threshold of each region unit, and calculate the deviation to be optimized for each region unit; Step S300 includes the following steps: Step S301: Establish a three-dimensional coordinate system with the center point of the TV screen as the origin, the horizontal direction from left to right as the x-axis, the vertical direction from top to bottom as the y-axis, and the direction perpendicular to the screen as the z-axis; Step S302: Use a visual sensor to monitor and identify all user facial feature information, extract user gaze information, and analyze user gaze focus point and user gaze direction information; wherein, the user gaze focus point is the direction of intersection of the user's eyes, and the user gaze direction is the ray direction from the center point of the user's eyes to the user gaze focus point; Step S303: Use a clustering algorithm to calculate the spatial center point of the pixels in each region unit in the image frame data, and determine the user's focusing coefficient k_x for any region unit x according to the formula: k_x=R_e×exp(-l); where R_e is the ratio of the distance from the user's eye center point to the user's gaze focus point to the distance from the user's eye center point to the intersection of the user's gaze direction and the plane where the TV screen is located, and l is the distance between the user's eye center point to the intersection of the user's gaze direction and the plane where the TV screen is located and the spatial center point of region unit x; Step S303: Calculate the focusing coefficients of all users for each region unit, and take the maximum value as the focusing coefficient of each region unit. Set the image optimization feature deviation threshold th_x for any region unit x as: th_x=(1-α)×k_x×C_dev; where α is the image optimization adjustment coefficient, k_x is the focusing coefficient of any region unit, and C_dev is the maximum deviation of the image optimization feature. Step S304: Then calculate the difference between the center of the feature deviation interval to which each region unit belongs and the image optimization feature deviation threshold, as the deviation to be optimized for each region unit; Step S400: Obtain historical data of TV screen casting image optimization, extract image feature parameters, optimization strategy data, feature deviation between optimized image frame data and image frame data before casting, optimization strategy resource consumption data, construct image optimization feature dataset, and train image optimization strategy decision model; Step S500: Based on the optimization deviation of each region unit of the image frame data to be transmitted, the image optimization strategy decision model is used to determine the image optimization strategy for the image frame data to be transmitted for TV screen projection.

2. The television image processing method based on artificial intelligence according to claim 1, characterized in that, The method for annotating the pixel points of each image frame in the historical data of the projected image in step S100 is as follows: For any pixel p in the image frame data, the projection transmission feature is labeled as: p[(x_p,y_p),F_before(p),F_after(p),S_tran(p)]; where x_p is the left-to-right number of pixel p in the image frame, y_p is the top-to-bottom number of pixel p in the image frame, F_before(p) is the set of image feature parameters of pixel p before transmission, F_after(p) is the set of image feature parameters of pixel p after transmission, and S_tran(p) is the set of transmission link state parameters of the image frame data to which pixel p belongs during projection transmission.

3. The television image processing method based on artificial intelligence according to claim 1, characterized in that, Step S400 includes the following: The image optimization strategy decision model takes as input image feature parameters, the optimization strategy adopted, and the feature deviation between the image frame data optimized by the current optimization strategy and the image frame data before projection. The output is the image optimization strategy time consumption data; the optimization strategy resource consumption data is the image optimization strategy time consumption data.

4. The television image processing method based on artificial intelligence according to claim 1, characterized in that, Step S500 includes the following: Obtain the inter-frame time interval of the image to be transmitted, and set the image optimization time threshold after the image frame data is projected and transmitted based on the inter-frame time interval of the image to be transmitted. Using the image optimization strategy decision model, the deviation to be optimized in each region unit of the image frame data is taken as input. The time consumption information of optimizing each region unit to meet the image optimization feature deviation threshold using different optimization strategies is analyzed, and the optimization strategy for the image frame data is set: When the overall time taken by each image unit in the image frame data to adopt each optimization strategy is greater than the image optimization time threshold, the deviation to be optimized in each region unit is proportionally amplified until there is an optimization strategy that satisfies that the overall time taken by each image unit in the image frame data to adopt each optimization strategy is less than or equal to the image optimization time threshold. This optimization strategy is then used as the final optimization strategy for the image frame data. When each image unit in the image frame data adopts an optimization strategy, if there is an optimization strategy whose overall time consumption is less than or equal to the image optimization time threshold, the deviation to be optimized in each region unit is reduced proportionally until there is one and only one optimization strategy that satisfies this condition. If the overall time consumption of each image unit in the image frame data adopting this optimization strategy is less than or equal to the image optimization time threshold, then this optimization strategy is taken as the final optimization strategy for the image frame data.

5. An artificial intelligence-based television image processing system applying the artificial intelligence-based television image processing method according to any one of claims 1-4, characterized in that, The system includes: a screen projection analysis module, a user gaze analysis module, and an optimization strategy decision module; The screen projection transmission analysis module annotates the pixels of each image frame in the historical data of the screen projection image with screen projection transmission features, trains a screen projection transmission image prediction neural network model, calculates the feature deviation of each pixel before and after the screen projection transmission of the image frame data to be transmitted, and divides the image frame data to be transmitted into regional units; the user gaze analysis module monitors the eye data of all viewing users, analyzes the user's gaze focus point and user gaze direction information, determines the image optimization feature deviation threshold of each regional unit, and calculates the deviation to be optimized for each regional unit; the optimization strategy decision module determines the image optimization strategy for the image frame data to be transmitted based on the deviation to be optimized for each regional unit using the trained image optimization strategy decision model.

6. The television image processing system based on artificial intelligence according to claim 5, characterized in that, The screen projection transmission analysis module includes: a transmission feature annotation unit, a feature deviation analysis unit, and a region unit division unit; The transmission feature annotation unit annotates the pixel points of each image frame in the historical data of the projected image with projection transmission features; the feature deviation analysis unit uses the projection transmission image prediction neural network model to predict the image feature parameters of the image frame data to be transmitted after projection transmission, and calculates the feature deviation of each pixel point before and after projection transmission of the image frame data to be transmitted; the region unit division unit divides the image frame data to be transmitted into region units according to the feature deviation of each pixel point before and after projection transmission of the image frame data to be transmitted.

7. The television image processing system based on artificial intelligence according to claim 5, characterized in that, The user gaze analysis module includes: a user monitoring unit, an orientation analysis unit, and a deviation setting unit; The user monitoring unit is used to monitor the eye data of all viewing users; the orientation analysis unit is used to construct a three-dimensional coordinate system and analyze the orientation information of the user's gaze focus point, the user's gaze direction and the intersection of the plane where the TV screen is located in the three-dimensional coordinate system; the deviation setting unit is used to determine the image optimization feature deviation threshold of each region unit in the image frame data to be transmitted, and calculate the deviation to be optimized for each region unit.

8. The television image processing system based on artificial intelligence according to claim 5, characterized in that, The optimization strategy decision-making module includes: a decision model construction unit and an optimization strategy decision-making unit; The decision model building unit trains the image optimization strategy decision model by constructing an image optimization feature dataset; the optimization strategy decision unit adjusts the deviation to be optimized in each region of the image frame data according to the overall time consumed by each optimization strategy to optimize each region of the image frame data, thereby determining the final optimization strategy for image frame optimization.

Images