Intelligent graphical interface operation optimization method based on image processing

Abstract

The invention discloses an intelligent graphical interface operation optimization method based on image processing, and relates to the field of image processing and graphical interface operation optimizing.The intelligent graphical interface operation optimization method comprises the steps that the ranking of functional elements is obtained by calculating the comprehensive use index of each functional element, and the functional elements are grouped according to functions; according to the sum value, the interface area and the area are allocated to each group, and the icon area is allocated to the functional elements in the group by setting the icon factors in the group, so that the intellectualization and individuation of the graphical interface are improved; meanwhile, by analyzing the fatigue degrees of the users, different colors are set for the interface background according to different fatigue degrees of the users, the satisfaction degree of the users is improved, and the stickiness of the users is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing and image interface operation optimization, and in particular to an intelligent graphic interface operation optimization method based on image processing. Background Art

[0002] With the rapid development of information technology, the graphical interfaces of various software and smart devices have become more and more complex and diverse to meet the growing functional needs of users. However, this also brings a series of operational challenges:

[0003] The numerous menu options, buttons and interactive elements often require users to spend a lot of time browsing and locating when looking for specific functions or performing operations. Existing graphical interface designs are often designed according to the designer's own ideas. The interface is single and cannot reasonably personalize the layout of interface elements based on the user's historical operations. The degree of intelligence is low; it cannot identify the user's fatigue and automatically adjust the background color of the graphical interface;

[0004] Therefore, an intelligent graphical interface operation optimization method based on image processing is proposed. Summary of the invention

[0005] In view of this, the present invention provides an intelligent graphical interface operation optimization method based on image processing to solve the problems raised by the above background technology.

[0006] The purpose of the present invention can be achieved by the following technical solution: an intelligent graphic interface operation optimization method based on image processing, comprising:

[0007] Data acquisition module: used to acquire user historical behavior data and real-time interface usage data; user historical behavior data includes the number of clicks on each functional element in the interface, the usage time of each functional element and the most recent usage time of each functional element; real-time interface usage data includes the response time of interface functional elements, interface frame rate and memory usage rate; and send user historical behavior data and real-time interface usage data to the data analysis module;

[0008] Data analysis module: used to analyze user historical behavior data and real-time interface usage data, so as to obtain the user's comprehensive usage index ZY for each functional element i and interface usage feedback index FK, i = 1, 2...x, x is the total number of functional elements; the comprehensive usage index ZY of each functional element i It is sent to the interface layout module, and the interface usage feedback index FK is sent to the adaptive adjustment module;

[0009] Interface layout module: based on the comprehensive usage index ZY of each functional element received i, adjust the interface layout accordingly;

[0010] Adaptive adjustment module: adjusts the interface accordingly based on the received interface usage feedback index FK;

[0011] User compensation module: obtains the current user's eye video and sitting posture video within the set time interval, and splits them into frames. The user's fatigue level is obtained through comprehensive analysis of the eye pictures and sitting posture pictures, and the interface background color is set according to the user's fatigue level.

[0012] In some embodiments, the comprehensive usage index ZY of each functional element in the interface is obtained. i , specifically:

[0013] Extract the most recent usage time of each functional element from the user's historical behavior data in the set time interval, subtract the most recent usage time of each functional element from the current time, and thus obtain the inactivity time MT of each functional element. i ; Set different time detection nodes in the set time interval, divide the set time interval into different time windows according to different time detection nodes, and record the number of clicks on each functional element in each time window as The usage duration of each functional element in each time window is j=1, 2...n, n is the total number of detection nodes;

[0014] For each time window, the number of clicks on each functional element is divided by the total number of clicks on each functional element in the time window to obtain the click rate of each functional element in the corresponding time window; the usage time of each functional element is divided by the total usage time of each functional element to obtain the usage ratio of each functional element in the corresponding time window;

[0015] Extract the click rate of each functional element in each time window, and select any functional element to perform the following operations:

[0016] S1: Calculate the average click rate of the current functional element in each time window, recorded as the point mean P1, extract the maximum and minimum click rates of the current functional element in each time window, recorded as the point peak value P2 and the point valley value P3 respectively; starting from the second time window, subtract the click rate corresponding to the previous node from the click rate corresponding to the current time window. If the result is a positive value, mark the time window, count the number of all marked time windows, and record it as the point appreciation value P4;

[0017] S2: Preset the weight influence factors corresponding to the point mean P1, point peak P2, point valley P3 and point appreciation P4. Normalize the point mean P1, point peak P2, point valley P3 and point appreciation P4, multiply them by the preset weight influence factors and sum them up. The obtained value is used as the click rate index DJX corresponding to the current functional element;

[0018] S1-S2 operations are performed on all functional elements to obtain the click rate index DJX corresponding to each functional element. i .

[0019] In some embodiments, the comprehensive usage index ZY of each functional element in the interface is obtained. i , further comprising:

[0020] Select any functional element, extract the usage ratio of the corresponding functional element in each time window, draw the usage ratio of the corresponding functional element in each time window into a line graph, perform straight line fitting on the image, and obtain the straight line expression after the line graph fitting. Perform the same operation on all functional elements to obtain the straight line expressions corresponding to each functional element, and extract the slope value in each straight line expression as the usage ratio trend value of each functional element, which is recorded as XL i ;

[0021] For each time window, the usage ratio of each functional element is ranked, and a score is set for each ranking. The higher the ranking, the higher the score. The scores of each functional element in each time window are accumulated to obtain the cumulative score FZ corresponding to each functional element. i ;

[0022] Calculate the average of the scores of each functional element at each time node to obtain the average value YJ of the corresponding functional element i ;

[0023] Preset usage ratio trend value XL i , cumulative score FZ i and use the mean value YJ i The corresponding weight impact factor will use the ratio trend value XL i , cumulative score FZ i and use the mean value YJ i Multiply them with their corresponding weight factors and add them together. The final result is the utilization index SYX corresponding to each functional element. i ;

[0024] The utilization index SYX of each functional element is obtained i , click rate index DJX i And the suspension time of each functional element MT i After normalization, enter the formula: Thus, the comprehensive usage index ZY corresponding to each functional element is obtained i , where u1, u2, and u3 are the utilization index SYX i , click rate index DJX i And the suspension time of each functional element MT i The corresponding weight influence factor.

[0025] In some embodiments, the interface usage feedback index FK is obtained, specifically:

[0026] Preset a time waiting value and the interface usage feedback index FK corresponding to the interface of the functional element that is not fully opened within the time waiting value;

[0027] If it is detected that the user clicks on a certain functional element, the timing starts from the time of the click. If the timing display reaches the preset time waiting value and the functional element interface is not fully opened, the interface usage feedback index FK is obtained according to the preset rules;

[0028] If the corresponding functional element interface is fully opened within the time waiting value, the functional element click time and the functional element opening time are marked, and the time interval between the two points is used as the analysis interval. The duration of the time interval is used as the response duration MKX of the corresponding functional element. The real-time usage data of the interface in the analysis interval is analyzed as follows:

[0029] Extract the frame rate value at each time node in the analysis interval, calculate the frame rate value at each time node using the mean formula and the standard deviation formula, and record the obtained values ​​as the frame average rate ZJ and the frame rate ZB respectively. Preset a frame rate low value allowable value, and count the number of frame rate values ​​at each time node in the analysis interval that are lower than the frame rate low value allowable value, and mark the obtained result as the low value number ZD;

[0030] The obtained average frame rate ZJ, frame rate ZB and number of low values ​​ZD are normalized and then input into the formula: The frame rate evaluation index ZLP is obtained, where v1, v2 and v3 are weighted influencing factors corresponding to the average frame rate ZJ, the frame rate ZB and the number of low values ​​ZD respectively.

[0031] In some embodiments, obtaining the interface usage feedback index FK further includes:

[0032] The memory usage rate of each time node in the analysis interval is calculated using the mean formula to obtain the internal mean rate N1; the maximum value of the memory usage rate of each node in the analysis interval is extracted as the memory peak value N2;

[0033] Substitute the obtained internal average rate N1 and storage peak value N2 into the formula: Get the memory usage index NC, where N1 阈值, N2 阈值 are the internal average rate threshold and storage peak value threshold extracted from the database, a1 and a2 are the weight influence factors corresponding to the internal average rate N1 and storage peak value N2 respectively;

[0034] The obtained module response time MKX, frame rate evaluation index ZLP and memory usage index NC are normalized and then entered into the formula: Thus, the interface feedback index FK is obtained, where ρ1, ρ2 and ρ3 are the weighted influencing factors corresponding to the module response time MKX, the frame rate evaluation index ZLP and the memory usage index NC respectively;

[0035] In some embodiments, the interface layout module adjusts the interface layout as follows:

[0036] Based on the different functions of interface functional elements, the interface functional elements are first grouped. Assuming the number of groups is v, the comprehensive usage index of the functional elements in each group is accumulated and summed, and the result is taken as the sum value HZ of the corresponding group. g , g=1,2...v, the total number of functional elements in each group is the group modulus SL g , preset and comprehensive value HZ g and group module SL g The corresponding weight influence factor will obtain the sum value HZ g And group module SL g After normalization, the corresponding weight influence factors are multiplied and added together, and the final result is used as the area allocation index MJ of each group. g , the area distribution index MJ of each group g Divide all group area distribution index MJ g The area allocation rate of each group is obtained by summing the area allocation rate and multiplying the area available for functional elements on the interface to obtain the corresponding interface usage area of ​​each group. g Ranking, put the first-ranked group at the top of the interface, the second-ranked group below the first-ranked group, and so on;

[0037] For each group's interface usage area, get its area value and set an icon growth factor λ g , the area of ​​the functional element with the smallest area in each group is L g Indicated by, the area of ​​the second smallest functional element in each region is L g *λ g , and so on, we can get the area of ​​the corresponding functional element in each group with respect to L g The expression of L is obtained by adding up the corresponding areas of all functional elements to get the interface usage area corresponding to the current group, thus solving L gThe value of , and then the area corresponding to all functional elements can be obtained, based on the obtained area corresponding to the functional element, the area of ​​all functional elements is modified to the corresponding size;

[0038] In some embodiments, the adaptive adjustment module adjusts the interface as follows:

[0039] Preset an interface feedback index threshold, compare the obtained interface feedback index FK with the interface feedback index threshold, if the interface feedback index FK is greater than the preset interface feedback index threshold, determine that the current interface is stuck; subtract the interface feedback index FK threshold from the interface feedback index FK to obtain a difference value, two intervals of the preset difference value correspond to two levels of stuck, the stuck levels are divided into slight stuck and severe stuck, match the difference value interval corresponding to the difference value, and thus obtain the interface stuck level corresponding to the interface feedback index FK; make different adjustments to the interface based on different levels of interface stuck;

[0040] Slight lag: reduce screen brightness, close all programs except the current one, and reduce screen resolution;

[0041] Severe lag: In addition to performing operations with minor lag, based on the comprehensive usage index ranking of each functional element, select the c functional elements with the lowest ranking to hide the functional elements, c>5, and the specific value of c can be set by the user according to the actual situation;

[0042] In some embodiments, the user compensation module adjusts the interface background color in the following steps:

[0043] The obtained user eye pictures are preprocessed, including but not limited to enhancement and noise reduction, and the pupil diameter value in each eye photo is obtained. The pupil diameters in all eye pictures are calculated using the mean formula and the standard deviation formula, and the obtained results are respectively used as the hole mean KJ and the hole wave value KB; the obtained hole mean KJ and hole wave value KB are normalized and then inserted into the formula: The eye fatigue index ZY is obtained, where γ1 and γ2 are the weighted influencing factors corresponding to the hole mean value KJ and the hole wave value KB, respectively. 标准 and KB 标准 They are the standard values ​​corresponding to the hole mean value KJ and the hole wave value KB extracted from the preset database;

[0044] Preprocess the user's sitting posture picture, extract features from the picture through a convolutional neural network, and further delete the features through the principal component analysis method, so as to obtain features in the user's sitting posture that are highly correlated with fatigue; preset four levels of fatigue, namely, energetic, general fatigue, moderate fatigue and very tired, and preset twenty standard pictures for each level, and perform feature matching between the obtained user's sitting posture picture and the preset eighty pictures, and preset a matching degree threshold. When the matching degree between the user's picture and the preset eighty pictures is lower than the preset threshold, the current picture is determined to be an invalid picture; if the picture is a valid picture, the fatigue level corresponding to the picture with the highest matching degree is selected as the fatigue level of the current picture; count the number of pictures corresponding to each fatigue level, select the fatigue level with the largest number of pictures in each fatigue level as the user's fatigue level in the current set time interval, set each fatigue level to correspond to each group of fatigue points, the higher the fatigue level, the higher the corresponding fatigue point, match the user's fatigue level with the set fatigue points corresponding to each group of fatigue levels, so as to obtain the fatigue point PL corresponding to the current user;

[0045] The obtained eye fatigue index ZY and fatigue score PL are normalized and then put into the formula: Thus, the user fatigue comprehensive index PLD within the set time interval is obtained, wherein λ1 and λ2 are weight influencing factors corresponding to the eye fatigue index ZY and the fatigue score PL respectively, each interval of the user fatigue comprehensive index PLD is set, various background colors corresponding to each interval of the user fatigue comprehensive index are preset, and the user fatigue comprehensive index interval corresponding to the user fatigue comprehensive index PLD is matched to obtain the background color corresponding to the user fatigue comprehensive index PLD, and the background color corresponding to the current fatigue level is compared with the background color of the current interface. If the background color of the current interface is not the background color corresponding to the current fatigue level, the background color of the current interface is adjusted to the background color corresponding to the current fatigue level.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention obtains the comprehensive usage index of all functional elements by analyzing the historical behavior data of users, groups the functional elements according to their functions, ranks the functional elements in each group according to the sum and comprehensive values, and allocates the corresponding interface area and area in combination with the number of functional elements in the group. For each functional element in each group, a suitable icon growth factor is set to allocate a corresponding area to each functional element; thus, the interface area is rationally arranged, the intelligence and personalization of the user interface are improved, and the user experience is improved;

[0048] The present invention obtains the user's comprehensive fatigue index by comprehensively analyzing the user's eye image and sitting posture image, and presets different interface background colors for the user's comprehensive fatigue index interval, so that a suitable interface background color can be set, thereby improving user satisfaction, and increasing user stickiness and satisfaction; BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0050] Figure 1 It is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0051] Several embodiments of the present application will be described in more detail below with reference to the accompanying drawings so that those skilled in the art can implement the present application. The present application can be embodied in many different forms and purposes and should not be limited to the embodiments described herein. These embodiments are provided to make the present application comprehensive and complete, and to fully convey the scope of the present application to those skilled in the art. The embodiments do not limit the present application.

[0052] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this application belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and / or this specification, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined as such herein.

[0053] See also Figure 1 As shown, an intelligent graphical interface operation optimization method based on image processing includes: a data acquisition module, a data analysis module, an interface layout module, an adaptive adjustment module and a user compensation module;

[0054] Data acquisition module: used to acquire user historical behavior data and real-time interface usage data; user historical behavior data includes the number of clicks on each functional element in the interface, the usage time of each functional element and the most recent usage time of each functional element; real-time interface usage data includes the response time of interface functional elements, interface frame rate and memory usage rate; and send user historical behavior data and real-time interface usage data to the data analysis module;

[0055] Data analysis module: used to analyze user historical behavior data and real-time interface usage data, so as to obtain the user's comprehensive usage index ZY for each functional element iand interface usage feedback index FK, i = 1, 2...x, x is the total number of functional elements; the comprehensive usage index ZY of each functional element i It is sent to the interface layout module, and the interface usage feedback index FK is sent to the adaptive adjustment module;

[0056] Extract the most recent usage time of each functional element from the user's historical behavior data in the set time interval, subtract the most recent usage time of each functional element from the current time, and thus obtain the inactivity time MT of each functional element. i ; Set different time detection nodes in the set time interval, divide the set time interval into different time windows according to different time detection nodes, and record the number of clicks on each functional element in each time window as The usage duration of each functional element in each time window is j=1, 2...n, n is the total number of detection nodes;

[0057] For each time window, the number of clicks on each functional element is divided by the total number of clicks on each functional element in the time window to obtain the click rate of each functional element in the corresponding time window; the usage time of each functional element is divided by the total usage time of each functional element to obtain the usage ratio of each functional element in the corresponding time window;

[0058] Extract the click rate of each functional element in each time window, and select any functional element to perform the following operations:

[0059] S1: Calculate the average click rate of the current functional element in each time window, recorded as the point mean P1, extract the maximum and minimum click rates of the current functional element in each time window, recorded as the point peak value P2 and the point valley value P3 respectively; starting from the second time window, subtract the click rate corresponding to the previous node from the click rate corresponding to the current time window. If the result is a positive value, mark the time window, count the number of all marked time windows, and record it as the point appreciation value P4;

[0060] S2: Preset the weight influence factors corresponding to the point mean P1, point peak P2, point valley P3 and point appreciation P4. Normalize the point mean P1, point peak P2, point valley P3 and point appreciation P4, multiply them by the preset weight influence factors and sum them up. The obtained value is used as the click rate index DJX corresponding to the current functional element;

[0061] Through comprehensive analysis of point average, point peak, point valley and point appreciation, we can get a user's click trend for all functional elements. The higher the click rate index of a functional element, the more frequently the user uses the current functional element.

[0062] S1-S2 operations are performed on all functional elements to obtain the click rate index DJX corresponding to each functional element. i ;

[0063] Select any functional element, extract the usage ratio of the corresponding functional element in each time window, draw the usage ratio of the corresponding functional element in each time window into a line graph, perform straight line fitting on the image, and obtain the straight line expression after the line graph fitting. Perform the same operation on all functional elements to obtain the straight line expressions corresponding to each functional element, and extract the slope value in each straight line expression as the usage ratio trend value of each functional element, which is recorded as XL i ;

[0064] The slope value represents a trend of the usage ratio of functional elements. A positive slope value indicates that the usage time of users on functional elements is on an upward trend, and the larger the slope value, the more obvious the upward trend; a negative slope value indicates that the usage time of users on functional elements is on a downward trend, and the smaller the slope value, the more obvious the downward trend;

[0065] For each time window, the usage ratio of each functional element is ranked, and a score is set for each ranking. For example, the first place can be set to 20 points, the second place to 18 points, and the third place to only 17 points. The higher the ranking, the greater the score. The scores of each functional element in each time window are accumulated to obtain the cumulative score FZ corresponding to each functional element i ;

[0066] Calculate the average of the scores of each functional element at each time node to obtain the average value YJ of the corresponding functional element i ;

[0067] Preset usage ratio trend value XL i , cumulative score FZ i and use the mean value YJ i The corresponding weight impact factor will use the ratio trend value XL i , cumulative score FZ i and use the mean value YJ i Multiply them with their corresponding weight factors and add them together. The final result is the utilization index SYX corresponding to each functional element. i ;

[0068] The utilization index SYX of each functional element is obtained i , click rate index DJX i And the suspension time of each functional element MT i After normalization, enter the formula: Thus, the comprehensive usage index ZY corresponding to each functional element is obtained i, where u1, u2, and u3 are the utilization index SYX i , click rate index DJX i And the suspension time of each functional element MT i The corresponding weight impact factor;

[0069] The greater the comprehensive usage index of a functional element, the higher the user's demand for the functional element, thus providing data support for subsequent interface design;

[0070] Preset a time waiting value and the interface usage feedback index FK corresponding to the interface of the functional element that is not fully opened within the time waiting value;

[0071] If it is detected that the user clicks on a certain functional element, the timing starts from the time of the click. If the timing display reaches the preset time waiting value and the functional element interface is not fully opened, the interface usage feedback index FK is obtained according to the preset rules;

[0072] Since interface elements may not be opened or may take a long time to open, a time waiting value and the corresponding interface feedback index FK for the functional element interface that is not fully opened within the time waiting value are preset. When a functional element takes too long to open, the current interface is directly judged to be stuck, so that corresponding operations are performed on the interface to avoid long waiting time and affect the user experience;

[0073] If the corresponding functional element interface is fully opened within the time waiting value, the functional element click time and the functional element opening time are marked, and the time interval between the two points is used as the analysis interval. The time interval duration is used as the corresponding functional element response duration MKX, and the real-time interface usage data of the analysis interval is analyzed, as follows:

[0074] Extract the frame rate value at each time node in the analysis interval, calculate the frame rate value at each time node using the mean formula and the standard deviation formula, and record the obtained values ​​as the frame average rate ZJ and the frame rate ZB respectively. Preset a frame rate low value allowable value, and count the number of frame rate values ​​at each time node in the analysis interval that are lower than the frame rate low value allowable value, and mark the obtained result as the low value number ZD;

[0075] The obtained average frame rate ZJ, frame rate ZB and number of low values ​​ZD are normalized and then input into the formula: The frame rate evaluation index ZLP is obtained, where v1, v2 and v3 are weighted influencing factors corresponding to the average frame rate ZJ, the frame rate ZB and the number of low values ​​ZD respectively.

[0076] The larger the frame rate evaluation index ZLP is, the higher and more stable the frame rate of the current interface is, and the smoother the picture is;

[0077] The memory usage rate of each time node in the analysis interval is calculated using the mean formula to obtain the internal mean rate N1; the maximum value of the memory usage rate of each node in the analysis interval is extracted as the memory peak value N2;

[0078] Substitute the obtained internal average rate N1 and storage peak value N2 into the formula: Get the memory usage index NC, where N1 阈值 , N2 阈值 are the internal average rate threshold and storage peak value threshold extracted from the database, a1 and a2 are the weight influence factors corresponding to the internal average rate N1 and storage peak value N2 respectively;

[0079] The obtained module response time MKX, frame rate evaluation index ZLP and memory usage index NC are normalized and then entered into the formula: Thus, the interface feedback index FK is obtained, where ρ1, ρ2 and ρ3 are the weighted influencing factors corresponding to the module response time MKX, the frame rate evaluation index ZLP and the memory usage index NC respectively;

[0080] Interface layout module: based on the comprehensive usage index ZY of each functional element received i , adjust the interface layout accordingly;

[0081] Based on the different functions of interface functional elements, the interface functional elements are first grouped. Assuming the number of groups is v, the comprehensive usage index of the functional elements in each group is accumulated and summed, and the result is taken as the sum value HZ of the corresponding group. g , g=1,2...v, the total number of functional elements in each group is the group modulus SL g , preset and comprehensive value HZ g and group module SL g The corresponding weight influence factor will obtain the sum value HZ g And group module SL g After normalization, the corresponding weight influence factors are multiplied and added together, and the final result is used as the area allocation index MJ of each group. g , the area distribution index MJ of each group g Divide all group area distribution index MJ g The area allocation rate of each group is obtained by summing the area allocation rate and multiplying the area available for functional elements on the interface to obtain the corresponding interface usage area of ​​each group. g Perform ranking and place the top-ranked groups at the top of the interface;

[0082] For each group's interface usage area, get its area value and set an icon growth factor λ g , the area of ​​the functional element with the smallest area in each group is L gIndicated by, the area of ​​the second smallest functional element in each region is L g *λ g , and so on, we can get the area of ​​the corresponding functional element in each group with respect to L g The expression of L is obtained by adding up the corresponding areas of all functional elements to get the interface usage area corresponding to the current group, thus solving L g The value of , and then the area corresponding to all functional elements can be obtained, based on the obtained area corresponding to the functional element, the area of ​​all functional elements is modified to the corresponding size;

[0083] For example: the icon growth factor of a group is 1.05, and the smallest icon area is L g , the interface usage area of ​​this group is 10, and there are four functional elements in this group. The area of ​​the second smallest functional element is 1.05*L g , the third smallest functional element area is 1.05 2 *L g , the largest functional element area is 1.05 3 *L g , then we can write equation L g +1.05*L g +1.05 2 *L g +1.05 3 *L g =10, thus solving for L g The value is 2.32, the area of ​​the second smallest functional element is 2.44, the area of ​​the third smallest functional element is 2.56, and the area of ​​the largest functional element is 2.69;

[0084] Adaptive adjustment module: adjusts the interface accordingly based on the received interface usage feedback index FK;

[0085] Preset an interface feedback index threshold, compare the obtained interface feedback index FK with the interface feedback index threshold, if the interface feedback index FK is greater than the preset interface feedback index threshold, determine that the current interface is stuck; subtract the interface feedback index FK threshold from the interface feedback index FK to obtain a difference value, two intervals of the preset difference value correspond to two levels of stuck, the stuck levels are divided into slight stuck and severe stuck, match the difference value interval corresponding to the difference value, and thus obtain the interface stuck level corresponding to the interface feedback index FK; make different adjustments to the interface based on different levels of interface stuck;

[0086] Slight lag: reduce screen brightness, close all programs except the current one, and reduce screen resolution;

[0087] Severe lag: In addition to performing operations with minor lag, based on the comprehensive usage index ranking of each functional element, select the c functional elements with the lowest ranking to hide the functional elements, where c>5, and the specific value of c can be set by the user according to the specific situation;

[0088] User compensation module: obtains the current user's eye video and sitting posture video within the set time interval, and splits them into frames. The user's fatigue level is obtained through comprehensive analysis of the eye pictures and sitting posture pictures, and the interface background color is set according to the user's fatigue level;

[0089] The obtained user eye pictures are preprocessed, including but not limited to enhancement and noise reduction, and the pupil diameter value in each eye photo is obtained. The pupil diameters in all eye pictures are calculated using the mean formula and the standard deviation formula, and the obtained results are respectively used as the hole mean KJ and the hole wave value KB; the obtained hole mean KJ and hole wave value KB are normalized and then inserted into the formula: The eye fatigue index ZY is obtained, where γ1 and γ2 are the weighted influencing factors corresponding to the hole mean value KJ and the hole wave value KB, respectively. 标准 and KB 标准 They are the standard values ​​corresponding to the hole mean value KJ and the hole wave value KB extracted from the preset database;

[0090] Preprocess the user's sitting posture picture, extract features from the picture through a convolutional neural network, and further delete the features through a principal component analysis method, so as to obtain features in the user's sitting posture that are highly correlated with fatigue. Four levels of fatigue are preset, namely, energetic, general fatigue, moderate fatigue, and very tired. Twenty standard pictures are preset for each level. The obtained user's sitting posture picture is feature matched with eighty preset pictures, and a matching degree threshold is preset. When the matching degree between the user picture and the eighty preset pictures is lower than the preset threshold, the current picture is determined to be an invalid picture; if the picture is a valid picture, the fatigue level corresponding to the picture with the highest matching degree is selected as the fatigue level of the current picture; the number of pictures corresponding to each fatigue level is counted, and the fatigue level with the largest number of pictures in each fatigue level is selected as the fatigue level of the user in the current set time interval, and each fatigue level is set to correspond to each group of fatigue points. The higher the fatigue level, the higher the corresponding fatigue score. The user's fatigue level is matched with the set fatigue points corresponding to each group of fatigue levels, so as to obtain the fatigue score PL corresponding to the current user;

[0091] The obtained eye fatigue index ZY and fatigue score PL are normalized and then put into the formula: Thus, the user fatigue comprehensive index PLD within the set time interval is obtained, wherein λ1 and λ2 are weighted influencing factors corresponding to the eye fatigue index ZY and the fatigue score PL respectively, each interval of the user fatigue comprehensive index PLD is set, and various background colors corresponding to each interval of the user fatigue comprehensive index are preset, and the user fatigue comprehensive index interval corresponding to the user fatigue comprehensive index PLD is matched, so as to obtain the background color corresponding to the user fatigue comprehensive index PLD, and the background color corresponding to the current fatigue level is compared with the background color of the current interface. If the background color of the current interface is not the background color corresponding to the current fatigue level, the control system adjusts the background color of the current interface to the background color corresponding to the current fatigue level;

[0092] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. An intelligent graphical interface operation optimization method based on image processing, characterized in that: include: Data acquisition module: used to obtain user historical behavior data and real-time interface usage data; The user's historical behavior data includes the number of clicks on each functional element in the interface, the usage time of each functional element, and the most recent usage time of each functional element; the real-time interface usage data includes the response time of the interface functional elements, the interface frame rate, and the memory usage rate; and the user's historical behavior data and the real-time interface usage data are sent to the data analysis module; Data analysis module: used to analyze user historical behavior data and real-time interface usage data, so as to obtain the user's comprehensive usage index ZY for each functional element i and interface usage feedback index FK, i = 1, 2...x, x is the total number of functional elements; the comprehensive usage index ZY of each functional element i It is sent to the interface layout module, and the interface usage feedback index FK is sent to the adaptive adjustment module; Interface layout module: based on the comprehensive usage index ZY of each functional element received i , adjust the interface layout accordingly; Adaptive adjustment module: adjusts the interface accordingly based on the received interface usage feedback index FK; User compensation module: obtains the current user's eye video and sitting posture video within the set time interval, and splits them into frames. The user's fatigue level is obtained through comprehensive analysis of the eye pictures and sitting posture pictures, and the interface background color is set according to the user's fatigue level.

2. The method for optimizing intelligent graphical interface operation based on image processing according to claim 1, characterized in that: Get the comprehensive usage index ZY of each functional element in the interface i , specifically: Extract the most recent usage time of each functional element from the user's historical behavior data in the set time interval, subtract the most recent usage time of each functional element from the current time, and thus obtain the inactivity time MT of each functional element. i ; Set different time detection nodes in the set time interval, divide the set time interval into different time windows according to different time detection nodes, and record the number of clicks on each functional element in each time window as The usage duration of each functional element in each time window is j=1, 2...n, n is the total number of detection nodes; For each time window, the number of clicks on each functional element is divided by the total number of clicks on each functional element in the time window to obtain the click rate of each functional element in the corresponding time window; the usage time of each functional element is divided by the total usage time of each functional element to obtain the usage ratio of each functional element in the corresponding time window; Extract the click rate of each functional element in each time window, and select any functional element to perform the following operations: S1: Calculate the average click rate of the current functional element in each time window, recorded as the point mean P1, extract the maximum and minimum click rates of the current functional element in each time window, recorded as the point peak value P2 and the point valley value P3 respectively; starting from the second time window, subtract the click rate corresponding to the previous node from the click rate corresponding to the current time window. If the result is a positive value, mark the time window, count the number of all marked time windows, and record it as the point appreciation value P4; S2: Preset the weight influence factors corresponding to the point mean P1, point peak P2, point valley P3 and point appreciation P4. Normalize the point mean P1, point peak P2, point valley P3 and point appreciation P4, multiply them by the preset weight influence factors and sum them up. The obtained value is used as the click rate index DJX corresponding to the current functional element; S1-S2 operations are performed on each functional element to obtain the click rate index DJX corresponding to each functional element i .

3. The method for optimizing intelligent graphical interface operation based on image processing according to claim 2, characterized in that: Get the comprehensive usage index ZY of each functional element in the interface i , further comprising: Select any functional element, extract the usage ratio of the corresponding functional element in each time window, draw the usage ratio of the corresponding functional element in each time window into a line graph, perform straight line fitting on the image, and obtain the straight line expression after the line graph fitting. Perform the same operation on all functional elements to obtain the straight line expressions corresponding to each functional element, and extract the slope value in each straight line expression as the usage ratio trend value of each functional element, which is recorded as XL i ; For each time window, the usage ratio of each functional element is ranked, a score is set for each ranking, and the scores of each functional element in each time window are accumulated to obtain the cumulative score FZ corresponding to each functional element i ; Calculate the average of the scores of each functional element at each time node to obtain the average value YJ of the corresponding functional element i ; Preset usage ratio trend value XL i , cumulative score FZ i and use the mean value YJ i The corresponding weight impact factor will use the ratio trend value XL i , cumulative score FZ i and use the mean value YJ i Multiply them with their corresponding weight factors and add them together. The final result is the utilization index SYX corresponding to each functional element. i ; The utilization index SYX of each functional element is obtained i , click rate index DJX i And the suspension time of each functional element MT i After normalization, enter the formula: Thus, the comprehensive usage index ZY corresponding to each functional element is obtained i , where u1, u2, and u3 are the utilization index SYX i , click rate index DJX i And the suspension time of each functional element MT i The corresponding weight influence factor.

4. The method for optimizing intelligent graphical interface operation based on image processing according to claim 3, characterized in that: Get the interface usage feedback index FK, specifically: Preset a time waiting value and the interface usage feedback index FK corresponding to the interface of the functional element that is not fully opened within the time waiting value; If it is detected that the user clicks on a certain functional element, the timing starts from the time of the click. If the timing display reaches the preset time waiting value and the functional element interface is not fully opened, the interface usage feedback index FK is obtained according to the preset rules; If the corresponding functional element interface is fully opened within the time waiting value, the functional element click time and the functional element opening time are marked, and the time interval between the two points is used as the analysis interval. The time interval duration is used as the corresponding functional element response duration MKX, and the real-time interface usage data of the analysis interval is analyzed, as follows: Extract the frame rate value at each time node in the analysis interval, calculate the frame rate value at each time node using the mean formula and the standard deviation formula, and record the obtained values ​​as the frame average rate ZJ and the frame rate ZB respectively. Preset a frame rate low value allowable value, and count the number of frame rate values ​​at each time node in the analysis interval that are lower than the frame rate low value allowable value, and mark the obtained result as the low value number ZD; The obtained average frame rate ZJ, frame rate ZB and number of low values ​​ZD are normalized and then input into the formula: The frame rate evaluation index ZLP is obtained, where v1, v2 and v3 are weighted influencing factors corresponding to the average frame rate ZJ, the frame rate ZB and the number of low values ​​ZD respectively.

5. The method for optimizing intelligent graphical interface operation based on image processing according to claim 4, characterized in that: Get the interface usage feedback index FK, further including: The memory usage rate of each time node in the analysis interval is calculated using the mean formula to obtain the internal mean rate N1; the maximum value of the memory usage rate of each node in the analysis interval is extracted as the memory peak value N2; Substitute the obtained internal average rate N1 and storage peak value N2 into the formula: Get the memory usage index NC, where N1 阈值 , N2 阈值 are the internal average rate threshold and storage peak value threshold extracted from the database, a1 and a2 are the weight influence factors corresponding to the internal average rate N1 and storage peak value N2 respectively; The obtained module response time MKX, frame rate evaluation index ZLP and memory usage index NC are normalized and then input into the formula: Thus, the interface feedback index FK is obtained, where ρ1, ρ2 and ρ3 are the weight influencing factors corresponding to the module response time MKX, the frame rate evaluation index ZLP and the memory usage index NC respectively.

6. The method for optimizing intelligent graphical interface operation based on image processing according to claim 5, characterized in that: The interface layout module adjusts the interface layout as follows: Based on the different functions of interface functional elements, the interface functional elements are first grouped, and the number of groups is represented by v. The comprehensive usage index of the functional elements in each group is accumulated and summed, and the result is taken as the sum value HZ of the corresponding group. g , g=1,2...v, the total number of functional elements in each group is the group modulus SL g , preset and comprehensive value HZ g and group module SL g The corresponding weight influence factor will obtain the sum value HZ g And group module SL g After normalization, the corresponding weight influence factors are multiplied and added together, and the final result is used as the area allocation index MJ of each group. g , the area distribution index MJ of each group g Divide all group area distribution index MJ g The area allocation rate of each group is obtained by summing the area allocation rate and multiplying the area available for functional elements on the interface to obtain the corresponding interface usage area of ​​each group. g Ranking, put the first-ranked group at the top of the interface, the second-ranked group below the first-ranked group, and so on; For each group's interface usage area, get its area value and set an icon growth factor λ g , the area of ​​the functional element with the smallest area in each group is L g Indicated by, the area of ​​the second smallest functional element in each region is L g *λ g , and so on, we can get the area of ​​the corresponding functional element in each group with respect to L g The expression of L is obtained by adding up the corresponding areas of all functional elements to get the interface usage area corresponding to the current group, thus solving L g The value of , and then the areas corresponding to all functional elements can be obtained, and based on the obtained areas corresponding to the functional elements, the areas of all functional elements are modified to corresponding sizes.

7. The method for optimizing intelligent graphical interface operation based on image processing according to claim 6, characterized in that: The adaptive adjustment module adjusts the interface as follows: Preset an interface feedback index threshold, compare the obtained interface feedback index FK with the interface feedback index threshold, if the interface feedback index FK is greater than the preset interface feedback index threshold, determine that the current interface is stuck; subtract the interface feedback index FK threshold from the interface feedback index FK to obtain a difference value, two intervals of the preset difference value correspond to two levels of stuck, the stuck levels are divided into slight stuck and severe stuck, match the difference value interval corresponding to the difference value, and thus obtain the interface stuck level corresponding to the interface feedback index FK; make different adjustments to the interface based on different levels of interface stuck; Slight lag: reduce screen brightness, close all programs except the current one, and reduce screen resolution; Severe lag: In addition to performing operations with slight lag, based on the comprehensive usage index ranking of each functional element, the bottom c functional elements are selected for hiding, where c>5.

8. The method for optimizing intelligent graphical interface operation based on image processing according to claim 7, characterized in that: The steps for the user compensation module to adjust the interface background color are as follows: The obtained user eye pictures are preprocessed to obtain the pupil diameter value in each eye photo, and the pupil diameter in all eye pictures is calculated using the mean formula and the standard deviation formula. The results are used as the hole mean KJ and the hole wave value KB respectively; the obtained hole mean KJ and hole wave value KB are normalized and then inserted into the formula: The eye fatigue index ZY is obtained, where γ1 and γ2 are the weighted influencing factors corresponding to the hole mean value KJ and the hole wave value KB, respectively. 标准 and KB 标准 They are the standard values ​​corresponding to the hole mean value KJ and the hole wave value KB extracted from the preset database; Preprocess the user's sitting posture pictures, extract features from the pictures through convolutional neural networks, and further reduce the features through principal component analysis to obtain features in the user's sitting posture that are highly correlated with fatigue; Four levels of fatigue are preset, namely, energetic, general fatigue, moderate fatigue and very tired. Twenty standard pictures are preset for each level. The obtained user sitting posture picture is feature matched with the preset eighty pictures. A matching degree threshold is preset. When the matching degree between the user picture and the preset eighty pictures is lower than the preset threshold, the current picture is judged as an invalid picture; if the picture is a valid picture, the fatigue level corresponding to the picture with the highest matching degree is selected as the fatigue level of the current picture; Count the number of pictures corresponding to each fatigue level, select the fatigue level with the largest number of pictures in each fatigue level as the fatigue level of the user in the current set time interval, set each fatigue level to correspond to each group of fatigue points, match the user's fatigue level with the set fatigue points corresponding to each group of fatigue levels, and thus obtain the fatigue score PL corresponding to the current user; The obtained eye fatigue index ZY and fatigue score PL are normalized and then put into the formula: Thus, the user fatigue comprehensive index PLD within the set time interval is obtained, wherein λ1 and λ2 are weight influencing factors corresponding to the eye fatigue index ZY and the fatigue score PL respectively, each interval of the user fatigue comprehensive index PLD is set, each interval of the user fatigue comprehensive index PLD is set, various background colors are preset for each interval of the user fatigue comprehensive index PLD, and the user fatigue comprehensive index interval corresponding to the user fatigue comprehensive index PLD is matched, so as to obtain the background color corresponding to the user fatigue comprehensive index PLD, and compare the background color corresponding to the current fatigue level with the background color of the current interface. If the background color of the current interface is not the background color corresponding to the current fatigue level, the background color of the current interface is adjusted to the background color corresponding to the current fatigue level.

Images