A method, device and storage medium for intelligent split-screen control of a tablet computer

Abstract

The present invention discloses a method, device and storage medium for intelligent split-screen control of a tablet computer, relating to the technical field of split-screen control. The present invention includes a register, a screen recording module, a synchronization module, a monitoring module, a display module, a processing module and a camera module. Ports of the register are respectively connected to ports of the synchronization module, the screen recording module and the monitoring module. Ports of the synchronization module, the screen recording module and the monitoring module are all connected to ports of the display module. Ports of the display module are connected to ports of the processing module. An output end of the camera module is connected to an input end of the processing module. The processing module obtains the length L and width W of the display module. The present invention automatically adjusts the size of the main window and the font size for easy viewing by the user, making the display content of the display module more suitable for the user's eyesight. Improving the local frame rate according to the user's focus can enhance the user experience, and the monitoring module can broaden the usage scenarios of the tablet computer.

Description

Technical Field

[0001] The present invention relates to the technical field of split - screen control, and particularly to an intelligent split - screen control method, device and storage medium for a tablet computer. Background Art

[0002] With the improvement of the hardware level of tablet computers in the market, the multi - task processing requirements of tablet computers have gradually increased. The split - screen technology enables users to operate multiple application programs simultaneously on the same screen, improving the efficiency of work and study.

[0003] The existing split - screen technology for tablet computers has the following defects: 1. It cannot automatically adjust the split - screen ratio according to user habits; 2. After split - screen, it cannot achieve dynamic frame rate adjustment and cannot automatically increase the frame rate of the corresponding display area according to the user's focus; 3. It cannot help users automatically monitor the programs hanging in the background; 4. It cannot achieve synchronous linkage of multiple windows after split - screen; 5. It cannot achieve automatic repetition of the user's operations in the background during screen recording. Therefore, improving the functional defects of the above - mentioned split - screen technology is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the present invention provides an intelligent split - screen control method, device and storage medium for a tablet computer, solving the problems raised in the above - mentioned background art.

[0005] To achieve the above object, the present invention is realized through the following technical solutions: An intelligent split - screen control method for a tablet computer includes a register, a screen recording module, a synchronization module, a monitoring module, a display module, a processing module and a camera module. The ports of the register are respectively connected to the ports of the synchronization module, the screen recording module and the monitoring module. The ports of the synchronization module, the screen recording module and the monitoring module are all connected to the ports of the display module. The ports of the display module are connected to the ports of the processing module. The output end of the camera module is connected to the input end of the processing module;

[0006] The processing module obtains the length L and width W of the display module. The camera module obtains the image information of the user's head and transmits it to the processing module. The display module transmits the user's touch behavior to the processing module, the screen recording module and the monitoring module in real time. The control method includes the following steps:

[0007] The window display weight. The processing module presets a main window and auxiliary windows. The processing module executes a head analysis program to obtain the user's attention direction A, the left deviation degree xy1, and the right deviation degree xy2. The processing module automatically adjusts the layout of the main window and the auxiliary windows according to the user's attention direction A. The layout includes the window size and the arrangement direction. The processing module executes an eye analysis program to obtain the user's squinting times j. The higher the squinting times j, the less clear the user can see the display content of the display module, and it is necessary to correspondingly increase the size of the main window and the text content in the main window. The processing module presets a magnification factor k. The magnification factor k is associated with the size of the main window and the font size in the main window. The processing module automatically adjusts the magnification factor k according to the squinting times j, and automatically adjusts the size of the main window and the font size to facilitate the user's viewing, making the display content of the display module more suitable for the user's eyesight;

[0008] Switching between the main window and the auxiliary windows. When the user clicks on any auxiliary window in the display module, the processing module swaps the clicked auxiliary window with the main window. The auxiliary window moves to the original position of the main window and is enlarged to the size of the original main window. The main window moves to the original position of the auxiliary window and is reduced to the size of the original auxiliary window. The user can only perform touch operations in the main window;

[0009] Focus frame rate optimization. The processing module presets a high-refresh window. At least one side edge of the high-refresh window is adjacent to the edge of the display module. The high-refresh window is an edge-attached window and overlaps with the main window or the auxiliary windows. The processing module automatically adjusts the size and position of the high-refresh window according to the attention direction A. Improving the local frame rate can enhance the user experience;

[0010] Screen monitoring. The user establishes a monitoring task through the display module. The monitoring module obtains the display information of each auxiliary window in real time. The monitoring module controls each auxiliary window to display interactive animations according to the display information. The monitoring module can help the user automatically monitor the changes of background programs and broaden the usage scenarios of the tablet computer;

[0011] Window synchronization. The user establishes a synchronization task through the display module. The synchronization module connects several auxiliary windows to the main window according to the synchronization task. The synchronization module synchronizes the touch operations of the main window to each auxiliary window. The synchronization operation can help the user control multiple identical programs simultaneously by one person and improve the usage efficiency of the tablet;

[0012] Automatic loop execution of screen recording. The user establishes an automation task through the display module. During the task establishment stage, the screen recording module obtains the touch operations of the main window and saves them in the register. During the task execution stage, the screen recording module transmits the touch operations saved in the register to the specified auxiliary window for execution. The automatic execution of screen recording can help the user achieve unattended automatic execution and improve the utilization rate of the tablet computer when it is idle.

[0013] Furthermore, the window display weight specifically includes the following steps:

[0014] The processing module sets the length of the main window to L and the width to 0.8W, and the auxiliary windows are arranged in a row along the length of the main window. The number of auxiliary windows is n, and the length of each auxiliary window is L / n and the width is 0.2W;

[0015] The processing module aligns the main window with the edge of the display module along the user's focus direction, and aligns the arranged n auxiliary windows with the edge of the display module on the opposite side of the user's focus direction;

[0016] When L / n≤0.1W, the auxiliary windows are arranged into two columns along the length of the main window. After the auxiliary windows are arranged into two columns, the processing module resets the length and width of each auxiliary window. The length of each auxiliary window is L / 2n and the width is 0.1W.

[0017] When the application in the auxiliary window does not support the free change of the aspect ratio of the application interface, the processing module calculates the product of the length and width of each auxiliary window, and arranges the auxiliary windows in descending order according to the product results. The auxiliary windows are arranged from top to bottom. The auxiliary windows with the same product size are arranged again in descending order according to the number of pixel colors.

[0018] The processing module initializes the magnification k, and the value of the magnification k after initialization is 1. The processing module uses the formula Calculate the magnification k of the main window, j is the number of squinting times, the processing module counts the number of squinting times j every fixed time of 1 minute, and the processing module enlarges the width of the main window and the text content in the main window according to the magnification k. When the magnification k is less than 1, the width of the main window and the text content in the main window are reduced;

[0019] When the width of the main window reaches 0.9W, it will no longer be enlarged. When the text content in the main window is enlarged, the distance between the characters is reduced, and the text layout remains unchanged. When the character spacing is zero, the processing module reduces the image in the main window and expands the layout area of ​​the text content.

[0020] Furthermore, the focus frame rate optimization specifically includes the following steps:

[0021] The processing module marks the intersection of the user's focus direction A and the display module as a focus;

[0022] Determine the size of the high refresh window, and the processing module uses the formula Calculate the size of the high refresh window, xy1 and xy2 are both deviations, xy1 is the deviation of the face direction, xy2 is the deviation of the eye direction, L is the length of the display module, and W is the width of the display module;

[0023] The position of the high - refresh window changes synchronously according to the user's attention direction A. When the attention direction A is to the left, the high - refresh window unfolds along the left side of the display module. When the attention direction A is to the right, the high - refresh window unfolds along the right side of the display module. The unfolded high - refresh window stops at the focus. At present, the screen local high - refresh technology cannot achieve a completely free - floating window. Instead, a side - attached window is used to replace the floating window to provide a high refresh rate.

[0024] Further, the screen monitoring specifically includes the following steps:

[0025] Display information monitoring: The monitoring module sets a preset change cycle of 1 second. The display information of each window is compared pixel - by - pixel in terms of RGB values with the display information of the previous change cycle. If the RGB value of any pixel changes, it means that the picture of the auxiliary window has changed, and then the interactive animation is triggered. On the contrary, if the RGB values of all pixels have not changed, the comparison is repeated in a loop.

[0026] Triggering the interactive animation: The monitoring module controls the auxiliary window that triggers the interactive animation to move back and forth three times in the change cycle in the direction close to the main window. The moving distance is half of the maximum side length of the auxiliary window, which is convenient to attract the user's attention.

[0027] Further, the window synchronization specifically includes the following steps:

[0028] The synchronization module establishes a touch coordinate system in the main window. The synchronization module records the user's touch behavior in the main window. The touch behavior includes the click time and click coordinates.

[0029] The synchronization module obtains the lengths and widths of the main window and the auxiliary window connected to the main window. The synchronization module obtains the average RGB value of the pixels in the click coordinate area of the main window.

[0030] The synchronization module converts the click coordinates of the main window into synchronous coordinates of the same ratio in the auxiliary window according to the length ratio and width ratio of the main window and the auxiliary window. The synchronization module compares the average RGB value of the pixels in the synchronous coordinate area with the average RGB value of the pixels in the click coordinate area for verification.

[0031] If the average RGB values of the main window and the auxiliary window are the same, it means that the synchronous coordinate verification is passed. The synchronization module executes the synchronous coordinates in the auxiliary window at the click time. If the average RGB values of the main window and the auxiliary window are different, it means that the synchronous coordinate verification fails. The synchronization module does not execute the synchronous coordinates in the auxiliary window, and the synchronization module outputs a pop - up window indicating the failure of the synchronization task and transmits it to the main window for the user to view.

[0032] Further, the automatic loop execution of screen recording specifically includes the following steps:

[0033] Record touch behavior. The screen recording module records the click coordinates of the user's each click on the main window. The screen recording module records the change in the average RGB value of the pixels within the click coordinate area between two click behaviors of the user. The screen recording module plots the average RGB value into an RGB curve in the order of the time axis. The screen recording module saves the click coordinates and the corresponding RGB curve to the register;

[0034] Click coordinate transformation. When the main window becomes the auxiliary window, the screen recording module retrieves the click coordinates from the register. The screen recording module calculates the length ratio and width ratio of the main window and the auxiliary window. According to the length ratio and width ratio, the click coordinates are converted into recording coordinates in equal proportion;

[0035] Execute the automated task. Each time the screen recording module executes a recording coordinate, the screen recording module converts the average RGB value of the pixels in the recording coordinate area into a recording curve in the order of the time axis. The screen recording module retrieves the RGB curve from the register and compares it with the recording curve. If the numerical changes of the two curves are the same, the screen recording module executes the next recording coordinate. On the contrary, if the numerical changes of the two curves are different, it means that the interface generated by executing the recording coordinate is different from the interface generated by the click coordinates, and the automated task execution is stopped. The screen recording module outputs a pop-up window indicating the failure of the automated task execution to the main window for the user to view.

[0036] Furthermore, the head analysis program specifically includes the following steps:

[0037] Locate the face and facial features. There is a planar face model preset inside the processing module. The processing module calculates the similarity between the pixels in the image information and the planar face model. The processing module selects the region with the highest similarity from the image information and marks it as the user's face. The processing module splits the planar face model into an eye model, a nose model, and a mouth model. The processing module calculates the similarity between the pixels in the user's face and the eye model, nose model, and mouth model respectively. The similarity calculation uses the MSE formula to calculate the pixel similarity of the two images. The processing module selects the midpoints of the two sides of the region with the highest similarity of the eye model from the user's face and marks them as the left and right eye corners. The processing module selects the intersection point of the diagonals of the region with the highest similarity of the nose model from the user's face and marks it as the tip of the nose. The processing module selects the midpoints of the two sides of the region with the highest similarity of the mouth model from the user's face and marks them as the left and right mouth corners;

[0038] Analyze the face orientation. The processing module connects the left eye corner, right eye corner, tip of the nose, left mouth corner, and right mouth corner with line segments. The processing module calculates the length z1 of all line segments on the left side of the tip of the nose and the length y1 of all line segments on the right side of the tip of the nose. If z1 - y1 < 0, the processing module outputs that the face orientation is to the left, and the degree of deviation to the left xy1 is , if z1 - y1 = 0, the processing module outputs that the face orientation is centered and the deviation degree xy1 is 0. If z1 - y1 > 0, the processing module outputs that the face orientation is to the right and the right deviation degree xy1 is ;

[0039] Locate the eye sockets. The processing module extracts the area between the left and right eye corners and marks it as the eye sockets. The processing module presets a circular vector diagram and calculates the similarity between the circular vector diagram and the eye sockets. The pixels with the highest and second highest similarities in the eye sockets are marked as pupils;

[0040] Analyze the eye orientation. The processing module counts the number of white pixels z2 on the left side and the number of white pixels y2 on the right side of the two pupils. The processing module defines the gray value range of each pixel as 0 - 255, and marks the pixels with a gray value greater than 247 as white pixels. If z2 - y2 < 0, the processing module outputs that the eye orientation is to the left and the left deviation degree xy2 is , if z2 - y2 = 0, the processing module outputs that the eye orientation is centered and the deviation degree xy2 is 0. If z2 - y2 > 0, the processing module outputs that the eye orientation is to the right and the right deviation degree ;

[0041] Output the attention direction. The processing module calculates the user's attention direction according to the formula . L is the length of the display module, W is the width of the display module, dis is the distance between the user's face and the display module. The camera module is located directly above the display module. Therefore, the distance dis between the user's face and the display module is proportional to the proportion of the user's face in the image information. , Fidp is the number of pixels of the user's face, Picp is the number of pixels of the image information, ds is the equivalent distance, and the value is a constant. The value range is generally within one foot plus or minus one inch, and the specific value is determined according to the pixel parameters of the camera module. The processing module calculates the distance dis according to the proportion of the number of pixels of the user's face to the number of pixels of the image information. w1 and w2 are weight coefficients, and the specific values are determined according to the number of times the user twists their neck and blinks.

[0042] Further, the eye analysis program specifically includes the following steps:

[0043] The processing module internally presets a planar face model. The processing module extracts the eye model and eyebrow model from the planar face model. The processing module calculates the similarity between the pixels in the image information and the eye model and eyebrow model in the planar face model. The similarity calculation uses the MSE formula to calculate the pixel similarity of the two images. The processing module marks the area with the highest similarity to the eye model in the image information as the eye reference, and marks the area with the highest similarity to the eyebrow model in the image information as the eyebrow reference. Both the eye reference and the eyebrow reference are two;

[0044] The processing module creates Condition 1. The processing module calculates the pixel ratio pix between two eyebrow references. The calculation formula is pix = Dm / Dz, where Dm is the average value of the maximum number of pixels in the horizontal direction between two eyebrow references, and Dz is the number of pixels between two eyebrow references. The processing module calculates the pixel ratio pix once per second. When the value of pix is less than or equal to 80% of the pix of the previous second, Condition 1 is triggered. Otherwise, the processing module continues to repeat the calculation of the pixel ratio pix. When the user can't see the content on the display module clearly, they will squint and subconsciously frown, and the distance between two eyebrow references will decrease.

[0045] The processing module creates Condition 2. The processing module calculates the aspect ratio ck of two eye references. The calculation formula is ck = Dh / Dw, where Dh is the average value of the maximum number of pixels in the vertical direction between two eye references, and Dw is the average value of the maximum number of pixels in the horizontal direction between two eye references. The processing module calculates the aspect ratio ck once per second. When the value of ck is less than or equal to 75% of the ck of the previous second, Condition 2 is triggered. Otherwise, the processing module continues to repeat the calculation of the aspect ratio ck. When the user squints, the height of two eye references in the vertical direction will decrease.

[0046] When both Condition 1 and Condition 2 are triggered, the processing module records the number of blinks j once. If either Condition 1 or Condition 2 is triggered, the processing module does not record the number of blinks j.

[0047] A split-screen intelligent control device for a tablet computer includes a display module, a processing module, and a camera module. The display module is a capacitive touch screen, and the display module is used to display a main window and several auxiliary windows. The processing module is a mobile microprocessor, and the processing module is used to adjust the display size, arrangement method, and focus frame rate of the windows. The camera module is an image sensor, and the camera module acquires the image information of the area directly opposite the display module.

[0048] A split-screen intelligent control storage medium for a tablet computer includes a register. The register internally stores the monitoring tasks, synchronization tasks, and automation tasks established by the user. The register internally stores the touch behavior of the user on the display module. The register also internally stores the computer programs or instructions for the synchronization module, screen recording module, and monitoring module to execute. When the computer programs or instructions are executed, the method described in any one of the above contents is implemented.

[0049] The present invention has the following beneficial effects:

[0050] 1. Automatically adjust the size of the main window and the font size for easy viewing by the user, making the display content of the display module more adaptable to the user's eyesight. Improving the local frame rate according to the user's focus can enhance the user experience.

[0051] 2. The monitoring module can help users automatically monitor the changes of background programs, broaden the usage scenarios of the tablet computer. The synchronization operation can help users control multiple identical programs simultaneously by one person, improving the usage efficiency of the tablet. The automatic execution of screen recording can help users achieve unattended automatic execution, improving the utilization rate when the tablet is idle.

[0052] Of course, it is not necessary for any product implementing the present invention to simultaneously achieve all the above-mentioned advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for describing the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0054] Figure 1 It is a system block diagram of an intelligent split-screen control device for a tablet computer according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0056] Please refer to Figure 1 , the present invention provides a technical solution: an intelligent split-screen control method for a tablet computer, including a register, a screen recording module, a synchronization module, a monitoring module, a display module, a processing module, and a camera module. The ports of the register are respectively connected to the ports of the synchronization module, the screen recording module, and the monitoring module. The ports of the synchronization module, the screen recording module, and the monitoring module are all connected to the port of the display module. The port of the display module is connected to the port of the processing module. The output end of the camera module is connected to the input end of the processing module;

[0057] The processing module obtains the length L and width W of the display module. The camera module obtains the image information of the user's head and transmits it to the processing module. The display module transmits the user's touch behavior to the processing module, the screen recording module, and the monitoring module in real time. The control method includes the following steps:

[0058] The window displays weights. The processing module presets a main window and auxiliary windows. The processing module executes a head analysis program to obtain the user's attention direction A, the left deviation degree xy1, and the right deviation degree xy2. The processing module automatically adjusts the arrangement mode of the main window and the auxiliary windows according to the user's attention direction A. The arrangement mode includes the window size and the arrangement direction. The processing module executes an eye analysis program to obtain the user's squinting times j. The higher the squinting times j, the less clear the user can see the display content of the display module, and the corresponding main window size and the font size of the text content in the main window need to be increased. The processing module presets a magnification factor k. The magnification factor k is associated with the main window size and the font size in the main window. The processing module automatically adjusts the magnification factor k according to the squinting times j;

[0059] Switching between the main window and the auxiliary windows. When the user clicks on any auxiliary window in the display module, the processing module swaps the clicked auxiliary window with the main window. The auxiliary window moves to the original position of the main window and is enlarged to the size of the original main window. The main window moves to the original position of the auxiliary window and is reduced to the size of the original auxiliary window. The user can only perform touch operations in the main window;

[0060] Focus frame rate optimization. The processing module presets a high-refresh window. At least one side edge of the high-refresh window is adjacent to the edge of the display module. The high-refresh window is an edge-attached window and overlaps with the main window or the auxiliary windows. The processing module automatically adjusts the size and position of the high-refresh window according to the attention direction A;

[0061] Screen monitoring. The user establishes a monitoring task through the display module. The monitoring module obtains the display information of each auxiliary window in real time. The monitoring module controls each auxiliary window to display an interactive animation according to the display information;

[0062] Window synchronization. The user establishes a synchronization task through the display module. The synchronization module connects several auxiliary windows to the main window according to the synchronization task. The synchronization module synchronizes the touch operations of the main window to each auxiliary window;

[0063] Automatic loop execution of screen recording. The user establishes an automation task through the display module. During the task establishment phase, the screen recording module obtains the touch operations of the main window and saves them in the register. During the task execution phase, the screen recording module transfers the touch operations saved in the register to the specified auxiliary window for execution.

[0064] Among them, the window display weight specifically includes the following steps:

[0065] The processing module sets the length of the main window to L and the width to 0.8W. The auxiliary windows are arranged in a column along the length of the main window. The number of auxiliary windows is n. The length of each auxiliary window is L / n, and the width is 0.2W;

[0066] The processing module aligns the main window along the user's attention direction with the edge of the display module, and aligns the arranged n auxiliary windows with the edge of the display module on the side opposite to the user's attention direction;

[0067] When L / n ≤ 0.1W, the auxiliary windows are arranged in two columns along the length of the main window. After the auxiliary windows are arranged in two columns, the processing module re-sets the length and width of each auxiliary window. The length of each auxiliary window is L / 2n, and the width is 0.1W;

[0068] When the applications in the auxiliary windows do not support freely changing the aspect ratio of the application interface, the processing module calculates the product of the length and width of each auxiliary window, and arranges the auxiliary windows in descending order according to the product results. The arrangement direction of the auxiliary windows is from top to bottom. For the auxiliary windows with the same product size, a secondary arrangement is performed in descending order according to the number of pixel colors;

[0069] The processing module initializes the magnification factor k. After initialization, the value of the magnification factor k is 1. The processing module calculates the magnification factor k of the main window according to the formula where j is the number of times of squinting. The processing module counts the number of times of squinting j every fixed time of 1 minute. The processing module magnifies the width of the main window and the text content in the main window according to the magnification factor k. When the magnification factor k is less than 1, it reduces the width of the main window and the text content in the main window;

[0070] When the width of the main window reaches 0.9W, it stops magnifying. When the text content in the main window is magnified, the character spacing between the characters is reduced, and the text layout remains unchanged. When the character spacing is zero, the processing module reduces the pictures in the main window and expands the typesetting area of the text content.

[0071] Among them, the focus frame rate optimization specifically includes the following steps:

[0072] The processing module marks the intersection of the user's attention direction A and the display module as the focus;

[0073] Determine the size of the high-refresh window. The processing module calculates the size of the high-refresh window according to the formula where xy1 and xy2 are both the deviation degrees. xy1 is the deviation degree of the face orientation, xy2 is the deviation degree of the eyeball orientation, L is the length of the display module, and W is the width of the display module;

[0074] The position of the high-refresh window changes synchronously according to the user's attention direction A. When the attention direction A is to the left, the high-refresh window expands along the left side of the display module. When the attention direction A is to the right, the high-refresh window expands along the right side of the display module. The expanded high-refresh window stops at the focus. At present, the screen local high-refresh technology cannot achieve a completely free floating window, and a border window is used instead of a floating window to provide a high refresh rate.

[0075] Among them, the screen monitoring specifically includes the following steps:

[0076] Display information monitoring. The monitoring module sets the change cycle to 1 second. It compares the display information of each window pixel by pixel in terms of RGB values with the display information in the previous change cycle. If the RGB value of any pixel changes, it means the screen of the auxiliary window has changed, and then the interactive animation is triggered. On the contrary, if the RGB values of all pixels have not changed, the comparison is repeated in a loop;

[0077] Trigger the interactive animation. The monitoring module controls the auxiliary window that triggers the interactive animation to move back and forth three times towards the direction close to the main window within the change cycle, and the moving distance is half of the maximum side length of the auxiliary window, which is convenient to attract the user's attention.

[0078] Among them, window synchronization specifically includes the following steps:

[0079] The synchronization module establishes a touch coordinate system in the main window. The synchronization module records the touch behavior of the user in the main window, and the touch behavior includes the click time and click coordinates;

[0080] The synchronization module obtains the lengths and widths of the main window and the auxiliary window connected to the main window. The synchronization module obtains the average RGB value of the pixels within the click coordinate area of the main window;

[0081] The synchronization module converts the click coordinates of the main window into synchronous coordinates with equal ratios in the auxiliary window according to the length ratio and width ratio of the main window and the auxiliary window. The synchronization module compares the average RGB value of the pixels within the synchronous coordinate area with the average RGB value of the pixels within the click coordinate area for verification;

[0082] If the average RGB values of the main window and the auxiliary window are the same, it means the synchronous coordinate verification is passed. The synchronization module executes the synchronous coordinates in the auxiliary window at the click time. If the average RGB values of the main window and the auxiliary window are different, it means the synchronous coordinates have not passed the verification. The synchronization module does not execute the synchronous coordinates in the auxiliary window, and the synchronization module outputs a pop-up window indicating the failure of the synchronization task and transmits it to the main window for the user to view.

[0083] Among them, the screen recording automatically loops and executes specifically includes the following steps:

[0084] Record the touch behavior. The screen recording module records the click coordinates of the user's each click on the main window. The screen recording module records the change in the average RGB value of the pixels within the click coordinate area between two click behaviors of the user. The screen recording module plots the average RGB value into an RGB curve according to the time axis sequence. The screen recording module saves the click coordinates and the corresponding RGB curve to the register;

[0085] Click coordinate conversion. When the main window becomes the auxiliary window, the screen recording module retrieves the click coordinates from the register. The screen recording module calculates the length ratio and width ratio of the main window and the auxiliary window, and converts the click coordinates into recording coordinates in equal proportion according to the length ratio and width ratio.

[0086] Execute the automated task. Each time the screen recording module executes a recording coordinate, the screen recording module converts the average RGB value of the pixels in the recording coordinate area into a recording curve in chronological order. The screen recording module retrieves the RGB curve from the register and compares it with the recording curve. If the numerical changes of the two curves are the same, the screen recording module executes the next recording coordinate. Otherwise, if the numerical changes of the two curves are different, it means that the interface generated by executing the recording coordinate is different from the interface generated by the click coordinate, and the automated task execution is stopped. The screen recording module outputs a pop-up window indicating the failure of the automated task execution to the main window for the user to view.

[0087] Among them, the head analysis program specifically includes the following steps:

[0088] Locate the face and facial features. The processing module internally presets a planar face model. The processing module calculates the similarity between the pixels in the image information and the planar face model. The processing module selects the region with the highest similarity in the image information and marks it as the user's face. The processing module splits the planar face model into an eye model, a nose model, and a mouth model. The processing module calculates the similarity between the pixels in the user's face and the eye model, nose model, and mouth model respectively. The similarity calculation uses the MSE formula to calculate the pixel similarity of the two images. The processing module selects the midpoints of the two sides of the region with the highest similarity to the eye model in the user's face and marks them as the left and right eye corners. The processing module selects the intersection point of the diagonal of the region with the highest similarity to the nose model in the user's face and marks it as the tip of the nose. The processing module selects the midpoints of the two sides of the region with the highest similarity to the mouth model in the user's face and marks them as the left and right mouth corners.

[0089] Analyze the face orientation. The processing module connects the left eye corner, right eye corner, tip of the nose, left mouth corner, and right mouth corner with line segments. The processing module calculates the length z1 of all line segments on the left side of the tip of the nose and the length y1 of all line segments on the right side of the tip of the nose. If z1 - y1 < 0, the processing module outputs that the face orientation is to the left, and the deviation degree to the left xy1 is , if z1 - y1 = 0, the processing module outputs that the face orientation is centered, and the deviation degree xy1 is 0. If z1 - y1 > 0, the processing module outputs that the face orientation is to the right, and the deviation degree to the right xy1 is ;

[0090] Locate the eye sockets. The processing module extracts the region between the left and right eye corners and marks it as the eye socket. The processing module presets a circular vector diagram and calculates the similarity between the circular vector diagram and the eye socket. The pixels with the highest and second highest similarity in the eye socket are marked as the pupils.

[0091] Analyze the eye orientation. The processing module counts the number of white pixels z2 on the left side and the number of white pixels y2 on the right side of the two pupils. The processing module defines the gray value range of each pixel as 0 - 255, and marks the pixels with a gray value greater than 247 as white pixels. If z2 - y2 < 0, the processing module outputs that the eye orientation is to the left, and the left deviation degree xy2 is , if z2 - y2 = 0, the processing module outputs that the eye orientation is centered, and the deviation degree xy2 is 0. If z2 - y2 > 0, the processing module outputs that the eye orientation is to the right, and the right deviation degree ;

[0092] Output the attention direction. The processing module calculates the user's attention direction according to the formula . L is the length of the display module, W is the width of the display module, dis is the distance between the user's face and the display module. The camera module is located directly above the display module. Therefore, the distance dis between the user's face and the display module is proportional to the proportion of the user's face in the image information. , Fidp is the number of pixels of the user's face, Picp is the number of pixels of the image information, ds is the equidistant distance, and the value is a constant. The value range is generally within plus or minus one inch of one foot, and the specific value is determined according to the pixel parameters of the camera module. The processing module calculates the distance dis according to the proportion of the number of pixels of the user's face to the number of pixels of the image information. w1 and w2 are weight coefficients, and the specific values are determined according to the number of times the user twists their neck and blinks.

[0093] Among them, the eye analysis program specifically includes the following steps:

[0094] The processing module internally presets a planar face model. The processing module extracts the eye model and the eyebrow model from the planar face model. The processing module calculates the similarity between the pixels in the image information and the eye model and the eyebrow model in the planar face model. The similarity calculation uses the MSE formula to calculate the pixel similarity of the two images. The processing module marks the area with the highest similarity to the eye model in the image information as the eye reference, and marks the area with the highest similarity to the eyebrow model in the image information as the eyebrow reference. There are two eye references and two eyebrow references.

[0095] The processing module creates condition one. The processing module calculates the pixel ratio pix between the two eyebrow references. The calculation formula is pix = Dm / Dz, where Dm is the average value of the maximum number of pixels in the horizontal direction between the two eyebrow references, and Dz is the number of pixels between the two eyebrow references. The processing module calculates the pixel ratio pix once per second. When the value of pix is less than or equal to 80% of the pix of the previous second, condition one is triggered. Otherwise, the processing module continues to repeat the calculation of the pixel ratio pix. When the user can't see the content of the display module clearly, they will squint and subconsciously frown, and the distance between the two eyebrow references will decrease.

[0096] The processing module creates Condition 2. The processing module calculates the aspect ratio ck of the two-eye reference, and the calculation formula is ck = Dh / Dw, where Dh is the average value of the maximum number of pixels in the vertical direction of the two-eye reference, and Dw is the average value of the maximum number of pixels in the horizontal direction of the two-eye reference. The processing module calculates the aspect ratio ck once per second. When the value of ck is less than or equal to 75% of the ck of the previous second, Condition 2 is triggered. Otherwise, the processing module continues to repeat the calculation of the aspect ratio ck. When the user squints, the height of the two-eye reference in the vertical direction will decrease;

[0097] When both Condition 1 and Condition 2 are triggered, the processing module records the number of blinks j once. If either Condition 1 or Condition 2 is triggered, the processing module does not record the number of blinks j.

[0098] A split-screen intelligent control device for a tablet computer includes a display module, a processing module, and a camera module. The display module is a capacitive touch screen and is used to display a main window and several auxiliary windows. The processing module is a mobile microprocessor and is used to adjust the display size, arrangement method, and focus frame rate of the windows. The camera module is an image sensor and is used to obtain the image information of the area directly opposite the display module.

[0099] A split-screen intelligent control storage medium for a tablet computer includes a register. The register internally stores the monitoring tasks, synchronization tasks, and automation tasks established by the user. The register internally stores the touch behavior of the user on the display module. The register also internally stores the computer programs or instructions for the synchronization module, screen recording module, and monitoring module to execute. When the computer programs or instructions are executed, the method described in any one of the above embodiments is implemented.

[0100] As mentioned above, the above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and all should be covered by the protection scope of the present invention.

Claims

1. A method for intelligent split-screen control of a tablet computer, comprising a register, a screen recording module, a synchronization module, a monitoring module, a display module, a processing module, and a camera module, characterized in that: The processing module obtains the length L and width W of the display module. The camera module obtains the image information of the user's head and transmits it to the processing module. The display module transmits the user's touch behavior to the processing module, the screen recording module, and the monitoring module in real time. The control method includes the following steps: The processing module preset the main window and the auxiliary window. The processing module executes the head analysis program to obtain the user's attention direction A, deviation degree xy1, and deviation degree xy2. The processing module automatically adjusts the arrangement mode of the main window and the auxiliary window according to the user's attention direction A. The arrangement mode includes window size and arrangement direction. The processing module executes the eye analysis program to obtain the number of times j the user squints. The processing module presets the magnification factor k. The magnification factor k is associated with the size of the main window and the font size in the main window. The processing module automatically adjusts the magnification factor k according to the number of times j of squinting; Switch between the main window and the auxiliary window. When the user clicks on any auxiliary window in the display module, the processing module swaps the clicked auxiliary window with the main window; The processing module presets a high-refresh window. At least one side edge of the high-refresh window is adjacent to the edge of the display module. The high-refresh window is a side-attached window and overlaps with the main window or the auxiliary window. The processing module automatically adjusts the size and position of the high-refresh window according to the attention direction A; The processing module marks the intersection of the user's attention direction A and the display module as the focus; Determine the size of the high-refresh window, and the processing module calculates it according to the formula Calculate the size of the high-refresh window, where both xy1 and xy2 are the deviation degrees, L is the length of the display module, and W is the width of the display module; The position of the high-refresh window changes synchronously according to the user's attention direction A. When the attention direction A is to the left, the high-refresh window unfolds along the left side of the display module. When the attention direction A is to the right, the high-refresh window unfolds along the right side of the display module. The unfolded high-refresh window stops at the focus; Screen monitoring. The user establishes a monitoring task through the display module. The monitoring module obtains the display information of each auxiliary window in real time. The monitoring module controls each auxiliary window to display an interactive animation according to the display information; Window synchronization. The user establishes a synchronization task through the display module. The synchronization module connects several auxiliary windows with the main window according to the synchronization task. The synchronization module synchronizes the touch behavior of the main window to each auxiliary window; Screen recording is automatically executed in a loop. The user establishes an automation task through the display module. In the task establishment stage, the screen recording module obtains the touch behavior of the main window and saves it in the register. In the task execution stage, the screen recording module transmits the touch behavior saved in the register to the specified auxiliary window for execution.

2. The method for intelligent split-screen control of a tablet computer according to claim 1, wherein, The ports of the register are respectively connected to the ports of the synchronization module, the screen recording module, and the monitoring module. The ports of the synchronization module, the screen recording module, and the monitoring module are all connected to the ports of the display module. The ports of the display module are connected to the ports of the processing module. The output end of the camera module is connected to the input end of the processing module; The processing module sets the length of the main window to L and the width to 0.8W. The auxiliary windows are arranged in a column along the length of the main window. The number of auxiliary windows is n. The length of each auxiliary window is L / n and the width is 0.2W; The processing module aligns the main window with the edge of the display module along the user's attention direction, and aligns the arranged n auxiliary windows with the edge of the display module on the side opposite to the user's attention direction; When L / n ≤ 0.1W, the auxiliary windows are arranged in two columns along the length of the main window. After the auxiliary windows are arranged in two columns, the processing module resets the length and width of each auxiliary window. The length of each auxiliary window is L / 2n, and the width is 0.1W; When the applications in the auxiliary windows do not support freely changing the aspect ratio of the application interface, the processing module calculates the product of the length and width of each auxiliary window and arranges the auxiliary windows in descending order according to the product results. The arrangement direction of the auxiliary windows is from top to bottom. For the auxiliary windows with the same product size, a secondary arrangement is performed in descending order according to the number of pixel colors; The processing module initializes the magnification factor k. The processing module calculates the magnification factor k of the main window according to the formula where j is the number of times of squinting. The processing module counts the number of times of squinting j every fixed time. The processing module magnifies the width of the main window and the text content within the main window according to the magnification factor k. When the magnification factor k is less than 1, it reduces the width of the main window and the text content within the main window. When the width of the main window reaches 0.9W, it will no longer be enlarged. When the text content in the main window is enlarged, the character spacing between the characters is reduced, and the text layout remains unchanged. When the character spacing is zero, the processing module reduces the pictures in the main window and enlarges the layout area of the text content.

3. A method for intelligent split-screen control of a tablet computer according to claim 1, characterized in that, The screen monitoring specifically includes the following steps: Display information monitoring. The monitoring module presets a change period and compares the display information of each window pixel by pixel with the RGB values of the display information in the previous change period. If the RGB value of any pixel changes, an interactive animation is triggered. Conversely, if the RGB values of all pixels have not changed, the comparison is repeated in a loop; Triggering the interactive animation. The monitoring module controls the auxiliary window that triggers the interactive animation to move back and forth three times in the change period in the direction close to the main window, and the moving distance is half of the maximum side length of the auxiliary window.

4. A method for intelligent control of split screen of a tablet computer according to claim 1, characterized in that, Window synchronization specifically includes the following steps: The synchronization module establishes a touch coordinate system in the main window. The synchronization module records the touch behavior of the user in the main window, and the touch behavior includes the click time and the click coordinates; The synchronization module obtains the length and width of the main window and the auxiliary windows. The synchronization module obtains the average RGB value of the pixels in the click coordinate area of the main window; According to the length ratio and width ratio of the main window and the auxiliary windows, the synchronization module converts the click coordinates of the main window into synchronous coordinates of the same ratio in the auxiliary window. The synchronization module compares the average RGB value of the pixels in the synchronous coordinate area with the average RGB value of the pixels in the click coordinate area for verification; If the average RGB values of the main window and the auxiliary window are the same, the synchronous coordinate verification passes, and the synchronization module executes the synchronous coordinates in the auxiliary window at the click time. If the average RGB values of the main window and the auxiliary window are different, the synchronous coordinate verification fails, and the synchronization module does not execute the synchronous coordinates in the auxiliary window. The synchronization module outputs a pop-up window indicating the failure of the synchronization task and transmits it to the main window for display.

5. A method for intelligent control of split screen of a tablet computer according to claim 1, characterized in that, The screen recording automatically loops and executes specifically includes the following steps: Recording touch behavior. The screen recording module records the click coordinates of the user's each click on the main window. The screen recording module records the change in the average RGB value of the pixels in the click coordinate area between two click behaviors of the user. The screen recording module plots the average RGB value into an RGB curve. The screen recording module saves the click coordinates and the RGB curve to the register; Click coordinate conversion. When the main window becomes the auxiliary window, the screen recording module retrieves the click coordinates from the register. The screen recording module calculates the length ratio and width ratio of the main window and the auxiliary window, and converts the click coordinates into recording coordinates in equal proportion according to the length ratio and width ratio. Execute the automated task. Each time the screen recording module executes the recording coordinates, the screen recording module converts the average RGB value of the pixels in the recording coordinate area into a recording curve. The screen recording module retrieves the RGB curve from the register and compares it with the recording curve. If the numerical changes of the two curves are the same, the screen recording module executes the next recording coordinate. Otherwise, if the numerical changes of the two curves are different, the automated task execution is stopped, and the screen recording module outputs a pop-up window indicating the failure of the automated task execution to the main window for display.

6. The method for intelligent split-screen control of a tablet computer according to claim 1, wherein, The head analysis program specifically includes the following steps: Locate the face and facial features. There is a planar face model preset inside the processing module. The processing module calculates the similarity between the pixels in the image information and the planar face model. The processing module selects the region with the highest similarity in the image information and marks it as the user's face. The processing module splits the planar face model into an eye model, a nose model, and a mouth model. The processing module calculates the similarity between the pixels in the user's face and the eye model, nose model, and mouth model respectively. The processing module selects the midpoints of the two sides of the region with the highest similarity to the eye model in the user's face and marks them as the left and right eye corners. The processing module selects the intersection point of the diagonals of the region with the highest similarity to the nose model in the user's face and marks it as the tip of the nose. The processing module selects the midpoints of the two sides of the region with the highest similarity to the mouth model in the user's face and marks them as the left and right mouth corners. Analyze the face orientation. The processing module connects the left eye corner, right eye corner, nose tip, left mouth corner, and right mouth corner with line segments. The processing module calculates the length z1 of all line segments on the left side of the nose tip and the length y1 of all line segments on the right side of the nose tip. If z1 - y1 < 0, the processing module outputs that the face orientation is to the left, and the deviation degree xy1 to the left is , if z1 - y1 = 0, the processing module outputs that the face orientation is centered, and the deviation degree xy1 is 0. If z1 - y1 > 0, the processing module outputs that the face orientation is to the right, and the deviation degree xy1 to the right is ; Locate the eye sockets. The processing module extracts the region between the left and right eye corners and marks it as the eye socket. The processing module presets a circular vector diagram and calculates the similarity between the circular vector diagram and the eye socket. The pixels with the highest and second highest similarity in the eye socket are marked as the pupils. Analyze the eye orientation. The processing module counts the number of white pixels z2 on the left side and the number of white pixels y2 on the right side of the two pupils. If z2 - y2 < 0, the processing module outputs that the eye orientation is to the left, and the degree of deviation to the left xy2 is , if z2 - y2 = 0, the processing module outputs that the eye orientation is centered, and the degree of deviation xy2 is 0. If z2 - y2 > 0, the processing module outputs that the eye orientation is to the right, and the degree of deviation to the right ; Output the attention direction, and the processing module calculates according to the formula to obtain the user's attention direction. L is the length of the display module, W is the width of the display module, dis is the distance between the user's face and the display module. The camera module is located directly above the display module. Therefore, the distance dis between the user's face and the display module is directly proportional to the proportion of the user's face in the image information. , Fidp is the number of user face pixels, Picp is the number of image information pixels, ds is the equidistant distance, and the value is a constant. The value range is generally within one foot plus or minus one inch, and the specific value is determined according to the pixel parameters of the camera module. The processing module calculates the distance dis according to the proportion of the number of user face pixels to the number of image information pixels. w1 and w2 are weight coefficients, and the specific values are determined according to the number of times the user twists their neck and blinks.

7. A method for intelligent split - screen control of a tablet computer according to claim 1, characterized in that, The eye analysis program specifically includes the following steps: There is a planar face model preset inside the processing module. The processing module extracts the eye model and the eyebrow model from the planar face model. The processing module calculates the similarity between the pixels in the image information and the eye model and the eyebrow model in the planar face model. The processing module marks the region with the highest similarity to the eye model in the image information as the eye reference, and marks the region with the highest similarity to the eyebrow model in the image information as the eyebrow reference. There are two eye references and two eyebrow references. The processing module creates condition one. The processing module calculates the pixel ratio pix between the two eyebrow references. The calculation formula is pix = Dm / Dz, where Dm is the average of the maximum number of pixels in the horizontal direction between the two eyebrow references, and Dz is the number of pixels between the two eyebrow references. The processing module calculates the pixel ratio pix once per second. When the value of pix is less than or equal to 80% of the pix of the previous second, condition one is triggered. Otherwise, the processing module continues to repeat the calculation of the pixel ratio pix. The processing module creates Condition 2. The processing module calculates the aspect ratio ck of the two-eye reference, and the calculation formula is ck = Dh / Dw, where Dh is the average value of the maximum number of pixels in the vertical direction of the two-eye reference, and Dw is the average value of the maximum number of pixels in the horizontal direction of the two-eye reference. The processing module calculates the aspect ratio ck once per second. When the value of ck is less than or equal to 75% of the ck of the previous second, Condition 2 is triggered. Otherwise, the processing module continues to repeat the calculation of the aspect ratio ck; When both Condition 1 and Condition 2 are triggered, the processing module records a blink count j. When either Condition 1 or Condition 2 is triggered, the processing module does not record the blink count j.

8. An intelligent control device for split-screen of a tablet computer, characterized in that Applied to implement a split-screen intelligent control method for a tablet computer as described in Claim 1, including a display module, a processing module, and a camera module. The display module is a capacitive touch screen, and the display module is used to display a main window and several auxiliary windows. The processing module is a mobile microprocessor, and the processing module is used to adjust the display size, arrangement mode, and focus frame rate of the windows. The camera module is an image sensor, and the camera module acquires image information of the area directly opposite the display module.

9. A storage medium for intelligent split-screen control of a tablet computer, characterized in that Applied to implement a split-screen intelligent control method for a tablet computer as described in Claim 1, including a register. The register internally stores monitoring tasks, synchronization tasks, and automation tasks established by the user. The register internally stores the touch behavior of the user on the display module. The register also internally stores computer programs or instructions for the synchronization module, screen recording module, and monitoring module to execute.

Images