Tablet computer split screen intelligent control method and device and storage medium

Abstract

The invention discloses an intelligent split-screen control method and device for a tablet computer and a storage medium, and relates to the technical field of split-screen control. The system comprises a register, a screen recording module, a synchronization module, a monitoring module, a display module, a processing module and a camera module, the port of the register is connected with the ports of the synchronization module, the screen recording module and the monitoring module, and the ports of the synchronization module, the screen recording module and the monitoring module are connected with the port of the display module. The port of the display module is connected with the port of the processing module, the output end of the camera module is connected with the input end of the processing module, and the processing module obtains the length L and the width W. According to the invention, the size of the main window and the font size are automatically adjusted for a user to watch, so that the display content of the display module is more adaptive to the vision of the user; the user impression can be improved by improving the local frame rate according to the focus of the user, and the use scene of the tablet computer can be widened by the monitoring module.

Description

Technical Field

[0001] The present invention relates to the technical field of split-screen control, and in particular to a method, device and storage medium for intelligent split-screen control of a tablet computer. Background Art

[0002] With the improvement of the hardware level of tablet computers on the market, the demand for multi-tasking of tablet computers has gradually increased. Split-screen technology allows users to operate multiple applications on the same screen at the same time, improving the efficiency of work and study.

[0003] The existing tablet split-screen technology has the following defects: 1. It cannot automatically adjust the size of the split screen according to the user's habits; 2. It cannot realize dynamic frame rate adjustment after splitting the screen, and cannot automatically increase the frame rate of the corresponding display area according to the user's focus; 3. It cannot help users automatically monitor the programs hanging in the background; 4. It cannot realize the synchronous linkage of multiple windows after splitting the screen; 5. It cannot realize screen recording to automatically repeat user operations in the background; therefore, improving the functional defects of the above split-screen technology is a technical problem that technical personnel in this field need to solve. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a method, device and storage medium for intelligent control of split screen of a tablet computer, which solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a tablet computer split-screen intelligent control method, including a register, a screen recording module, a synchronization module, a monitoring module, a display module, a processing module and a camera module, the ports of the register are respectively connected to the ports of the synchronization module, the screen recording module and the monitoring module, the ports of the synchronization module, the screen recording module and the monitoring module are all connected to the port of the display module, the port of the display module is connected to the port of the processing module, and the output end of the camera module is connected to the input end of the processing module; The processing module obtains the length L and width W of the display module, the camera module obtains the image information of the user's head and transmits it to the processing module, and the display module transmits the user's touch behavior to the processing module, the screen recording module and the monitoring module in real time. The control method includes the following steps: Window display weight, the processing module presets the main window and the auxiliary window, the processing module executes the head analysis program to obtain the user's focus direction A, the left deviation degree xy1 and the right deviation degree xy2, the processing module automatically adjusts the arrangement of the main window and the auxiliary window according to the user's focus direction A, the arrangement method includes the window size and arrangement direction, the processing module executes the eye analysis program to obtain the user's squinting times j, the higher the squinting times j, the user cannot see the display content of the display module clearly, and the size of the main window and the size of the text content in the main window need to be increased accordingly, the processing module presets the magnification k, the magnification k is associated with the main window size and the font size in the main window, the processing module automatically adjusts the magnification k according to the squinting times j, and automatically adjusts the main window size and the font size for the user to watch, so that the display content of the display module is more suitable for the user's eyesight; Switching between the main window and the auxiliary window: when the user clicks on any auxiliary window in the display module, the processing module swaps the clicked auxiliary window with the main window. The auxiliary window moves to the original main window position and is enlarged to the original main window size. The main window moves to the original auxiliary window position and is reduced to the original auxiliary window size. The user can only perform touch actions in the main window. Focus frame rate optimization: the processing module presets a high refresh window, at least one side of the high refresh window is in contact with the edge of the display module, the high refresh window is an edge-attached window and overlaps with the main window or auxiliary window. The processing module automatically adjusts the size and position of the high refresh window according to the focus direction A. Improving the local frame rate can improve the user's viewing experience. Screen monitoring: users establish monitoring tasks through the display module. The monitoring module obtains the display information of each auxiliary window in real time. The monitoring module controls each auxiliary window to display interactive animations based on the display information. The monitoring module can help users automatically monitor changes in background programs and expand the use scenarios of tablet computers. Window synchronization: users establish synchronization tasks through the display module. The synchronization module connects several auxiliary windows to the main window according to the synchronization tasks. The synchronization module synchronizes the touch behavior of the main window to each auxiliary window. The synchronization operation can help users control multiple identical programs at the same time, thereby improving the efficiency of tablet use. The screen recording is automatically executed in a loop. The user establishes an automated task through the display module. During the task establishment phase, the screen recording module obtains the touch behavior of the main window and saves it in the register. During the task execution phase, the screen recording module transfers the touch behavior saved in the register to the specified auxiliary window for execution. Automatic execution of screen recording can help users achieve unmanned automatic execution and improve the utilization rate of the tablet computer when it is idle.

[0006] Furthermore, the window display weight specifically includes the following steps: The processing module sets the length of the main window to L and the width to 0.8W, and the auxiliary windows are arranged in a row along the length L of the main window. The number of auxiliary windows is n, and the length of each auxiliary window is L / n and the width is 0.2W; The processing module aligns the main window with the edge of the display module along the user's focus direction, and aligns the arranged n auxiliary windows with the edge of the display module on the opposite side of the user's focus direction; When L / n≤0.1W, the auxiliary windows are arranged into two columns along the length L of the main window. After the auxiliary windows are arranged into two columns, the processing module resets the length and width of each auxiliary window. The length of each auxiliary window is L / 2n and the width is 0.1W. When the application in the auxiliary frame does not support the free change of the aspect ratio of the application interface, the processing module calculates the product of the length and width of each auxiliary window, and arranges the auxiliary windows in descending order according to the product results. The auxiliary windows are arranged from top to bottom. The auxiliary windows with the same product size are arranged again in descending order according to the number of pixel colors. The processing module initializes the magnification k, and the value of the magnification k after initialization is 1. The processing module uses the formula Calculate the magnification k of the main window, j is the number of squinting times, the processing module counts the number of squinting times j every fixed time of 1 minute, and the processing module enlarges the width of the main window and the text content in the main window according to the magnification j. When the magnification j is less than 1, the width of the main window and the text content in the main window are reduced; When the width of the main window reaches 0.9W, it will no longer be enlarged. When the text content in the main window is enlarged, the distance between the characters is reduced, and the text layout remains unchanged. When the character spacing is zero, the processing module reduces the image in the main window and expands the layout area of ​​the text content.

[0007] Furthermore, the focus frame rate optimization specifically includes the following steps: The processing module marks the intersection of the user's focus direction A and the display module as a focus; Determine the size of the high refresh window, and the processing module uses the formula Calculate the size of the high refresh window, xy1 and xy2 are both deviations, xy1 is the deviation of the face direction, xy2 is the deviation of the eye direction, L is the length of the display module, and W is the width of the display module; The position of the high refresh window changes synchronously with the user's attention direction A. When the attention direction A is to the left, the high refresh window expands along the left side of the display module. When the attention direction A is to the right, the high refresh window expands along the right side of the display module. The expanded high refresh window stops at the focus. The local high refresh technology of the screen cannot currently achieve a completely free floating window, and a border window is used instead of a floating window to provide a high refresh rate.

[0008] Furthermore, the screen monitoring specifically includes the following steps: Display information monitoring: The monitoring module presets a change cycle of 1 second, and compares the RGB values ​​of each window's display information with the display information of the previous change cycle pixel by pixel. If the RGB value of any pixel changes, it means that the screen of the auxiliary window has changed, and the interactive animation is triggered. On the contrary, if the RGB values ​​of all pixels change slightly, the comparison is repeated in a cycle. Trigger the interactive animation. The monitoring module controls the auxiliary window that triggers the interactive animation to reciprocate three times toward the main window within the change cycle. The movement distance is half of the maximum side length of the auxiliary window, so as to attract the user's attention.

[0009] Furthermore, the window synchronization specifically includes the following steps: The synchronization module establishes a touch coordinate system in the main window. The synchronization module records the user's touch behavior in the main window. The touch behavior includes click time and click coordinates. The synchronization module obtains the length and width of the main window and the auxiliary window connected to the main window, and the synchronization module obtains the average RGB value of the pixel in the click coordinate area of ​​the main window; The synchronization module converts the click coordinates of the main window into synchronization coordinates of the auxiliary window with the same ratio according to the length ratio and width ratio of the main window and the auxiliary window, and compares the RGB value average of the pixels in the synchronization coordinate area with the RGB value average of the pixels in the click coordinate area for verification; If the average RGB values ​​of the main window and the auxiliary window are the same, it means that the synchronization coordinate verification has passed, and the synchronization module executes the click time and synchronization coordinates in the auxiliary window. If the average RGB values ​​of the main window and the auxiliary window are different, it means that the synchronization coordinates have not passed the verification, and the synchronization module does not execute the click time and synchronization coordinates in the auxiliary window. The synchronization module outputs a pop-up window indicating that the synchronization task has failed and transmits it to the main window for display by the user.

[0010] Furthermore, the automatic loop execution of screen recording specifically includes the following steps: Recording touch behaviors: the screen recording module records the click coordinates of each click of the main window by the user. The screen recording module records the change of the average RGB value of the pixels in the click coordinate area between two clicks of the user. The screen recording module plots the average RGB value into an RGB curve in the order of the time axis. The screen recording module saves the click coordinates and the RGB curve corresponding to the click coordinates to the register. Click coordinate conversion: when the main window becomes the auxiliary window, the screen recording module retrieves the click coordinates from the register, calculates the length ratio and width ratio of the main window and the auxiliary window, and converts the click coordinates into recording coordinates in proportion according to the length ratio and width ratio; Execute the automated task. Each time the screen recording module executes the recording coordinates, the screen recording module converts the RGB value average of the pixels in the recording coordinate area into a recording curve in the time axis order. The screen recording module retrieves the RGB curve from the register and compares it with the recording curve. If the value changes of the two curves are the same, the screen recording module executes the next recording coordinate. On the contrary, if the value changes of the two curves are the same, it means that the interface generated by executing the recording coordinates is different from the interface generated by clicking the coordinates, and the automated task is stopped. The screen recording module outputs a pop-up window indicating that the automated task execution failed to the main window for the user to view.

[0011] Furthermore, the header analysis procedure specifically includes the following steps: Locate the face and facial features. A plane face model is preset inside the processing module. The processing module calculates the similarity between the pixels in the image information and the plane face model. The processing module selects the area with the highest similarity from the image information and marks it as the user's face. The processing module splits the plane face model into an eye model, a nose model and a mouth model. The processing module calculates the similarity between the pixels in the user's face and the eye model, the nose model and the mouth model respectively. The similarity calculation uses the MSE formula to calculate the pixel similarity of the two images. The processing module selects the midpoints of both sides of the area with the highest similarity of the eye model from the user's face and marks them as the left eye corner and the right eye corner. The processing module selects the diagonal intersection of the area with the highest similarity of the nose model from the user's face and marks it as the nose tip. The midpoints of both sides of the area with the highest similarity of the mouth model from the user's face are marked as the left mouth corner and the right mouth corner. Analyze the face orientation, the processing module connects the left eye corner, right eye corner, nose tip, left mouth corner and right mouth corner with line segments, and calculates the length z1 of all line segments on the left side of the nose tip and the length y1 of all line segments on the right side of the nose tip. If z1-y1<0, the processing module outputs that the face orientation is biased to the left, and the degree of left deviation xy1 is If z1-y1=0, the processing module outputs that the face is centered and the deviation degree xy1 is 0. If z1-y1>0, the processing module outputs that the face is biased to the right and the deviation degree xy1 is ; Locate the eye sockets, the processing module extracts the area between the left and right eye corners and marks it as the eye sockets, the processing module presets a circular vector graph, calculates the similarity between the circular vector graph and the eye sockets, and marks the pixels with the highest and second highest similarity in the eye sockets as pupils; Analyze the eyeball direction. The processing module counts the number of white pixels on the left side of the two pupils, z2, and the number of white pixels on the right side, y2. The processing module defines the grayscale value range of each pixel as 0-255, and marks each pixel with a grayscale value greater than 247 as a white pixel. If z2-y2<0, the processing module outputs that the eyeball direction is biased to the left, and the degree of left deviation xy2 is If z2-y2=0, the processing module outputs that the eyeball is centered and the deviation degree xy2 is 0. If z2-y2>0, the processing module outputs that the eyeball is biased to the right and the deviation degree to the right is ; Output the attention direction, and the processing module uses the formula Calculate the user's attention direction, where L is the length of the display module, W is the width of the display module, and dis is the distance between the user's face and the display module. The camera module is located directly above the display module, so the distance dis between the user's face and the display module is proportional to the proportion of the user's face in the image information. , Fidp is the number of pixels of the user's face, Picp is the number of pixels of the image information, ds is the geometric distance, the value is a constant, and the value range is generally within the range of plus or minus one inch of one foot. The specific value is determined according to the pixel parameters of the camera module. The processing module calculates the distance dis according to the ratio of the number of pixels of the user's face to the number of pixels of the image information. w1 and w2 are weight coefficients, and the specific value is determined according to the number of times the user twists his neck and moves his eyes.

[0012] Furthermore, the eye analysis procedure specifically includes the following steps: A plane face model is preset inside the processing module, and the processing module extracts an eye model and an eyebrow model from the plane face model. The processing module calculates the similarity between pixels in the image information and the eye model and the eyebrow model in the plane face model. The similarity calculation uses the MSE formula to calculate the pixel similarity of the two images. The processing module marks the area in the image information that is most similar to the eye model as an eye reference, and the processing module marks the area in the image information that is most similar to the eyebrow model as an eyebrow reference. There are two eye references and two eyebrow references. The processing module creates condition one. The processing module calculates the pixel ratio pix between the two eyebrow references. The calculation formula is pix=Dm / Dz, where Dm is the average of the maximum number of pixels in the horizontal direction of the two eyebrow references, and Dz is the number of pixels between the two eyebrow references. The processing module calculates the pixel ratio pix once per second. When the value of pix is ​​less than or equal to 80% of the pix of the previous second, condition one is triggered. Otherwise, the processing module continues to repeatedly calculate the pixel ratio pix. When the user cannot see the content of the display module clearly, he will squint and frown subconsciously, and the distance between the two eyebrow references will decrease. The processing module creates condition 2. The processing module calculates the aspect ratio ck of the two eyes. The calculation formula is ck=Dh / Dw, where Dh is the average value of the maximum number of pixels in the vertical direction of the two eyes, and Dw is the average value of the maximum number of pixels in the horizontal direction of the two eyes. The processing module calculates the aspect ratio ck once per second. When the value of ck is less than or equal to 75% of ck of the previous second, condition 2 is triggered. Otherwise, the processing module continues to repeatedly calculate the aspect ratio ck. When the user squints, the height of the two eyes in the vertical direction will decrease. When condition 1 and condition 2 are triggered at the same time, the processing module records the number of blinks j. When either condition 1 or condition 2 is triggered, the processing module does not record the number of blinks j.

[0013] A tablet computer split-screen intelligent control device comprises a display module, a processing module and a camera module. The display module is a capacitive touch screen, the display module is used to display a main window and a plurality of auxiliary windows, the processing module is a mobile microprocessor, the processing module is used to adjust the window display size, arrangement mode and focus frame rate, and the camera module is an image sensor, which obtains image information of the area facing the display module.

[0014] A tablet computer split-screen intelligent control storage medium includes a register, wherein the register stores monitoring tasks, synchronization tasks and automation tasks established by a user, the register stores the user's touch behavior on a display module, and the register also stores computer programs or instructions for execution by a synchronization module, a screen recording module and a monitoring module. When the computer program or instruction is executed, any method described in the above content is implemented.

[0015] The present invention has the following beneficial effects: 1. Automatically adjust the main window size and font size for user viewing convenience, so that the display content of the display module is more suitable for the user's eyesight. Increasing the local frame rate according to the user's focus can improve the user's viewing experience.

[0016] 2. The monitoring module can help users automatically monitor changes in background programs and broaden the usage scenarios of tablet computers. Synchronous operation can help users control multiple identical programs at the same time and improve the efficiency of tablet use. Automatic screen recording can help users achieve unmanned automatic execution and improve the utilization rate of tablet computers when they are idle.

[0017] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0019] Figure 1 The present invention is a system block diagram of a tablet computer split-screen intelligent control device. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] See also Figure 1 The present invention provides a technical solution: a tablet computer split-screen intelligent control method, comprising a register, a screen recording module, a synchronization module, a monitoring module, a display module, a processing module and a camera module, the ports of the register are respectively connected to the ports of the synchronization module, the screen recording module and the monitoring module, the ports of the synchronization module, the screen recording module and the monitoring module are all connected to the port of the display module, the port of the display module is connected to the port of the processing module, and the output end of the camera module is connected to the input end of the processing module; The processing module obtains the length L and width W of the display module, the camera module obtains the image information of the user's head and transmits it to the processing module, and the display module transmits the user's touch behavior to the processing module, the screen recording module and the monitoring module in real time. The control method includes the following steps: Window display weight, the processing module presets the main window and the auxiliary window, the processing module executes the head analysis program to obtain the user's focus direction A, the left deviation degree xy1 and the right deviation degree xy2, the processing module automatically adjusts the arrangement of the main window and the auxiliary window according to the user's focus direction A, the arrangement method includes the window size and arrangement direction, the processing module executes the eye analysis program to obtain the user's squinting times j, the higher the squinting times j, the user cannot see the display content of the display module clearly, and the size of the main window and the size of the text content in the main window need to be increased accordingly, the processing module presets the magnification k, the magnification k is associated with the main window size and the font size in the main window, and the processing module automatically adjusts the magnification k according to the squinting times j; Switching between the main window and the auxiliary window: when the user clicks on any auxiliary window in the display module, the processing module swaps the clicked auxiliary window with the main window. The auxiliary window moves to the original main window position and is enlarged to the original main window size. The main window moves to the original auxiliary window position and is reduced to the original auxiliary window size. The user can only perform touch actions in the main window. Focus frame rate optimization, the processing module presets a high refresh window, at least one side edge of the high refresh window is attached to the edge of the display module, the high refresh window is an edge-attached window and overlaps with the main window or the auxiliary window, and the processing module automatically adjusts the size and position of the high refresh window according to the focus direction A; Screen monitoring: users establish monitoring tasks through the display module. The monitoring module obtains the display information of each auxiliary window in real time. The monitoring module controls each auxiliary window to display interactive animations according to the display information. Window synchronization: the user establishes a synchronization task through the display module. The synchronization module connects several auxiliary windows to the main window according to the synchronization task. The synchronization module synchronizes the touch behavior of the main window to each auxiliary window. The screen recording is executed automatically in a loop. The user establishes an automated task through the display module. During the task establishment phase, the screen recording module obtains the touch behavior of the main window and saves it in the register. During the task execution phase, the screen recording module transfers the touch behavior saved in the register to the specified auxiliary window for execution.

[0022] The window display weight specifically includes the following steps: The processing module sets the length of the main window to L and the width to 0.8W, and the auxiliary windows are arranged in a row along the length L of the main window. The number of auxiliary windows is n, and the length of each auxiliary window is L / n and the width is 0.2W; The processing module aligns the main window with the edge of the display module along the user's focus direction, and aligns the arranged n auxiliary windows with the edge of the display module on the opposite side of the user's focus direction; When L / n≤0.1W, the auxiliary windows are arranged into two columns along the length L of the main window. After the auxiliary windows are arranged into two columns, the processing module resets the length and width of each auxiliary window. The length of each auxiliary window is L / 2n and the width is 0.1W. When the application in the auxiliary frame does not support the free change of the aspect ratio of the application interface, the processing module calculates the product of the length and width of each auxiliary window, and arranges the auxiliary windows in descending order according to the product results. The auxiliary windows are arranged from top to bottom. The auxiliary windows with the same product size are arranged again in descending order according to the number of pixel colors. The processing module initializes the magnification k, and the value of the magnification k after initialization is 1. The processing module uses the formula Calculate the magnification k of the main window, j is the number of squinting times, the processing module counts the number of squinting times j every fixed time of 1 minute, and the processing module enlarges the width of the main window and the text content in the main window according to the magnification j. When the magnification j is less than 1, the width of the main window and the text content in the main window are reduced; When the width of the main window reaches 0.9W, it will no longer be enlarged. When the text content in the main window is enlarged, the distance between the characters is reduced, and the text layout remains unchanged. When the character spacing is zero, the processing module reduces the image in the main window and expands the layout area of ​​the text content.

[0023] Among them, the focus frame rate optimization specifically includes the following steps: The processing module marks the intersection of the user's focus direction A and the display module as a focus; Determine the size of the high refresh window, and the processing module uses the formula Calculate the size of the high refresh window, xy1 and xy2 are both deviations, xy1 is the deviation of the face direction, xy2 is the deviation of the eye direction, L is the length of the display module, and W is the width of the display module; The position of the high refresh window changes synchronously with the user's attention direction A. When the attention direction A is to the left, the high refresh window expands along the left side of the display module. When the attention direction A is to the right, the high refresh window expands along the right side of the display module. The expanded high refresh window stops at the focus. The local high refresh technology of the screen cannot currently achieve a completely free floating window, and a border window is used instead of a floating window to provide a high refresh rate.

[0024] Among them, the screen monitoring specifically includes the following steps: Display information monitoring: The monitoring module presets a change cycle of 1 second, and compares the RGB values ​​of each window's display information with the display information of the previous change cycle pixel by pixel. If the RGB value of any pixel changes, it means that the screen of the auxiliary window has changed, and the interactive animation is triggered. On the contrary, if the RGB values ​​of all pixels change slightly, the comparison is repeated in a cycle. Trigger the interactive animation. The monitoring module controls the auxiliary window that triggers the interactive animation to reciprocate three times toward the main window within the change cycle. The movement distance is half of the maximum side length of the auxiliary window, so as to attract the user's attention.

[0025] The window synchronization specifically includes the following steps: The synchronization module establishes a touch coordinate system in the main window. The synchronization module records the user's touch behavior in the main window. The touch behavior includes click time and click coordinates. The synchronization module obtains the length and width of the main window and the auxiliary window connected to the main window, and the synchronization module obtains the average RGB value of the pixel in the click coordinate area of ​​the main window; The synchronization module converts the click coordinates of the main window into synchronization coordinates of the auxiliary window with the same ratio according to the length ratio and width ratio of the main window and the auxiliary window, and compares the RGB value average of the pixels in the synchronization coordinate area with the RGB value average of the pixels in the click coordinate area for verification; If the average RGB values ​​of the main window and the auxiliary window are the same, it means that the synchronization coordinate verification has passed, and the synchronization module executes the click time and synchronization coordinates in the auxiliary window. If the average RGB values ​​of the main window and the auxiliary window are different, it means that the synchronization coordinates have not passed the verification, and the synchronization module does not execute the click time and synchronization coordinates in the auxiliary window. The synchronization module outputs a pop-up window indicating that the synchronization task has failed and transmits it to the main window for display by the user.

[0026] The automatic loop execution of screen recording specifically includes the following steps: Recording touch behaviors: the screen recording module records the click coordinates of each click of the main window by the user. The screen recording module records the change of the average RGB value of the pixels in the click coordinate area between two clicks of the user. The screen recording module plots the average RGB value into an RGB curve in the order of the time axis. The screen recording module saves the click coordinates and the RGB curve corresponding to the click coordinates to the register. Click coordinate conversion: when the main window becomes the auxiliary window, the screen recording module retrieves the click coordinates from the register, calculates the length ratio and width ratio of the main window and the auxiliary window, and converts the click coordinates into recording coordinates in proportion according to the length ratio and width ratio; Execute the automated task. Each time the screen recording module executes the recording coordinates, the screen recording module converts the RGB value average of the pixels in the recording coordinate area into a recording curve in the time axis order. The screen recording module retrieves the RGB curve from the register and compares it with the recording curve. If the value changes of the two curves are the same, the screen recording module executes the next recording coordinate. On the contrary, if the value changes of the two curves are the same, it means that the interface generated by executing the recording coordinates is different from the interface generated by clicking the coordinates, and the automated task is stopped. The screen recording module outputs a pop-up window indicating that the automated task execution failed to the main window for the user to view.

[0027] The header analysis procedure specifically includes the following steps: Locate the face and facial features. A plane face model is preset inside the processing module. The processing module calculates the similarity between the pixels in the image information and the plane face model. The processing module selects the area with the highest similarity from the image information and marks it as the user's face. The processing module splits the plane face model into an eye model, a nose model and a mouth model. The processing module calculates the similarity between the pixels in the user's face and the eye model, the nose model and the mouth model respectively. The similarity calculation uses the MSE formula to calculate the pixel similarity of the two images. The processing module selects the midpoints of both sides of the area with the highest similarity of the eye model from the user's face and marks them as the left eye corner and the right eye corner. The processing module selects the diagonal intersection of the area with the highest similarity of the nose model from the user's face and marks it as the nose tip. The midpoints of both sides of the area with the highest similarity of the mouth model from the user's face are marked as the left mouth corner and the right mouth corner. Analyze the face orientation, the processing module connects the left eye corner, right eye corner, nose tip, left mouth corner and right mouth corner with line segments, and calculates the length z1 of all line segments on the left side of the nose tip and the length y1 of all line segments on the right side of the nose tip. If z1-y1<0, the processing module outputs that the face orientation is biased to the left, and the degree of left deviation xy1 is If z1-y1=0, the processing module outputs that the face is centered and the deviation degree xy1 is 0. If z1-y1>0, the processing module outputs that the face is biased to the right and the deviation degree xy1 is ; Locate the eye sockets, the processing module extracts the area between the left and right eye corners and marks it as the eye sockets, the processing module presets a circular vector graph, calculates the similarity between the circular vector graph and the eye sockets, and marks the pixels with the highest and second highest similarity in the eye sockets as pupils; Analyze the eyeball direction. The processing module counts the number of white pixels on the left side of the two pupils, z2, and the number of white pixels on the right side, y2. The processing module defines the grayscale value range of each pixel as 0-255, and marks each pixel with a grayscale value greater than 247 as a white pixel. If z2-y2<0, the processing module outputs that the eyeball direction is biased to the left, and the degree of left deviation xy2 is If z2-y2=0, the processing module outputs that the eyeball is centered and the deviation degree xy2 is 0. If z2-y2>0, the processing module outputs that the eyeball is biased to the right and the deviation degree to the right is ; Output the attention direction, and the processing module uses the formula Calculate the user's attention direction, where L is the length of the display module, W is the width of the display module, and dis is the distance between the user's face and the display module. The camera module is located directly above the display module, so the distance dis between the user's face and the display module is proportional to the proportion of the user's face in the image information. , Fidp is the number of pixels of the user's face, Picp is the number of pixels of the image information, ds is the geometric distance, the value is a constant, and the value range is generally within the range of plus or minus one inch of one foot. The specific value is determined according to the pixel parameters of the camera module. The processing module calculates the distance dis according to the ratio of the number of pixels of the user's face to the number of pixels of the image information. w1 and w2 are weight coefficients, and the specific value is determined according to the number of times the user twists his neck and moves his eyes.

[0028] The eye analysis procedure specifically includes the following steps: A plane face model is preset inside the processing module, and the processing module extracts an eye model and an eyebrow model from the plane face model. The processing module calculates the similarity between pixels in the image information and the eye model and the eyebrow model in the plane face model. The similarity calculation uses the MSE formula to calculate the pixel similarity of the two images. The processing module marks the area in the image information that is most similar to the eye model as an eye reference, and the processing module marks the area in the image information that is most similar to the eyebrow model as an eyebrow reference. There are two eye references and two eyebrow references. The processing module creates condition one. The processing module calculates the pixel ratio pix between the two eyebrow references. The calculation formula is pix=Dm / Dz, where Dm is the average of the maximum number of pixels in the horizontal direction of the two eyebrow references, and Dz is the number of pixels between the two eyebrow references. The processing module calculates the pixel ratio pix once per second. When the value of pix is ​​less than or equal to 80% of the pix of the previous second, condition one is triggered. Otherwise, the processing module continues to repeatedly calculate the pixel ratio pix. When the user cannot see the content of the display module clearly, he will squint and frown subconsciously, and the distance between the two eyebrow references will decrease. The processing module creates condition 2. The processing module calculates the aspect ratio ck of the two eyes. The calculation formula is ck=Dh / Dw, where Dh is the average value of the maximum number of pixels in the vertical direction of the two eyes, and Dw is the average value of the maximum number of pixels in the horizontal direction of the two eyes. The processing module calculates the aspect ratio ck once per second. When the value of ck is less than or equal to 75% of ck of the previous second, condition 2 is triggered. Otherwise, the processing module continues to repeatedly calculate the aspect ratio ck. When the user squints, the height of the two eyes in the vertical direction will decrease. When condition 1 and condition 2 are triggered at the same time, the processing module records the number of blinks j. When either condition 1 or condition 2 is triggered, the processing module does not record the number of blinks j.

[0029] A tablet computer split-screen intelligent control device comprises a display module, a processing module and a camera module. The display module is a capacitive touch screen, the display module is used to display a main window and a plurality of auxiliary windows, the processing module is a mobile microprocessor, the processing module is used to adjust the window display size, arrangement mode and focus frame rate, and the camera module is an image sensor, which obtains image information of the area facing the display module.

[0030] A tablet computer split-screen intelligent control storage medium includes a register, wherein the register stores monitoring tasks, synchronization tasks and automation tasks established by a user, the register stores the user's touch behavior on a display module, and the register also stores computer programs or instructions for execution by a synchronization module, a screen recording module and a monitoring module. When the computer program or instruction is executed, the method described in any one of the above embodiments is implemented.

[0031] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A tablet computer split screen intelligent control method, comprising a register, a screen recording module, a synchronization module, a monitoring module, a display module, a processing module and a camera module, characterized in that: The processing module obtains the length L and width W of the display module, the camera module obtains the image information of the user's head and transmits it to the processing module, and the display module transmits the user's touch behavior to the processing module, the screen recording module and the monitoring module in real time. The control method includes the following steps: Window display weight, the processing module presets the main window and the auxiliary window, the processing module executes the head analysis program to obtain the user's focus direction A, the deviation degree xy1 and the deviation degree xy2, the processing module automatically adjusts the arrangement of the main window and the auxiliary window according to the user's focus direction A, the arrangement method includes the window size and arrangement direction, the processing module executes the eye analysis program to obtain the number of times the user squints j, the processing module presets the magnification k, the magnification k is associated with the main window size and the font size in the main window, and the processing module automatically adjusts the magnification k according to the number of times the user squints j; Switching between the main window and the auxiliary window: when the user clicks any auxiliary window in the display module, the processing module will swap the clicked auxiliary window with the main window; Focus frame rate optimization, the processing module presets a high refresh window, at least one side edge of the high refresh window is attached to the edge of the display module, the high refresh window is an edge-attached window and overlaps with the main window or the auxiliary window, and the processing module automatically adjusts the size and position of the high refresh window according to the focus direction A; Screen monitoring: users establish monitoring tasks through the display module. The monitoring module obtains the display information of each auxiliary window in real time. The monitoring module controls each auxiliary window to display interactive animations according to the display information. Window synchronization: the user establishes a synchronization task through the display module. The synchronization module connects several auxiliary windows to the main window according to the synchronization task. The synchronization module synchronizes the touch behavior of the main window to each auxiliary window. The screen recording is executed automatically in a loop. The user establishes an automated task through the display module. During the task establishment phase, the screen recording module obtains the touch behavior of the main window and saves it in the register. During the task execution phase, the screen recording module transfers the touch behavior saved in the register to the specified auxiliary window for execution.

2. A tablet computer split screen intelligent control method according to claim 1, characterized in that: The ports of the register are connected to the ports of the synchronization module, the screen recording module and the monitoring module respectively, the ports of the synchronization module, the screen recording module and the monitoring module are all connected to the port of the display module, the port of the display module is connected to the port of the processing module, and the output end of the camera module is connected to the input end of the processing module; The window display weight specifically includes the following steps: The processing module sets the length of the main window to L and the width to 0.8W, and the auxiliary windows are arranged in a row along the length L of the main window. The number of auxiliary windows is n, and the length of each auxiliary window is L / n and the width is 0.2W; The processing module aligns the main window with the edge of the display module along the user's focus direction, and aligns the arranged n auxiliary windows with the edge of the display module on the opposite side of the user's focus direction; When L / n≤0.1W, the auxiliary windows are arranged into two columns along the length L of the main window. After the auxiliary windows are arranged into two columns, the processing module resets the length and width of each auxiliary window. The length of each auxiliary window is L / 2n and the width is 0.1W. When the application in the auxiliary frame does not support the free change of the aspect ratio of the application interface, the processing module calculates the product of the length and width of each auxiliary window, and arranges the auxiliary windows in descending order according to the product results. The auxiliary windows are arranged from top to bottom. The auxiliary windows with the same product size are arranged again in descending order according to the number of pixel colors. The processing module initializes the magnification factor k, and the processing module uses the formula Calculate the magnification k of the main window, j is the number of squinting times, the processing module counts the number of squinting times j every fixed time, and the processing module enlarges the width of the main window and the text content in the main window according to the magnification j. When the magnification j is less than 1, the width of the main window and the text content in the main window are reduced; When the width of the main window reaches 0.9W, it will no longer be enlarged. When the text content in the main window is enlarged, the distance between the characters is reduced, and the text layout remains unchanged. When the character spacing is zero, the processing module reduces the image in the main window and expands the layout area of ​​the text content.

3. The method for intelligently controlling split screen of a tablet computer according to claim 1, characterized in that: Focus frame rate optimization specifically includes the following steps: The processing module marks the intersection of the user's focus direction A and the display module as a focus; Determine the size of the high refresh window, and the processing module uses the formula Calculate the size of the high refresh window, xy1 and xy2 are both deviations, L is the length of the display module, and W is the width of the display module; The position of the high refresh window changes synchronously according to the user's focus direction A. When the focus direction A is to the left, the high refresh window expands along the left side of the display module. When the focus direction A is to the right, the high refresh window expands along the right side of the display module. The expanded high refresh window stops at the focus.

4. The method for intelligently controlling split screen of a tablet computer according to claim 1, characterized in that: Screen monitoring specifically includes the following steps: Display information monitoring: The monitoring module presets a change cycle and compares the RGB values ​​of each window's display information with the display information of the previous change cycle pixel by pixel. If the RGB value of any pixel changes, the interactive animation is triggered. Conversely, if the RGB values ​​of all pixels change slightly, the comparison is repeated in a cycle. The interactive animation is triggered, and the monitoring module controls the auxiliary window that triggers the interactive animation to reciprocate three times toward the direction close to the main window within the change cycle, and the movement distance is half of the maximum side length of the auxiliary window.

5. The method for intelligently controlling split screen of a tablet computer according to claim 1, characterized in that: Window synchronization specifically includes the following steps: The synchronization module establishes a touch coordinate system in the main window. The synchronization module records the user's touch behavior in the main window. The touch behavior includes click time and click coordinates. The synchronization module obtains the length and width of the main window and the auxiliary window, and the synchronization module obtains the average RGB value of the pixel in the click coordinate area of ​​the main window; The synchronization module converts the click coordinates of the main window into synchronization coordinates of the auxiliary window with the same ratio according to the length ratio and width ratio of the main window and the auxiliary window, and compares the RGB value average of the pixels in the synchronization coordinate area with the RGB value average of the pixels in the click coordinate area for verification; If the average RGB values ​​of the main window and the auxiliary window are the same, the synchronization coordinate verification is passed, and the synchronization module executes the click time and synchronization coordinates in the auxiliary window. If the average RGB values ​​of the main window and the auxiliary window are different, the synchronization coordinates fail to pass the verification, and the synchronization module does not execute the click time and synchronization coordinates in the auxiliary window. The synchronization module outputs a pop-up window indicating that the synchronization task has failed and transmits it to the main window for display.

6. The method for intelligently controlling split screen of a tablet computer according to claim 1, characterized in that: The automatic loop execution of screen recording includes the following steps: Recording touch behavior: the screen recording module records the click coordinates of each click of the main window by the user. The screen recording module records the change of the RGB value average of the pixels in the click coordinate area between two clicks of the user. The screen recording module plots the RGB value average into an RGB curve. The screen recording module saves the click coordinates and RGB curve to the register. Click coordinate conversion: when the main window becomes the auxiliary window, the screen recording module retrieves the click coordinates from the register, calculates the length ratio and width ratio of the main window and the auxiliary window, and converts the click coordinates into recording coordinates in proportion according to the length ratio and width ratio; Execute the automation task. Each time the screen recording module executes the recording coordinates, the screen recording module converts the RGB value average of the pixels in the recording coordinate area into a recording curve. The screen recording module retrieves the RGB curve from the register and compares it with the recording curve. If the value changes of the two curves are the same, the screen recording module executes the next recording coordinate. Otherwise, if the value changes of the two curves are the same, the automation task is stopped. The screen recording module outputs a pop-up window indicating that the automation task execution failed to be displayed in the main window.

7. The method for intelligently controlling split screen of a tablet computer according to claim 1, characterized in that: The head analysis procedure specifically includes the following steps: Locate the face and facial features. A plane face model is preset inside the processing module. The processing module calculates the similarity between the pixels in the image information and the plane face model. The processing module selects the area with the highest similarity from the image information and marks it as the user's face. The processing module splits the plane face model into an eye model, a nose model and a mouth model. The processing module calculates the similarity between the pixels in the user's face and the eye model, the nose model and the mouth model respectively. The processing module selects the midpoints of both sides of the area with the highest similarity of the eye model from the user's face and marks them as the left eye corner and the right eye corner. The processing module selects the diagonal intersection of the area with the highest similarity of the nose model from the user's face and marks it as the nose tip. The processing module selects the midpoints of both sides of the area with the highest similarity of the mouth model from the user's face and marks them as the left mouth corner and the right mouth corner. Analyze the face orientation, the processing module connects the left eye corner, right eye corner, nose tip, left mouth corner and right mouth corner with line segments, and calculates the length z1 of all line segments on the left side of the nose tip and the length y1 of all line segments on the right side of the nose tip. If z1-y1<0, the processing module outputs that the face orientation is biased to the left, and the degree of left deviation xy1 is If z1-y1=0, the processing module outputs that the face is centered and the deviation degree xy1 is 0. If z1-y1>0, the processing module outputs that the face is biased to the right and the deviation degree xy1 is ; Locate the eye sockets, the processing module extracts the area between the left and right eye corners and marks it as the eye sockets, the processing module presets a circular vector graph, calculates the similarity between the circular vector graph and the eye sockets, and marks the pixels with the highest and second highest similarity in the eye sockets as pupils; Analyze the eyeball direction, the processing module counts the number of white pixels on the left side of the two pupils z2 and the number of white pixels on the right side y2. If z2-y2<0, the processing module outputs that the eyeball direction is biased to the left, and the degree of left deviation xy2 is If z2-y2=0, the processing module outputs that the eyeball is centered and the deviation degree xy2 is 0. If z2-y2>0, the processing module outputs that the eyeball is biased to the right and the deviation degree to the right is ; Output the attention direction, and the processing module uses the formula Calculate the user's attention direction, where L is the length of the display module, W is the width of the display module, and dis is the distance between the user's face and the display module. The camera module is located directly above the display module, so the distance dis between the user's face and the display module is proportional to the proportion of the user's face in the image information. , Fidp is the number of pixels of the user's face, Picp is the number of pixels of the image information, ds is the geometric distance, the value is a constant, and the value range is generally within the range of plus or minus one inch of one foot. The specific value is determined according to the pixel parameters of the camera module. The processing module calculates the distance dis according to the ratio of the number of pixels of the user's face to the number of pixels of the image information. w1 and w2 are weight coefficients, and the specific value is determined according to the number of times the user twists his neck and moves his eyes.

8. The method for intelligently controlling split screen of a tablet computer according to claim 1, characterized in that: The eye analysis procedure specifically includes the following steps: A plane face model is preset inside the processing module, and the processing module extracts an eye model and an eyebrow model from the plane face model. The processing module calculates similarity between pixels in the image information and the eye model and the eyebrow model in the plane face model, and the processing module marks the area in the image information that is most similar to the eye model as an eye reference, and the processing module marks the area in the image information that is most similar to the eyebrow model as an eyebrow reference, and there are two eye references and two eyebrow references; The processing module creates condition one. The processing module calculates the pixel ratio pix between the two eyebrow references. The calculation formula is pix=Dm / Dz, where Dm is the average of the maximum number of pixels in the horizontal direction of the two eyebrow references, and Dz is the number of pixels between the two eyebrow references. The processing module calculates the pixel ratio pix once per second. When the value of pix is ​​less than or equal to 80% of the pix of the previous second, condition one is triggered. Otherwise, the processing module continues to repeatedly calculate the pixel ratio pix. The processing module creates condition 2. The processing module calculates the aspect ratio ck of the two eyes' references. The calculation formula is ck=Dh / Dw, where Dh is the average value of the maximum number of pixels in the vertical direction of the two eyes' references, and Dw is the average value of the maximum number of pixels in the horizontal direction of the two eyes' references. The processing module calculates the aspect ratio ck once per second. When the value of ck is less than or equal to 75% of ck of the previous second, condition 2 is triggered. Otherwise, the processing module continues to repeatedly calculate the aspect ratio ck. When condition 1 and condition 2 are triggered at the same time, the processing module records the number of blinks j. When either condition 1 or condition 2 is triggered, the processing module does not record the number of blinks j.

9. A tablet computer split screen intelligent control device, characterized in that: It includes a display module, a processing module and a camera module. The display module is a capacitive touch screen, the display module is used to display a main window and several auxiliary windows, the processing module is a mobile microprocessor, the processing module is used to adjust the window display size, arrangement and focus frame rate, and the camera module is an image sensor, which obtains image information of the area facing the display module.

10. A tablet computer split screen intelligent control storage medium, characterized in that: It includes a register, which stores monitoring tasks, synchronization tasks and automation tasks established by the user, the register stores the user's touch behavior on the display module, and the register also stores computer programs or instructions for execution by the synchronization module, the screen recording module and the monitoring module. When the computer program or instruction is executed, the method described in any one of claims 1 to 8 is implemented.

Images