Program interface anti-shake method, device, equipment and medium based on deep learning

Abstract

The present application relates to artificial intelligence technology, and provides a method, device, equipment and medium for anti-shake of a program interface based on deep learning, the method comprising: when it is determined that a mobile terminal is shaking and an application is running in the foreground of the mobile terminal, obtaining the current jitter parameters of the mobile terminal; performing initial compensation on the display interface of the application according to the jitter parameters; obtaining the compensation effect of the display interface after the initial compensation; when the compensation effect is not good, establishing a deep learning model according to the compensation effect of the initial compensation and relevant information of the jitter parameters; performing optimization compensation on the display interface after the initial compensation according to the deep learning model; after the mobile terminal is shaken, through initial compensation and optimization compensation of the display interface, the display interface is made still relative to the user, the anti-shake of the display interface of the application is more intelligent and stable, and the visual fatigue of the user caused by the shaking is reduced.

Description

Technical Field

[0001] The present application relates to an artificial intelligence technology and provides a method, device, equipment and medium for program interface anti-shake based on deep learning. Background Art

[0002] Currently, when mobile phones, tablets and other devices are used and shaken, the interface display of the application on the mobile phones, tablets and other devices shakes dynamically. In order to clearly see the content displayed on the application display interface, users need to constantly adjust their eye muscles to adapt to the constantly changing position of the display interface, which causes users to experience visual fatigue, dizziness and other discomforts. Summary of the invention

[0003] The purpose of this application is to provide a method, device, equipment and medium for program interface anti-shake based on deep learning. After the mobile terminal shakes, the display interface is compensated so that the display interface is still relative to the user, and the interface anti-shake is more intelligent and stable, reducing the user's visual fatigue caused by shaking.

[0004] The present application proposes a program interface anti-shake method based on deep learning, comprising: when it is determined that a mobile terminal is shaking and an application is running in the foreground of the mobile terminal, obtaining the current jitter parameters of the mobile terminal; performing initial compensation on the display interface of the application according to the jitter parameters; obtaining the compensation effect of the display interface after the initial compensation, wherein the compensation effect is used to characterize the degree of deviation of the display interface after the initial compensation relative to the original static display interface; when the degree of deviation of the display interface after the initial compensation relative to the original static display interface is greater than a preset deviation threshold, establishing a deep learning model according to the compensation effect of the initial compensation and relevant information of the jitter parameters; the relevant information at least includes the jitter parameters; and optimizing and compensating the display interface after the initial compensation according to the deep learning model.

[0005] Furthermore, determining that the mobile terminal is shaking includes: obtaining usage habits and usage scenarios of the user to which the mobile terminal belongs; and determining that the mobile terminal is shaking when the user is in a special state based on the usage habits and usage scenarios, wherein the special state includes that the user suffers from shaking symptoms or the user is in a shaking scenario.

[0006] Furthermore, the jitter parameters include jitter amplitude, jitter direction and jitter speed, and the initial compensation of the display interface of the application according to the jitter parameters includes: using the opposite direction of the jitter direction and the jitter amplitude as reverse compensation parameters; re-rendering the display interface according to the reverse compensation parameters, and jittering the display interface according to the jitter speed.

[0007] Further, before taking the reverse direction of the jitter direction and the jitter amplitude as reverse compensation data, it includes: determining that the jitter amplitude is smaller than a preset amplitude threshold, and the jitter frequency is smaller than a preset frequency threshold.

[0008] Furthermore, the compensation effect of obtaining the display interface after initial compensation includes: obtaining a first position of the original static display interface relative to the user; obtaining a second position of the display interface after compensation relative to the user; and taking the difference between the first position and the second position as the degree of offset of the display interface after initial compensation relative to the original static display interface.

[0009] Furthermore, the establishment of a deep learning model based on the compensation effect of the initial compensation and the related information of the jitter parameters includes: taking the compensation effect corresponding to each image frame of the display interface in the initial compensation process as a label, and taking the jitter parameters of each image frame of the display interface and the reverse compensation parameters corresponding to the jitter parameters as dimensional data; inputting multiple dimensional data and multiple labels into a deep residual network ResNet model for training to obtain the deep learning model.

[0010] Furthermore, the optimizing compensation of the display interface after the initial compensation according to the deep learning model includes: inputting the compensation effect to be achieved and the jitter parameters into the deep learning model to obtain optimized compensation parameters; and optimizing and adjusting the display interface after the initial compensation according to the optimized compensation parameters.

[0011] The present application also proposes a program interface anti-shake device based on deep learning, including: a parameter acquisition module, used to determine that a mobile terminal is shaking and an application is running in the foreground of the mobile terminal, and to obtain the current jitter parameters of the mobile terminal; an initial compensation module, used to perform initial compensation on the display interface of the application according to the jitter parameters; an effect acquisition module, used to obtain the compensation effect of the display interface after the initial compensation, wherein the compensation effect is used to characterize the degree of deviation of the display interface after the initial compensation relative to the original static display interface; a model establishment module, used to establish a deep learning model according to the compensation effect of the initial compensation and the relevant information of the jitter parameters when the degree of deviation of the display interface after the initial compensation relative to the original static display interface is greater than a preset deviation threshold; the relevant information includes at least the jitter parameters; an optimization compensation module, used to optimize the compensation of the display interface after the initial compensation according to the deep learning model.

[0012] The present application also proposes a computer device, which includes: a memory storing computer-readable instructions; and a processor reading the computer-readable instructions stored in the memory to execute the method described above.

[0013] The present application also proposes a computer-readable storage medium on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the method as described above.

[0014] Compared with the prior art, this application has the following beneficial effects:

[0015] In the technical solution provided by the present application, it is determined that a mobile terminal is jittering and an application is running in the foreground of the mobile terminal to ensure that the user is currently viewing a display interface, the current jitter parameters of the mobile terminal are obtained, and the display interface of the application is initially compensated according to the jitter parameters. When the anti-shake compensation effect is not good, that is, the offset of the compensated display interface relative to the original static display interface is greater than a preset threshold, a deep learning model is established through the compensation effect of the initial compensation and the relevant information of the jitter parameters. The compensation of the display interface is optimized based on the deep learning model. Through two compensations, it is ensured that when jitter occurs, the display interface is stationary relative to the user. In addition, the establishment of the deep learning model makes the interface anti-shake more intelligent and stable, reducing the user's visual fatigue caused by jitter. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a program interface anti-shake method based on deep learning according to an embodiment of the present application is shown;

[0017] Figure 2 Shows Figure 1 A flowchart of step S110 in the illustrated embodiment in an exemplary embodiment;

[0018] Figure 3 Shows Figure 1 A flowchart of step S120 in the illustrated embodiment in an exemplary embodiment;

[0019] Figure 4 Shows Figure 1 A flowchart of step S130 in the illustrated embodiment in an exemplary embodiment;

[0020] Figure 5 Shows Figure 1 A flowchart of step S140 in the illustrated embodiment in an exemplary embodiment;

[0021] Figure 6 A schematic diagram of the architecture of a program interface anti-shake device based on deep learning according to an embodiment of the present application is shown;

[0022] Figure 7 A schematic structural diagram of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0024] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0025] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0026] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. Deep Learning (DL) is a new research direction in the field of machine learning. The concept of deep learning originates from the study of artificial neural networks. Multilayer perceptrons with multiple hidden layers are a type of deep learning structure. Deep learning combines low-level features to form more abstract high-level representations of attribute categories or features to discover distributed feature representations of data. The motivation for studying deep learning is to establish a neural network that simulates the human brain for analysis and learning. It imitates the mechanism of the human brain to interpret data.

[0027] The solution provided in the embodiments of the present application involves deep learning technology of artificial intelligence, which is specifically illustrated by the following embodiments.

[0028] See also Figure 1 , Figure 1 1 is a flowchart of a method for anti-shaking a program interface based on deep learning according to an exemplary embodiment. The method for anti-shaking a program interface based on deep learning includes steps S110 to S150, which are described in detail as follows:

[0029] S110: When it is determined that the mobile terminal is jittering and an application is running in the foreground of the mobile terminal, a current jitter parameter of the mobile terminal is obtained.

[0030] The method proposed in this embodiment is to perform anti-shake on the interface of the application program when the mobile terminal shakes during use of the mobile terminal. Therefore, it is necessary to first determine whether the mobile terminal has shaken. Shaking means that the position of the mobile terminal has changed.

[0031] In one exemplary embodiment, Figure 2 yes Figure 1 The step S110 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 2 As shown, determining that the mobile terminal is shaking includes the following steps:

[0032] S111, obtaining usage habits and usage scenarios of a user of a mobile terminal;

[0033] It is understandable that the mobile terminal is operated by the user, and whether the mobile terminal is shaken is determined based on the user's usage habits and usage scenarios of the mobile terminal.

[0034] Acquiring the usage habits of the user of the mobile terminal may be determining the usage habits of the user according to the historical usage habits of the user during the process of the user using the mobile terminal; for example, acquiring the number of jitters of the mobile terminal when the user uses the mobile terminal within a period of time, if jitters occur every time the user uses the mobile terminal, the user's usage habit is a constant jitter operation habit; when the number of jitters is less than a preset jitter threshold, the user's usage habit is an occasional jitter;

[0035] Obtaining the usage scenario of the user belonging to the mobile terminal can be determining the user's usage scenario based on the GPS (Global Positioning System) and / or the camera; for example, capturing the environment in which the mobile terminal is located through the camera, such as a bus, train, or other environment that is prone to shaking; and determining the location of the mobile terminal through GPS. For example, when the GPS location is at sea, the mobile terminal shakes due to the shaking of the sea surface.

[0036] In one example, obtaining the usage habits and usage scenarios of the mobile terminal may also be that the user inputs his own usage habits or the current usage scenarios into the mobile terminal when using the mobile terminal.

[0037] S112: When it is determined according to the usage habits and usage scenarios that the user is in a special state, it is determined that the mobile terminal is shaking, and the special state includes that the user suffers from shaking symptoms or the user is in a shaking scenario.

[0038] In this embodiment, when the user's usage habit is to perform shaking operations all the time regardless of the usage scenario, it is determined that the user suffers from shaking symptoms and the mobile terminal shakes; when the usage scenario is a specific location such as a bus, train, etc. that is prone to shaking environments, and the usage habit is occasional shaking, it is determined that the user is in a shaking scene and the mobile terminal shakes.

[0039] In another example, determining that a mobile terminal is shaking includes the following steps: when the mobile terminal is detected to be shaking, obtaining displacement information of the mobile terminal relative to the user, and when the displacement information is greater than a preset threshold displacement, determining that the mobile terminal is shaking by a preset manner. The displacement information of the mobile terminal relative to the user includes a horizontal or vertical distance, and the displacement information can be obtained by a displacement sensor. When the displacement information is greater than a preset threshold position, it is determined that the mobile terminal is shaking, so as to avoid the situation where the shaking is generated by the mobile terminal message; optionally, when the displacement information is not regular, it can also indicate that the mobile terminal is shaking, and when the displacement information is regular, it indicates that the shaking is generated by the mobile terminal message.

[0040] In this embodiment, since the display interface of the application is subjected to anti-shake processing, it is necessary to determine whether the application is running in the foreground of the mobile terminal. When the application is running in the foreground of the mobile terminal, it means that the user is currently viewing the display interface of the application, and then the jitter parameters are obtained to facilitate the subsequent reverse compensation of the display interface. The application can be any application installed on the mobile terminal; and when the application is running in the background of the mobile terminal but not in the foreground, it indicates that the user may not be using the mobile terminal at present. Optionally, before obtaining the current jitter parameters, it is also possible to determine whether the application running in the foreground of the mobile terminal is the target application. The target application is an application that requires the user to view the display interface, such as an application that requires video, page browsing, etc.; and for the call interface, the user only needs to listen to the voice and does not need to view the interface, so there is no need to perform anti-shake processing on the application.

[0041] In this embodiment, a gyroscope is provided on the mobile terminal. When it is determined that the mobile terminal is shaking and the application is running in the foreground, the current shaking parameters of the mobile terminal are obtained through the gyroscope. The shaking parameters include: when the mobile terminal is shaking, the shaking direction, shaking amplitude and shaking frequency detected by the gyroscope.

[0042] S120 , performing initial compensation on the display interface of the application program according to the jitter parameter.

[0043] In this embodiment, the display interface of the application is initially compensated according to the jitter parameters so that the display interface appears relatively still in the eyes of the user; the compensation means that the display interface may be re-rendered or adjusted.

[0044] In one example, Figure 3 yes Figure 2 The step S120 in the embodiment shown is a flow chart in an exemplary embodiment. Figure 3 As shown, the jitter parameters include jitter direction, jitter amplitude and jitter speed, and step S120 includes:

[0045] S121. Use the reverse direction of the jitter direction and the jitter amplitude as reverse compensation parameters.

[0046] S122: re-render the display interface according to the reverse compensation parameter, and dither the display interface according to the dithering speed.

[0047] The jitter amplitude is the displacement of the mobile terminal jitter; for example, the jitter direction and the jitter amplitude are an upward displacement of one unit length, and the reverse compensation parameter is a downward displacement of one unit length; when other display parameters of the display interface remain unchanged, the display position parameter of the display interface is adjusted to a downward displacement of one unit, and then the display interface is re-rendered based on the other display parameters and the display position parameters, and the rendered display interface is jittered according to the jitter frequency.

[0048] Exemplarily, when the display interface is re-rendered according to the reverse compensation parameters, the display interface can be rendered in 3D through a rendering program; wherein 3D rendering is to present the model from the 3D model mesh to a 2D image with high realism through calculation, and the calculation process includes comprehensive variables such as light and auxiliary light, material material and texture, and related settings; the rendering program includes the rendering engine that comes with CG (computer graphics) software, For 3DsMax (V-Ray For 3DsMax is a super renderer of 3DsMax, which is a renderer designed by a professional rendering engine company Chaos Software with Raytracing and Global Illumination), etc.

[0049] In order to ensure the reliability of initial compensation for display, before using the shaking direction, reverse direction and shaking amplitude as reverse compensation parameters, this embodiment further includes: determining that the shaking amplitude is less than a preset amplitude threshold and the shaking speed is less than a preset frequency threshold.

[0050] It is understandable that when the jitter amplitude is large, the displacement of the display interface relative to the user is large, and the impact on the user is greater. When the jitter amplitude is small, the display interface may not change for the user; but if the jitter speed is large, even if the jitter amplitude is large and the display interface is rendered and jittered at a high speed, the display interface will be blurred for the user; therefore, when the jitter amplitude is large and the jitter speed is small, the display interface is adjusted through subsequent reverse compensation parameters.

[0051] S130: Obtain a compensation effect of the display interface after initial compensation, wherein the compensation effect is used to characterize a degree of deviation of the display interface after initial compensation relative to the original static display interface.

[0052] In this embodiment, after the initial compensation of the display interface, there will be a final compensation effect of the display interface. The quality of the compensation effect is characterized by the degree of offset of the display interface after the initial compensation relative to the original static display interface; the smaller the offset, the better the compensation effect.

[0053] Since the original static display interface is absolutely static for the user, that is, the offset between the original static display interface and the user infinitely tends to 0, the compensation effect of the compensated display interface can be determined by the change in the positional relationship between the compensated display interface and the user; Figure 4 yes Figure 1 The step S130 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 4 As shown, specifically including:

[0054] S131, obtaining a first position of an original static display interface relative to a user;

[0055] S132, obtaining a second position of the display interface relative to the user after compensation;

[0056] S133: Taking the difference between the first position and the second position as the offset degree of the display interface after initial compensation relative to the original static display interface.

[0057] In this embodiment, the original static display interface refers to the display interface when the mobile terminal is not shaking. Since the offset between the original static display interface and the user tends to 0 infinitely, at a certain moment, the first position of the original static display interface relative to the user's eyes can be obtained by calling the front camera or infrared sensor of the mobile terminal; the second position of the compensated display interface relative to the user's eyes can also be obtained through the front camera or infrared sensor; when the first position and the second position are different, it means that the compensated display interface and the user are not relatively static, and a relative position is generated, so the difference between the first position and the second position is used as the degree of offset.

[0058] S140. When the degree of deviation of the display interface after initial compensation relative to the original static display interface is greater than a preset deviation threshold, a deep learning model is established according to the compensation effect of the initial compensation and relevant information of the jitter parameters; the relevant information includes at least the jitter parameters.

[0059] In this embodiment, the display interface is first initially reverse compensated by the jitter parameters to obtain the compensation effect of the display interface after the initial compensation, and based on the compensation effect, it is determined whether to perform compensation optimization. When the offset of the display interface after the initial compensation relative to the original static display interface is greater than a preset offset threshold, it indicates that the compensation effect is not good. A deep learning model is established through the compensation effect of the initial compensation, the jitter parameters and the related information of the jitter parameters. The relationship between the compensation effect of the initial compensation, the jitter parameters and the related information of the jitter parameters is learned through the deep learning model, so as to optimize and compensate the display interface.

[0060] The jitter parameter related information includes not only the jitter parameters, but also the reverse compensation parameters corresponding to the jitter parameters. Figure 5 yes Figure 1 The step S140 in the illustrated embodiment is a flow chart in an exemplary embodiment. Figure 5 As shown in Figure 1, the process of building a deep learning model includes:

[0061] S141, using the compensation effect corresponding to each image frame of the display interface in the initial compensation process as a label, and using the jitter parameter of each image frame of the display interface and the reverse compensation parameter corresponding to the jitter parameter as dimensional data;

[0062] In this embodiment, during the initial compensation of the display interface, each image frame of the display interface may be rendered, and the compensated display interface may be obtained through the rendering changes of multiple consecutive frames; therefore, during the initial process, the compensation effect of each image frame of the display interface after rendering may be recorded, and the corresponding compensation effect of each image frame of the display interface during the initial compensation process may be used as a label, and the jitter direction and jitter speed of each image frame of the display interface, as well as the speed and direction of the reverse compensation of the current frame may be used as dimensional data. In an exemplary embodiment, an effect video of the initial compensation of the display interface may be obtained, such as by shooting a video through an auxiliary device, and then obtaining the video from the auxiliary device, and then separating the video from the video to obtain the compensation effect of each image frame of the display interface after rendering.

[0063] S142. Input multiple dimensional data and multiple labels into the deep residual network ResNet model for training to obtain a deep learning model.

[0064] The compensation effect, jitter direction and jitter speed of image frames of multiple display interfaces, as well as the speed and direction of reverse compensation are input into the deep residual network ResNet model for training to improve the accuracy of the model; the deep residual network ResNet model can solve the disadvantage of large errors of deep neural networks; specifically, the ResNet18 model includes 17 convolutional layers (conv) + 1 fully connected layer (fc), and the dimensional data and labels are output through 17 convolutional layers, and each dimensional data and label corresponds to the feature data respectively. The relationship between the feature data of each dimensional data and label is learned through the fully connected layer, and then the trained deep learning model can determine the corresponding relationship between the compensation effect, jitter parameters, and the corresponding reverse compensation parameters.

[0065] S150: Optimize and compensate the display interface after initial compensation according to the deep learning model.

[0066] In this embodiment, a deep learning model is used to optimize the display interface after the initial compensation, so that the compensation effect is more intelligent and stable.

[0067] In an exemplary embodiment, optimizing and compensating the display interface after initial compensation according to the deep learning model includes:

[0068] Input the expected compensation effect and jitter parameters into the deep learning model to obtain the optimized compensation parameters;

[0069] The display interface after initial compensation is optimized and adjusted according to the optimization compensation parameters.

[0070] The expected compensation effect is that the display interface is absolutely still relative to the user, that is, the offset of the display interface relative to the original still display interface is 0; the expected compensation effect and jitter parameters are passed through the deep learning model, and the optimized compensation parameters are output. For example, when jitter occurs, the display interface is offset to the left by 7 units. After the initial compensation, the display interface is offset to the left by 4 units, which is greater than the preset offset threshold of 3 units. A deep learning model is established based on the compensation effect, jitter parameters and reverse jitter parameters of the initial compensation. If the deep learning model determines that the best reverse compensation parameter is 3 units, then based on the display interface after the initial compensation, the display interface is optimized and adjusted according to the displacement of 3 units, so that the final offset of the display interface relative to the user is 0, which can make the compensation effect more intelligent, more stable and more humane.

[0071] In an exemplary embodiment, after obtaining the deep learning model, the mobile terminal may be provided with a corresponding anti-shake mode, specifically including a primary anti-shake mode and a secondary anti-shake mode. The primary anti-shake mode is to initially compensate the display interface through the jitter parameters, and the secondary anti-shake mode is based on the primary anti-shake mode. When the anti-shake effect compensation effect is not good, the display interface is optimized and adjusted by optimizing the compensation parameters based on the optimized compensation parameters predicted by the constructed deep learning model on the basis of the compensated display interface. The user can click to trigger the corresponding command, and then choose to turn on the primary anti-shake mode or the secondary anti-shake mode to meet the different needs of users.

[0072] The following describes an apparatus embodiment of the present application, which can be used to execute the program interface anti-shake method based on deep learning in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the program interface anti-shake method based on deep learning in the present application.

[0073] like Figure 6 As shown, Figure 6 A block diagram of a program interface anti-shake device based on deep learning according to an embodiment of the present application is shown, and the device includes:

[0074] The parameter acquisition module 610 is used to determine that the mobile terminal is jittering and an application is running in the foreground of the mobile terminal, and to acquire the current jitter parameters of the mobile terminal;

[0075] An initial compensation module 620, used to perform initial compensation on the display interface of the application according to the jitter parameter;

[0076] An effect acquisition module 630, used to acquire a compensation effect of the display interface after initial compensation, wherein the compensation effect is used to characterize the degree of deviation of the display interface after initial compensation relative to the original static display interface;

[0077] A model building module 640 is used to build a deep learning model according to the compensation effect of the initial compensation and relevant information of the jitter parameters when the offset degree of the display interface after the initial compensation relative to the original static display interface is greater than a preset offset threshold; the relevant information includes at least the jitter parameters;

[0078] The optimization compensation module 650 is used to optimize and compensate the display interface after initial compensation according to the deep learning model.

[0079] The parameter acquisition module 610 is used to obtain the usage habits and usage scenarios of the user of the mobile terminal; when it is determined that the user is in a special state based on the usage habits and usage scenarios, it is determined that the mobile terminal is shaking, and the special state includes that the user suffers from shaking symptoms or the user is in a shaking scenario.

[0080] The initial compensation module 620 is used to use the reverse direction of the shaking direction and the shaking amplitude as reverse compensation parameters; re-render the display interface according to the reverse compensation parameters, and shake the display interface according to the shaking speed.

[0081] The initial compensation module 620 is used to determine whether the jitter amplitude is smaller than a preset amplitude threshold and the jitter speed is smaller than a preset speed threshold before using the reverse direction of the jitter direction and the jitter amplitude as reverse compensation data.

[0082] The effect acquisition module 630 is used to obtain the first position of the original static display interface relative to the user; obtain the second position of the compensated display interface relative to the user; and use the difference between the first position and the second position as the offset degree of the initial compensated display interface relative to the original static display interface.

[0083] The model building module 640 is used to use the compensation effect corresponding to each image frame of the display interface in the initial compensation process as a label, and use the jitter parameters of each image frame of the display interface and the reverse compensation parameters corresponding to the jitter parameters as dimensional data; multiple dimensional data and multiple labels are input into the deep residual network ResNet model for training to obtain a deep learning model.

[0084] The optimization compensation module 650 is used to input the expected compensation effect and jitter parameters into the deep learning model to obtain the optimized compensation parameters; and optimize and adjust the display interface after the initial compensation according to the optimized compensation parameters.

[0085] In this embodiment, by performing anti-shake processing on the display interface of the application of the mobile terminal, the intelligent development of the software program is promoted. At the same time, the application can be normally accessed, viewed, and used by specific groups of people and in specific scenarios, thereby reducing the user's visual fatigue or cognitive errors caused by shaking, and has a significant improvement in particular for some patients with hand tremors who have trouble using mobile phones and other devices.

[0086] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0087] In an exemplary embodiment, a computer device includes a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above method is implemented.

[0088] Figure 7 The diagram is a schematic structural diagram of a computer device according to an exemplary embodiment.

[0089] It should be noted that the computer device is only an example adapted for the present application and cannot be considered to provide any limitation on the scope of use of the present application. The computer device cannot be interpreted as requiring dependence on or having Figure 7 One or more components of the exemplary computer device shown in .

[0090] like Figure 7 As shown, in an exemplary embodiment, the computer device includes a processing component 701, a memory 702, a power component 703, a multimedia component 704, an audio component 705, a sensor component 707, and a communication component 708. Among them, the above components are not all necessary, and the computer device can add other components or reduce certain components according to its own functional requirements, which is not limited in this embodiment.

[0091] The processing component 701 generally controls the overall operation of the computer device, such as operations associated with display, data communication, and log data processing. The processing component 701 may include one or more processors 709 to execute instructions to complete all or part of the steps of the above operations. In addition, the processing component 701 may include one or more modules to facilitate the interaction between the processing component 701 and other components. For example, the processing component 701 may include a multimedia module to facilitate the interaction between the multimedia component 704 and the processing component 701.

[0092] The memory 702 is configured to store various types of data to support operations on the computer device, examples of which include instructions for any application or method for operating on the computer device. The memory 702 stores one or more modules, which are configured to be executed by the one or more processors 709 to complete all or part of the steps in the method described in the above embodiment.

[0093] The power supply component 703 provides power to various components of the computer device. The power supply component 703 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the computer device.

[0094] The multimedia component 704 includes a screen that provides an output interface between the computer device and the user. In some embodiments, the screen may include a TP (Touch Panel) and an LCD (Liquid Crystal Display). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0095] The audio component 705 is configured to output and / or input audio signals. For example, the audio component 705 includes a microphone, and when the computer device is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive external audio signals. In some embodiments, the audio component 705 also includes a speaker for outputting audio signals.

[0096] The sensor component 707 includes one or more sensors for providing status evaluation of various aspects for the computer device. For example, the sensor component 707 can detect the on / off status of the computer device and can also detect the temperature change of the computer device.

[0097] The communication component 708 is configured to facilitate wired or wireless communication between the computer device and other devices. The computer device can access a wireless network based on a communication standard, such as Wi-Fi (Wireless-Fidelity).

[0098] Understandably, Figure 7 The structure shown is for illustration only. The computer device may include Figure 7 More or fewer components as shown, or with Figure 7 The different components are shown. Figure 7 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0099] In an exemplary embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0100] It should be noted that the computer-readable storage medium shown in the embodiments of the present application may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0101] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0102] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A program interface anti-shake method based on deep learning, characterized in that: include: Obtain usage habits and usage scenarios of users of mobile terminals; When it is determined according to the usage habits and usage scenarios that the user is in a special state, determining that the mobile terminal is shaking, wherein the special state includes that the user suffers from a shaking symptom or the user is in a shaking scenario; Determine that an application is running in the foreground of the mobile terminal and the application is a target application, and obtain current jitter parameters of the mobile terminal, wherein the target application is an application that requires the user to view a display interface, and the jitter parameters include jitter amplitude, jitter direction, and jitter speed; Using the reverse direction of the shaking direction and the shaking amplitude as reverse compensation parameters; Re-rendering the display interface according to the reverse compensation parameter, and shaking the display interface at a shaking speed; Acquire a compensation effect of the display interface after initial compensation, wherein the compensation effect is used to characterize a degree of deviation of the display interface after initial compensation relative to the original static display interface; When the offset degree of the display interface after the initial compensation relative to the original static display interface is greater than a preset offset threshold, the compensation effect corresponding to the image frame of each display interface in the initial compensation process is used as a label, and the jitter parameter of the image frame of each display interface and the reverse compensation parameter corresponding to the jitter parameter are used as dimensional data; Input multiple dimensional data and multiple labels into the deep residual network ResNet model for training to obtain a deep learning model; The display interface after initial compensation is optimized and compensated according to the deep learning model.

2. The method according to claim 1, characterized in that The method of using the reverse direction of the jitter direction and the jitter amplitude as reverse compensation data comprises: It is determined that the shaking amplitude is less than a preset amplitude threshold, and the shaking speed is less than a preset speed threshold.

3. The method according to claim 1, characterized in that The compensation effect of the display interface after obtaining the initial compensation includes: Obtaining a first position of the original static display interface relative to the user; Obtaining a second position of the display interface relative to the user after compensation; The difference between the first position and the second position is used as the offset degree of the display interface after initial compensation relative to the original static display interface.

4. The method according to claim 1, characterized in that: The optimizing and compensating the display interface after the initial compensation according to the deep learning model comprises: Inputting the compensation effect to be achieved and the jitter parameters into the deep learning model to obtain optimized compensation parameters; The display interface after the initial compensation is optimized and adjusted according to the optimization compensation parameters.

5. A program interface anti-shake device based on deep learning, characterized in that: include: Parameter acquisition module, used for: Obtain usage habits and usage scenarios of users of mobile terminals; When it is determined according to the usage habits and usage scenarios that the user is in a special state, determining that the mobile terminal is shaking, wherein the special state includes that the user suffers from a shaking symptom or the user is in a shaking scenario; Determine that an application is running in the foreground of the mobile terminal and the application is a target application, and obtain current jitter parameters of the mobile terminal, wherein the target application is an application that requires the user to view a display interface, and the jitter parameters include jitter amplitude, jitter direction, and jitter speed; Initial compensation module for: Using the reverse direction of the shaking direction and the shaking amplitude as reverse compensation parameters; Re-rendering the display interface according to the reverse compensation parameter, and shaking the display interface at a shaking speed; An effect acquisition module, used to acquire a compensation effect of the display interface after initial compensation, wherein the compensation effect is used to characterize the degree of deviation of the display interface after initial compensation relative to the original static display interface; Model building modules for: When the offset degree of the display interface after the initial compensation relative to the original static display interface is greater than a preset offset threshold, the compensation effect corresponding to the image frame of each display interface in the initial compensation process is used as a label, and the jitter parameter of the image frame of each display interface and the reverse compensation parameter corresponding to the jitter parameter are used as dimensional data; Input multiple dimensional data and multiple labels into the deep residual network ResNet model for training to obtain a deep learning model; The optimization compensation module is used to optimize and compensate the display interface after initial compensation according to the deep learning model.

6. A computer device, characterized in that: The computer device comprises: a memory storing computer-readable instructions; A processor reads the computer-readable instructions stored in the memory to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 4.

Images