A software display method and system based on artificial intelligence

Abstract

The present invention provides an artificial intelligence-based software display method and system, which relates to the field of software display technology. The method comprises obtaining a user's facial gaze video and a screen recording interface image from a front camera; determining multiple blank areas of the screen using a blank area determination model based on the screen recording interface image; determining the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue using a facial video processing model based on the user's facial gaze video from the front camera; determining a target display position of the software based on the multiple blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; and displaying the software based on the target display position of the software. The method can accurately determine a reasonable display position of the software.

Description

Technical Field

[0001] The present invention relates to the technical field of software display, and in particular to a software display method and system based on artificial intelligence. Background Art

[0002] In the field of software development, the display position of software content is crucial to the user experience. Traditional software display methods usually rely on a fixed display position, which may cause users to feel uncomfortable during use, especially after long-term use, the user's eyes may feel tired. In addition, different users may have different usage habits and preferences, and a fixed display position may not meet the needs of all users. Existing solutions often lack consideration of the user's real-time status, such as the user's facial orientation, the distance between the eyes and the screen, and the degree of eye fatigue, all of which will affect the user's comfort with the software display position. Therefore, how to accurately determine the reasonable display position of the software is an urgent problem to be solved. Summary of the invention

[0003] The main technical problem solved by the present invention is how to accurately determine the reasonable display position of the software.

[0004] According to a first aspect, the present invention provides a software display method based on artificial intelligence, comprising: when detecting that a user performs an operation of opening a software page, opening the front camera and the screen recording operation; obtaining a user's facial video and a screen recording interface image from the front camera; determining multiple blank areas of the screen using a blank area determination model based on the screen recording interface image; determining the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue using a facial video processing model based on the user's facial video from the front camera; determining the target display position of the software based on the multiple blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; and displaying the software based on the target display position of the software.

[0005] In a possible implementation, the target display position of the software is determined based on the multiple blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue, including: constructing a knowledge graph, the knowledge graph includes multiple nodes and edges between nodes, each node represents a blank area of ​​the screen, the node features of each node include a blank area position information of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue, and the edges between nodes represent the positional relationship between the blank areas; processing the knowledge graph based on a graph neural network model to determine the software display area area, software display brightness, and a target area for software display; a plurality of preliminary target software display positions are generated based on the software display area, the software display brightness, and the target area for software display by using a variational autoencoder, and each preliminary target software display position is located within the target area for software display; a support vector machine is used to process and determine the comfort level of each preliminary target software display position based on each preliminary target software display position, the software display brightness, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; and the preliminary target software display position with the highest comfort level is used as the target display position of the software.

[0006] In a possible implementation, the facial video processing model is a gated recurrent unit, the input of the facial video processing model is the user's facial video of the front camera, and the output of the video processing model is the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue.

[0007] In a possible implementation, the input of the variational autoencoder is the software display area, the software display brightness, and the target area of ​​the software display, and the output of the variational autoencoder is a plurality of preliminarily selected target software display positions.

[0008] According to a second aspect, the present invention provides a software display system based on artificial intelligence, comprising: The detection module is used to detect that the user opens the software page, and then turns on the front camera and screen recording; An acquisition module is used to acquire the user's gaze facial video and screen recording interface image from the front camera; A blank area determination module, configured to determine a plurality of blank areas of the screen using a blank area determination model based on the screen recording interface image; A facial video processing module, used for determining the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue using a facial video processing model based on the user's gaze facial video captured by the front camera; A target display position determination module, used to determine the target display position of the software based on a plurality of blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; The display module is used to display the software based on the target display position of the software.

[0009] In a possible implementation, the target display position determination module is also used to: construct a knowledge graph, the knowledge graph includes multiple nodes and edges between nodes, each node represents a blank area of ​​the screen, the node features of each node include the position information of a blank area of ​​the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue, and the edges between nodes represent the positional relationship between the blank areas; based on the graph neural network model, the knowledge graph is processed to determine the software display area, the software display brightness, and the target area of ​​the software display; based on the software display area, the software display brightness, and the target area of ​​the software display, a plurality of preliminary target software display positions are generated using a variational autoencoder, and each preliminary target software display position is located within the target area of ​​the software display; based on each preliminary target software display position, the software display brightness, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue, a support vector machine is used to process and determine the comfort of each preliminary target software display position; and the preliminary target software display position with the highest comfort is used as the target display position of the software.

[0010] In a possible implementation, the facial video processing model is a gated recurrent unit, the input of the facial video processing model is the user's facial video of the front camera, and the output of the video processing model is the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue.

[0011] In a possible implementation, the input of the variational autoencoder is the software display area, the software display brightness, and the target area of ​​the software display, and the output of the variational autoencoder is a plurality of preliminarily selected target software display positions.

[0012] According to a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor; a memory; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method as described above, the method comprising: when detecting that a user performs a software page opening operation, turning on the front camera and the screen recording operation; obtaining the user's facial video and the screen recording interface image from the front camera; using a blank area determination model to determine multiple blank areas of the screen based on the screen recording interface image; using a facial video processing model to determine the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue based on the user's facial video from the front camera; determining the target display position of the software based on the multiple blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; and displaying the software based on the target display position of the software.

[0013] According to the fourth aspect, the present embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned artificial intelligence-based software display method, the method comprising: when detecting that a user performs a software page opening operation, opening the front camera and the screen recording operation; obtaining the user's facial video and the screen recording interface image from the front camera; using a blank area determination model to determine multiple blank areas of the screen based on the screen recording interface image; using a facial video processing model to determine the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue based on the user's facial video from the front camera; determining the target display position of the software based on the multiple blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; and displaying the software based on the target display position of the software.

[0014] The present invention provides an artificial intelligence-based software display method and system, which includes: when detecting that a user performs a software page opening operation, opening a front camera and a screen recording operation; obtaining a user's facial video and a screen recording interface image from the front camera; using a blank area determination model to determine multiple blank areas of the screen based on the screen recording interface image; using a facial video processing model to determine the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue based on the user's facial video from the front camera; determining a target display position of the software based on the multiple blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; and displaying the software based on the target display position of the software. The method can accurately determine a reasonable display position of the software. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1A schematic diagram of an application scenario of an artificial intelligence-based software display method provided by an embodiment of the present invention; Figure 2 A flowchart of a software display method based on artificial intelligence provided by an embodiment of the present invention; Figure 3 A schematic diagram of a process for determining a target display position of software provided by an embodiment of the present invention; Figure 4 A schematic diagram of an artificial intelligence-based software display system provided by an embodiment of the present invention; Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present invention; Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present invention better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present invention are not shown or described in the specification, this is to avoid the core part of the present invention being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0017] Figure 1 A schematic diagram of an application scenario of an artificial intelligence-based software display method provided in an embodiment of the present invention. Figure 1 The application scenario of the artificial intelligence-based software display method may include a server 11, a network 12, a terminal 13 and a storage device 14.

[0018] In some embodiments, the server 11 may be a single server or a server group. The server 11 may access information and / or data stored in the terminal 13 or the storage device 14 through the network 12. In some embodiments, the server 11 may be used to execute Figure 2 The artificial intelligence-based software display method shown in .

[0019] The network 12 may facilitate the exchange of information and / or data. In some embodiments, the network 12 may be any form of wired or wireless network, or any combination thereof.

[0020] Terminal 13 may refer to one or more terminal devices used by a user. In some embodiments, terminal 13 may include one or more combinations of a mobile device, a tablet computer, a laptop computer, and the like.

[0021] The storage device 14 may store data and / or instructions. For example, the storage device 14 may store data instructions of a software display method based on artificial intelligence.

[0022] In an embodiment of the present invention, there is provided Figure 2 An artificial intelligence-based software display method is shown, and the artificial intelligence-based software display method includes steps S1 to S6: Step S1, when it is detected that the user opens the software page, the front camera and the screen recording operation are turned on; In some embodiments, when a user triggers the opening operation of a software page by clicking, touching or other means, the system will detect the user's behavior, which can be achieved through the API of the operating system and the trigger inside the software. The software page opening operation includes clicking and touching.

[0023] The front camera is used to capture the user's facial information, which includes the user's gaze direction and eye state. The screen recording operation records the current screen display content for subsequent analysis of the blank area on the screen. The front camera and screen recording are started through the interface provided by the operating system.

[0024] Step S2, obtaining the user's facial video and screen recording interface image from the front camera; The user gaze facial video is a video of the user's face that is continuously recorded by the front camera. By analyzing the video frames in the facial video, data such as the user's gaze direction, the distance between the eyes and the screen, and the degree of eye fatigue can be determined.

[0025] The screen recording interface image is a static image or video frame of the current screen captured by the screen recording function. The screen recording interface image records all the contents currently displayed on the screen, including UI layout, text, icons, images and other information. The screen recording interface image provides complete visual information of the screen. By analyzing the screen recording interface image, the position and size of the blank area can be determined.

[0026] Step S3, determining a plurality of blank areas of the screen using a blank area determination model based on the screen recording interface image; The blank area is the area that is not occupied when the currently opened software is displayed on the screen. By placing the software content in the blank area, you can avoid conflicts with the text, icons or images already on the screen, ensuring that users can clearly see all the content.

[0027] The blank area determination model is a convolutional neural network model (CNN). The input of the blank area determination model is the screen recording interface image, and the output of the blank area determination model is the position and size information of multiple blank areas on the screen.

[0028] Convolutional neural networks (CNNs) can identify unoccupied blank areas on the screen by analyzing screen recording interface images, detecting text, icons and other content. Blank areas usually have features such as uniform color or texture. Convolutional neural networks can accurately distinguish between areas with content and blank areas by learning these features, and can provide key information for subsequent display optimization, ensuring that the adjusted content layout is reasonable and does not block important information. Convolutional neural networks are a way to implement artificial intelligence.

[0029] Step S4, determining the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue using a facial video processing model based on the user's gaze facial video captured by the front camera; The user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue are key visual state reference data and can be used to optimize the content display position of the software.

[0030] The user's facial orientation is the direction of the user's head relative to the screen, and the user's facial orientation can show the user's current visual focus area.

[0031] The distance between the user's eyes and the screen is the straight-line distance between the user's eyes and the screen. The distance between the user's eyes and the screen may be used for processing later to adjust the size and brightness of displayed content.

[0032] User eye fatigue reflects the degree of user eye fatigue. The facial video processing model can be used to determine whether the user is fatigued by analyzing features such as blinking frequency and pupil size.

[0033] The facial video processing model is a gated recurrent unit, the input of the facial video processing model is the user's gaze facial video of the front camera, and the output of the video processing model is the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue.

[0034] The Gated Recurrent Unit (GRU) is used to process sequence data and time series information. The GRU has a simple structure and high computational efficiency, and is suitable for processing continuous time series data such as video and speech. The GRU consists of three components: a memory unit, an update gate, and a reset gate. Through the gating mechanism, the GRU can better capture the feature information in the user's facial gaze video sequence data, so that it can more accurately determine important information such as the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue.

[0035] In some embodiments, the gated loop unit includes a facial video processing layer, an eye feature extraction layer, and a fatigue analysis layer. The input of the facial video processing layer is the user's facial gaze video from the front camera, and the output of the facial video processing layer is the user's eye segmentation video, the user's facial orientation, and the distance between the user's eyes and the screen. The input of the eye feature extraction layer is the user's eye segmentation video, and the output of the eye feature extraction layer is the user's eye movement speed, the frequency of gaze point changes, and the blinking frequency. The input of the fatigue analysis layer is the user's eye movement speed, the frequency of gaze point changes, the blinking frequency, the user's facial orientation, and the distance between the user's eyes and the screen, and the output of the fatigue analysis layer is the user's eye fatigue.

[0036] Eye movement velocity refers to how quickly the eyes move from one location to another in space. It can be calculated by dividing the distance the eyes move by the time. Higher eye movement velocities may be associated with greater alertness and attention.

[0037] The frequency of gaze point changes refers to the frequency at which the user's gaze point changes per unit time. By tracking the changes in eye gaze points, the number or frequency of gaze point switches can be calculated. A higher frequency of gaze point changes may be associated with higher attention and task switching.

[0038] Blink frequency refers to the number of times the eye blinks per unit time. By detecting changes in eye movement, the frequency of blinks can be calculated. An increase in blink frequency may be associated with an increase in fatigue.

[0039] Eye movement speed, frequency of gaze point changes, and blinking frequency can be used to analyze the user's eye fatigue.

[0040] The facial video processing layer is responsible for extracting key features from the user's facial gaze video. These features are the basic information of the user's visual state. Through feature extraction, the video information can be converted into more concise and useful data for subsequent analysis. The eye feature extraction layer extracts the dynamic features of the eye, such as eye movement speed, frequency of gaze point changes, and blinking frequency, by analyzing the user's eye segmentation video. The dynamic features of the eye are important indicators for judging the user's eye state, which can reflect the user's concentration and visual fatigue. The fatigue analysis layer combines the dynamic features of the eye, facial orientation, and the distance between the eye and the screen to calculate the user's eye fatigue. This layered design not only improves the accuracy and efficiency of the system, but also enhances the interpretability of the model, making the monitoring of the user's visual state and fatigue analysis more intelligent and refined.

[0041] By breaking down the task into multiple steps and refining each step based on the results of the previous step, the accuracy of the information can be gradually improved. This layered processing helps to enhance the accuracy of the final output data because each layer can focus on optimizing the accuracy of its specific task, thereby improving the stability and reliability of the entire model.

[0042] Step S5, determining a target display position of the software based on a plurality of blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; In some embodiments, Figure 4 A schematic diagram of a process for determining a target display position of software provided by an example of the present invention, wherein the process for determining the target display position of software comprises steps S21 to S25: Step S21, constructing a knowledge graph, the knowledge graph includes a plurality of nodes and edges between the nodes, each node represents a blank area of ​​the screen, the node features of each node include the position information of a blank area of ​​the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue, and the edges between the nodes represent the positional relationship between the blank areas; Knowledge graph is a structured data representation method, which consists of nodes (vertices) and edges (edges) to represent the relationship between nodes. Each node represents a blank area on the screen. The characteristics of the node include the location information of the blank area, the user's facial orientation, the distance between the eyes and the screen, and the degree of eye fatigue. These features can be used to calculate the user's current visual state and screen layout. Edges are used to connect different nodes to represent the positional relationship between blank areas. By constructing this knowledge graph, the blank area of ​​the screen can be combined with the user's visual state to form a structured data representation, which is convenient for subsequent analysis and processing using graph neural networks (GNNs).

[0043] Step S22, processing the knowledge graph based on the graph neural network model to determine the software display area, software display brightness, and target area of ​​software display; The software display area is calculated by the graph neural network model by analyzing the location information of the blank area in the knowledge graph and the distance between the user's eyes and the screen. For example, if the user's eyes are far away from the screen, the graph neural network model can increase the display area during processing to ensure that the content is clearly visible. If the user's eye fatigue is high, the graph neural network model can reduce the display area during processing to reduce the visual burden.

[0044] Software display brightness is the appropriate software display brightness output by the TuShen network model.

[0045] The target area displayed by the software is the target area that the Graph Neural Network model determines best by analyzing the positional relationship of the blank areas in the knowledge graph and the user's facial orientation. For example, if the user's head is tilted to the left and the gaze is focused on the left side of the screen, the Graph Neural Network (GNN) will adjust the target area displayed by the software to the blank area on the left to ensure that the user can easily view it.

[0046] The input of the graph neural network model is the knowledge graph, and the output of the graph neural network model is the software display area, software display brightness, and the target area of ​​the software display.

[0047] Graph Neural Network (GNN) is a deep learning model specifically designed for processing graph structure data. By transmitting and aggregating information between nodes and edges in the knowledge graph, Graph Neural Network (GNN) can comprehensively consider the relationship between various blank areas and the user's visual state, such as facial orientation and fatigue, from the perspective of the overall graph structure, thereby determining the optimal display area, brightness, and target area of ​​the software. Through the processing of the graph neural network, the characteristics of the nodes and edges in the knowledge graph can be fully utilized to generate the optimal display parameters, which not only improves the adaptability of the software, but also significantly improves the user's visual comfort and usage experience.

[0048] Step S23, using a variational autoencoder to generate a plurality of preliminary target software display positions based on the software display area, the software display brightness, and the target area of ​​the software display, each of which is located within the target area of ​​the software display; The multiple preliminary target software display positions are a set of candidate positions generated by the variational autoencoder, each of which is located within the target area of ​​the software display and meets the constraints of display area and brightness. The preliminary target software display position is generated based on the user's visual state and the position of the blank area. By generating multiple candidate positions, they can be used for further screening and optimization in subsequent steps to select the target position that best suits the user's current state.

[0049] The input of the variational autoencoder is the software display area, the software display brightness, and the target area of ​​the software display, and the output of the variational autoencoder is a plurality of preliminarily selected target software display positions.

[0050] Variational Autoencoder (VAE) is a generative model that combines the features of autoencoders and probabilistic graphical models. By learning the latent distribution of data, the variational autoencoder can generate new samples similar to the input data. The variational autoencoder consists of two parts: the encoder and the decoder. The encoder maps the input data to a latent space and learns the latent distribution of the data. The decoder samples points from the latent space and generates new samples similar to the input data. By introducing variational inference, the variational autoencoder can generate diverse samples while maintaining the structure and characteristics of the data. This makes the variational autoencoder particularly suitable for generation tasks such as image generation, text generation, etc.

[0051] In some embodiments, the variational autoencoder uses the constraints of software display area, brightness and target area to generate multiple candidate positions within the target area, each of which meets the display parameter requirements. The generation capability of the variational autoencoder can provide a variety of candidate solutions, providing a basis for subsequent screening and optimization, ensuring that the generated display position not only meets user needs but also optimizes the user's visual experience.

[0052] Step S24, determining the comfort level of each of the display positions of the initially selected target software by processing using a support vector machine based on the display positions of each of the initially selected target software, the display brightness of the software, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue level; The input of the support vector machine is the display position of each preliminary target software, the software display brightness, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue. The output of the support vector machine is the comfort level of each preliminary target software display position.

[0053] Support vector machine (SVM) is a classic machine learning model that excels at handling classification and regression problems. The support vector machine maps the input data to a high-dimensional space through its kernel function and finds an optimal hyperplane in the space to distinguish display positions of different comfort levels. For example, the support vector machine can learn that when the user's eye fatigue is high, lower brightness and display positions close to the user's line of sight will lead to higher comfort. Ultimately, the support vector machine is able to output a comfort score for each preliminary target software display position, with a higher score indicating that the position is more comfortable for the user. By using a support vector machine (SVM) to evaluate the comfort level of each preliminary target software display position, the target position that best suits the user's current state can be selected from multiple candidate positions to ensure that the displayed content is presented to the user in the most comfortable way.

[0054] Step S25, taking the initially selected target software display position with the highest comfort level as the target display position of the software.

[0055] The support vector machine (SVM) evaluates the comfort score of each preliminary target software display position, and finally selects the position with the highest score as the target display position. The comfort score takes into account factors such as display position, brightness, user's facial orientation, distance between eyes and screen, and eye fatigue, to ensure that the selected position can best adapt to the user's current visual state. By selecting the position with the highest comfort, the system can optimize the layout of the displayed content, reduce the user's visual burden, and improve the comfort of reading or operation.

[0056] S6, displaying the software based on the target display position of the software.

[0057] After determining the target display position of the software, the software content is controlled to be displayed based on the target display position.

[0058] Based on the same inventive concept, Figure 4 A schematic diagram of a software display system based on artificial intelligence provided by an embodiment of the present invention, the software display system based on artificial intelligence includes: A detection module 41 is used to detect that a user opens a software page, and then turns on the front camera and screen recording operations; An acquisition module 42 is used to acquire a user's gaze facial video and a screen recording interface image from a front camera; A blank area determination module 43, configured to determine a plurality of blank areas of the screen using a blank area determination model based on the screen recording interface image; A facial video processing module 44, configured to determine the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue using a facial video processing model based on the user's gaze facial video captured by the front camera; A target display position determination module 45, configured to determine a target display position of the software based on a plurality of blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; The display module 46 is used to display the software based on the target display position of the software.

[0059] Based on the same inventive concept, an embodiment of the present invention provides an electronic device, such as Figure 5 As shown, including: It includes: a processor 51; a memory 52; and a computer program; wherein the computer program is stored in the memory 52 and is configured to be executed by the processor 51 to implement the artificial intelligence-based software display method provided above, the method comprising: when detecting that a user performs a software page opening operation, opening the front camera and the screen recording operation; obtaining the user's facial video and the screen recording interface image from the front camera; using a blank area determination model to determine multiple blank areas of the screen based on the screen recording interface image; using a facial video processing model to determine the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue based on the user's facial video from the front camera; determining the target display position of the software based on the multiple blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; and displaying the software based on the target display position of the software.

[0060] Based on the same inventive concept, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by the processor 51, implements the aforementioned artificial intelligence-based software display method, the method comprising: when detecting that a user performs a software page opening operation, opening the front camera and the screen recording operation; obtaining the user's facial video and the screen recording interface image from the front camera; using a blank area determination model to determine multiple blank areas of the screen based on the screen recording interface image; using a facial video processing model to determine the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue based on the user's facial video from the front camera; determining the target display position of the software based on the multiple blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; and displaying the software based on the target display position of the software.

[0061] The artificial intelligence-based software display method provided in the embodiments of the present application can be applied to terminal devices (such as mobile phones), tablet computers, laptops, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), wearable devices (such as smart watches, smart glasses or smart helmets, etc.), augmented reality (AR) and virtual reality (VR) devices, smart home devices, car computers and other electronic devices, and the embodiments of the present application do not impose any restrictions on this.

[0062] Taking the mobile phone 100 as an example of the electronic device, Figure 6 A schematic structural diagram of the mobile phone 100 is shown.

[0063] like Figure 6 As shown, the mobile phone 100 may include a processing module 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0064] The processing module 110 can be used to: when it is detected that a user performs an operation of opening a software page, open the front camera and perform screen recording operation; obtain the user's facial video and the screen recording interface image from the front camera; determine multiple blank areas of the screen using a blank area determination model based on the screen recording interface image; determine the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue using a facial video processing model based on the user's facial video from the front camera; determine the target display position of the software based on the multiple blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; and display the software based on the target display position of the software.

[0065] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0066] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A software display method based on artificial intelligence, characterized in that: include: If it is detected that the user opens the software page, the front camera and screen recording operations are turned on; Get the user's facial video and screen recording interface image from the front camera; Determine a plurality of blank areas of the screen using a blank area determination model based on the screen recording interface image; Determine the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue using a facial video processing model based on the user's gaze facial video from the front camera; Determining a target display position of the software based on a plurality of blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; The software is displayed based on the target display position of the software.

2. The software display method based on artificial intelligence according to claim 1, characterized in that: The determining of the target display position of the software based on the plurality of blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue comprises: Constructing a knowledge graph, the knowledge graph includes multiple nodes and edges between the nodes, each node represents a blank area of ​​the screen, the node features of each node include the position information of a blank area of ​​the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue, and the edges between the nodes represent the positional relationship between the blank areas; Processing the knowledge graph based on a graph neural network model to determine the software display area, software display brightness, and a target area of ​​the software display; Generate multiple preliminary target software display positions using a variational autoencoder based on the software display area, the software display brightness, and the target area of ​​the software display, each of which is located within the target area of ​​the software display; Determine the comfort level of each of the initially selected target software display positions based on the display position of each of the initially selected target software, the display brightness of the software, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue by using a support vector machine; The initially selected target software display position with the highest comfort level is used as the target display position of the software.

3. The software display method based on artificial intelligence according to claim 1, characterized in that: The facial video processing model is a gated recurrent unit, the input of the facial video processing model is the user's gaze facial video of the front camera, and the output of the video processing model is the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue.

4. The software display method based on artificial intelligence as claimed in claim 2, characterized in that: The input of the variational autoencoder is the software display area, the software display brightness, and the target area of ​​the software display, and the output of the variational autoencoder is a plurality of preliminarily selected target software display positions.

5. A software display system based on artificial intelligence, characterized in that: include: The detection module is used to detect that the user opens the software page, and then turns on the front camera and screen recording; An acquisition module is used to acquire the user's gaze facial video and screen recording interface image from the front camera; A blank area determination module, configured to determine a plurality of blank areas of the screen using a blank area determination model based on the screen recording interface image; A facial video processing module, used for determining the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue using a facial video processing model based on the user's gaze facial video captured by the front camera; A target display position determination module, used to determine the target display position of the software based on a plurality of blank areas of the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue; The display module is used to display the software based on the target display position of the software.

6. The artificial intelligence-based software display system according to claim 5, characterized in that: The target display position determination module is also used for: Constructing a knowledge graph, the knowledge graph includes multiple nodes and edges between the nodes, each node represents a blank area of ​​the screen, the node features of each node include the position information of a blank area of ​​the screen, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue, and the edges between the nodes represent the positional relationship between the blank areas; Processing the knowledge graph based on a graph neural network model to determine the software display area, software display brightness, and a target area of ​​the software display; Generate multiple preliminary target software display positions using a variational autoencoder based on the software display area, the software display brightness, and the target area of ​​the software display, each of which is located within the target area of ​​the software display; Determine the comfort level of each of the initially selected target software display positions based on the display position of each of the initially selected target software, the display brightness of the software, the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue by using a support vector machine; The initially selected target software display position with the highest comfort level is used as the target display position of the software.

7. The artificial intelligence-based software display system according to claim 5, characterized in that: The facial video processing model is a gated recurrent unit, the input of the facial video processing model is the user's gaze facial video of the front camera, and the output of the video processing model is the user's facial orientation, the distance between the user's eyes and the screen, and the user's eye fatigue.

8. The artificial intelligence-based software display system according to claim 6, characterized in that: The input of the variational autoencoder is the software display area, the software display brightness, and the target area of ​​the software display, and the output of the variational autoencoder is a plurality of preliminarily selected target software display positions.

9. An electronic device, characterized in that: include: processor; Memory; And a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the artificial intelligence-based software display method as described in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the artificial intelligence-based software display method as described in any one of claims 1 to 4 is implemented.

Images