User interaction management for reducing hysteresis in user interaction applications

Abstract

Embodiments herein provide a method for user interaction management in a user interaction application in an electronic device. The method includes: generating and displaying artificial frames among a plurality of real frames of a user interaction application, and receiving user input on the artificial frames displayed among the plurality of real frames of the user interaction application; detecting an object in the artificial frame input by the user; determining a correlation between the user input, coordinates of the object in a real frame of the plurality of real frames, and coordinates of the object in the artificial frame based on the object detected in the artificial frame; and displaying an interaction of the object in the artificial frame based on the determined correlation.

Description

Technical Field

[0001] The present disclosure relates to an electronic device. For example, the present disclosure relates to a system and method for user interaction management to reduce lag in user interaction applications. Background Art

[0002] Mobile devices currently support computationally intensive applications with high graphics demands. In some conventional systems, interpolated (artificial) frames are added to increase frame rates, providing smoother motion. However, when a user input is given on an artificial frame, the artificial frame introduces lag by ignoring the user input. Therefore, when user input is ignored within the artificial frame, the immediate result on the artificial frame can result in a poor response from the user's perspective.

[0003] The frame interpolation feature predicts one or more intermediate or artificial frames at defined timestamps between two real frames. This feature increases the frame rate of videos and games, or creates slow-motion videos. It automatically increases the frames per second (fps) of moving images when playing games or watching videos. When enabled, it adds latency to user input for applications such as gaming. Furthermore, it improves the frame rate when playing games or watching videos on various mobile devices. Frame interpolation is performed by adding additional frames between real frames, improving the overall user experience.

[0004] Unlike movie playback or slow-motion video creation, interpolated frames for user-interactive applications (e.g., gaming applications) can vary based on user interaction and input. When frame interpolation is enabled, artificial frames are inserted between real frames, increasing the frame rate from, for example, 60 fps to 90 fps or 120 fps. However, frame interpolation also introduces lag (delay) associated with user interactions performed on the artificial frames.

[0005] It would therefore be desirable to address the above-mentioned or other disadvantages or at least provide a useful alternative. Summary of the Invention

[0006] Technical Solution According to an exemplary embodiment, a method for user interaction management in a user interaction application is provided. The method includes: generating and displaying an artificial frame between multiple real frames of the user interaction application; receiving input on the artificial frame displayed between the multiple real frames of the user interaction application; detecting an object in the artificial frame in which the input was received, and determining a correlation between the input, coordinates of the object in a real frame of the multiple real frames, and the coordinates of the object in the artificial frame based on the object detected in at least one artificial frame; and enabling interaction with the object in the artificial frame based on the determined correlation.

[0007] In an example embodiment, interacting with at least one object in at least one artificial frame is achieved by applying a transformation matrix based on the determined correlation.

[0008] In an example embodiment, the method includes: generating or training an AI model; monitoring multiple inputs on multiple objects in multiple artificial frames and multiple real frames of a user interaction application, and using the trained AI model to determine the correlation between the coordinates of the inputs, the coordinates of the objects in the real frames, and the coordinates of the objects in the artificial frames.

[0009] In an exemplary embodiment, the artificial frame is generated using at least any one of an interpolation method and an extrapolation method.

[0010] In an example embodiment, the real frame includes discontinuous motion regions and / or continuous motion regions.

[0011] In an example embodiment, the method includes: performing interaction of the input by applying an input transformation matrix in response to the received input; determining whether the input is received on a discontinuous motion region of the artificial frame or on a continuous motion region of the artificial frame; applying the input transformation matrix to the continuous motion region based on receiving the input on the continuous motion region, or ignoring the received input by skipping the AI ​​model based on receiving the input on the discontinuous motion region.

[0012] In an example embodiment, the method includes: based on receiving the input on the artificial frame, inputting the received input to the AI ​​model, or based on receiving the input on the real frame, sending the received input to the user interaction application.

[0013] According to an example embodiment, an electronic device for user interaction management in a user interaction application is provided. The electronic device includes: a memory; at least one processor, including a processing circuit, coupled to the memory; and a user interaction controller, communicatively coupled to the memory and the processor. The user interaction controller is configured to: display an artificial frame between multiple real frames of the user interaction application; receive input on the artificial frame displayed between the multiple real frames of the user interaction application; detect an object in the artificial frame in which the user input is received; determine a correlation between the input, the coordinates of the object in the real frame among the multiple real frames, and the coordinates of the object in the artificial frame based on the object detected in the artificial frame; and enable interaction with the object in the artificial frame based on the determined correlation.

[0014] These and other aspects of the various example embodiments herein will be better appreciated and understood when considered in conjunction with the following description and accompanying drawings. However, it should be understood that the following description, while indicating various example embodiments and many specific details thereof, is given by way of illustration and not limitation. Many changes and modifications may be made within the scope of the embodiments herein without departing from the spirit thereof, and the embodiments herein include all such modifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] These and other features, aspects, and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, wherein like reference numerals designate corresponding parts throughout the various figures, and wherein: Figure 1 is a diagram illustrating a scenario in which user input is ignored in an artificial frame according to the prior art; Figure 2 is a diagram illustrating a scenario of supporting or considering user input performed on an artificial frame according to various embodiments; Figure 3A and Figure 3B is a block diagram illustrating an example configuration of an electronic device for user interaction management to reduce lag in user interaction applications according to various embodiments; Figure 4 is a flow chart illustrating an example method for user interaction management to reduce lag in user interaction applications according to various embodiments; Figure 5 is a diagram illustrating an example scenario of analyzing properties of a real frame and extracting information of areas of a continuous motion region and a discontinuous motion region in the real frame according to various embodiments; Figure 6 is a diagram illustrating an example of an artificial frame generated based on a real frame according to various embodiments; Figure 7 is a diagram illustrating analysis of a real frame and generation of a binary mask for distinguishing continuous motion regions from discontinuous motion regions according to various embodiments; Figure 8 is a diagram illustrating an example of motion information between an artificial frame and a real frame according to various embodiments; Figure 9 is a diagram illustrating the linear displacement of an arbitrary object between two real frames according to various embodiments; and Figure 10 is a diagram illustrating a comparison of frame interpolation lag reduction according to various embodiments.

[0016] It is noted that, to the extent possible, like reference numerals are used to represent like elements in the drawings. Furthermore, one of ordinary skill in the art will appreciate that the elements in the drawings are illustrated for simplicity and may not necessarily be drawn to scale. For example, the dimensions of some of the elements in the drawings may be exaggerated relative to other elements to help improve understanding of various aspects of the present disclosure. Furthermore, elements may be represented in the drawings by conventional symbols, and the drawings may show only those specific details that are relevant to understanding the embodiments of the present disclosure, so as not to obscure the drawings with details that would be readily understood by one of ordinary skill in the art having the benefit of the description herein. DETAILED DESCRIPTION

[0017] The various example embodiments of the present invention and their various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the accompanying drawings and described in detail in the following description. Descriptions of well-known components and processing technologies may be omitted so as not to unnecessarily obscure the embodiments of the present invention. In addition, the various example embodiments described herein are not necessarily mutually exclusive, as the various embodiments can be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" used herein refers to a non-exclusive "or". The examples used herein are intended only to facilitate understanding of the ways in which the embodiments of the present invention can be practiced, and also to enable those skilled in the art to practice the embodiments of the present invention. Therefore, the examples should not be interpreted as limiting the scope of the disclosure herein.

[0018] Various example embodiments are described and illustrated in terms of blocks that perform the described functions. These blocks (referred to herein as managers, units, modules, hardware components, etc.) are physically implemented by analog and / or digital circuitry (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuitry, etc.), and are optionally driven by firmware and software. For example, the circuitry is embodied in one or more semiconductor chips or on a substrate support (such as a printed circuit board). The circuitry of a block is implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware for performing some of the block's functions and a processor for performing other functions of the block. Without departing from the scope of the disclosed method, each block of an embodiment may be physically divided into two or more interacting and discrete blocks. Similarly, without departing from the scope of the disclosed method, the blocks of an embodiment may be physically combined into more complex blocks.

[0019] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. Therefore, the disclosed method is interpreted as extending to any changes, equivalents and alternative forms other than those specifically set forth in the accompanying drawings. Although the terms "first", "second" and the like are used herein to describe various elements, these elements are not limited by these terms. These terms are generally used to distinguish one element from another.

[0020] The terms "discontinuous motion region" and "static region" are used interchangeably throughout this disclosure.

[0021] The terms "continuous motion region" and "non-static region" are used interchangeably throughout this disclosure.

[0022] The term "user interactive application" may refer to, for example, an accelerated application that allows a user to interact with various content, such as audiovisual information, visualizations, VR / AR content, etc. Such accelerated applications may be computationally intensive on an electronic device, requiring multiple computations (using, for example, a GPU, CPU, and other hardware components of the electronic device) to run as designed.

[0023] The term "transformation matrix" may refer to, for example, a mapping of coordinates of an object in an artificial frame to coordinates of the object in a real frame. The transformation matrix represents the pixel-to-pixel correspondence between the artificial frame and the previous real frame displayed to the user.

[0024] Therefore, various embodiments disclose a method for user interaction management in a user interaction application in an electronic device. The method may include: generating and displaying an artificial frame between multiple real frames of the user interaction application, receiving input on the artificial frame displayed between the multiple real frames of the user interaction application; detecting an object in the artificial frame in which the input was received, and determining a correlation between the input, coordinates of the object in a real frame of the multiple real frames, and the coordinates of the object in the artificial frame based on the object detected in at least one artificial frame; and enabling interaction with the object in the artificial frame based on the determined correlation.

[0025] Therefore, various example embodiments disclose an electronic device for user interaction management in a user interaction application. The electronic device includes: a memory; at least one processor, including a processing circuit, coupled to the memory; and a user interaction controller, communicatively coupled to the memory and the processor. The user interaction controller can be configured to: generate and display an artificial frame between multiple real frames of the user interaction application; receive input on the artificial frame displayed between multiple real frames of the user interaction application; detect an object in the artificial frame in which the input is received; determine a correlation between the input, the coordinates of the object in the real frame of multiple real frames, and the coordinates of the object in the artificial frame based on the object detected in the artificial frame; and enable interaction with the object in the artificial frame based on the determined correlation.

[0026] Conventional methods evaluate the quality of interpolated frames by considering the user's head movement, velocity, and head angle while wearing a virtual reality (VR) headset. Furthermore, conventional methods include a head-mounted display / VR headset and motion sensors such as an accelerometer (e.g., a gravity sensor) and a gyroscope (e.g., a gyroscopic sensor). Furthermore, conventional methods track the VR headset's linear movement and direction of linear movement or rotational movement (e.g., rotational angular velocity or rotational angle). Conventional methods aim to enhance the viewing experience by providing smooth transitions between video frames based on the user's head movement. Interpolated frames are used to bridge the visual gaps between real frames, resulting in a more immersive and seamless viewing experience for the user wearing the head-mounted display. However, conventional methods fail to accommodate user input to create future frames (or upcoming / subsequent frames) when input (e.g., user input) is given in a previous artificial frame, resulting in lag for the user.

[0027] Unlike conventional methods, user input information is used to create future frames. Future frames are generated based on user input given for previous artificial frames. When the user input is touch, touch transformation technology helps find accurate coordinates using pixel-to-pixel correspondence techniques. These coordinates are then used to create future frames, reducing lag in user-interactive applications.

[0028] Referring now to the drawings, and more particularly to the Figure 1 to Figure 1 2, wherein like reference numerals denote corresponding features consistently throughout the drawings, which illustrate various example embodiments.

[0029] Figure 1 is a diagram (100) illustrating a scenario in which user input is ignored in an artificial frame according to the prior art. Figure 1As shown, frames 1-6 are displayed on the display system, where frames 1, 3, and 5 are real frames generated by a user interaction application, and frames 2, 4, and 6 are artificial frames generated using a frame interpolation function. In conventional methods, when a user performs touch input on artificial frames 2, 4, and 6, the display system ignores the user input. In conventional methods, because the user performs actions based on what the user sees on the display device, it is not always possible to accurately identify the intention of the user's touch map coordinates. The display device can display real frames or artificial frames, but the user does not know whether the real frame or the artificial frame is displayed. Therefore, the conventional method only accepts user input in real frames 1, 3, and 5. Ignoring user input in artificial frames causes lag / delay when the frames are displayed by the user interaction application.

[0030] Figure 2 is a diagram (200) illustrating a scenario in which user input performed on an artificial frame is supported or considered according to various embodiments. Unlike conventional methods, according to the present disclosure, user input is accepted and corrected on an artificial frame. Accurate identification of coordinates is essential in user interaction applications, otherwise, it may lead to inaccurate, erroneous actions, or lags in displaying frames. A transformation of touch coordinates is performed to reflect the real action that the user wants. A transformation matrix with respect to the user touch coordinates is estimated based on the coordinates of the artificial frame. The artificial frame is mapped back to a previous real frame based on a pixel-to-pixel correspondence. The pixel-to-pixel correspondence refers to the matching of individual pixels between the artificial frame and the real frame. In this matching process, the transformation matrix is ​​used to achieve accurate alignment and synchronization of visual elements between the artificial frame and the real frame. For example, referring to Figure 2 , when the user gives an input (xartificial, yartificial) on frame 1.5, the present disclosure causes interaction with the objects of the artificial frame.

[0031] Figure 3A and Figure 3B is a block diagram illustrating an example configuration of an electronic device ( 300 ) for user interaction management in a user interaction application according to various embodiments.

[0032] The electronic device (300) includes a memory (310), a processor (e.g., including processing circuitry) (320) coupled to the memory (310), and a user interaction controller (e.g., including various circuits and / or executable program instructions) (330) communicatively coupled to the memory (310) and the processor (320).

[0033] The memory (310) is configured to store instructions to be executed by the processor (320). For example, the memory (310) stores a user interaction application (310a) to be executed by the processor (320). The memory (310) includes a non-volatile storage element. Examples of such non-volatile storage elements include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or a form of electrically programmable memory (EPROM) or electrically erasable programmable memory (EEPROM). In addition, in some examples, the memory (310) is considered to be a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is not embodied in a carrier wave or propagating signal. The term "non-transitory" is not to be interpreted as meaning that the memory (310) is non-removable. In various examples, the memory (310) is configured to store larger amounts of information. In a specific example, the non-transitory storage medium stores data that can change over time (e.g., in random access memory (RAM) or cache).

[0034] The processor (320) includes one or more processors. The processor (320) according to an embodiment of the present disclosure may include various processing circuits and / or multiple processors. For example, as used herein (including the claims), the term "processor" may include various processing circuits including at least one processor, wherein one or more processors in the at least one processor may be configured individually and / or collectively in a distributed manner to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform a number of functions, these terms encompass, for example, but not limited to, the following situations: one processor performs some of the described functions and another processor performs other of the described functions, and a single processor may perform all of the described functions. In addition, the at least one processor may include, for example, a combination of processors that perform the various described / disclosed functions in a distributed manner. The at least one processor may execute program instructions to implement or execute the various functions. The one or more processors are general-purpose processors (such as a central processing unit (CPU), an application processor (AP), etc.), a graphics processing unit (such as a graphics processing unit (GPU), a visual processing unit (VPU)), and / or an AI-specific processor (such as a neural processing unit (NPU)). The processor (320) includes multiple cores and is configured to execute instructions stored in the memory (310).

[0035] The user interaction controller (330) may include various circuits and / or executable program instructions, and generates and displays an artificial frame between a plurality of real frames of the user interaction application (310a), and receives user input on the artificial frame displayed between the plurality of real frames of the user interaction application (310a). The user interaction controller (330) detects an object in the artificial frame in which the user input is received, and determines a correlation between the user input, the coordinates of the object in the real frame among the plurality of real frames, and the coordinates of the object in the artificial frame based on the object detected in the artificial frame. In addition, the user interaction controller (330) implements interaction with the object in the artificial frame in response to the user input based on the correlation.

[0036] The user interaction controller (330) includes a user input detector (340), an AI model (350) and a correlation determiner (360). The user interaction application (310a) generates a plurality of real frames and determines two or more real frames from the generated plurality of real frames. The electronic device (300) generates and displays an artificial frame between the two or more real frames. When a user gives an input on the artificial frame displayed between the two or more real frames, the user input detector (340) of the user interaction controller (330) detects an object in the artificial frame in which the user input is received from the user. The AI ​​model (350) of the user interaction controller (330) includes correlation information of the coordinates of the user input, the coordinates of the object in the real frame and the coordinates of the object in the artificial frame. The electronic device (300) inputs the user input received on the artificial frame into the AI ​​model (350) to determine the correlation between the user input, the coordinates of the object in the real frame and the coordinates of the object in the artificial frame based on the objects detected in the artificial frame using the correlation determiner (360).

[0037] The user interaction controller (330) is implemented by processing circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.) and is optionally driven by firmware. For example, the circuits are embodied in one or more semiconductor chips, or on a substrate support (such as a printed circuit board, etc.). The user interaction controller (330) according to an embodiment of the present disclosure may include various processing circuits and / or multiple processors. For example, as used herein (including the claims), the term "processor" may include various processing circuits including at least one processor, wherein one or more processors of the at least one processor may be configured individually and / or collectively in a distributed manner to perform various functions described herein. As used herein, when "processor", "at least one processor" and "one or more processors" are described as being configured to perform a number of functions, these terms cover, for example, but not limited to, the following situations: one processor performs some of the described functions and another processor performs other of the described functions, as well as the situation where a single processor can perform all of the described functions. In addition, the at least one processor may include, for example, a combination of processors that perform various described / disclosed functions in a distributed manner. At least one processor may execute program instructions to implement or perform various functions.

[0038] The AI ​​model (350) may include multiple neural network layers. Each layer has multiple weight values, and layer operations are performed by calculating the previous layer and operating on the multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0039] A learning process may refer to, for example, a process for training a predetermined target device (e.g., a robot) using a plurality of learning data to enable, allow, or control the target device to make a determination or prediction. Examples of learning processes include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0040] The electronic device (300) monitors a plurality of user inputs on a plurality of objects in a plurality of artificial frames and a plurality of real frames of a user interaction application (310a), and uses an AI model (350) to determine a correlation among coordinates of the user inputs, coordinates of the objects in the real frames, and coordinates of the objects in the artificial frames.

[0041] Reference Figure 3BFor example, at S1, a user interaction application (310a) generates real frames (f1 and f2), wherein the user interaction application is not limited to a game application. At S2, a frame analyzer (331) of a user interaction controller (330) analyzes the real frames generated by the user interaction application (310a) to determine an area where a UI / static (331b) / dynamic (331c) element exists, and then classifies the area in the form of a binary mask using a UI element scanner (331a). In addition, the frame analyzer (331) sends information on the area of ​​the determined UI / static (331b) / dynamic (331c) element to an AI model (350) to determine a continuous motion area / discontinuous motion area in the real frames (f1 and f2).

[0042] In S3, the artificial frame generator (332) of the user interaction controller (330) includes a frame synthesizer (332a), a forward / backward motion analyzer (332b), a scene analyzer (332c), a feature extractor (332d), a forward warper (332e), and an optical flow determiner (332f). The frame synthesizer (332a) takes existing frames (f1 and f2) as input and uses the existing frames (f1 and f2) information to generate new frames, and furthermore, the frame synthesizer (332a) plays a vital role in generating artificial frames by combining input information to create visually appealing and contextually relevant image frames with respect to continuous motion regions and discontinuous motion regions.

[0043] The forward / backward motion analyzer (332b) analyzes motion information in a sequence of frames (f1 and f2) and determines the direction and magnitude of motion between consecutive frames (f1 and f2). In addition, the forward / backward motion analyzer (332b) is typically used to determine displacement vectors that describe the movement of objects or pixels between frames (f1 and f2). By analyzing these motion vectors, the forward / backward motion analyzer (332b) identifies the direction (forward or backward) and speed of motion. This information is valuable in generating artificial frames to maintain the coherence and authenticity of the generated content. By understanding the motion in the input frames (f1 and f2), the generator predicts appropriate motion for the generated frames, ensuring smooth and natural transitions between consecutive frames, wherein the artificial frames are generated to fill the gaps between existing frames (f1 and f2).

[0044] The scene analyzer (332c) captures and interprets the scene information required to generate realistic and contextually appropriate frames. By analyzing the input data (f1 and f2), the scene analyzer (332c) provides valuable insights into the visual, auditory, or other sensory aspects of the scene, which are leveraged by the artificial frame generator to create more accurate and immersive artificial frames.

[0045] The feature extractor (332d) extracts visual features to capture and reproduce the important characteristics of the input frames (f1 and f2) and enables the generation of artificial frames. The forward warper (332e) creates artificial frames by applying specific transformations to the input frames (f1 and f2). These transformations are used to simulate camera movement, change perspective, or create visual effects, and ensure that each pixel in the transformed frame is generated based on a known position in the original frame, making it easier to control and manipulate the resulting image or video.

[0046] The optical flow (332f) generates an artificial frame by estimating the motion between two consecutive frames (f1 and f2), and generates a reasonable intermediate frame that promotes a seamless transition between the consecutive frames (f1 and f2).

[0047] The artificial frame generator (332) generates artificial frames that are inserted between real frames to smooth the movement of game characters that are part of the game world, including but not limited to objects, NPCs, etc. Artificial frame generation avoids discontinuous motion areas to prevent and / or reduce updates in text areas (such as player positions, game statistics, etc.). Therefore, the created frames are as realistic as possible.

[0048] After generating the artificial frames, the artificial frame generator (332) sends the generated artificial frames to the frame sequencer (333), wherein, at S4, the frame sequencer (333) uses the real / artificial frame order determiner (333a) to arrange the real frames and the artificial frames in a specific order to create a coherent sequence or animation of images. At S5, the real frames and the artificial frames are displayed on the display (370) of the electronic device (300) in a sorted manner.

[0049] When a user gives an input on the display (370), for example, the user input is not limited to a touch input, a gesture, a motion input, a voice input, a virtual reality input, a user input from an external controller connected to the electronic device (300), and for example, when the user input is a touch input on the display, the electronic device (300) detects whether the touch input is received on the artificial frame or the real frame based on determining the touch coordinates x1, y1. At S6, the electronic device (300) determines that the touch input coordinates x1, y1 are received on the real frame, and the electronic device (300) sends the coordinates (x1, y1) of the touch input as output (x real frame and y real frame) to the user interaction application (310a) to perform interaction according to the touch input (x1, y1) received from the user.

[0050] At S6, when the electronic device (300) determines the coordinates (x1, y1) of the touch input received on the artificial frame, the electronic device (300) feeds the artificial frame touch as input (xartificial frame, yartificial frame) to the AI ​​model (350) to detect the object in the artificial frame, wherein the AI ​​model (350) includes correlation information between the touch input from the user, the coordinates of the object in the real frame, and the coordinates of the object in the artificial frame. When the AI ​​model (350) receives the coordinates (xartificial frame, yartificial frame) of the touch input, the AI ​​model (350) determines the correlation between the touch input given by the user, the coordinates of the object in the real frame, and the coordinates of the object in the artificial frame based on the object detected in the artificial frame.

[0051] The AI ​​model (350) includes a coordinate analyzer (350a), a transformation matrix block (350b), and a forward warp analyzer (350c). The coordinate analyzer (350a) determines the coordinates of the touch input received from the user on the artificial frame. The transformation matrix block (350b) is used to map the coordinates of the object in the artificial frame to the coordinates of the object in the real frame. The forward warp analyzer (350c) is used to determine the movement of the object from the real frame to the artificial frame, wherein the information of the movement of the object from the real frame to the artificial frame is received from the artificial frame generator (332). Once the AI ​​model (350) determines the correlation between the touch input, the coordinates of the object in the real frame, and the coordinates of the object in the artificial frame, the AI ​​model (350) sends the determined correlation to the user interaction application (310a) for causing interaction with the object in the artificial frame.

[0052] In an embodiment, the real frame includes at least one discontinuous motion area and / or at least one continuous motion area. Examples of UI elements may include, for example, but not limited to, player statistics, menu buttons, on-screen joystick controls, mission windows, or any other element with discontinuous motion. The generation of the artificial frame is accomplished in such a way that when the artificial frame is generated from multiple real frames, these static areas or areas with discontinuous motion are kept intact. Additionally, the touch transformation matrix is ​​applied to areas with continuous motion and not to static areas or areas with discontinuous motion.

[0053] In an embodiment, the AI ​​model (350) operates on dense optical flow to produce displacements that can be linear or nonlinear. The displacement represents the spatial shift or movement of pixels between the artificial frame and the real frame, and also indicates that the pixels in the artificial frame have moved from their original positions in the corresponding real frame. In addition, the displacement provides the necessary information for understanding the motion and changes that occur between the real frame and the artificial frame. Optical flow is commonly used to estimate the motion of objects in a video sequence by tracking the movement of pixels in consecutive frames of the video. The basic assumption is that the pixels in the video sequence move smoothly from frame to frame. The optical flow model estimates the motion of each pixel in the video sequence by calculating the displacement vector between the corresponding pixels in two consecutive frames. These resulting motion vectors are then used to generate a flow field that describes the motion of objects in the scene. Some commonly used methods are the Lucas-Kannade method, the Horn-Schunck method, and the frameback method. The AI ​​model (350) estimates the optical flow to determine the motion between the real frame and the artificial frame. In addition, the AI ​​model (350) uses the optical flow information to synthesize new artificial frames or interpolated frames between existing real frames. In addition, the AI ​​model (350) establishes a pixel-to-pixel correspondence between the artificial frame and the real frame to accurately locate each pixel in the artificial frame relative to the real frame. The AI ​​model (350) takes the displacement vector provided by the artificial frame generator as input and generates a mapping of coordinates from the artificial frame to the real frame. The user touch coordinates and the displacement vector are provided as input to the AI ​​model (350) along with the binary mask of the UI element to determine the correlation between the user input, the coordinates of the object in the real frame, and the coordinates of the object in the artificial frame to enable interaction with the object in the artificial frame based on the determined correlation. In the presence of discontinuous motion areas, the transformation matrix is ​​not applied.

[0054] In an embodiment, the artificial frame generator (332) is an AI model (350) that generates frames based on input from real frames and a frame analyzer. The AI ​​model (350) performs user action analysis and converts user actions into actual real frames. These coordinates are also used to take actions on the real frames. The AI ​​model (350) will use the correlation between the real frame and the artificial frame to map the coordinates in the artificially generated frame and take actions on the future frame by transforming the mapped coordinates of the artificially generated frame in the real frame.

[0055] Finally, the touches performed by the user on the artificial frame are mapped to the real frame using backward motion estimation and warping methods. The transformation of the user touch coordinates is performed in the frame generation application to update the frame according to the user input.

[0056] Figure 4 is a flow chart ( 400 ) illustrating an example method for user interaction management to reduce lag in user interaction applications according to various embodiments.

[0057] At S401 , the method includes generating a real frame in a user interactive application.

[0058] At S402 , the method includes analyzing, by a frame analyzer, a real frame.

[0059] At S403 , the method includes deriving, by the frame analyzer, a mask (the mask may be binary or non-binary) for distinguishing (continuous / discontinuous motion) regions.

[0060] At S404 , the method includes acquiring, by an artificial frame generator, a mask and generating an interpolated frame.

[0061] At S405 , the method includes displaying an artificial frame between a plurality of real frames of the user interactive application.

[0062] At S406 , the method includes sequencing the real frames and the artificial frames by a frame sequencer and sending the real frames and the artificial frames to a display.

[0063] At S407 , the method includes receiving user input on the artificial frame displayed between the plurality of real frames of the user interactive application. In addition, the method includes the user interacting with the display while accessing the user interactive application and recording coordinates (x1, y1) in screen space.

[0064] At S408 , the method includes determining, using the frame sequencer based on the user interaction, whether user input is received on the artificial frame.

[0065] At S409 , the method includes identifying whether the user input is on an artificial frame.

[0066] At S410 , the method includes detecting an object in an artificial frame in which a user input is received.

[0067] At S411 , the method includes determining a correlation between a user input, coordinates of the object in a real frame among a plurality of real frames, and coordinates of the object in the artificial frame based on the object detected in the artificial frame.

[0068] At S412 , the method includes determining whether the user input is in a discontinuous motion region or in a continuous motion region (coordinate analysis).

[0069] At S413 , the method includes identifying whether the user input is on a static area.

[0070] At S414 , the method includes applying a user input transform when the user input is on a non-static area.

[0071] At S415 , the method includes disabling user input transformation for a portion of the frame when the user input is on a static area.

[0072] At S416 , the method includes mapping the user input coordinates to a real frame.

[0073] At S417 , the method includes providing the mapping information to a user interaction application (as shown in S418 ) to enable interaction with the object in the artificial frame in response to user input.

[0074] The various actions, acts, blocks, steps, etc. in the method may be performed in the order presented, in a different order, or simultaneously. Furthermore, in various embodiments, some of the actions, acts, blocks, steps, etc. may be omitted, added, modified, skipped, etc. without departing from the scope of the disclosed method.

[0075] Figure 5 5 is a diagram illustrating an example scenario of analyzing properties of a real frame and extracting information about regions of continuous motion and regions of discontinuous motion in the real frame according to various embodiments. Furthermore, the technique specifies the distinction between regions of continuous motion and regions of discontinuous motion in the segmentation output, which is used to generate an interpolated frame.

[0076] Reference Figure 5 , an artificial frame (1.5) is generated by a frame interpolator, where the frame interpolator obtains real frame 1 and real frame 2 as input.

[0077] Figure 6 is a diagram ( 600 ) illustrating an example of an artificial frame generated based on a real frame according to various embodiments.

[0078] Reference Figure 6 , when the user gives input on the artificial frame (1.5), the electronic device (300) determines the corresponding relationship in the real frame (eg, real frame 1, real frame 2) to perform interaction on the artificial frame (1.5).

[0079] Figure 7 is a diagram ( 700 ) illustrating an example analysis of a real frame and generating a binary mask that distinguishes between continuous motion regions and discontinuous motion regions according to various embodiments.

[0080] Reference Figure 7 , S1 represents real frames f1 and f2, at S2, the AI-based frame analyzer (331) analyzes and predicts various frame attributes in the real frame and identifies whether UI elements, text areas, player statistics, etc. exist in the real frame. At S3, the frame analyzer (331) segments discontinuous elements and continuous elements, wherein the transformation matrix is ​​not enabled in the discontinuous area and the transformation matrix is ​​enabled in the continuous area. When the UI element exists in the real frame, the frame analyzer also generates a binary mask that acts as a notifier by activating subsequent frame decisions based on the frame attributes.

[0081] Figure 8is a diagram ( 800 ) illustrating an example of motion information between an artificial frame and a real frame according to various embodiments.

[0082] Reference Figure 8 , when the user gives input in the artificial frame, the AI ​​model (350) determines the correlation between the coordinates of the user input and the coordinates of the object in the artificial frame (Xart, Yart=950, 350), and uses the displacement vector (dmvx, dmvy) as shown in Table 1 to map the determined user input and the coordinates of the object in the artificial frame (Xart, Yart=950, 350) to the real frame (Xreal, Yreal=940, 350) to achieve interaction on the artificial frame.

[0083] [Table 1]

[0084] Figure 9 is a diagram (900) illustrating an example linear displacement of an arbitrary object between two real frames, according to various embodiments.

[0085] Reference Figure 9 , for the linear displacement example, in s1, the ball (object) is at position x1, y1 at time t1 in the real frame, and in s2, the ball is at position x2=x1+dx1, y2=y1+dy1 in the other real frame. The movement of the ball between the two real frames is determined as dx1 and dy1. When the user gives input on the artificial frame (Xartificial and Yartificial), the AI ​​model (350) determines the displacement vector between the artificial frame and the real frame to determine the correlation between the coordinates of the object in the artificial frame and the coordinates of the object in the real frame to cause an interaction in the artificial frame in response to the user input. Similarly, if the coordinates of the object in the other real frame are determined as x2=x1+dx1 and y2=y1+dy1, the coordinates of the object in the artificial frame are determined as xart=dx1 / 2 and yart=dy1 / 2, as shown in s2.

[0086] Figure 10 is a diagram ( 1000 ) illustrating example frame interpolation lag reduction according to various embodiments.

[0087] In current systems, when a user gives input on an artificial frame, the existing system either ignores the input on the artificial frame or adopts misguided input due to the motion added in the artificial frame generation. In either scenario, the existing system faces lag issues (e.g., more than 32ms) in example user interaction applications.

[0088] In the disclosed method, when the user gives input on the artificial frame, the disclosed method accepts and corrects the input on the artificial frame, which reduces the lag (about 16ms for the given example) in the performance of the user interactive application.

[0089] While the present disclosure has been shown and described with reference to various exemplary embodiments, it should be understood that the various exemplary embodiments are intended to be illustrative rather than restrictive. Those skilled in the art will also understand that various changes in form and details may be made without departing from the true spirit and full scope of the present disclosure, including the appended claims and their equivalents. It will also be understood that any embodiment described herein may be used in combination with any other embodiment described herein.

Claims

1. A method for user interaction management in a user interaction application of an electronic device, the method comprising: displaying, by the electronic device, at least one artificial frame between a plurality of real frames of the user interactive application; receiving, by the electronic device, an input on the at least one artificial frame displayed between the plurality of real frames of the user interactive application; detecting, by the electronic device, at least one object in the at least one artificial frame in which the input is received; determining, by the electronic device, a correlation between the input, coordinates of the at least one object in at least one real frame of the plurality of real frames, and coordinates of the at least one object in the at least one artificial frame based on the at least one object detected in the at least one artificial frame; as well as The electronic device enables interaction with the at least one object in the at least one artificial frame based on the determined correlation.

2. The method according to claim 1, wherein Interacting with the at least one object in the at least one artificial frame includes applying a transformation matrix based on the determined correlation.

3. The method according to claim 1, wherein The method includes generating or training an AI model by the electronic device, wherein the generating or training operation includes: monitoring, by the electronic device, a plurality of inputs on a plurality of objects in a plurality of artificial frames and the plurality of real frames of the user interactive application; and The electronic device uses the pre-trained AI model to determine the correlation between the coordinates of the at least one input, the coordinates of the at least one object in the at least one real frame, and the coordinates of the at least one object in the at least one artificial frame.

4. The method according to claim 1, wherein The at least one artificial frame is generated using at least any one of an interpolation method and an extrapolation method.

5. The method according to claim 1, wherein The plurality of real frames include at least one discontinuous motion region and / or at least one continuous motion region.

6. The method of claim 2, wherein: The interacting of performing the input by the electronic device by applying an input transformation matrix in response to the received input includes: determining, by the electronic device, whether the input is received on the at least one discontinuous motion region of the at least one artificial frame or the input is received on the at least one continuous motion region of the at least one artificial frame; The electronic device performs one of the following: Based on receiving the input on the at least one continuous motion region, applying, by the electronic device, the input transformation matrix to the at least one continuous motion region; or Based on receiving the input over the at least one discontinuous motion region, ignoring the received input by the electronic device by skipping the AI ​​model.

7. The method of claim 1, wherein: The method comprises: Based on receiving the input on the at least one artificial frame, the electronic device inputs the received input into the AI ​​model; or Based on receiving the input on the at least one real frame, the received input is sent by the electronic device to the user interaction application.

8. An electronic device configured for user interaction management in a user interaction application, wherein: The electronic device comprises: Memory; at least one processor, including processing circuitry, coupled to said memory; A user interaction controller, comprising a processing circuit, communicatively coupled to the memory and the at least one processor, wherein the user interaction controller is configured to control the electronic device to perform the following operations: displaying at least one artificial frame between a plurality of real frames of the user interactive application; receiving input on the at least one artificial frame displayed between the plurality of real frames of the user interactive application; detecting at least one object in the at least one artificial frame receiving the input; determining a correlation between the input, coordinates of the at least one object in at least one of the plurality of real frames, and coordinates of the at least one object in the at least one artificial frame based on the at least one object detected in the at least one artificial frame; and Interaction with the at least one object in the at least one artificial frame is achieved based on the determined correlation.

9. The electronic device according to claim 8, wherein: Interaction with the at least one object in the at least one artificial frame is achieved by applying a transformation matrix based on the determined correlation.

10. The electronic device according to claim 8, wherein: The electronic device is configured to: monitoring a plurality of inputs on a plurality of objects in a plurality of artificial frames and the plurality of real frames of the user interactive application; as well as The pre-trained AI model is used to determine the correlation between the coordinates of the at least one input, the coordinates of the at least one object in the at least one real frame, and the coordinates of the at least one object in the at least one artificial frame.

11. The electronic device according to claim 8, wherein: The at least one artificial frame is configured to be generated using at least any one of an interpolation method and an extrapolation method.

12. The electronic device according to claim 8, wherein The plurality of real frames include at least one discontinuous motion region and / or at least one continuous motion region.

13. The electronic device according to claim 9, wherein: The electronic device is configured to: determining whether the input is received on the at least one discontinuous motion region of the at least one artificial frame or the input is received on the at least one continuous motion region of the at least one artificial frame; and do one of the following: applying the user transformation matrix to the at least one continuous motion region based on receiving the input on the at least one continuous motion region; or Based on receiving the input over the at least one discontinuous motion region, ignoring the received input by skipping the AI ​​model.

14. The electronic device according to claim 8, wherein: The electronic device is configured to: Based on receiving the input on the at least one artificial frame, inputting the received input into the AI ​​model; or Based on receiving the input on the at least one real frame, the received input is sent to the user interaction application.

15. A non-transitory computer-readable recording medium comprising a program for executing a control method of an electronic device, the method comprising: displaying, by the electronic device, at least one artificial frame between a plurality of real frames of the user interactive application; receiving, by the electronic device, an input on the at least one artificial frame displayed between the plurality of real frames of the user interactive application; detecting, by the electronic device, at least one object in the at least one artificial frame in which the input is received; determining, by the electronic device, a correlation between the input, coordinates of the at least one object in at least one real frame of the plurality of real frames, and coordinates of the at least one object in the at least one artificial frame based on the at least one object detected in the at least one artificial frame; as well as The electronic device enables interaction with the at least one object in the at least one artificial frame based on the determined correlation.

Images