Visual experience modulation based on stroboscopic effects

The use of GANs in real-time strobe effect modification addresses the issue of stationary objects in video displays by dynamically adjusting visibility, enhancing the viewing experience through AR-based control of moving objects and backgrounds.

JP2024544862AActive Publication Date: 2024-12-05INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
JP2024526489
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-11
Filing Date
2022-10-25
Publication Date
2024-12-05
Estimated Expiration
2042-10-25

AI Technical Summary

Technical Problem

Existing video display technologies struggle with strobe effects that cause moving objects to appear stationary, leading to undesirable visual illusions and distorted viewing experiences, particularly in scenarios where visibility of background or objects is crucial.

Method used

A computer-implemented method using generative adversarial networks (GAN) for real-time modification of strobe effects, leveraging augmented reality (AR) and machine learning to adjust visibility of moving objects and backgrounds based on environmental and user data, enabling dynamic control of strobe effects.

Benefits of technology

Enhances or removes strobe effects to improve the viewing experience by making moving objects transparent or opaque, reducing visual illusions, and ensuring accurate representation of moving objects and backgrounds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024544862000001_ABST
    Figure 2024544862000001_ABST
Patent Text Reader

Abstract

Techniques are disclosed for modifying in real time by removing or enhancing stroboscopic effects from images associated with a viewing experience. The techniques include identifying a video clip, detecting environmental parameters, and calculating display settings. The techniques also include analyzing display settings using recommendations from the GAN, outputting the display settings on an AR display, and receiving feedback from a user.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates generally to the field of displays, and more particularly to adjusting strobe effects to the viewing experience. [Background technology]

[0002] The visual phenomenon of stroboscopic effect causes a moving object to appear stationary when viewed in non-sequential and discrete sampling views. The effect is not the same for everyone and depends on the person's physiological state, geographic location, etc.

[0003] In real life, the presence of stroboscopic effects can make the comprehensibility of a displayed scene better or worse. On the other hand, the removal of stroboscopic effects can improve the experience. In pre-recorded videos (e.g., movies, or camera captures), the difference between the rotation frequency of the wheel and the frequency of the light can unintentionally create a wagon wheel effect, which worsens the experience. In most cases, the absence of the optical illusion is preferable, but in some cases, the absence of the optical illusion is uncomfortable for people in some specific scenarios, and an enhancement is necessary. Summary of the Invention

[0004] According to one aspect, a computer-implemented method for modifying strobe effects from images in real time by leveraging machine learning is provided, the computer-implemented method includes identifying one or more video clips by a user, determining visual data, determining strobe effect settings, performing the strobe effect settings on the one or more video clips, outputting the video clips based on the updated strobe effect settings, and receiving feedback from the user.

[0005] According to another aspect, a computer program product for modifying a strobe effect from an image in real time is provided, the computer program product including one or more computer readable storage media and program instructions stored on the one or more computer readable storage media, the program instructions including program instructions for identifying one or more video clips by a user, program instructions for determining visual data, program instructions for determining a strobe effect setting, program instructions for performing the strobe effect setting on the one or more video clips, program instructions for outputting the video clip based on the updated strobe effect setting, and program instructions for receiving feedback from the user.

[0006] According to another aspect, a computer system for modifying a strobe effect from an image in real time is provided, the computer system including one or more computer processors, one or more computer readable storage media, and program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions including program instructions for identifying one or more video clips by a user, program instructions for determining visual data, program instructions for determining strobe effect settings, program instructions for performing the strobe effect settings on the one or more video clips, program instructions for outputting the video clips based on the updated strobe effect settings, and program instructions for receiving feedback from the user.

[0007] Preferred embodiments of the present invention disclose a computer-implemented method, computer system, and computer program product for modifying in real time by removing or enhancing stroboscopic effects from images associated with a viewing experience. The computer-implemented method may be performed by one or more computer processors and may include identifying a video clip, detecting environmental parameters, calculating display settings, analyzing the display settings using recommendations from the GAN, outputting the display settings on an AR display, and receiving feedback from a user.

[0008] According to another embodiment of the present invention, there is provided a computer system including a processing unit and a memory coupled to the processing unit for storing instructions, the instructions, when executed by the processing unit, perform operations of a method according to an embodiment of the present invention.

[0009] According to yet another embodiment of the present invention, there is provided a computer program product tangibly stored on a non-transitory machine-readable medium and including machine-executable instructions that, when executed on a device, cause the device to perform actions of a method according to an embodiment of the present invention.

[0010] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which: [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a functional block diagram illustrating a display environment, designated 100, according to one embodiment of the present invention. [Diagram 2] FIG. 1 is a functional block diagram illustrating high level schematic steps of a display component 111 according to one embodiment of the present invention. [Diagram 3] FIG. 1 illustrates a video generation sequence associated with display environment 100, according to one embodiment of the present invention. [Figure 4]FIG. 1 illustrates how a display environment 100 captures visual illusion losses using a video generator and a video discriminator, according to another embodiment of the invention. [Diagram 5] 5 is a high level flow diagram illustrating the operation of a display component 111, designated 500, in accordance with another embodiment of the present invention. [Figure 6] 6 is a block diagram, designated 600, of components of a server computer capable of executing the display component 111 within the display environment 100, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The current state of the art in video display, especially the stroboscopic effect associated with moving objects, can pose some challenges for the viewer. For example, scenes from a video image may need to be rendered clear of the stroboscopic effect of moving objects to allow the background to be seen. If uncorrected, this can lead to undesirable effects, such as in the medical field where the presence of a lesion / tumor may be obscured from an X-ray chart. Conversely, it may be desirable to make a moving object opaque by completely blocking the visibility of the background.

[0013] Other challenges associated with the consequences of stroboscopic effects can include the example that in a video clip with stroboscopic effects, it becomes difficult to prioritize the importance of moving objects and the background if the entire context is not taken into account. In another example with capturing video, the entire viewing experience can become boring in the absence of the obvious stroboscopic effects that the human brain is normally trained to have.

[0014] An embodiment of the present invention recognizes deficiencies in the current state of the art regarding stroboscopic effects associated with moving objects and provides a technique. The technique consists of real-time correction by removing or enhancing the stroboscopic effects associated with the viewing experience. The technique leverages Augmented Reality (AR) and uses machine learning, specifically Generative Adversarial Networks (GANs), to correct / correct images in real-time. The technique can also be applied to pre-recorded videos to correct portions of specific frames. The technique can be a deployable, downloadable, and shareable model. Additionally, the technique can be integrated into a system that can be developed and updated on a user interface (UI).

[0015] The embodiments of the present invention may provide advantages in the following scenarios: (i) industries where fatal accidents may occur due to the stroboscopic effect of the moving blades being obscured (i.e. scenarios where it is more important to see the background than the moving object), (ii) videos or commercial advertisements that distort the viewing experience by showing something that is not possible due to the wagon wheel effect or another wagon wheel effect where the wheels appear to be moving in the opposite direction, (iii) various applications in computer vision using AI for accurate image representation, (iv) medical image analysis and solutions, which are areas of research focused on healthcare, informatics, etc., and (v) can be very useful for AR experiences where the user relies primarily on visual cues.

[0016] Some embodiments may include techniques that identify the context of the scene and modulate the visibility of moving objects and backgrounds. The same embodiments may enhance the stroboscopic background by making the objects visible, thus making it easier for the user to experience the transparency of moving objects.

[0017] Some embodiments may include techniques that can enhance or correct existing strobe effects for a better viewing experience by making moving objects completely opaque, allowing the user to be aware of their presence (without seeing the background), and avoiding scenarios such as the wagon-wheel effect, where a visual illusion causes an object to contradict its physical properties.

[0018] Some embodiments may include the following advantageous features to minimize stereoscopic effects: (i) modulating the visibility of the moving object and the background simultaneously, (ii) enhancing the background and adding transparency, (iii) making the moving object opaque and distinguishing it from the background, (iv) using a light source to control the strobe effect, but this is only applicable to the display device and does not work immediately in real-world scenarios, (v) making the moving object opaque and distinguishing it from the background, (vi) enhancing the background and adding transparency, (vii) modifying the object's movement in the video, but this is done as a post-processing step on the media file and is not done immediately (and does not consider the background effect).

[0019] References herein to "one embodiment," "an embodiment," "an exemplary embodiment," and the like indicate that the embodiment being described may include a particular feature, structure, or characteristic, but each embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Moreover, when a particular feature, structure, or characteristic is described in connection with one embodiment, it is believed to be within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0020] It should be understood that the figures are only schematic and are not drawn to scale, and that the same reference numbers have been used throughout the figures to denote the same or similar parts.

[0021] Figure 1 is a functional block diagram illustrating a display environment 100 in accordance with one embodiment of the present invention. Figure 1 is intended only to provide an illustration of one implementation and is not intended to suggest any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by one of ordinary skill in the art without departing from the scope of the invention as set forth in the claims.

[0022] The display environment 100 includes a network 101, an IoT device 102, a display device 103, a generative adversarial network (GAN) server 104, a video source 105, and a server 110.

[0023] The network 101 may be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the three, and may include wired, wireless, or fiber optic connections. The network 101 may include one or more wired and / or wireless networks capable of receiving and transmitting data signals, voice signals, or video signals, or combinations thereof, including multimedia signals including voice, data, and video information. In general, the network 101 may be any combination of connections and protocols capable of supporting communication between the server 110, the display device 103, the IoT device 102, the GAN server 104, the video source 105, and other computing devices (not shown) in the display environment 100. It should be noted that the other computing devices may include, but are not limited to, the IoT device 102, and any electromechanical device capable of executing a sequence of computing instructions.

[0024] The IoT device 102 can be any smart / Internet of Things (IoT) device (e.g., wearable smart devices, smart phones, wireless cameras, etc.) that includes various sensors (e.g., temperature sensors / imaging, heart rate monitors, microphones, etc.) capable of collecting real-time data (e.g., video images, temperature, humidity, etc.). For example, the IoT device 102 can collect personal user data along with environmental data. The user data can be associated with the user's medical profile (using wearable sensors). The environmental data can be collected using light sensors, speed sensors, etc. mapped to an environmental profile such as lighting conditions, light frequency, rotational speed of an object, etc.

[0025] The display device 103 can be any display device that allows a user to view an image with stroboscopic effects modified / corrected. The display device 103 can be, but is not limited to, an LCD / LED display, a virtual reality (VR) goggle, an augmented reality (AR) goggle, a projector, and a tablet.

[0026] The GAN server 104 is an artificial intelligence (AI) server that leverages generative adversarial network (GAN) techniques to perform image manipulation (e.g., editing, redaction, etc.).

[0027] What is GAN technology? GAN technology is a type of machine learning (e.g. supervised learning, unsupervised learning, reinforcement learning, etc.) that has broad applicability in training models. Given a training set, GAN technology can learn to generate new data with the same accuracy as the training set. The main idea of ​​GAN is based on "indirect" training via a discriminator, which itself is also dynamically updated. This basically means that the generator is not trained to minimize the gap with a particular image, but rather to fool the discriminator. This allows the model to learn in an unsupervised manner. GAN uses two neural networks in its architecture (e.g. generator, discriminator). The goal of the generator network is to generate fake outputs, then take random noise as input and create outputs as similar as possible to the real output. For example, if counterfeit money is used, the generator tries to create outputs that look like real money. Conversely, the discriminator network plays the role of the police. The discriminator network is trained with images of real money so that it has a good understanding of what real money should look like. Fake images from the generator are also fed to the discriminator. Initially, the discriminator has no problem distinguishing real from fake in the early stages of training. Furthermore, the discriminator also provides feedback to the generator on how good the job it is doing. Based on this feedback, the generator modifies its approach (e.g., loss function, etc.) to create more authentic outputs in the next iteration.

[0028] Video source 105 is the source of video (eg, still image, computer file, moving image, live, pre-recorded, etc.) that a user wishes to modify due to a strobe effect.

[0029] The server 110 and / or the GAN server 104 may be a standalone computing device, an administration server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In other embodiments, the server 110 may represent a server computing system utilizing multiple computers as a server system, such as in a cloud computing environment. In another embodiment, the server 110 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any other programmable electronic device capable of communicating with other computing devices (not shown) in the display environment 100 via the network 101. In another embodiment, the server 110 represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single pool of seamless resources when accessed within the display environment 100.

[0030] An embodiment of the present invention may reside on a server 110. The server 110 includes a display component 111 and a database .

[0031] The display component 111 provides the ability to avoid or enhance the strobe effect associated with video / still images by leveraging generative adversarial network (GAN) techniques. Additionally, an augmented reality (AR) device can be used to provide additional filters to the image.

[0032] A user case scenario is used to illustrate the display component 111 of the preferred embodiment. A video clip contains an image of a ceiling fan. The speed of the ceiling fan is approximately the same as the frequency of the ceiling light. Due to the stroboscopic effect, the human eye perceives a moving object as stationary. The image processing performed by the smart glasses (i.e., AR goggles) identifies subtle frequency differences and continues to capture images at different times. After identifying the stroboscopic effect, the present embodiment uses GANs to generate a modified image.

[0033] The display component 111 has the following features and / or capabilities: (i) to identify the scene conditions and modulate the visibility of moving objects and background accordingly, (ii) to create a strobe effect background enhancement to make the object visible - the user experiences the moving object becoming transparent, (iii) to make the moving object completely opaque to make the user aware of its presence - the user does not see the background, and (iv) to enhance or correct the existing strobe effect for a better viewing experience - avoiding scenarios like the wagon wheel effect where a visual illusion causes an object to contradict its physical nature.

[0034] Some embodiments of the display component 111 may include the following steps and / or features: (i) taking various IoT sensor data as input along with the video to generate the augmented video using a generative adversarial network, (ii) encoding various spatio-temporal data into a latent space using various temporal encoders, (iii) determining various factors such as environmental conditions, user's medical profile, lighting conditions, light frequency, object rotation speed, spatio-temporal object visibility scores, (iv) learning to reduce the amount of noise by combining multiple data sources with controllable parameters, and (v) learning discriminators and generators using various loss functions to obtain possible controllable parameters (e.g., enable / disable effects, enhance effect resolution, cancel effects, merge / split effects) to generate the augmented video. Loss functions include visual illusion loss, effect counteracting loss, effect merging / splitting loss, etc.; (vi) the output from the previous step can be integrated with an augmented reality system to overlay the GAN-generated video as an additional layer; and (vii) dynamically update model parameters and use reinforcement learning to fine-tune the parameters based on reward scores provided by the user. Possible reward scores are provided at the effect level, or in labeling spatiotemporal video frames, or in marking side effects for the strobe effect. In general, "spatiotemporal data" corresponds to a data set that gives information about the spatial and temporal parameters contained in that data. For example, tracking a moving object in a video, or occupying a single position at a given time.

[0035] To illustrate the functionality of the display component 111, an example is provided below. Consider a scenario where the moving object and the light frequency are the same (e.g., a ceiling fan). Due to the strobe effect, the human eye perceives the moving object as stationary. As part of the display component 111, the image processing performed by the smart glasses identifies subtle frequency differences and continues to capture images at different times. After identifying the strobe effect, the display component 111 uses a GAN to generate the strobe effect and an augmented reality (AR) device to introduce the strobe effect as expected.

[0036] The database 116 is a repository for data used by the display component 111. The database 116 may be implemented using any type of storage device capable of storing data and configuration files that can be accessed and utilized by the server 110, such as a database server, hard disk drive, or flash memory. The database 116 stores information using one or more of a number of techniques known in the art. In the illustrated embodiment, the database 116 resides on the server 110. In alternative embodiments, the database 116 may reside elsewhere within the display environment 100, provided that the display component 111 has access to the database 116. The database 116 may store information related to IoT device specifications, including, but not limited to, a knowledge corpus, training datasets, training models, a library of images, feedback via reinforcement learning (RL), augmented reality (AR) display settings / profiles, VR display settings / profiles, display settings, loss functions, data interfaces, and video editing techniques.

[0037] FIG. 2 is a functional block diagram showing high level schematic steps of the display component 111 according to an embodiment of the present invention. Block 201 shows an input phase by identifying a video clip of interest. Block 201 leads to block 202. Block 202 shows functionality for detecting environmental parameters and visual illusions. Other features in block 202 include (i) detecting light frequency and rotating objects, and (ii) detecting visibility of the effect to the naked eye. Block 202 flows to block 203. Block 203 includes features of data conditioning and noise reduction of the video data (from block 201). Block 203 leads to block 204, which is a recommendation engine. As a recommendation engine, block 204 has features of (i) enabling / disabling strobe effect, (ii) enhancing stereoscopic effect, and (iii) canceling stereoscopic effect. Based on the decision made by block 204, block 205 executes the recommended action. The recommended actions may be described in blocks 206, 207, and 208. Block 206 represents that the embodiment can apply the effect to a scene where the effect is needed but not present, and vice versa. Block 207 represents that the embodiment can enhance the effect if the raw visibility of the effect is not noticeable, and block 208 represents that the embodiment can evaluate the importance between the background and the moving object and modulate the visibility. All decision blocks (e.g., 206, 207, and 208) lead to block 209, which represents pattern matching with an existing knowledge base KB (knowledge base). The next phase after the image generation / modification is the reinforcement learning (RL) phase. Block 210 represents using AR glasses to enhance the visibility of the modified image. Block 211 represents feedback from the user being fed back to the system for future adjustments.

[0038] FIG. 3 illustrates a video generation sequence associated with the display environment 100, according to an embodiment of the present invention. Data from the IoT device 102 is combined with the video data and then fed to the video generation component of the GAN system. The data collected by the IoT device can include personal user data along with environmental data. The user data can be associated with the user's medical profile (via a wearable watch / sensor). The environmental data can be collected using light sensors, speed sensors, etc., mapped to an environmental profile such as lighting conditions, light frequency, rotational speed of an object, etc. For example, an environmental condition / data is the amount of lighting in a space. The light intensity is detected and fed to the system. Note that the environmental parameters can be associated with controllable parameters. These are trained and learned by the model.

[0039] Note that the user has access to adjust the controllable parameters, including enabling / disabling effects, enhancing the resolution of effects, canceling effects, and merging / splitting effects to generate enhanced video.

[0040] FIG. 4 is a diagram illustrating how display environment 100 captures visual illusion losses using a video generator and a video discriminator, according to another embodiment of the present invention.

[0041] FIG. 5 is a high level flow diagram illustrating the operation of display component 111, designated 500, in accordance with another embodiment of the present invention.

[0042] The presentation component 111 identifies one or more video clips (step 502). In one embodiment, the presentation component 111 selects, based on a selection from a user, video and / or still images (e.g., live stream, camera recording, pre-recorded, etc.) that will be processed to enhance the user's viewing experience. For example, the user selects a pre-recorded clip from a video source 105 (i.e., a file stored on a personal computer).

[0043] The display component 111 determines visual data (step 504). In one embodiment, the display component 111 collects visual data (including environmental data and user data) through machine learning and IoT devices. Environmental conditions are collected using light sensors, speed sensors, etc. that are mapped to an environmental profile such as lighting conditions, light frequency, rotational speed of an object, etc.

[0044] The IoT device 102 collects personal user data along with environmental data. The user data is associated with the user's medical profile using wearable sensors. Once all the visual data is collected, the data is then combined with selected video clips. The selected video clips are encoded into latent space along with spatio-temporal data using various temporal encoders. The spatio-temporal data includes tracking moving objects in the video, which may occupy a single position at a given time.

[0045] The display component 111 determines the settings for the strobe effect (step 506). In one embodiment, the display component 111 determines the optimal recommendation (for the strobe effect settings) based on various factors by leveraging AI (machine learning). For example, the factors can include environmental conditions, the user's medical profile, lighting conditions, light frequency, object rotation speed, spatiotemporal object visibility score, etc.

[0046] Thus, the display component 111 can make optimal (user / viewer specific) recommendations based on various factors, such as (i) enabling / disabling strobe effects, (ii) enhancing stereoscopic effects, and (iii) canceling stereoscopic effects.

[0047] In some embodiments, the display component 111 can learn to reduce the amount of noise by combining multiple data sources together with controllable parameters.

[0048] The display component 111 executes the strobe effect settings (step 508). In one embodiment, the display component 111 starts with the recommended settings from the previous step. The display component 111 generates the augmented video using the recommended settings by training a discriminator and generator (via GAN) using various loss functions. The loss functions include visual illusion loss, effect cancellation loss, effect merging / splitting loss, etc. For example, if the recommendation is to enable the strobe effect, the system executes that action.

[0049] The display component 111 outputs the video clip (step 510). In one embodiment, the display component 111 outputs the video clip to a display based on the updated strobe effect settings. For example, the output can be integrated with an augmented reality (AR) system (i.e., 103) to overlay the GAN-generated video as an additional layer.

[0050] The display component 111 receives feedback from the user (step 512). In one embodiment, the display component 111 can dynamically update model parameters and use reinforcement learning to fine-tune the parameters based on reward scores provided by the user. Possible reward scores can be provided at the effect level, or in labeling of spatio-temporal video frames, or in marking of side effects of the strobe effect.

[0051] 6 illustrates a block diagram of components of the display component 111 application, designated 600, in accordance with an exemplary embodiment of the invention. It should be understood that FIG. 6 is merely provided as an illustration of one implementation and is not intended to suggest any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made.

[0052] 6 includes a processor 601, a cache 603, a memory 602, a persistent storage 605, a communication unit 607, an input / output (input / output) interface 606, and a communication fabric 604. The communication fabric 604 provides communication between the cache 603, the memory 602, the persistent storage 605, the communication unit 607, and the input / output (input / output) interface 606. The communication fabric 604 may be implemented using any architecture designed to pass data and / or control information between a processor (such as a microprocessor, communication and network processor, etc.), system memory, peripheral devices, and any other hardware components in the system. For example, the communication fabric 604 may be implemented using one or more buses or crossbar switches.

[0053] Memory 602 and persistent storage 605 are computer-readable storage media. In this embodiment, memory 602 includes random access memory (RAM). In general, memory 602 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 603 is a high-speed memory that improves the performance of processor 601 by holding recently accessed and near recently accessed data from memory 602.

[0054] Program instructions and data (e.g., software and data x10) used to practice embodiments of the present invention may be stored in persistent storage 605 and memory 602 for execution by one or more of the respective processors 601 via cache 603. In one embodiment, persistent storage 605 includes a magnetic hard disk drive. As an alternative to, or in addition to, a magnetic hard disk drive, persistent storage 605 may include a solid state hard drive, a semiconductor storage device, a read only memory (ROM), an erasable programmable read only memory (EPROM), a flash memory, or any other computer readable storage medium capable of storing program instructions or digital information.

[0055] The media used by persistent storage 605 may also be removable. For example, a removable hard drive may be used for persistent storage 605. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer readable storage medium that is also part of persistent storage 605. Display component 111 may be stored in persistent storage 605 for access and / or execution by one or more of the respective processors 601 via cache 603.

[0056] Communications unit 607, in these examples, provides for communication with other data processing systems or devices. In these examples, communications unit 607 includes one or more network interface cards. Communications unit 607 may provide for communication by using either or both physical and wireless communications links. Program instructions and data used to practice embodiments of the present invention (e.g., display component 111) may be downloaded to persistent storage 605 via communications unit 607.

[0057] The input / output interface 606 allows for the input and output of data with other devices that may be connected to each computer system. For example, the input / output interface 606 may provide a connection to an external device 608, such as a keyboard, a keypad, a touch screen, or some other suitable input device, or a combination thereof. The external device 608 may also include a portable computer-readable storage medium, such as, for example, a thumb drive, a portable optical or magnetic disk, and a memory card. Program instructions and data used to practice embodiments of the present invention (e.g., the display component 111) may be stored on such a portable computer-readable storage medium and loaded into the persistent storage 605 via the input / output interface 606. The input / output interface 606 also connects to a display 609.

[0058] Display 609 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0059] The programs described herein are identified based on the application for which they are implemented in a particular embodiment of the invention, however, it will be understood that the nomenclature of specific programs herein is used merely for convenience, and thus the invention should not be limited to use only with the particular application identified and / or implied by such nomenclature.

[0060] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer readable storage medium (or multiple computer readable storage media) having computer readable program instructions for causing a processor to implement aspects of the present invention.

[0061] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0062] The computer readable program instructions described herein may be downloaded from a computer readable storage medium into each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage.

[0063] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions to perform aspects of the invention by utilizing state information of the computer readable program instructions to personalize the electronic circuitry.

[0064] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0065] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable medium, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams, and can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0066] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to produce a computer-implemented process that causes a series of operational steps to be performed on the computer, other programmable apparatus, or other device, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0067] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions or a combination of dedicated hardware and computer instructions.

[0068] Although the description of various embodiments of the present invention has been presented for illustrative purposes, the description is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0069] The corresponding structures, materials, acts, and equivalents of all means-plus-function or step-plus-function elements within the scope of the appended claims are intended to include any structures, materials, or acts for performing a function in combination with other claim elements that are specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present disclosure. The present embodiments have been selected and described in order to best explain the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in various embodiments with various modifications as suited to the particular uses contemplated.

[0070] Finally, the proposed concepts according to the preferred embodiments can be succinctly summarized in the following clauses. 1) The AI ​​system takes various IoT sensor data along with the video as input to generate the augmented video using a generative adversarial network. 2) The AI ​​system uses different temporal encoders to encode different spatiotemporal data into a latent space. 3) The proposed AI system determines various factors such as environmental conditions, user's medical profile, lighting conditions, light frequency, object rotation speed, and spatiotemporal object visibility scores. 4) The proposed AI system learns to reduce the amount of noise by combining multiple data sources with controllable parameters. 5) The proposed AI obtains possible controllable parameters, such as effect enable / disable, effect resolution enhancement, effect cancellation, effect merging / splitting, etc., for generating augmented videos by training discriminators and generators using various loss functions. The loss functions include visual illusion loss, effect cancellation loss, effect merging / splitting loss, etc. 6) The output of the proposed system can be integrated with an augmented reality system to overlay the GAN-generated video as an additional layer. 7) The proposed system dynamically updates model parameters and uses reinforcement learning to fine-tune the parameters based on reward scores provided by the user. Possible reward scores are provided at the effect level, or in labeling spatio-temporal video frames, or in marking side effects of the strobe effect.

Claims

1. 1. A computer-implemented method for correcting stroboscopic effects from an image in real time by leveraging machine learning, comprising: Identifying one or more video clips by a user; Determining visual data; determining a strobe effect setting; applying a strobe effect setting to the one or more video clips; outputting the video clip based on the updated strobe effect settings; Receiving feedback from users 4. A computer-implemented method comprising:

2. Determining visual data includes: collecting visual data, the visual data including environmental data and user data; Encoding visual data into a latent space The computer-implemented method of claim 1 , further comprising:

3. 3. The computer-implemented method of claim 1, wherein determining the strobe effect settings is based on factors including environmental conditions, a user's medical profile, lighting conditions, light frequency, rotational speed of the object, and a visibility score of the spatiotemporal object.

4. 4. The computer-implemented method of claim 1, 2, or 3, wherein the strobe effect settings further include enable / disable strobe effect, enhance stereoscopic effect, and cancel stereoscopic effect.

5. Executing a strobe effect setting on the one or more video clips includes: Generating augmented videos by training discriminators and generators using different loss functions The computer-implemented method of claim 1 , further comprising:

6. outputting the video clip based on updated strobe effect settings; Overlaying GAN-generated video as an additional layer with an augmented reality (AR) system The computer-implemented method of claim 1 , further comprising:

7. Receiving feedback from users Updating the model parameters; using reinforcement learning to fine-tune the model parameters based on reward scores provided by the users; The computer-implemented method of claim 1 , further comprising:

8. 1. A computer program product for modifying stroboscopic effects from an image in real time, the computer program product comprising: one or more computer readable storage media; and program instructions stored on the one or more computer readable storage media, the program instructions comprising: program instructions for identifying one or more video clips by a user; program instructions for determining visual data; program instructions for determining a strobe effect setting; program instructions for implementing a strobe effect setting on said one or more video clips; program instructions for outputting the video clip based on updated strobe effect settings; program instructions for receiving feedback from a user; 2. A computer program product comprising:

9. The program instructions for determining the visual data include: program instructions for collecting the visual data, the visual data including environmental data and user data; Program instructions for encoding visual data into a latent space; 9. The computer program product of claim 8, further comprising:

10. 10. The computer program product of claim 8 or 9, wherein the program instructions for determining the strobe effect settings are based on factors including environmental conditions, a user's medical profile, lighting conditions, light frequency, rotational speed of the object, and a visibility score of the spatiotemporal object.

11. 11. The computer program product of claim 8, 9 or 10, wherein the strobe effect settings further include enable / disable strobe effect, enhance stereoscopic effect, and cancel stereoscopic effect.

12. The program instructions for implementing a strobe effect setting on the one or more video clips include: Program instructions for generating augmented videos by training discriminators and generators using various loss functions 12. A computer program product according to any one of claims 8 to 11, further comprising:

13. and program instructions for outputting the video clip based on updated strobe effect settings, the program instructions comprising: Program instructions for overlaying a GAN-generated video as an additional layer with an augmented reality (AR) system 13. A computer program product according to any of claims 8 to 12, further comprising:

14. The program instructions for receiving feedback from the user include: program instructions for updating the model parameters; program instructions for using reinforcement learning to fine-tune the model parameters based on a reward score provided by the user; 14. A computer program product according to any one of claims 8 to 13, further comprising:

15. 1. A computer system for real-time correction of stroboscopic effects from an image, the computer system comprising: one or more computer processors; one or more computer readable storage media; program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors; the program instructions comprising: program instructions for identifying one or more video clips by a user; program instructions for determining visual data; program instructions for determining a strobe effect setting; program instructions for implementing a strobe effect setting on said one or more video clips; program instructions for outputting the video clip based on updated strobe effect settings; program instructions for receiving feedback from a user; 1. A computer system comprising:

16. The program instructions for determining the visual data include: program instructions for collecting the visual data, the visual data including environmental data and user data; Program instructions for encoding visual data into a latent space; 16. The computer system of claim 15, further comprising:

17. 17. The computer system of claim 15 or 16, wherein the program instructions for determining the strobe effect settings are based on factors including environmental conditions, a user's medical profile, lighting conditions, light frequency, rotational speed of the object, and visibility scores of spatiotemporal objects.

18. 18. The computer system of claim 15, wherein the strobe effect settings further include enabling / disabling the strobe effect, enhancing the stereoscopic effect, and canceling the stereoscopic effect.

19. The program instructions for implementing a strobe effect setting on the one or more video clips include: Program instructions for generating augmented videos by training discriminators and generators using various loss functions 19. The computer system of claim 15, further comprising:

20. and program instructions for outputting the video clip based on updated strobe effect settings, the program instructions comprising: Program instructions for overlaying a GAN-generated video as an additional layer with an augmented reality (AR) system 20. The computer system of claim 15, further comprising:

21. A computer program comprising program code means adapted to perform the method according to any one of claims 1 to 7 when the computer program is run on a computer.

Citation Information

Patent Citations

  • Intelligent, dynamic, long-term digital surveillance media storage system

    US20060182357A1

  • System and method for effectively implementing a stroboscopic visual effect

    US20120242779A1

  • Polishing agent, stock solution for polishing agent, and polishing method

    US20200017718A1

  • Computer Method and Apparatus Making Screens Safe for Those with Photosensitivity

    US20200021718A1

  • Pushing items to users based on a reinforcement learning model

    US20200342268A1