Visual experience modulation based on the strobe effect

Abstract

Techniques are disclosed for modifying in real time by removing or enhancing stroboscopic effects from images associated with a viewing experience. The techniques include identifying a video clip, detecting environmental parameters, and calculating display settings. The techniques also include analyzing display settings using recommendations from the GAN, outputting the display settings on an AR display, and receiving feedback from a user.

Description

Technical Field

[0001] The present invention generally relates to the field of displays, and more particularly to adjusting the stroboscopic effect on the viewing experience.

Background Art

[0002] Due to the visual phenomenon of the stroboscopic effect, a moving object appears to be stationary when viewed in a non - continuous and discrete sampling view. The effect is not the same for all people and depends on the person's physiological state, geographical location, etc.

[0003] In real life, due to the presence of the stroboscopic effect, the legibility of the display scene may become better or worse. On the other hand, removing the stroboscopic effect may improve the experience. In pre - recorded videos (e.g., movies or camera captures), due to the difference between the rotational frequency of the wheels and the frequency of the light, the wagon - wheel effect occurs unintentionally and degrades the experience. In most cases, it is preferable to eliminate visual illusions. However, in some specific scenarios, the absence of visual illusions may make people feel uncomfortable, so enhancement is required.

Summary of the Invention

[0004] According to one aspect, a computer - implemented method for real - time correction of the stroboscopic effect from an image by leveraging machine learning is provided. The computer - implemented method includes identifying one or more video clips by a user, determining visual data, determining a stroboscopic effect setting, performing the stroboscopic effect setting on one or more video clips, outputting the video clip based on the updated stroboscopic effect setting, and receiving feedback from the user.

[0005] In another embodiment, a computer program product is provided for correcting a strobe effect from an image in real time, the computer program product comprising one or more computer-readable storage media and program instructions stored in one or more computer-readable storage media, the program instructions comprising program instructions for identifying one or more video clips by the user, program instructions for determining visual data, program instructions for determining strobe effect settings, program instructions for applying the strobe effect settings to one or more video clips, program instructions for outputting video clips based on the updated strobe effect settings, and program instructions for receiving feedback from the user.

[0006] In another embodiment, a computer system is provided for correcting a strobe effect from an image in real time, the computer system comprising one or more computer processors, one or more computer-readable storage media, and program instructions stored on one or more computer-readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising program instructions for identifying one or more video clips by a user, program instructions for determining visual data, program instructions for determining strobe effect settings, program instructions for applying the strobe effect settings to one or more video clips, program instructions for outputting video clips based on the updated strobe effect settings, and program instructions for receiving feedback from the user.

[0007] A preferred embodiment of the present invention discloses a computer implementation method, a computer system, and a computer program product for real-time modification by removing or enhancing stroboscopic effects from images related to a viewing experience. The computer implementation method may be carried out by one or more computer processors and may include identifying a video clip, detecting environmental parameters, calculating display settings, analyzing the display settings using recommendations from a GAN, outputting the display settings onto an AR display, and receiving user feedback.

[0008] According to another embodiment of the present invention, a computer system is provided. The computer system includes a processing unit and a memory coupled to the processing unit for storing instructions. When the instructions are executed by the processing unit, they perform the action of the method according to an embodiment of the present invention.

[0009] According to yet another embodiment of the present invention, a computer program product is provided which includes machine-executable instructions that are tangibly stored on a non-temporary machine-readable medium. When executed on a device, the instructions cause the device to perform the action of the method according to an embodiment of the present invention.

[0010] Next, preferred embodiments of the present invention will be described as mere examples with reference to the following drawings. [Brief explanation of the drawing]

[0011] [Figure 1] This is a functional block diagram showing a display environment designated as 100, according to one embodiment of the present invention. [Figure 2] This is a functional block diagram showing high-level schematic steps of a display component 111 according to one embodiment of the present invention. [Figure 3] This figure shows a video generation sequence related to a display environment 100 according to one embodiment of the present invention. [Figure 4]This figure shows how the display environment 100 uses a video generator and a video discriminator to capture visual illusion loss, according to another embodiment of the present invention. [Figure 5] This is a high-level flowchart illustrating the operation of a display component 111 designated as 500 according to another embodiment of the present invention. [Figure 6] This is a block diagram of 600 components of a server computer capable of executing a display component 111 within a display environment 100, according to one embodiment of the present invention. [Modes for carrying out the invention]

[0012] Current state-of-the-art technology for video display, particularly regarding stroboscopic effects related to moving objects, can present several challenges for viewers. For example, scenes from video images may need to have the stroboscopic effect of moving objects removed to allow the background to be visible. If not corrected, this can have undesirable effects, such as potentially obscuring lesions / tumors from X-ray charts in medical fields. Conversely, there are also cases where it is desirable to make moving objects opaque by completely blocking the visibility of the background.

[0013] Other challenges associated with the strobe effect include the difficulty in prioritizing the importance of moving objects and backgrounds in video clips with a strobe effect, when the overall context is not considered. Another example related to capturing video is that the overall viewing experience can become boring if there is no obvious strobe effect, which the human brain is normally trained to react to.

[0014] Embodiments of the present invention recognize and provide a method for addressing shortcomings in current state-of-the-art technology regarding stroboscopic effects associated with moving objects. The method comprises real-time correction by removing or enhancing stroboscopic effects relevant to the viewing experience. The method leverages augmented reality (AR) and uses machine learning, specifically generative adversarial networks (GANs), to correct / correct images in real time. The method can also be applied to pre-recorded video to correct specific portions of frames. The method can be made into a deployable, downloadable, and shareable model. Furthermore, the method can be integrated into a system that can be developed and updated on a user interface (UI).

[0015] Embodiments of the present invention may offer advantages in the following scenarios: (i) industries where a stroboscopic effect, where a moving blade becomes invisible, could lead to life-threatening accidents (i.e., scenarios where it is more important to see the background than the moving object); (ii) videos or commercial advertisements that distort the viewing experience by showing things that are impossible due to the wagon-wheel effect, or another wagon-wheel effect that makes the wheels appear to be moving in the opposite direction; (iii) various computer vision applications that use AI for accurate image representation; (iv) medical image analysis and solutions, which are research fields that focus on healthcare, informatics, etc.; and (v) AR experiences where the user relies primarily on visual stimuli.

[0016] Some embodiments may include techniques for identifying the context of a scene and modulating the visibility of moving objects and the background. The same embodiments may extend the strobe effect background by making objects visible, thereby allowing the user to easily experience that moving objects are transparent.

[0017] Some embodiments may include techniques that can enhance or correct existing strobe effects for a better viewing experience by making moving objects completely opaque so that the user is aware of their presence (without the user seeing the background), thereby avoiding scenarios such as the wagon-wheel effect where an object contradicts its physical properties due to a visual illusion.

[0018] Some embodiments may include the following advantageous features for minimizing stereoscopic effects: (i) simultaneously modulating the visibility of moving objects and the background; (ii) expanding the background and adding transparency; (iii) making moving objects opaque and distinguishing them from the background; (iv) using a light source to control the stroboscopic effect, however this is only applicable to display devices and does not work immediately in real-world scenarios; (v) making moving objects opaque and distinguishing them from the background; (vi) expanding the background and adding transparency; (vii) changing the movement of objects in the video, however this is done as a post-processing step of the media file and not immediately (and does not take background effects into account).

[0019] References in this specification to “one embodiment,” “embodiment,” and “exemplary embodiment” indicate that the embodiments described may include certain features, structures, or characteristics, but each embodiment does not necessarily have to include those features, structures, or characteristics. Furthermore, such phrases do not necessarily refer to the same embodiment. Moreover, when certain features, structures, or characteristics are described in relation to one embodiment, it is considered within the knowledge of those skilled in the art that they may affect such features, structures, or characteristics in relation to other embodiments, whether or not they are explicitly described.

[0020] It should be understood that the diagrams are merely schematic and are not drawn to a specific scale. It should also be understood that the same reference numbers are used throughout the diagrams to indicate the same or similar parts.

[0021] FIG. 1 is a functional block diagram showing a display environment 100 according to an embodiment of the present invention. FIG. 1 merely provides an example of one implementation form and does not imply any limitation regarding an environment in which different embodiments can be implemented. Without departing from the scope of the present invention described in the claims, many modifications may be added to the illustrated environment by those skilled in the art.

[0022] The display environment 100 includes a network 101, an IoT device 102, a display device 103, an adversarial generation network (GAN) server 104, a video source 105, and a server 110.

[0023] The network 101 can be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the three, and can include a wired connection, a wireless connection, or an optical fiber connection. The network 101 can include one or more wired networks, wireless networks, or both capable of receiving and transmitting data signals, voice signals, or video signals or combinations thereof, including multimedia signals containing voice, data, and video information. Generally, the network 101 can be any combination of connections and protocols capable of supporting communication between the server 110, the display device 103, the IoT device 102, the GAN server 104, the video source 105, and other computing devices (not shown) within the display environment 100. Note that other computing devices can include, but are not limited to, the IoT device 102 and any electromechanical device capable of executing a series of computing instructions.

[0024] The IoT device 102 can be any smart / IoT (Internet of Things) device (such as a wearable smart device, a smartphone, a wireless camera, etc.) that includes various sensors (such as temperature sensors / imaging, heart rate monitors, microphones, etc.) capable of collecting real-time data (such as video images, temperature, humidity, etc.). For example, the IoT device 102 can collect personal user data along with environmental data. The user data can be associated with the user's medical profile (using wearable sensors). The environmental data can be collected using light sensors, speed sensors, etc. that are mapped to environmental profiles such as lighting conditions, light frequency, rotational speed of objects, etc.

[0025] The display device 103 can be any display device that enables a user to view an image with a corrected / strobe effect. The display device 103 can be, but is not limited to, an LCD / LED display, a virtual reality (VR) headset, an augmented reality (AR) headset, a projector, and a tablet.

[0026] The GAN server 104 is an artificial intelligence (AI) server that utilizes adversarial generative network (GAN) technology to perform image operations (such as editing, correction, etc.).

[0027] What is GAN technology? GAN technology is a type of machine learning (e.g., supervised learning, unsupervised learning, reinforcement learning) with broad applicability in training models. Given a training set, GAN technology can learn to generate new data with the same accuracy as the training set. The main idea behind GANs is based on "indirect" training through a discriminator, which is also dynamically updated. This essentially means that the generator is not trained to minimize the deviation from a particular image, but rather to deceive the discriminator. This allows the model to learn in an unsupervised manner. GANs use two neural networks (e.g., a generator and a discriminator) in their architecture. The goal of the generator network is to generate a false output, then take random noise as input, and create an output that is as similar as possible to a real output. For example, if counterfeit money is used, the generator will try to create an output that looks like real money. Conversely, the discriminator network plays the role of the police. The discriminator network is trained with images of real money so that it has a good understanding of what real money should look like. The discriminator also receives fake images from the generator. Initially, the discriminator has no problem distinguishing real from fake during the early stages of training. Furthermore, the discriminator also provides feedback to the generator on how well it is doing its job. Based on this feedback, the generator modifies its technique (e.g., loss function) to produce more authentic output in the next iteration.

[0028] Video source 105 is the source of the video (e.g., still image, computer file, video, live, pre-recorded, etc.) that the user wishes to modify due to the strobe effect.

[0029] Server 110 or GAN Server 104, or both, can be a standalone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In another embodiment, Server 110 can represent a server computing system that utilizes multiple computers as a server system, such as in a cloud computing environment. In yet another embodiment, Server 110 can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smartphone, or any other programmable electronic device capable of communicating with other computing devices (not shown) within the display environment 100 via the network 101. In yet another embodiment, Server 110 can represent a computing system that utilizes clustered computers and components (e.g., a database server computer, an application server computer, etc.) that act as a single pool of seamless resources when accessed within the display environment 100.

[0030] Embodiments of the present invention can reside on a server 110, which includes a display component 111 and a database 116.

[0031] The display component 111 provides the ability to avoid or enhance stroboscopic effects associated with video / still images by leveraging Generative Adversarial Network (GAN) technology. Furthermore, it can provide additional filters to images using augmented reality (AR) devices.

[0032] A user case scenario is used to illustrate a display component 111 of a preferred embodiment. The video clip includes an image of a ceiling fan. The speed of the ceiling fan is approximately the same as the frequency of the ceiling light. Due to the stroboscopic effect, the human eye perceives a moving object as stationary. Image processing performed by smart glasses (i.e., AR goggles) identifies subtle frequency differences and continues to capture the image at different points in time. After identifying the stroboscopic effect, this embodiment uses a GAN to generate a corrected image.

[0033] The display component 111 has the following features or capabilities, or both: (i) to identify the scene context and modulate the visibility of moving objects and the background accordingly; (ii) to extend the background with a strobe effect to make objects visible—the user experiences moving objects becoming transparent; (iii) to make moving objects completely opaque to allow the user to be aware of their presence—the user does not see the background; and (iv) to enhance or correct existing strobe effects for a better viewing experience—avoiding scenarios such as the wagon-wheel effect where visual illusions cause objects to contradict their physical properties.

[0034] Some embodiments of the display component 111 may include the following steps or features or both: (i) taking various IoT sensor data as input along with video to generate augmented video using a generative adversarial network; (ii) encoding various spatiotemporal data into latent space using various time encoders; (iii) determining various factors such as environmental conditions, user medical profile, lighting conditions, light frequency, object rotation speed, and spatiotemporal object visibility score; (iv) learning to reduce the amount of noise by combining multiple data sources with controllable parameters; and (v) acquiring possible controllable parameters (e.g., enabling / disabling effects, enhancing effect resolution, canceling effects, merging / splitting effects, for generating augmented video) by learning discriminators and generators using various loss functions. (vi) the loss function includes visual illusion loss, effect cancellation loss, effect merging / splitting loss, etc., the output from the previous step can be integrated with an augmented reality system to overlay the GAN-generated video as an additional layer, and (vii) the model parameters are dynamically updated and reinforcement learning is used to fine-tune the parameters based on reward scores provided by the user. Possible reward scores are provided by effect level, or by labeling spatiotemporal video frames, or by marking secondary effects of stroboscopic effects. In general, “spatiotemporal data” corresponds to a dataset that gives information about the spatial and temporal parameters contained in that data. For example, tracking a moving object in a video or occupying a single location at a given time.

[0035] To illustrate the functionality of the display component 111, an example is provided below. Consider a scenario where a moving object and light have the same frequency (e.g., a ceiling fan). Due to the stroboscopic effect, the human eye perceives the moving object as stationary. Image processing performed by smart glasses as part of the display component 111 identifies subtle frequency differences and continues to capture images at different points in time. After identifying the stroboscopic effect, the display component 111 uses a GAN to generate that stroboscopic effect and uses an augmented reality (AR) device to introduce that stroboscopic effect as expected.

[0036] Database 116 is a repository for data used by the display component 111. Database 116 can be implemented using any type of storage device capable of storing data and configuration files that can be accessed and utilized by server 110, such as a database server, hard disk drive, or flash memory. Database 116 stores multiple pieces of information using one or more of the techniques known in the art. In the illustrated embodiment, database 116 resides on server 110. In another embodiment, database 116 may reside elsewhere within the display environment 100, provided that the display component 111 can access database 116. Database 116 may, but is not limited to, store information related to IoT device specifications, including knowledge corpora, training datasets, training models, image libraries, reinforcement learning (RL) feedback, augmented reality (AR) display settings / profiles, VR display settings / profiles, display settings, loss functions, data interfaces, and video editing techniques.

[0037] Figure 2 is a functional block diagram showing high-level schematic steps of a display component 111 according to one embodiment of the present invention. Block 201 represents the input phase by identifying the target video clip. Block 201 leads to Block 202. Block 202 represents the functionality of detecting environmental parameters and visual illusions. Other features within Block 202 include (i) detecting the frequency of light and rotating objects, and (ii) detecting the visibility of the effect to the naked eye. Block 202 flows to Block 203. Block 203 includes the features of data condition and noise reduction of the video data (from Block 201). Block 203 leads to Block 204, which is the recommendation engine. As a recommendation engine, Block 204 has the features of (i) enabling / disabling the stroboscopic effect, (ii) enhancing the stereoscopic effect, and (iii) canceling the stereoscopic effect. Based on the decisions made by Block 204, Block 205 performs the recommended action. Recommended actions may be described in blocks 206, 207, and 208. Block 206 represents that the embodiment can apply an effect to a scene where the effect is needed but not present, and vice versa. Block 207 represents that the embodiment can enhance an effect when the raw visibility of the effect is not significant, and block 208 represents that the embodiment can modulate visibility by evaluating the importance between the background and the moving object. All decision blocks (e.g., 206, 207, and 208) lead to block 209, which represents pattern matching with an existing knowledge base (KB). The next phase after image generation / modification is the reinforcement learning (RL) phase. Block 210 represents enhancing the visibility of the modified image using AR glasses. Block 211 represents that user feedback is fed back to the system for future adjustments.

[0038] Figure 3 shows a video generation sequence related to a display environment 100 according to one embodiment of the present invention. Data from the IoT device 102 is combined with video data and then fed to the video generation component of the GAN system. Data collected by the IoT device may include personal user data along with environmental data. User data may be associated with the user's medical profile (via a wearable watch / sensor). Environmental data may be collected using light sensors, velocity sensors, etc., mapped to environmental profiles such as lighting conditions, light frequency, and rotational speed of objects. For example, environmental conditions / data may be the amount of lighting in a given space. Light intensity is detected and fed to the system. Note that environmental parameters may be associated with controllable parameters, which are trained and learned by the model.

[0039] It should be noted that users have access to adjust controllable parameters. These controllable parameters include enabling / disabling effects, enhancing effect resolution, canceling effects, and merging / splitting effects for generating enhanced video.

[0040] Figure 4 shows how, according to another embodiment of the present invention, the display environment 100 uses a video generator and a video discriminator to capture visual illusion loss.

[0041] Figure 5 is a high-level flowchart illustrating the operation of a display component 111 designated as 500 according to another embodiment of the present invention.

[0042] The display component 111 identifies one or more video clips (step 502). In one embodiment, the display component 111 selects, based on the user's selection, a video or still image or both (e.g., a live stream, camera recording, pre-recorded video, etc.) that will enhance the user's viewing experience. For example, the user selects a pre-recorded clip from a video source 105 (i.e., a file stored on a personal computer).

[0043] The display component 111 determines the visual data (step 504). In one embodiment, the display component 111 collects visual data (including environmental data and user data) through machine learning and IoT devices. Environmental conditions are collected using light sensors, velocity sensors, etc., mapped to environmental profiles such as lighting conditions, light frequency, and rotational speed of objects.

[0044] The IoT device 102 collects personal user data along with environmental data. The user data is associated with the user's medical profile using wearable sensors. Once all visual data is collected, it is then combined with selected video clips. The selected video clips are encoded into latent space along with spatiotemporal data using various time encoders. The spatiotemporal data includes tracking moving objects in the video, which can occupy a single location at a given time.

[0045] The display component 111 determines the strobe effect settings (step 506). In one embodiment, the display component 111 uses AI (machine learning) to determine the optimal recommendation (for strobe effect settings) based on various factors. For example, factors may include environmental conditions, the user's medical profile, lighting conditions, light frequency, object rotation speed, and spatiotemporal object visibility score.

[0046] Therefore, the display component 111 can make optimal (user / viewer-specific) recommendations based on various factors, such as (i) enabling / disabling the strobe effect, (ii) enhancing the stereoscopic effect, and (iii) canceling the stereoscopic effect.

[0047] In some embodiments, the display component 111 can learn to reduce the amount of noise by combining multiple data sources with controllable parameters.

[0048] The display component 111 performs strobo effect settings (step 508). In one embodiment, the display component 111 starts the recommended settings from the previous step. The display component 111 generates an augmented video using the recommended settings by training discriminators and generators (via GAN) using various loss functions. Loss functions include visual illusion loss, effect cancellation loss, effect merging / splitting loss, etc. For example, if the recommendation is to enable the strobo effect, the system performs that action.

[0049] The display component 111 outputs a video clip (step 510). In one embodiment, the display component 111 outputs the video clip to a display based on the updated strobe effect settings. For example, the output can be integrated with an augmented reality (AR) system (i.e., 103) to overlay the GAN-generated video as an additional layer.

[0050] The display component 111 receives feedback from the user (step 512). In one embodiment, the display component 111 can dynamically update model parameters and use reinforcement learning to fine-tune the parameters based on reward scores provided by the user. Possible reward scores are provided by effect level, or by labeling spatiotemporal video frames, or by marking secondary effects of the stroboscopic effect.

[0051] Figure 6, designated 600, shows a block diagram of the components of a display component 111 application according to an exemplary embodiment of the present invention. It should be understood that Figure 6 merely provides an example of one implementation and does not imply any limitations regarding environments in which different embodiments may be implemented. Many modifications may be made to the illustrated environment.

[0052] Figure 6 includes a processor 601, a cache 603, memory 602, persistent storage 605, a communication unit 607, an input / output (I / O) interface 606, and a communication fabric 604. The communication fabric 604 provides communication between the cache 603, the memory 602, the persistent storage 605, the communication unit 607, and the input / output (I / O) interface 606. The communication fabric 604 can be implemented using any architecture designed to pass data or control information, or both, between the processor (such as a microprocessor, communication and network processor), system memory, peripheral devices, and any other hardware components in the system. For example, the communication fabric 604 can be implemented using one or more buses or crossbar switches.

[0053] Memory 602 and persistent storage 605 are computer-readable storage media. In this embodiment, memory 602 includes random-access memory (RAM). Generally, memory 602 can include any suitable volatile or non-volatile computer-readable storage media. Cache 603 is a high-speed memory that improves the performance of processor 601 by holding recently accessed data and data close to recently accessed data from memory 602.

[0054] Program instructions and data (e.g., software and data x10) used to implement embodiments of the present invention may be stored in persistent storage 605 and memory 602 for execution by one or more of the respective processors 601 via cache 603. In one embodiment, persistent storage 605 includes a magnetic hard disk drive. As an alternative to, or in addition to, a magnetic hard disk drive, persistent storage 605 may include a solid-state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0055] The media used by the persistent storage 605 may also be removable. For example, a removable hard drive may be used for the persistent storage 605. Other examples include optical and magnetic disks, thumb drives, and smart cards inserted into a drive for transfer to another computer-readable storage medium, which is also part of the persistent storage 605. The display component 111 may be stored in the persistent storage 605 for access or execution, or both, by one or more of the respective processors 601 via the cache 603.

[0056] In these examples, the communication unit 607 provides communication with other data processing systems or devices. In these examples, the communication unit 607 includes one or more network interface cards. The communication unit 607 may provide communication by using either or both physical communication links and wireless communication links. Program instructions and data used to implement embodiments of the present invention (e.g., display component 111) may be downloaded to persistent storage 605 via the communication unit 607.

[0057] The input / output interface 606 enables the input and output of data to and from other devices that may be connected to each computer system. For example, the input / output interface 606 may provide a connection to an external device 608 such as a keyboard, keypad, touchscreen, or any other suitable input device, or a combination thereof. The external device 608 may also include, for example, a portable computer-readable storage medium such as a thumb drive, a portable optical or magnetic disk, and a memory card. Program instructions and data used to practice embodiments of the present invention (e.g., display component 111) may be stored on such a portable computer-readable storage medium and loaded into persistent storage 605 via the input / output interface 606. The input / output interface 606 also connects to a display 609.

[0058] The display 609 provides a mechanism for displaying data to the user, and may be, for example, a computer monitor.

[0059] The programs described herein are identified based on the application in which they are intended to be implemented in particular embodiments of the present invention. However, it should be understood that the nomenclature of specific programs herein is used merely for convenience, and therefore the present invention should not be limited to use only in the specific application identified, implied, or both by such nomenclature.

[0060] The present invention may be a system, method, or computer program product, or a combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium (or a plurality of computer-readable storage media) having computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0061] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes, namely, portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random-access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved raised structures on which instructions are recorded, and any preferred combination thereof. The computer-readable storage media used herein should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through optical fiber cables), or electrical signals transmitted through wires.

[0062] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and transfers those computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0063] The computer-readable program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk® and C++, and procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may be executed as a standalone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or a server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, to carry out aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute a computer-readable program instruction by personalizing the electronic circuit using state information of the computer-readable program instruction.

[0064] This specification describes aspects of the present invention with reference to flowcharts, block diagrams, or both, of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in the flowcharts, block diagrams, or both, and combinations of blocks in the flowcharts, block diagrams, or both, can be implemented by computer-readable program instructions.

[0065] These computer-readable program instructions may be provided to a general-purpose computer, a dedicated computer, or a processor of another programmable data processing device to create a machine, such that instructions executed via the processor of a computer or other programmable data processing device create means for performing functions / operations specified in one or more blocks of a flowchart or block diagram or both. These computer-readable program instructions may also be stored on a computer-readable storage medium on which the instructions are stored, such that the storage medium contains a product containing instructions that perform a manner of function / operation specified in one or more blocks of a flowchart or block diagram or both, and can instruct a computer, a programmable data processing device, or other device or a combination thereof to function in a particular manner.

[0066] Computer-readable program instructions may also be loaded into a computer, other programmable data processing device, or other device to create a computer execution process in which instructions executed on a computer, other programmable device, or other device perform a function / action specified in one or more blocks of a flowchart or block diagram, or both, causing the computer, other programmable device, or other device to execute a series of operational steps.

[0067] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in a block may be performed in an order different from the order shown in the figure. For example, depending on the functionality involved, two blocks shown consecutively may actually be executed substantially simultaneously, or they may be executed in reverse order. It should also be noted that each block in a block diagram or flowchart or both, and combinations of blocks in a block diagram or flowchart or both, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or performs a combination of dedicated hardware and computer instructions.

[0068] While various embodiments of the present invention have been presented for illustrative purposes, this description is not intended to be exhaustive or to limit the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The terminology used herein has been selected to best describe the principles of the embodiments, practical applications, or technical improvements beyond the technology available on the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0069] The corresponding structures, materials, actions, and equivalents of all means-plus-function elements or step-plus-function elements within the attached claims are intended to include any structures, materials, or actions for performing a function in combination with other specifically claimed elements. The description of the present invention is presented for illustrative and explanatory purposes only and is not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will become apparent to those skilled in the art without departing from the scope and spirit of this disclosure. These embodiments have been selected and described to best illustrate the principles and practical applications of the invention and to enable those skilled in the art to understand the invention in terms of various embodiments with various modifications to suit specific intended uses.

[0070] Finally, the proposed concept in a preferred embodiment can be concisely summarized in the following clauses. 1) The AI ​​system takes various IoT sensor data as input along with video in order to generate augmented video using a generative adversarial network. 2) The AI ​​system uses various time encoders to encode various spatiotemporal data into latent space. 3) The proposed AI system determines various factors such as environmental conditions, the user's medical profile, lighting conditions, light frequency, object rotation speed, and spatiotemporal object visibility score. 4) The proposed AI system learns to reduce the amount of noise by combining multiple data sources with controllable parameters. 5) The proposed AI acquires potentially controllable parameters for generating augmented videos, such as effect enable / disable, effect resolution enhancement, effect cancelout, and effect merge / split, by training discriminators and generators using various loss functions. Loss functions include visual illusion loss, effect cancelout loss, and effect merge / split loss. 6) The output of the proposed system can be integrated with an augmented reality system to overlay the GAN-generated video as an additional layer. 7) The proposed system dynamically updates model parameters and uses reinforcement learning to fine-tune them based on reward scores provided by the user. Possible reward scores are provided by effect level, spatiotemporal video frame labeling, or marking of secondary effects of the stroboscopic effect.

Claims

1. A method for correcting a stroboscopic effect in real time from one or more video clips by utilizing machine learning of generators and discriminators of a generative adversarial network (GAN) through computer information processing, Identifying one or more video clips from the user, Determining visual data and, Based on the aforementioned visual data, determine the strobe effect settings, which include, as recommended actions, either enabling / disabling the strobe effect, enhancing the stereoscopic effect, or canceling the stereoscopic effect. Perform the actions recommended in the strobe effect settings for one or more video clips, Outputting the video clip based on the updated strobe effect settings, Receiving user feedback and A method that includes this.

2. Determining visual data is The collection of the aforementioned visual data, wherein the visual data includes environmental data and user data. Encoding visual data into latent space and The method according to claim 1, further comprising:

3. The method according to claim 1 or 2, wherein determining the strobe effect settings is based on factors including environmental conditions, the user's medical profile, lighting conditions, light frequency, object rotation speed, and a visibility score which is an index indicating the spatiotemporal visibility of objects tracking moving objects in the video.

4. The method according to claim 1 or 2, wherein the strobe effect setting further includes enabling / disabling the strobe effect, enhancing the stereoscopic effect, and canceling the stereoscopic effect.

5. Applying a strobe effect setting to one or more video clips is: To generate augmented videos for use in augmented reality (AR) by training discriminators and generators using a loss function that produces a more authentic output in subsequent iterations based on the aforementioned feedback. The method according to claim 1 or 2, further comprising:

6. Outputting the video clip based on the updated strobe effect settings is, The process involves overlaying GAN-generated video as an additional layer with an augmented reality (AR) system. The method according to claim 1 or 2, further comprising:

7. Receiving feedback from users is Updating model parameters, Using reinforcement learning, the model parameters are fine-tuned based on the reward score provided by the user. The method according to claim 1 or 2, further comprising:

8. A computer-readable computer program for correcting a strobe effect in real time from one or more video clips, wherein the computer program comprises: The program includes program instructions stored in one or more computer-readable storage media, wherein the program instructions are: A program instruction for identifying one or more video clips from the user, Program instructions for determining visual data, A program instruction for determining strobe effect settings, which include, as recommended actions, enabling / disabling the strobe effect, enhancing the stereoscopic effect, or canceling the stereoscopic effect, based on the aforementioned visual data, A program instruction for performing the action recommended by the strobe effect setting for one or more video clips, A program instruction for outputting the video clip based on the updated strobe effect settings, Program instructions for receiving user feedback and A computer program that includes [this].

9. A computer-readable storage medium recording the computer program described in claim 8.

10. A computer system for correcting a strobe effect in real time from one or more video clips, wherein the computer system comprises: One or more computer processors, One or more computer-readable storage media, Program instructions stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors and The program instructions include, A program instruction for identifying one or more video clips from the user, Program instructions for determining visual data, A program instruction for determining strobe effect settings, which include, as recommended actions, enabling / disabling the strobe effect, enhancing the stereoscopic effect, or canceling the stereoscopic effect, based on the aforementioned visual data, A program instruction for performing the action recommended by the strobe effect setting for one or more video clips, A program instruction for outputting the video clip based on the updated strobe effect settings, Program instructions for receiving user feedback and A computer system that includes [a certain feature].

Images