Systems and methods for real-time data analysis and device remote control - Patents.com

Abstract

Disclosed are systems and computer software for an AR device having a display configured to display augmented, mixed, or virtual reality images to a user. The AR device may display, on the display, a virtual control panel including a representation of one or more targets. The AR device may render outputs of projection devices intended to be aimed at the one or more targets. The AR device may enable the user to control one or more operations of the projection devices based on inputs received from the user's interaction with the virtual control panel.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to and the benefit of U.S. patent application Ser. No. 17 / 894,958, filed Aug. 24, 2022, entitled "SYSTEM AND METHOD FOR REAL-TIME DATA ANALYSIS AND DEVICE REMOTE CONTROL," the contents of which are incorporated herein by reference. [Background technology]

[0002] Augmented reality (AR) blends the digital and real worlds, overlaying information onto the user's field of view. Virtual reality (VR) and mixed reality (MR) create virtual objects in the user's field of view. These technologies can enhance user experiences and are used in a variety of industries, including entertainment, travel, education, and healthcare. Summary of the Invention

[0003] In one aspect, the disclosed system comprises an AR device having a display configured to display augmented reality, mixed reality, or virtual reality images to a user, at least one programmable processor, and a non-transitory machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations including: displaying on the display a virtual control panel including depictions of one or more targets; rendering output of a projection device intended to be directed at the one or more targets; and controlling one or more operations of the projection device by the user based on input received from the user's interaction with the virtual control panel.

[0004] In some variations, the rendering step includes a representation of a region of directed energy from the radiating device, the rendered region including a main lobe and one or more side lobes of the directed energy. The rendering step includes obtaining settings for the radiating device and / or static parameters of the radiating device, calculating a radiation field resulting from the radiating device using a radiation simulator based on at least the settings and / or the static parameters, and displaying an electromagnetic field strength associated with the output.

[0005] In some variations, the one or more manipulations of the emitting device include controlling the emitting device to project the output to affect the one or more targets. The controlling may be based on the system interpreting one or more hand gestures, gaze or voice commands, or button presses on the AR device or control peripheral. The rendering may be updated based on the user's control of the emitting device. The manipulation of the emitting device includes one or more of positioning the emitting device, selecting an output type, setting a frequency of the output, setting an intensity of the output, setting a direction of the output, or setting a time for directing and projecting the output at the one or more targets.

[0006] In some variations, the operations include displaying identification information of the one or more targets on the display of the AR device, and the illumination of the output is based on the identification information of the one or more targets. The displayed identification information may include one or more of type, size, attached components, direction of movement, brand, friendly classification, or unfriendly classification. The operations further include obtaining map data of an area including the one or more targets, obtaining real-time target location information, and displaying a real-time overview including the map data and a representation of the one or more targets on the display. The system may display coordinate information of the one or more targets.

[0007] In some variations, the coordinate information may include one or more of a target's latitude, longitude, and elevation. The coordinate information may be obtained from GPS, RADAR, or LIDAR data. The coordinate information may be obtained from coordinate information of the one or more targets relative to a reconnaissance UAV. The coordinate information obtained from coordinate information of the one or more targets relative to the reconnaissance UAV may be combined with coordinate information obtained from GPS, RADAR, or LIDAR data.

[0008] In some embodiments, the system further comprises a sensor configured to detect one or more attributes of the one or more targets and an imaging device configured to image the one or more targets. The operations further comprise acquiring real-time sensor data and / or real-time image data from the sensor and identifying the targets from the real-time sensor data and / or real-time image data. The detected one or more attributes of the one or more targets include presence, environmental data surrounding the one or more targets, velocity, acceleration, or coordinates. The one or more targets include imaging of one or more of the one or more targets themselves, attached parts, or radiation.

[0009] In some embodiments, the system may determine information about the one or more targets based on the sensor data, the information including one or more of: presence, speed, acceleration, coordinates, guided route, satellite source, distance from the emitting device, environmental data surrounding the one or more targets, temperature, field of view (FOV). The system may determine information about the one or more targets based on the image data, the information including one or more of: information identifying a type of the one or more targets, information identifying one or more characteristics, information identifying one or more devices or systems associated with the one or more targets, battery information, power information, type, brand, size, or shape.

[0010] Examples of the presently disclosed subject matter may include, but are not limited to, methods consistent with what is described herein and articles including tangibly embodied machine-readable media configured to cause one or more machines (e.g., computers, etc.) to perform processes that implement one or more of the described features. Similarly, computer systems including one or more processors and one or more memories coupled to the one or more processors are contemplated. The memory may include a computer-readable storage medium and contain, encode, store, etc., one or more programs that cause the one or more processors to perform one or more of the processes described herein. A computer-implemented method consistent with one or more examples of the presently disclosed subject matter may be implemented by one or more data processors. The data processors may reside in a single computer system or across multiple computer systems. Such multiple computer systems may be connected to one another to exchange data and / or commands or other instructions, etc., via one or more connections or by direct connections between one or more of the multiple computer systems. The one or more connections may include, but are not limited to, connections over a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.).

[0011] Details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features and advantages of the subject matter described herein will become apparent from the specification and drawings, and from the claims. While features of the subject matter disclosed herein have been described for illustrative purposes in connection with certain embodiments, it will be readily understood that such features are not intended to be limiting. The claims following this disclosure define the scope of the subject matter for which protection is sought. [Brief explanation of the drawings]

[0012] [Figure 1]1 illustrates a system that utilizes an AR device to display information about a radiation device according to certain aspects of the present disclosure.

[0013] [Figure 2] 1 illustrates a system utilizing an AR device to control a radiation device according to an embodiment of the present disclosure.

[0014] [Figure 3] 1 illustrates an AR device and a virtual control panel interfaced to a controller according to an embodiment of the present disclosure.

[0015] [Figure 4] 1 illustrates a software architecture of a controller according to certain aspects of the present disclosure.

[0016] [Figure 5] 1 illustrates an exemplary AR display of target identification and target information according to certain aspects of the present disclosure.

[0017] [Figure 6] 1 illustrates an exemplary AR display of a radiation emission pattern and an identified target, according to certain aspects of the present disclosure.

[0018] [Figure 7] 1 illustrates a graphical user interface (GUI) of an AR device that displays an augmented field of view as a virtual control panel to a user, according to certain aspects of the present disclosure.

[0019] [Figure 8] 1 illustrates an example GUI for an AR device displaying control options and target information in accordance with certain aspects of the present disclosure.

[0020] [Figure 9] 1 illustrates an exemplary GUI for an AR device displaying overview, control options, and target information in accordance with certain aspects of the present disclosure.

[0021] [Figure 10A] 1 illustrates controlling a radiation device using an AR device to direct radiation to a target, according to an aspect of the present disclosure. [Figure 10B] 1 illustrates controlling a radiation device using an AR device to direct radiation to a target, according to an aspect of the present disclosure. [Figure 10C] 1 illustrates controlling a radiation device using an AR device to direct radiation to a target, according to an aspect of the present disclosure. [Figure 10D] 1 illustrates controlling a radiation device using an AR device to direct radiation to a target, according to an aspect of the present disclosure.

[0022] [Figure 11] FIG. 1 is a simplified flow diagram illustrating the display of information and control of an emission device in an AR device, according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0023] The present disclosure provides, among other things, systems, methods, and computer programs that utilize augmented reality, mixed reality, and / or virtual reality technologies to facilitate the display and / or control of graphics and information related to hardware configured to direct energy from an irradiation device to one or more targets. Such control can, for example, be more responsive to user input and enable the user to understand the real-time environment surrounding the controlled device, the target of the controlled device, or the user himself / herself. Alternatively, real-time control can be facilitated with continuously updated information presented to the user regarding the controlled device, target, etc.

[0024] Augmented reality generally refers to the overlay of virtual information or images onto a user's real-world view through a display device such as glasses, goggles, a head-up display (HUD), or a teleprompter. Mixed reality generally allows real-world actions to affect virtual elements; for example, graphics or information can change based on gaze or hand movements, or virtual graphics / information can change based on the user's position. Virtual reality generally refers to an immersive environment in which all displayed features are virtual, including the environment perceived by the user.

[0025] As used herein, the terms augmented reality (AR), mixed reality (MR), and virtual reality (VR) are collectively referred to herein (for shorthand purposes only) as augmented reality. However, the present disclosure contemplates that any of the features described with respect to AR may be used identically or similarly in an MR or VR environment. For example, embodiments involving the depiction of directed energy may be realized in a pure AR mode (depicted in a real-world environment), an MR mode (the energy is depicted against a real-world background and its appearance may change based on the user's movement), and / or a VR mode (the energy is depicted in an entirely virtual environment).

[0026] As used herein, the term "radiation device" may include directed energy sources such as microwave or other electromagnetic radiation emitters, lasers, radio detection and ranging (RADAR), sound, particles (ions / plasma / fluids), etc. Examples of radiation devices include high power microwave (HPM) systems, directed energy weapons, radio frequency (RF) systems, manufacturing lasers, etc.

[0027] As used herein, the term "target" may refer to any object at which energy from a radiation device is directed. Examples of targets may include unmanned aerial vehicles (UAVs), surfaces to be cut or melted as part of a manufacturing process, etc.

[0028] FIG. 1 illustrates a system utilizing an AR device 204 to display information related to an emission device according to certain aspects of the present disclosure. More specifically, FIG. 1 illustrates a system 100 showing an exemplary configuration of various devices in communication with the AR device 204. In some embodiments, the system 100 is comprised of one or more emission devices 201A, a control device 201B, and an AR device 204. FIG. 1 also illustrates a simplified depiction of directed energy emitted from 201A in the form of lobes 210 that may be directed toward one or more targets, as described further herein. The system may optionally include sensors 206 (e.g., optical sensors, environmental sensors, RADAR, laser range finders, etc.) that may facilitate display of the AR environment or other display information and facilitate target identification and / or tracking. Also illustrated in FIG. 1 are example targets 208-1, 208-2, ..., 208-n. In some embodiments, targets 208-1, 208-2, ..., 208-n may be any number (n) of unmanned aerial vehicles (UAVs), also referred to herein as a group of UAVs. In some other embodiments, targets 208-1, 208-2, ..., 208-n may be objects on a manufacturing site.

[0029] The emitting device 201A may be configured to emit radiation, such as electromagnetic radiation or ultrasound. The emitting device 201A may be a radio frequency (RF) emitter, microwave emitter, laser, ultrasound emitter, sound emitter, etc. The control device 201B may include a signaling device (e.g., a light, transponder, alarm, etc.), a device such as a switch that provides power to the emitting device, and a position control device (e.g., a device for steering the emitting device, such as by rotating it) that controls the position, azimuth, and elevation of the emitting device. The emitting device 201A, the control device 201B, and the AR device 204 may be interconnected via wired and / or wireless connections. Communication between the emitting device 201A, the control device 201B, and the AR device 204 may be achieved by an integrated communication device or one or more external processors or servers executing software programs.

[0030] The AR device 204 may include a display device for presenting AR information similar to real-world information. In some implementations, the display device may comprise a liquid crystal display, a light-emitting diode display, a plasma display, a projection display, or a holographic projector. In various implementations, the AR device 204 may comprise a head-mounted display, such as a near-eye display, glasses, or goggles. In various implementations, the AR device 204 may include a depiction of a mobile device. The AR device 204 may be configured in this manner. In addition to the display device, the AR device 204 may include additional computers, cables, control peripherals (e.g., joysticks, hand sensors, body movement / position sensors, etc.), and / or other devices configured to generate graphics and / or other virtual information for display.

[0031] The AR device 204 may be connected to the emission device 201A and other control devices 201B using one or more electronic processors and one or more application programs. The AR device 204 may be configured to communicate with one or more emission devices 201A and other control devices 201B via a wired or wireless network. A wired network may comprise a conventional Ethernet, a local area network (LAN), fiber optic cable, etc. A wireless network may comprise a cellular system, a wireless application protocol (WAP), a Wi-Fi access point (Wi-Fi), a near field connection (NFC), etc.

[0032] In some embodiments, the system may include one or more sensors 206 configured to detect one or more attributes of the targets 208-1, 108-2, ..., 208-n. The one or more sensors 206 may include imagers configured to image the targets 208-1, 108-2, ..., 208-n. The system 100 may be configured to acquire sensor data in real time and / or acquire image data from the sensors 206 in real time. The system 100 may include an electronic processor configured to identify the one or more targets based on the sensor data.

[0033] The sensors 206 can be configured to provide information about the targets 208-1, 208-2, ..., 208-n to the AR device 204 in sufficient real-time (e.g., within seconds, milliseconds, microseconds, etc.). The sensors 206 may be electrical / optical, infrared, radio, vibration, location, temperature, radio detection and ranging (RADAR), etc. The sensors 206 may be connected to the AR device 204 using a wired or wireless connection. A user of the AR device 204 can view information received from the sensors 206 on a display of the AR device 204 and cause the system 100 to perform functions using one or more controls associated with the AR device 204. For example, a user can direct the emission device 201A to emit radiation toward one or more of the targets 208-1, 208-2, ..., 208-n using hand gestures, voice commands, and / or physical controls (e.g., activating a button or moving a joystick) in association with the AR device 204, etc.

[0034] Attributes of detected targets 208-1, 108-2, ..., 208-n may include presence (e.g., detecting that a target is present), environmental data of or around one or more targets, velocity, acceleration, coordinates, etc. Imaging of a target may include imaging of the target itself, attached elements, emissions, etc.

[0035] In some implementations, one or more electronic processors associated with sensor 206 may be configured to execute computer algorithms / processes / software programs for detecting and identifying attributes of targets 208-1, 208-2, ..., 208-n. Detection may include determining and providing real-time information related to the targets. Such information may be utilized to update status information of the targets displayed by AR device 204. Based on the information displayed on the display device, a user may control options (e.g., using hand gestures) on a virtual control panel 204A shown in FIG. 2 displayed on the display to obtain or present status information of emitter device 201A, targets 208-1, 208-2, ..., 208-n, etc.

[0036] The system may also determine information about targets 208-1, 108-2, ..., 208-n based on the sensor data, which may include presence, speed, acceleration, coordinates, guided route, satellite source, distance from the emitter, environmental data around one or more targets, temperature, field of view (FOV), etc.

[0037] The system can also determine information about the targets 208-1, 108-2, ..., 208-n based on the image data. This information can include identifying the type of target (e.g., if one or more targets are UAVs), identifying one or more characteristics of the target, identifying one or more devices (e.g., cameras) or systems associated with the target, battery, power information, type, brand, size, shape, etc.

[0038] Sensors 206 are optional and are not necessarily physically included in all embodiments. For example, a particular system that does not have its own sensors may receive sensor data via the internet or other communication channel. In other embodiments, a system may not include sensors 206 or utilize sensor data. For example, such a system may rely on visual guidance from a user to direct the emitter rather than relying on automatic target tracking provided by a sensor.

[0039] The information about targets 208-1, 208-2, ..., 208-n may include the target's type, shape and color, dimensions, any accessories associated with the target, specifications, direction, coordinates, speed, and target identification information for determining an association between the target and the user, such as a passenger aircraft, a UAV, a bird, or any other flying object. The association of a target may be defined as the target being friendly or unfriendly. A friendly target is a target that does not appear to pose a potential risk to the user or that appears to belong to the user's territory. Conversely, an unfriendly target is a target that does not appear to belong to the user or that appears to be a potential risk to the user.

[0040] FIG. 2 illustrates a system utilizing an AR device to control emission devices according to certain aspects of the present disclosure. FIG. 2 shows a system 200 depicting an exemplary virtual control panel 204A generated by an AR device 204 having a display configured to display augmented, mixed, or virtual reality images to a user. Thus, an embodiment of the system 200 may be configured to display a virtual control panel on the display, which may include images of one or more targets 208-1, 208-2, ..., 208-n. In some embodiments, the AR device 204 may also provide the user with information related to the targets displayed on the display of the AR device 204. The user can use the information about the targets to control one or more emission devices 201 via the AR device 204 to take one or more actions. In some implementations, the output of emission devices intended to be (or currently being) aimed at one or more targets may be rendered on the display. Details of such rendering are provided herein, for example, with reference to FIGS. 6, 7, and 10.

[0041] Some embodiments of the AR device 204 may include a virtual control panel 204A that may display one or more control options for activating and deactivating the emission device 201A and / or other control devices 201B. The AR device 204 may communicate with the sensor 206 via a wired or wireless connection. The situation awareness sensor 206 may sense and receive target information using software or a program embedded in the sensor and may provide information about the targets 208-1, 208-2, ..., 208-n to the AR device 204 and / or the hardware controller 212. Information about the targets 208-1, 208-2, ..., 208-n may be provided to the AR device 204 and / or the hardware controller 212 via wired or wireless communication. A display of the AR device 204 may display target information alerts from the situation awareness sensor 206 to the user. Based on the information displayed on the display of the AR device 204, the user can activate a control panel menu on the control panel 204A by using hand gestures, voice commands, pressing any switch displayed on the display of the AR device 204, etc.

[0042] The control panel 204A displays one or more controls for displaying real-time status of the emitting device 201A, other control devices 201B, targets 208-1, 208-2, ..., 208-n, etc. The real-time status of the emitting device 201A and control devices 201B may include charge level, operating status / power status, frequency range, direction information, position information, altitude, control switch configuration, beam intensity level, beam position, etc.

[0043] In some embodiments, system 200 may include a hardware controller 212 that can receive information from connected devices, such as sensor 206, control device 201B, emission device 201A, and AR device 204. Hardware controller 212 may process the information and provide one or more commands to the connected devices. In another embodiment, hardware controller 212 can process information from the connected devices, generate control information, and provide such control information to a user on AR device 204.

[0044] The system may also allow a user to control one or more operations of the emission device 201A based on input received through the user's interaction with the virtual control panel 204A. For example, some operations may include controlling the emission device to emit output to deactivate a target. In various embodiments, control of the emission device may be based on the system's interpretation of one or more hand gestures, gaze or voice commands, button presses on an AR device or control peripheral, etc. As a specific example, optical sensors mounted on goggles or a headset may be used to detect and interpret gaze movements. Other peripherals, such as joysticks, controllers, keyboards, or gloves equipped with sensors or motion detection hardware, may also provide input that can be interpreted as commands by the system. In some embodiments, generating control instructions may include positioning the emission device, selecting the type of output (e.g., a frequency band such as microwave, infrared, or a laser wavelength band such as UV, DUV, or EUV), setting the frequency of the output, setting the intensity of the output, setting the direction of the output, or setting the time for irradiating and directing the output to one or more targets.

[0045] One example of how the virtual control panel 204A can be used to control the emission device 201A may include the AR device 204 performing eye gaze tracking and / or hand movement tracking, allowing the AR device 204 to interpret gaze / hand movements as interactions with virtual elements. Another example may include moving slider(s) representing the direction and / or vertical angle of the emission device 201A, detecting the point at which the user is looking in virtual or augmented space and using that point to indicate the desired point of radiation delivery, determining a similar point based on a hand / eye selected target in the augmented environment, etc. Such manipulation / selection of virtual elements can be translated into commands for the hardware controller 212 (e.g., rotate the emission device 201A by 10 degrees, orient the emission device 201A to direct radiation at the center of the selected targets, etc.). The controller may then translate this command into a control instruction that can be communicated to the emission device 201A. For example, such instructions may be to activate a particular servo in the radiating device to perform a requested 10-degree turn, or to modify the frequency, power, phase, or other settings of one or more antennas of the radiating device 201A so that radiation meeting the requested parameters is emitted. In some embodiments, determining the requested radiation output may include utilizing an accessible lookup table or other pre-existing calculation to provide the necessary instructions and / or render the radiating device 201A of the expected output. In another embodiment, calculations / simulations may be performed in response to a user request to determine the amount of adjustment needed to the radiating device 201A and / or a representation of the simulated radiation output. In embodiments in which the status of the radiating device 201A is provided to the user, the user may view a rendering or other representation of the radiating device 201A after or during execution of the instructions by the real radiating device 201A.

[0046] In another variation, a user of the AR device 204 can activate and / or control the radiation device 201A. The user can use various functions via the virtual control panel 204A to activate the radiation device 201A or other control devices 201B to affect one or more targets 208-1, 208-2, ..., 208-n. In a manufacturing environment, affecting a target can include cutting, welding, melting, or other manufacturing processes. In some embodiments, radiation from the radiation device 201A can deactivate one or more targets 208-1, 208-2, ..., 208-n.

[0047] As used herein, the term "deactivate" means effectively causing a target to cease one or more operations. For example, deactivating a target may include interfering with or overloading one or more circuits of the target to cause the target to land or cease a desired operation (e.g., surveillance, navigation, weapons employment, propulsion, etc.). Deactivation may also include physically damaging the target, but does not necessarily have to cause damage.

[0048] In one embodiment, a user can use the virtual control panel 204A of the AR device 204 to provide control commands to emit radiation from the irradiation device 201A and deactivate targets 208-1, 208-2, ..., 208-n. To control the direction and emission of radiation from the irradiation device 201A, the control commands can be processed and executed by the hardware controller 212. For example, the AR device 204 can provide various control functions for positioning / steering the irradiation device 201A so that the irradiation device 201A irradiates any (or any combination) of targets 208-1, 208-2, ..., 208-n. In some embodiments, the hardware controller 212 can generate available control commands based on information received from the sensor 206, the other control devices 201B, and the irradiation device 201A. The hardware controller 212 can then provide control options on the virtual control panel 204A of the AR device 204 for the user to control and position / steer the emission device 201A and any other control devices 201B to deactivate the targets 208-1, 208-2, 208-n.

[0049] The frequency (e.g., frequency in Hz) and direction of the radiation from emission device 201A can be controlled by a user from a virtual control panel 204A of AR device 204. For example, as seen in Figure 2, the radiation from light-emitting device 201A can be steered in various directions depending on the speed and position of targets 208-1, 208-2, and 208-n. These and other features are further described later in the specification with reference to other figures.

[0050] In some embodiments, the display of the AR device 204 can display a rendering of the radiation emitted from the emitting device 201A, where the radiation may be invisible to the human eye. The sensor 206 can be a microwave camera or other detector / camera sensitive to the wavelength / frequency range of the light emitting device 201A, and can image the actual radiation of the light emitting device and provide imaging data for rendering in the AR device 204.

[0051] 3 illustrates a virtual control panel 204A interfaced with an AR device and a controller. More specifically, FIG. 3 illustrates an example implementation in which a user 302 and an AR device 204 use the virtual control panel 204A to communicate with an emission device 201A, other control devices 201B, and a hardware controller 212 (e.g., via a wireless communication connection 306).

[0052] The AR device 204 may be configured to display one or more control applications, control information, status information, characteristic information, or information related to the emission device 201A and other control devices 201B on a GUI displayed on the display 304. The AR device 204 may generate a display 304 including a virtual control panel 204A in a virtual space (e.g., the virtual control panel 204A may be configured in a portion of the user's field of view). As shown in FIG. 3 , the virtual control panel 204A may be comprised of multiple sections that can display a variety of different information related to the emission device 201A, targets, etc. The information displayed on the display 304 can be accessed using applications, menus, and associated hardware / software to allow the user to control and / or interact with the operation of the emission device 201A and to access / manipulate the information via the AR device 204. Further details of various embodiments of the virtual control panel 204A are described throughout this disclosure.

[0053] FIG. 4 illustrates the software architecture of the hardware controller 212 according to certain aspects of the present disclosure. FIG. 4 illustrates a simplified diagram of the AR device 204, the hardware controller 212, the emission device 201A, the other control devices 201B, the sensor 206, and the imaging device 402. In some embodiments, the imaging device 402 can capture an image of a target, and the sensor 206 can acquire detection data of the target. In some embodiments, the imaging device 402 and the sensor 206 can be integrated into a device / unit. Information captured by the imaging device 402 and the sensor 206, along with the current state of the emission device 201A and the other control devices 201B, can be processed in the hardware controller 212 or other computer system. The controller or other computer system may include a database or repository accessed using one or more software programs. The controller may include or have access to software modules such as databases related to the beamformer 212A, drone resources 212B, GPS information 212C, sensor information 212D, or imaging database 212E. In some alternative embodiments, when the emitter 201A is configured as a manufacturing laser, the databases can contain information related to objects and systems within the manufacturing facility. For example, these databases, and various combinations of databases, can be employed to process real-time information received from either the emitter 201A, other controllers 201B, sensors 206, or imaging device 402. This processing can include identifying targets (e.g., UAVs, airplanes, birds, objects and systems within the manufacturing facility), characteristic information (e.g., type of target UAV, friendly or unfriendly, size, direction, speed, altitude, coordinate information, brand information, color, characteristics of objects and systems within the manufacturing facility, etc.), beam information (e.g., what type of emission is needed to address the target, radiation emission intensity and timing, steering and positioning coordinates, activation status, etc.).The processed information is then provided from the hardware controller 212 to the AR device 204, which displays graphics and control options on a virtual control panel 204A, allowing the user to view and control the operation of the emission device 201A.

[0054] 5 shows an exemplary AR display of target identification and target information. The imaging device 402 and sensor 206 (e.g., as shown in FIG. 4) can collect image information and sensor data, which may be processed by the hardware controller 212 to determine, for example, whether the target is a friendly or unfriendly UAV. The identification process may identify and classify the characteristics of the UAV using various software components, command scripts, UAV libraries / databases, GPS coordinates, image recognition applications, etc. Information about the target can then be provided to the user of the AR device 204. In some embodiments, the AR device and its applications can receive RADAR or other sensor data to update the target's location over time.

[0055] In some implementations, information about the target may be collected by a reconnaissance UAV. The reconnaissance UAV may collect information about targets in its vicinity and transmit the collected information, along with the location of the reconnaissance UAV, to a hardware controller 212 or other computer system associated with the emission device 201A. The hardware controller 212 or other computer system may combine the information from the reconnaissance UAV with data from other sensors (e.g., sensor 206), imagers (e.g., imager 402), and / or associated RADAR data to improve the accuracy of determining the target's location. The location of the target and / or the location of the reconnaissance UAV may be displayed on the AR device 204. The reconnaissance UAV may be designated as a friendly target that is not affected by radiation from the emission device 201A.

[0056] The top panel of FIG. 5 illustrates identifying potential targets. Three potential targets are shown, with the two potential targets on the left (510, 520) representing aerial elements, such as, for example, a bird, a friendly aircraft, or a friendly drone. The leftmost potential target 510 is depicted as unidentified. The center potential target 520 is depicted as, for example, a friendly aerial element. To assist the user, some embodiments may display a graphical indicator 522 that indicates the potential target 520 as a friendly element or otherwise an element that should not receive direct energy from the emitting device. In the illustrated example, the graphical indicator 522 is depicted as a box. However, any suitable graphical indicator may be utilized to indicate the classification of the potential target 520. For example, the targets 510 and 520 may be displayed in different colors based on their classification. Arrows, text, or other graphics may be displayed above the targets 510 and 520 based on their classification. The rightmost aerial element is depicted as target 530, which is an identified drone but does not currently display a friendly / unfriendly graphical indicator.

[0057] The bottom panel shows a graphical output that can be displayed to a user in which a target 530 has been identified as a drone with the addition of a second graphical indicator 532. Additionally, certain embodiments may include target data 534 that provides information about the target, such as its identification, location, vector, etc., either automatically or in response to a user command.

[0058] 6 shows an exemplary AR display of the radiation pattern and identified targets. As shown in FIG. 6, in some embodiments, the system is configured to render a region of directed energy from the emitting device 201A.

[0059] In some embodiments, to perform the rendering, the system can obtain radiating device settings, static parameters of the radiating device 201A, power system, etc., and environmental conditions (e.g., temperature, humidity, cloud cover, etc.), which can be set by a user (e.g., via the virtual control panel 204A). The radiating device settings, static parameters, environmental conditions, etc., can be input into a radiation simulator, which calculates the radiation field resulting from the radiating device. Calculations regarding the radiation field can include the radiation pattern, elevation and azimuth angles of the radiation field, radiated power, etc. The radiation simulator can be performed by an electronic processing system associated with the radiating device 201A or the AR device 204. Such simulations can be based on the power / direction of the antenna output and the resulting electromagnetic field. In some embodiments, the system can render 2D regions or 3D surfaces representing predetermined intensities, deactivation effects / probabilities, etc. The radiation simulator can also provide electromagnetic field vectors, where the direction of the electromagnetic field at a target may be related to the effectiveness of the electromagnetic field in affecting the target's circuitry or operation.

[0060] In various embodiments, a user can specify one or more targets or areas in the surrounding environment to be avoided. The radiation simulator can calculate a radiation field that reduces the electric field strength in one or more targets or areas specified by the user to a harmless level. For example, the radiation simulator can generate a radiation field with nulls (0) in one or more targets or areas specified by the user. In various embodiments, a user may use the AR device 204 to provide input regarding the frequency configuration of the emitted radiation, waveform parameters of the emitted radiation (e.g., pulse width, pulse duration, duty cycle, etc.). In some implementations, an electronic processing system associated with the AR device 204 or the emitting device 201A can employ computer vision algorithms in conjunction with other sensor data to classify one or more targets. The classification data can provide the emitting device 201A with information to assist it in generating a radiation pattern that best engages the target.

[0061] In embodiments in which the radiating device 201A is an RF emitter, a particular rendering may depict the particular structure of the radiation emission from the radiating device 201A. Such a rendered region may consist of a main lobe of directed energy and one or more side lobes. For example, the main lobe 610 may be the region where the emitted radiation or directed energy is generally greatest. Similarly, a side lobe 612 may represent a region where significant radiation may be emitted but may not be used for target deactivation or other primary use of the radiating device 201A. In contrast, such a side lobe 612 may indicate a region that should be optionally avoided to avoid inadvertently affecting friendly airborne elements. While not shown, the rendered region may include various side lobes or gaps between the main lobe 610 and the side lobes corresponding to nulls in the radiation pattern.

[0062] In some embodiments, the rendering may be updated based on the user's control of the radiating device. For example, based on steering the radiating device, the depiction of the radiation pattern may change (e.g., from a side view to a more end view). In other embodiments, the size, shape, intensity, etc. may be updated based on the provided or planned radiation emission. For example, if the frequency, power, or waveform characteristics of the RF signal generated by the radiating device 201A are adjusted to change the shape of the radiation pattern, the radiation intensity may also be changed in the graphically updated rendering for the user.

[0063] 6 illustrates an example of a radiation beam emitted from emission device 201A to deactivate a target (e.g., target 530). Such an exemplary rendering is generated on a display for viewing by a user. As illustrated, main lobe 610 and side lobes 612 are depicted as radiation with one graphical indicator (with second graphical indicator 532) indicating that target 530 is affected by the emitted radiation, and graphical indicator 522 indicating that another potential target 520 is to be avoided and therefore not affected by the emitted radiation.

[0064] In some embodiments, where a swarm of targets may exist, such as a swarm of UAVs, the emitter can be controlled to track the center region of the swarm based on the average position of various targets in the swarm, and can be adjusted to take into account targets far from the center region of the swarm, for example, by directing the radiation pattern to encompass as many targets as possible, even if some targets are outside the radiation pattern.

[0065] FIG. 7 illustrates how an AR device's graphical user interface (GUI) displays an expanded field of view to a user as a virtual control panel 204A. The features described herein can be combined in numerous ways in many embodiments to provide an intuitive, dynamic, and accurate display of the status of the emitting device, potential targets 510, 520, 530, and potential directed energy from the emitting device. The directed energy from the emitting device may consist of a main lobe 610 and side lobes 612, as described above with reference to FIG. 6. In the embodiment of the virtual control panel 204A depicted in the example of FIG. 7, such features can be combined to provide a view of the surrounding environment where the user can utilize information displayed on the display device to direct specific radiation patterns to selected targets. This allows the system to display or otherwise make available the real environment (or a virtual replica of the environment in a VR embodiment) so that the user remains aware of the local environment (around the emitting device) while performing the illumination.

[0066] Some embodiments may include a real-time overview 720 of an area in which projectiles, targets, allies, or other points of interest may be depicted. In this manner, the system may be configured to obtain map data of an area containing one or more targets 530 and obtain target locations in real time. The system may then display the real-time overview 720 on a display, including the map data and a representation of the one or more targets. An example of the real-time overview is depicted in the lower right corner of the virtual control panel. In some implementations, a map command prompt 710 may be displayed to allow the user to zoom (e.g., by the user performing a pinch motion), rotate, etc. the real-time overview 720.

[0067] As shown in FIG. 7 , some embodiments may include displaying identification information 732 of a target (e.g., unidentified target 510, friendly 520, or specific target 530) on the display of the AR device. Emission of output may be based on the target's identification. For example, if a friendly element / target is within the emission area, output emission may not be permitted (or specific disabling may be required). To assist the user, some embodiments may display identification information such as type, size, attachments, direction of movement, brand, friendly or unfriendly classification, etc. Coordinate information of one or more targets may also be displayed, such as the target's latitude, longitude, or altitude. In various embodiments, the coordinate information may be obtained from GPS, RADAR, or LIDAR data. In some embodiments, the displayed identification information may be partial, for example, if such identification is in progress or if only some information is known.

[0068] As shown in FIG. 7, the virtual control panel 204A may display information related to the emitting device 201A. For example, it may depict a virtual representation of the main lobe 610 and side lobes 612 as well as the calculated null zone. Options available for controlling the beam, such as hand gestures, may display a beam command prompt 740 for positioning the beam, side lobes, or null zone. The beam control options may be displayed as virtual buttons / status fields and may include add null 742, beam position 743, request launch 744, display approved launch status 745 or rejected launch status 746, and a launch button 746.

[0069] 7, some embodiments may include any of the following features in any combination: a button 750 to activate one or more operations with a gesture, a button 752 to display target information, a button 753A or 753B to increase or decrease the display font size, a button 754 to display the status of available external devices or units, a button 756 to enable / disable beam data from an emitting device, or a button 758 to enable / disable null data (representing a beam null). It should be noted that the example virtual control panel 204A is not limited to the specific control operations, information, data, and buttons depicted in FIG. 7.

[0070] 8 illustrates an example GUI 800 for the AR device 204 that displays control options and target information. The example GUI 800 illustrated in FIG. 8 can display information on the display of the AR device 204 as part of a virtual control panel 204A. This information can be associated with target data 802, an option 804 to access GPS information or map data to obtain more information about the target, an option 806 to display an overview of the target, an option 808 to display characteristic information about the target, or an option 810 to display the area of ​​the radiation beam from the radiation device.

[0071] As illustrated by exemplary toggles, buttons, etc., such information and control options can be activated by user movements, hand gestures, buttons displayed on the AR device 204, or voice commands to activate options presented in the virtual control panel 204A. In some embodiments, the GUI 800 may be modifiable or manipulable, such as scalable, rotatable, movable, or lockable. The system may dynamically modify any of the disclosed GUIs to provide real-time information, for example, based on changing target conditions. The system may be configured to allow a user to toggle back and forth to access information related to various devices used in the field. In another implementation, the hardware controller 212 may also provide and display information to the AR device 204 that the user may rely on when providing commands to any of the connected devices.

[0072] 9 illustrates an exemplary GUI 900 for an AR device displaying an overview, control options, and target information. In some embodiments, the AR device can display a rendering 904 of the emitter 201A and / or radiation output, a map and GPS coordinate information 906 of the emitter 201A, and / or a rendering of a field 902 (similar to overview 720 in FIG. 7) containing information related to targets 908A, 908B, 908C. In some embodiments, the map information displayed on the display of the AR device 204 can be controlled by a user to move, rotate, scale, lock, zoom, traverse, etc.

[0073] In this example, the AR device 204 may display the emitter 201A, a detected target 908A that is not affected by the emitted radiation, and a target 908B that is affected by the emitted radiation. The AR device 204 may also render the emitted output 904 and depict it being directed in the direction of the target 908B to cause an effect.

[0074] When target 908B is detected as unfriendly, the user can activate one or more control commands from the display of AR device 204 to perform an action with emission device 201A or other external device. Based on the control commands displayed on the display of AR device 204, the user can control and manipulate emission device 201A to emit a radiation output to impact target 908B, as described herein.

[0075] 10A-10D illustrate controlling an emission device 201A using an AR device 204 to direct radiation to a target, according to certain aspects of the present disclosure. FIG. 10A shows a virtual control panel 204A displaying exemplary control information that can be activated by a user using hand gestures. The user can similarly place a rendered GUI with the control information on the display of the AR device 204. Upon activating a control menu based on a user command (gesture, voice, button press, etc.), the GUI can display information related to the emission device 201A or one or more control devices 201B. The user can control such devices by selecting options displayed on the display of the AR device 204.

[0076] The example of FIG. 10A shows a virtual control panel 204A that may provide options such as "Outline Target" 1002, "Show Target Information" 1004, and "Show Emitted Beam" 1006. The virtual control panel may further display options related to various possible actions the user can take based on whether the target 530 has been identified as being affected by emitted radiation. Examples of such control operations may be displayed in a menu on the virtual control panel 204A, such as "Request Irradiation" 1012, status information such as "Approved" 1014 or "Rejected" 1016, or "Irradiate" 1018. The virtual control panel 204A may further provide various options for toggling tracking information 1022 between various external devices and for repositioning the emitter 201A using virtual "up" and "down" keys displayed on the virtual control panel 204A.

[0077] 10B shows a rendering on the display virtual control panel 204A of an AR device similar to that shown in FIG. 10A, but including a rendering of the radiation output (e.g., main lobe 610 and / or side lobes 612) being emitted (or can be emitted on command) from the emission device to affect a target (e.g., target 530). As shown in FIG. 10B, the user can see the real-time emission of radiation by the AR device, which may not otherwise be directly visible by the human eye.

[0078] FIG. 10C illustrates controlling the emitter device to direct radiation to deactivate a swarm of targets. For example, the hardware controller 212 can process information using one or more software programs based on sensor information (e.g., from the sensor 206) and image information (e.g., from the imager 4402) to determine information about the target (e.g., its identification, location, etc.). As shown in FIG. 10C, an identified target 530 may be indicated by a second graphical indicator 532 as a target affected by the applied radiation. For example, an AR device can be utilized to control the operation of the emitter device to affect the swarm of targets via a virtual control panel 204A in communication with the hardware controller 212. The virtual control panel 204A is displayed, and a control menu may be displayed to partially overlay the rendered radiation output depending on the positioning of the virtual control panel 204A (which may be manipulated as described herein). Similarly, if the virtual control panel 204A is deemed distracting to the user, the control menu can be manipulated to not obscure the rendered radiation output (including the main lobe 610 and side lobes 612), but still be fully or partially within the field of view of the display device. In one exemplary embodiment, when the target 530 reaches a predetermined distance (e.g., a minimum or optimal range) to the emitting device 201A or comes within the area of ​​effect of the radiation output, the emitting device can emit radiation based on the target's position.

[0079] FIG. 10D illustrates an exemplary deactivation of target 530. The figure shows target 530 descending in response to the radiation output (including main lobe 610 and side lobes 612) and the deactivation caused by the radiation. In this manner, a user can view the radiation emitted from radiation device 201A and its effect on target 530 in real time. The color and rendered appearance of the radiation may vary in different embodiments. For example, in some embodiments, different frequencies of radiation may be depicted in different colors. In some embodiments, different radiation colors may be used to indicate radiation with different waveform characteristics. For ease of visualization, the color of the rendered radiation may change when target 530 is deactivated.

[0080] 11 is a simplified flow diagram illustrating the display of information and control of emission devices in an AR device. Any of the embodiments disclosed herein may be implemented as a process 1100 that may be performed by one or more computers as part of one or more software modules or computer programs.

[0081] In one embodiment, the process 1100 may include, at 1110, displaying a virtual control panel 204A on a display, the virtual control panel 204A including a depiction of one or more targets.

[0082] The process 1100 may include, at 1120, rendering the output of the emitting device 201A intended to be directed at one or more targets.

[0083] The process 1100 may include, at 1130, a user controlling one or more operations of the emitting device 201A based on input received by the user's interaction with the virtual control panel 204A.

[0084] Further features, characteristics and exemplary technical solutions of the present disclosure are described below by listing items that may be claimed in any combination.

[0085] (Item 1) 1. A system comprising: an AR device having a display configured to display an augmented reality image, a mixed reality image, or a virtual reality image to a user; at least one programmable processor; and a non-transitory machine-readable medium storing instructions, the instructions, when executed by the at least one programmable processor, cause the at least one programmable processor to perform an operation; The system, wherein the operations include displaying on the display a virtual control panel including a depiction of one or more targets; rendering an output of an emission device intended to be directed at the one or more targets; and the user controlling one or more operations of the emission device based on input received by the user's interaction with the virtual control panel.

[0086] (Item 2) Item 10. The system of item 1, wherein the rendering step includes a region of directed energy from the emitting device.

[0087] (Item 3) 3. The system of claim 1, wherein the rendered region includes a main lobe and one or more side lobes of the directed energy.

[0088] (Item 4) 4. The system of any one of items 1 to 3, wherein the rendering step comprises obtaining settings of the emitting device and / or static parameters of the emitting device, calculating a radiation field resulting from the emitting device based on at least the settings of the emitting device and / or the static parameters using a radiation simulator, and displaying an intensity of the electromagnetic field associated with the output.

[0089] (Item 5) 5. The system of any one of items 1 to 4, wherein the operation of the one or more radiation devices controls the radiation device to irradiate the output to affect the one or more targets.

[0090] (Item 6) 6. The system of any one of items 1 to 5, wherein the controlling step is based on the system interpreting one or more hand gestures, gaze or voice commands, or button presses on the AR device or control peripheral.

[0091] (Item 7) 7. The system of any one of items 1 to 6, wherein the rendering step is updated based on the control of the emitting device by the user.

[0092] (Item 8) 8. The system of any one of items 1 to 7, wherein the one or more operations of the radiation device include one or more of positioning the radiation device, selecting a type of output, setting a frequency of the output, setting an intensity of the output, setting a direction of the output, or setting a time for directing and irradiating the output at the one or more targets.

[0093] (Item 9) 9. The system of any one of items 1 to 8, wherein the operation includes displaying identification information of the one or more targets on the display of the AR device, and the illumination of the output is based on the identification information of the one or more targets.

[0094] (Item 10) 10. The system of any one of items 1 to 9, wherein the displayed identification information includes one or more of type, size, attached components, direction of travel, brand, friendly classification or unfriendly classification.

[0095] (Item 11) 11. The system of any one of items 1 to 10, further comprising the steps of: acquiring map data of an area including the one or more targets; acquiring real-time target position information; and displaying a real-time overview including the map data and a representation of the one or more targets on the display.

[0096] (Item 12) 12. The system of any one of items 1 to 11, further comprising the step of displaying coordinate information of the one or more targets.

[0097] (Item 13) 13. The system of any one of items 1 to 12, wherein the coordinate information includes one or more of the latitude, longitude, and altitude of the target.

[0098] (Item 14) 14. The system of any one of items 1 to 13, wherein the coordinate information is obtained from GPS, RADAR or LIDAR data.

[0099] (Item 15) 15. A system described in any one of items 1 to 14, wherein the coordinate information is obtained from coordinate information of the one or more targets relative to a reconnaissance UAV.

[0100] (Item 16) 16. The system of any one of items 1 to 15, wherein the coordinate information obtained from the coordinate information of the one or more targets relative to the reconnaissance UAV is integrated with coordinate information obtained from GPS data, RADAR data, or LIDAR data.

[0101] (Item 17) a sensor configured to detect one or more attributes of the one or more targets; and an imaging device configured to image the one or more targets; 17. The system of any one of items 1 to 16, wherein the operations further comprise acquiring real-time sensor data and / or real-time image data from the sensor; and identifying the target from the real-time sensor data and / or the real-time image data.

[0102] (Item 18) 18. The system of any one of items 1 to 17, wherein the one or more attributes of the detected one or more targets include presence, environmental data surrounding the one or more targets, velocity, acceleration or coordinates.

[0103] (Item 19) 19. The system of any one of items 1 to 18, wherein imaging the one or more targets includes imaging one or more of the one or more targets themselves, attached components, or radiation.

[0104] (Item 20) 20. The system of any one of items 1 to 19, wherein the operations further comprise determining information about the one or more targets based on the sensor data, the information including one or more of presence, speed, acceleration, coordinates, guided route, satellite source, distance from the emitting device, environmental data surrounding the one or more targets, temperature, and field of view (FOV).

[0105] (Item 21) 21. The system of any one of items 1 to 20, wherein the operations further comprise determining information about the one or more targets based on the image data, the information including one or more of information identifying a type of the one or more targets, information identifying one or more characteristics, information identifying one or more devices or systems associated with the one or more targets, battery information, power information, type, brand, size or shape.

[0106] (Item 22) 22. A non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations including the steps of any one of items 1 to 21 above.

[0107] One or more aspects or features of the subject matter described herein may be embodied in digital electronic circuitry, integrated circuits, specially designed application-specific integrated circuits (ASICs), field-programmable gate array (FPGA) computer hardware, firmware, software, and / or combinations thereof. These various aspects or features may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, whether special-purpose or general-purpose, at least one input device, and at least one output device. The programmable processor is coupled to receive data and instructions from, and transmit data and instructions to, a storage system. The programmable system or computer system may include a client side and a server side. The client and server are typically remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0108] These computer programs, which may also be referred to as programs, software, software applications, applications, components, or code, comprise machine instructions for a programmable processor and may be implemented in high-level procedural languages, object-oriented programming languages, functional programming languages, logic programming languages, and / or assembly / machine languages. As used herein, the term "machine-readable medium" (or "computer-readable medium") refers to any computer program product, apparatus, and / or device, such as a magnetic disk, optical disk, memory, and programmable logic device (PLD), used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" (or "computer-readable signal") refers to any signal used to provide machine instructions and / or data to a programmable processor. The machine-readable medium may store such machine instructions in a non-transitory manner, such as a non-transitory solid-state memory or a magnetic hard drive or any equivalent storage medium. Alternatively or additionally, the machine-readable medium may temporarily store such machine instructions, such as in a manner similar to a processor cache or other random access memory associated with one or more physical processor cores.

[0109] To facilitate user interaction, one or more aspects or features of the subject matter described herein may be implemented in a computer that includes a display device, such as a cathode ray tube (CRT) or liquid crystal display (LCD) or light-emitting diode (LED) monitor, for displaying information to a user, and a keyboard and pointing device, such as a mouse or trackball, for providing input to the computer. Other types of devices may also be used to facilitate user interaction. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. Input from the user may be received in any form, such as, but not limited to, acoustic, voice, or tactile input. Other possible input devices include, but are not limited to, touchscreens or other touch-sensitive devices, such as, for example, single-point or multi-point resistive or capacitive trackpads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and corresponding interpretation software, etc.

[0110] In the above description and in the claims, multiple elements or features may be listed with a conjunction, such as "at least one of" or "one or more of." The term "and / or" may also be used when listing two or more elements or features. Unless the context indicates otherwise, implicitly or explicitly, these expressions are intended to refer to any of the listed elements or features individually, or any of the listed elements or features in combination with any remaining elements or features. For example, the expressions "at least one of A and B," "one or more of A and B," and "A and / or B" are intended to mean "A only," "B only," or "A and B," respectively. A similar interpretation should be used when more than two elements or features are listed. For example, the phrases "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, and / or C" are intended to mean "A only," "B only," "C only," "A and B," "A and C," "B and C," or "A, B, and C," respectively. The term "based on," when used above and in the claims, is intended to mean "based at least in part on," so that unrecited features or elements may also be considered.

[0111] The subject matter described herein may be embodied in systems, apparatus, methods, computer programs, and / or articles, depending on the desired configuration. Any method or logic flow illustrated in the accompanying drawings and / or described herein does not necessarily require the particular order shown or sequential order to achieve desired results. The implementations set forth in the above description are not intended to represent all implementations consistent with the subject matter described herein. Rather, they are merely examples consistent with various aspects of the described subject matter. While certain variations have been described in detail above, other variations or additions are possible. Additional features and / or variations may be implemented beyond those described herein. The implementations described above may relate to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of additional features described above. Furthermore, the advantages described above are not intended to limit the application of the claims to processes and structures achieving any or all of these advantages.

[0112] Additionally, the section titles do not limit or characterize the invention(s) described in any claims based on this disclosure. Furthermore, the description of technology in the "Background" section should not be construed as an admission that such technology is prior art to the invention(s) of this disclosure. The "Summary of the Invention" section should also not be construed as characterizing the invention(s) described in the claims. Furthermore, when referring to this disclosure as a whole or when using the word "invention" in the singular, no limitations on the scope of the claims set forth below are intended. Multiple inventions may be described according to the limitations of multiple claims based on this disclosure. These claims, therefore, define the invention(s) and their equivalents to be protected.

Claims

1. a device having a display configured to display an augmented reality image, a mixed reality image, or a virtual reality image to a user; at least one programmable processor; a non-transitory machine-readable medium storing instructions, The instructions, when executed by the at least one programmable processor, cause the at least one programmable processor to: causing the display to display a virtual control panel including a representation of one or more targets; Rendering an output of an emitting device configured to be directed at the one or more targets; A system that allows user interaction with the virtual control panel to control one or more operations of the emitting device.

2. The system of claim 1 , wherein the instructions, when executed, cause the at least one programmable processor to render a representation of a region of directed energy from the emitting device.

3. The system of claim 2 , wherein the rendered region includes a main lobe and one or more side lobes of the directed energy.

4. The instructions, when executed, cause the at least one programmable processor to: obtaining settings of the radiation device or static parameters of the radiation device; using a radiation simulator to calculate a radiation field resulting from the radiation device based on at least the settings of the radiation device or static parameters of the radiation device; The system of claim 2 , further comprising a display of an electromagnetic field strength associated with said output.

5. The instructions, when executed, cause the at least one programmable processor to: The system of claim 1 , further comprising: controlling irradiation of the output of the radiation device to affect the one or more targets.

6. The instructions, when executed, cause the at least one programmable processor to:

10. The system of claim 1, wherein the one or more operations are controlled based on one or more hand gestures, gaze commands, voice commands, button presses on the device, or inputs to a control peripheral of the device.

7. The instructions, when executed, cause the at least one programmable processor to: The system of claim 1 , further comprising: a display unit configured to display a display of the output of the user;

8. 10. The system of claim 1, wherein the one or more operations of the radiation device include one or more of: positioning the radiation device, selecting a type of output, setting a frequency of the output, setting an intensity of the output, setting a direction of the output, or setting a time for directing and irradiating the output at the one or more targets.

9. The instructions, when executed, cause the at least one programmable processor to: causing the display of the device to display identification information of the one or more targets; The system of claim 1 , wherein the output is illuminated based on the identity of the one or more targets.

10. The system of claim 9 , wherein the displayed identification information includes an indication of one or more of type, size, attached components, direction of travel, brand, friendly classification, or unfriendly classification.

11. The instructions, when executed, cause the at least one programmable processor to: obtaining map data of an area including the one or more targets; obtaining real-time target positions of the one or more targets; The system of claim 1 , causing the display to display a real-time overview including the map data and a representation of the one or more targets.

12. The system of claim 1 , wherein the instructions, when executed, cause the at least one programmable processor to display coordinate information for the one or more targets.

13. The system of claim 12 , wherein the coordinate information includes one or more of a latitude, a longitude, and an elevation of a target.

14. The system of claim 13 , wherein the coordinate information is obtained from a GPS, a RADAR device, or a LIDAR device.

15. The system of claim 13 , wherein the coordinate information is obtained from coordinate information of the one or more targets relative to a reconnaissance unmanned aerial vehicle (UAV).

16. The system of claim 15 , wherein the coordinate information of the one or more targets relative to the reconnaissance UAV is integrated with coordinate information obtained from a GPS, a RADAR device, or a LIDAR device.

17. a sensor configured to detect one or more attributes of the one or more targets; an imaging device configured to image the one or more targets; The instructions, when executed, cause the at least one programmable processor to: acquiring real-time sensor data or real-time image data from the sensor; The system of claim 1 , further comprising: identifying targets from the real-time sensor data or the real-time image data.

18. The system of claim 17 , wherein the one or more attributes of the detected one or more targets include presence, environmental data surrounding the one or more targets, velocity, acceleration, or coordinates.

19. The system of claim 17 , wherein the imaging device is configured to image one or more of the one or more targets themselves, attached components, or radiation.

20. The instructions, when executed, cause the at least one programmable processor to: determining information about the one or more targets based on the sensor data; 20. The system of claim 17, wherein the information includes one or more of the following: presence, speed, acceleration, coordinates, guided route, satellite source, distance from the emitting device, environmental data surrounding the one or more targets, temperature, and field of view (FOV).

21. The instructions, when executed, cause the at least one programmable processor to: determining information about the one or more targets based on the image data; 20. The system of claim 17, wherein the information includes one or more of the following: type of the one or more targets, one or more characteristics, one or more devices or systems associated with the one or more targets, battery information, power information, type, brand, size, or shape.

22. A non-transitory machine-readable medium storing instructions, comprising: The instructions, when executed by at least one programmable processor, cause the at least one programmable processor to: displaying on the display a virtual control panel including a representation of one or more targets; Rendering an output of an emitting device configured to be directed at the one or more targets; A machine-readable medium that allows user interaction with the virtual control panel to control one or more operations of the emitting device.

Images