System and method for real time viewwpoint video sharing

Abstract

A system and method for real-time sharing of viewpoint images are disclosed. The present invention streams viewpoint images or screen data captured between heterogeneous edge devices in real time with low latency of less than a certain time, and can provide security and real-time situational awareness required in a battlefield environment through multi-viewpoint monitoring, real-time encryption, hierarchical access control, emergency blocking and session data deletion, and recipient identification-based digital watermarking functions.

Description

Technology Field

[0001] The present invention relates to a system and method for real-time sharing of viewpoint images, and more specifically, to a system and method for real-time sharing of viewpoint images or screen data captured between edge devices, which streams the viewpoint images or screen data in real-time with low latency of less than a certain time, and provides security and real-time situational awareness required in a battlefield environment through multi-viewpoint monitoring, real-time encryption, hierarchical access control, emergency blocking and session data deletion, and recipient identification-based digital watermarking functions. Background Technology

[0002] Edge devices such as smart glasses, automotive smartphones, and automotive tablets can perform various computer vision tasks, such as object recognition, people recognition, vehicle recognition, and text recognition, based on camera images. These recognition functions are key to the wearer's situational awareness, driving assistance, and identification of battlefield targets.

[0003] Modern battlefields are based on the concept of Network-Centric Warfare (NCW), where the rapid sharing of information acquired by individual combatants across the entire unit has become the core of combat power, and in particular, the real-time sharing of visual information (video) is a key factor in determining the quality of situational awareness.

[0004] For example, if all squad members can view the point-of-view video of a leading tactical agent—such as a combatant—in real time as he enters a building, coordinated action becomes possible while the entire squad maintains a shared situational awareness.

[0005] Furthermore, if a commander can simultaneously monitor viewpoint footage from multiple tactical entities from the rear, it is possible to make an integrated assessment of the entire battlefield situation.

[0006] The U.S. Army’s IVAS (Integrated Visual Augmentation System) includes the capability to share data between tactical entities via network radio and transmits video from vehicle-mounted cameras to soldiers’ headsets, enabling them to perceive the external situation from inside the armored vehicle.

[0007] However, video sharing in IVAS is primarily focused on unidirectional transmission between vehicles and individuals, and there is a problem in that optimization for bidirectional real-time viewpoint sharing between individuals has not been achieved.

[0008] In addition, IVAS is a large goggle-based system weighing about 1.5 kg, and implementation in a lightweight device in the form of glasses has not been considered. Since devices in the form of glasses have significant limitations in battery, heat dissipation, and computing power, efficient optimization of video encoding and decoding and wireless transmission is essential.

[0009] Meanwhile, rugged smartphones or rugged tablets used by tactical entities on the battlefield are already connected to tactical networks (such as ATAK / Nett Warrior), but real-time video sharing through them is limited to receiving drone feeds or transmitting static images.

[0010] In addition, there is a problem in that the viewpoint video of the smart glasses cannot be streamed in real-time to a smartphone or tablet, or tactical information from the tablet cannot be transmitted to the smart glasses. Prior art literature

[0011] Korean Registered Patent Publication No. 10-1547927 (Title of Invention: Reconnaissance Image Sharing System, Registration Date: August 21, 2015) The problem to be solved

[0012] To solve these problems, the present invention aims to provide a real-time point-of-view video sharing system and method that streams point-of-view video or screen data captured between edge devices in real time with low latency of less than a certain time, and provides security and real-time situational awareness required in a battlefield environment through multi-point-of-view monitoring, real-time encryption, hierarchical access control, emergency blocking and session data deletion, and recipient identification-based digital watermarking functions. means of solving the problem

[0013] To achieve the above-mentioned purpose, one embodiment of the present invention is a real-time sharing system for point-of-view images, comprising: a first terminal that captures captured point-of-view images or screen data, performs encoding that adaptively adjusts image quality according to available network environment, and outputs the encoded image data after security authentication and encryption; a second terminal that receives the encoded and encrypted image data from the first terminal, decodes and decodes the received image data, and displays it on an image output means; a network layer that supports image data transmission between the first terminal and the second terminal, includes one or more of direct communication between terminals, a mesh network, a tactical communication network, and an edge server relay method, and selects a transmission method according to network availability; and a session management terminal that creates, maintains, terminates, controls access rights, manages resources, adjusts encoding parameters based on latency, performs emergency blocking and session data deletion, and performs watermarking based on recipient identification of an image streaming session between the first and second terminals.

[0014] In addition, the real-time sharing system for view-time images according to the embodiment is characterized by having an end-to-end delay time of 500ms or less from the first terminal to the second terminal.

[0015] In addition, the above security authentication and encryption are characterized by being performed using one or more of AES-256-GCM, ChaCha20-Poly1305, or SEED algorithms.

[0016] In addition, the first terminal and the second terminal according to the above embodiment are one of smart glasses, a rugged smartphone, or a rugged tablet, and the first terminal and the second terminal may be heterogeneous devices of different types.

[0017] In addition, the session management terminal according to the above embodiment is characterized by collecting the display size, resolution, and battery level of each receiving terminal and independently adjusting the video format and quality according to the type of receiving terminal.

[0018] In addition, when the second terminal according to the above embodiment is a multi-video output device, it is characterized by providing a multi-viewpoint monitoring mode that simultaneously displays viewpoint images received from one or more first terminals in an m×n multi-screen split form, overlaying identification information, status information, and network quality information of the corresponding participant in each screen split area, providing a function to expand the viewpoint image of the corresponding participant to the full screen when a specific screen split area is selected and to directly transmit instructions to the corresponding participant through an interaction channel, and providing a warning display in the corresponding screen split area when the connection of a specific device is disconnected.

[0019] In addition, the session management terminal according to the above embodiment is characterized by inserting protection information including a recipient-related identification element into video data distributed to the second terminal in an invisible form, and enabling the video receiving device and the recipient to be tracked from the protection information when the video is leaked externally.

[0020] In addition, the session management terminal according to the above embodiment generates and propagates an emergency block that is simultaneously transmitted to all connected terminals when an emergency situation occurs, and the first terminal and the second terminal immediately terminate the video streaming session and delete the session video data and session key stored locally when they receive the emergency control command, and the session management terminal collects responses regarding the completion of the emergency block from all target devices to verify whether the entire block has been completed.

[0021] In addition, one embodiment of the present invention is a method for real-time sharing of a viewpoint image, comprising: a) capturing a viewpoint image or screen data captured by a first terminal; b) the first terminal performing encoding that adaptively adjusts the image resolution, frame rate, and bit rate according to the available network environment, and performing security authentication and encryption on the encoded image data to generate a transmission packet; c) the first terminal transmitting the transmission packet to a second terminal through one or more transmission methods selected according to network availability among direct communication between terminals, a mesh network, a tactical communication network, and an edge server relay method; d) a session management terminal performing creation, maintenance, termination, access control, resource management, latency-based encoding parameter adjustment, emergency blocking and session data deletion, and recipient identification-based watermarking of a video streaming session between the first terminal and the second terminal; and e) the second terminal decoding and decoding the received transmission packet and displaying it on a video output means.

[0022] In addition, the real-time sharing method for viewpoint images according to the above embodiment is characterized by the fact that the end-to-end delay time from step a) to step e) is 500ms or less.

[0023] In addition, step d) according to the above embodiment is characterized in that the session management terminal collects the display size, screen resolution, and battery level of the second terminal, and independently adjusts the video format and quality according to the type of receiving terminal based on the collected information.

[0024] Additionally, step d) according to the above embodiment is characterized by further including: a step in which the session management terminal inserts protection information containing a recipient-related identification element into video data distributed to the second terminal in an invisible form; and a step of enabling tracking of the video receiving device and the recipient from the protection information when the video is leaked to the outside.

[0025] Additionally, step d) according to the above embodiment is characterized by further including: when an emergency situation occurs or an external command regarding an emergency situation is received, the session management terminal generates an emergency block and simultaneously propagates the generated emergency block to all connected terminals; the first terminal and the second terminal, each of which receives the emergency block, immediately terminate the video streaming session and delete the session video data and session key stored locally; and the session management terminal collects responses regarding the completion of the emergency block from all target devices to check whether the entire block is complete.

[0026] Additionally, step e) according to the above embodiment is characterized by including: a step of providing a multi-viewpoint monitoring mode that simultaneously displays viewpoint images received from one or more first terminals in an m×n multi-screen split form when the second terminal is a multi-video output device; a step of overlaying identification information, status information, and network quality information of the corresponding participant in each screen split area; a step of expanding the viewpoint image of the corresponding participant to the full screen and directly transmitting instructions to the corresponding participant through an interaction channel when a specific screen split area is selected; and a step of providing a warning display in the corresponding screen split area when the connection of a specific device is disconnected. Effects of the invention

[0027] The present invention has the advantage of being able to stream camera images or screen data of a specific device in real time to the displays of other devices between edge devices.

[0028] In addition, the present invention has the advantage of enabling video sharing between various edge devices such as smart glasses, smartphones, and tablets, which can dramatically improve the quality of battlefield situation awareness, and can be immediately utilized for tactical decision-making with a delay of 500ms or less, thereby maximizing command and control efficiency.

[0029] In addition, the present invention can provide the highest level of security required for military operational environments through real-time encryption, hierarchical access control, emergency blocking, digital watermarking, etc., and has the advantage of enabling comprehensive situational awareness of the entire battlefield by allowing simultaneous monitoring of multiple viewpoints.

[0030] In addition, the present invention has the advantage of being able to share a head-up display image in which a restored image or a tactical information overlay is synthesized and shared in a night or smoke environment, thereby enabling the immediate dissemination of enhanced information secured by the forward tactical entity to the entire unit. Brief explanation of the drawing

[0031] FIG. 1 is a block diagram illustrating a real-time sharing system for viewpoint images according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating the configuration of a first terminal of a real-time viewing image sharing system according to an embodiment of FIG. 1. FIG. 3 is a block diagram illustrating the configuration of a session management terminal of a real-time sharing system of a viewpoint image according to an embodiment of FIG. 1. FIG. 4 is a flowchart illustrating a method for real-time sharing of viewpoint images according to an embodiment of the present invention. FIG. 5 is an example diagram illustrating the sharing between heterogeneous terminals of the real-time sharing method of a viewpoint image according to the embodiment of FIG. 4. FIG. 6 is an example diagram illustrating the sharing of a real-time viewpoint image sharing method according to the embodiment of FIG. 4 in a mixed environment between heterogeneous terminals. Specific details for implementing the invention

[0032] Hereinafter, the present invention will be described in detail with reference to preferred embodiments of the invention and the accompanying drawings, under the premise that identical reference numerals in the drawings refer to identical components.

[0033] Before describing specific details for the implementation of the present invention, it should be noted that configurations not directly related to the technical essence of the present invention have been omitted to the extent that the technical essence of the present invention is not obscured.

[0034] Furthermore, terms or words used in this specification and claims should be interpreted in a meaning and concept consistent with the technical spirit of the invention, based on the principle that the inventor may define the concept of appropriate terms to best describe their invention.

[0035] In this specification, the expression that a part "includes" a certain component means that it does not exclude other components but may include additional components.

[0036] Additionally, terms such as "...part," "...unit," and "...module" refer to a unit that processes at least one function or operation, and this can be classified as hardware, software, or a combination of both.

[0037] In addition, the term "at least one" is defined as a term including both singular and plural forms, and it is self-evident that even if the term "at least one" does not exist, each component may exist in the singular or plural form and may mean the singular or plural.

[0038] Hereinafter, a preferred embodiment of a real-time viewpoint image sharing system and method according to an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0039] FIG. 1 is a block diagram illustrating a real-time viewpoint image sharing system according to an embodiment of the present invention, FIG. 2 is a block diagram illustrating a first terminal configuration of a real-time viewpoint image sharing system according to an embodiment of FIG. 1, and FIG. 3 is a block diagram illustrating a session management terminal configuration of a real-time viewpoint image sharing system according to an embodiment of FIG. 1.

[0040] As shown in FIGS. 1 to 3, a real-time sharing system for view-time images according to one embodiment of the present invention may be configured to include a plurality of terminals (100, 100a to 100n), a network layer (200), and a session management terminal (300).

[0041] The terminal (100, 100a to 100n) may be composed of a first terminal (100), a second terminal (100a) to an nth terminal (100n), and each terminal (100, 100a to 100n) may be composed of one of smart glasses, a rugged smartphone, or a rugged tablet.

[0042] In this embodiment, the first terminal (100) is described as smart glasses for convenience of explanation, but is not limited thereto.

[0043] The first terminal (100) can capture a captured image or screen data, perform encoding that adaptively adjusts the image quality according to the available network environment, and output the encoded image data after security authentication and encryption.

[0044] To this end, the first terminal (100) may be configured to include an image acquisition unit (110), a processor unit (120), a display unit (130), a communication unit (140), an encoder / decoder unit (150), and a security module unit (160).

[0045] The image acquisition unit (110) can be implemented as a camera module including a CMOS or CCD image sensor, and can perform basic ISP (Image Signal Processing) processing for sensor noise removal on the acquired image, and can perform automatic white balance (AWB) and automatic exposure (AE) adjustments as needed.

[0046] Additionally, the image acquisition unit (110) can continuously capture the point of view (POV) of a user, for example, a combatant who is a tactical subject, and transmit the captured screen data to the processor unit (120).

[0047] Here, the screen data may include all forms of image data that are displayed or processed on the screen of the first terminal in addition to the original image captured through the image acquisition unit (110), and the captured screen data types may specifically include the following four types.

[0048] The first is an unprocessed viewpoint image captured by the camera module, which may be the original camera image with no separate post-processing applied other than basic processing such as ISP.

[0049] The second may be an image to which image restoration processing, such as low-light restoration and dehaze, has been applied so that the receiving side can receive an enhanced image without separate processing in degraded environments such as nighttime or smoke and fog.

[0050] The third may be a composite image of an augmented reality overlay (friendly location, movement path, threat marker, etc.) displayed on a smart glasses HUD (Head Up Display) so that the receiver can share the same augmented reality screen as the sender.

[0051] The fourth is an image in the form of a transmission packet to which, for example, the AES-256-GCM authentication encryption algorithm is applied to the aforementioned original camera image, image restoration processed image, and synthesized image data.

[0052] The processor unit (120) determines video encoding parameters, monitors network status, and exchanges control signals with the session management terminal (300), and may prioritize the use of a hardware encoder, such as a DSP or NPU, to minimize battery consumption.

[0053] The display unit (130) overlays a picture of another user's viewpoint received in a picture-in-picture (PIP) form in a part of the user's actual field of view, for example, the lower left or lower right, and full screen switching can be performed through voice commands or gaze input, and a thumbnail list can be displayed on the edge of the HUD when multiple images are received.

[0054] The communication unit (140) can perform data communication using multiple wireless interfaces including Wi-Fi Direct, Bluetooth 5.0+ or ​​higher, UWB for direct communication between terminals, and tactical communication network interfaces such as TICN (Tactical Information Communication Network, Korean military tactical information communication system), WIN-T (Warfighter Information Network-Tactical, US military), and Nett Warrior (US military) for tactical communication network connection, and can automatically switch to an optimal transmission method depending on network availability.

[0055] The encoder / decoder unit (150) can be implemented, for example, with an H.265 / HEVC low-latency profile or a low-latency codec of AV1 real-time mode or a lightweight proprietary codec, and can perform Adaptive Bitrate (ABR) encoding that adaptively adjusts the resolution, frame rate, and bit rate according to the available network bandwidth.

[0056] For example, if there is sufficient bandwidth, high-quality encoding of 1080p / 30fps or higher is performed, and if there is insufficient bandwidth, it may automatically switch to low-quality encoding of 480p / 15fps or lower, and if the battery level is 20% or less, a low-resolution encoding mode may be automatically applied.

[0057] The security module (160) can, for example, encrypt video data encoded with an AES-256-GCM authentication encryption algorithm in real time, and the encryption key (or session key) can be securely exchanged using a Diffie-Hellman key exchange method when a session is created and can be periodically updated during the session.

[0058] AES-256-GCM is an authenticated encryption method that performs encryption and integrity verification simultaneously, and can prevent eavesdropping and data tampering in battlefield environments.

[0059] In addition, the security module (160) may optionally use one or more of the ChaCha20-Poly1305 or SEED (domestic standard block cipher algorithm) algorithms in addition to AES-256-GCM, and this may be determined according to the operating environment, device performance, and national encryption regulations.

[0060] Additionally, the security module (160) can immediately terminate the video streaming session and delete the session video data and session key stored locally upon receiving an emergency block (kill switch) command from the session management terminal (300).

[0061] Meanwhile, if the first terminal (100) is implemented as a rugged smartphone, it may have the same configuration as smart glasses, but the ABR automatic quality control in the encoder / decoder section may be performed to dynamically adjust the resolution and frame rate according to changes in network conditions.

[0062] In addition, the screen display can be selected between full-screen mode or split-screen mode (simultaneous view for two people), the view can be switched by tapping the screen, and voice messages can be sent to the corresponding tactical subject by long-pressing, and simultaneous support for tactical network and P2P communication is also possible.

[0063] The second terminal (100a) receives encoded and encrypted video data from the first terminal (100), decodes and decodes the received video data, and can display it on a video output means. The second terminal (100a) may be implemented as a rugged smartphone or a rugged tablet, for example, if the first terminal (100) is smart glasses, the second terminal (100a) may be implemented as a heterogeneous device of a different type from the first terminal (100).

[0064] The second terminal (100a) can perform the same function as the first terminal (100) and can be used for a commander. That is, when the second terminal (100a) functions as a multi-image output device, the viewpoint images of nine combat subjects can be viewed through a multi-viewpoint monitoring mode that simultaneously displays viewpoint images received from one or more first terminals (100) in an m×n multi-screen split format, for example, a 3×3 multi-screen split format.

[0065] In addition, identification information including the name, rank, and location of the corresponding participant, status information including the remaining battery level, and network quality information including signal strength can be overlaid and displayed in each screen split area.

[0066] Additionally, when a user (e.g., a commander) selects a specific screen split area, the second terminal (100a) can enlarge the viewpoint image of the corresponding participant (e.g., a combatant) to the full screen and directly transmit instructions to the corresponding participant through an interaction channel such as a voice instruction channel.

[0067] Additionally, the second terminal (100a) can output a warning display to the corresponding screen split area when the connection of a specific device is disconnected or the battery level drops below a preset threshold.

[0068] In this embodiment, for convenience of explanation, the first terminal (100) and the second terminal (100a) are used for explanation, but are not limited thereto. The system may be configured to include a plurality of n terminals (100n) to enable real-time sharing of viewpoint images captured from an autonomous vehicle or flying vehicle, such as a drone.

[0069] The network layer (200) can support video data transmission between the first terminal (100) and the second terminal (100a) and can be physically implemented as a communication interface including a terminal-embedded wireless interface, tactical communication equipment, field-deployed network infrastructure, and a transmission medium.

[0070] That is, the network layer (200) may include one or more of the following: direct peer-to-peer (P2P) communication between terminals such as Wi-Fi Direct, UWB, Bluetooth 5.0+ or ​​higher, relay-based mesh network, tactical communication network such as TICN (Korean military tactical information communication system), WIN-T (US military Warfighter Information Network-Tactical), Nett Warrior (US military tactical force tracking system), and vehicle or base-based edge server relay methods, and may select a transmission method depending on network availability.

[0071] The session management terminal (300) can be implemented as a distributed type embedded in each device or as a central type deployed on an edge server, and can perform the creation, maintenance, termination, access control, resource management, latency-based encoding parameter adjustment, emergency blocking and session data deletion, and recipient identification-based watermarking of a video streaming session between the first terminal (100) and the second terminal (100a).

[0072] To this end, the session management terminal (300) may be configured to include a session management unit (310), an access control unit (320), a bandwidth management unit (330), a parameter adjustment unit (340), an emergency blocking unit (350), and a watermarking unit (360).

[0073] The session management unit (310) can manage the entire lifecycle of video streaming, namely the creation, maintenance, and termination of sessions.

[0074] Additionally, when the session management unit (310) receives a session creation request from the first terminal (100), it checks whether the device is authenticated, and if authentication is complete, it performs hierarchical access control list (ACL) verification through the access control unit (320), and can establish a streaming session and allocate bandwidth priority only if the authority is approved.

[0075] Meanwhile, if the authority is not approved, the session creation request may be rejected, and after the session is established, a session request response signal is transmitted to the first terminal (100), which is the transmitting device, and a stream distribution signal is transmitted to the second terminal (100a), which is the receiving device, thereby maintaining the normal operation of the streaming session.

[0076] The access control unit (320) is configured to manage and verify a hierarchical access control list (ACL) according to the command system, and specifically, the commander may be granted the authority to view the viewpoint images of all combatants who are tactical subjects under his command.

[0077] Additionally, the access control unit (320) can restrict access so that only the viewpoint video of a combatant, who is a specific tactical subject designated through the session management terminal (300), can be viewed.

[0078] ACL verification is performed in conjunction with the device authentication process during session creation and can be updated in real-time according to changes in tactical situations even while the session is in progress.

[0079] The bandwidth management unit (330) can determine the combat status of each device in real time and perform priority-based bandwidth distribution.

[0080] For example, it can operate in such a way that the highest priority bandwidth is allocated to the video stream of a combatant engaged in combat, and the remaining bandwidth is distributed to tactical subjects in a non-combat state.

[0081] In addition, the bandwidth management unit (330) can collect information on the display size, screen resolution, and remaining battery level of the second terminal (100a) in real time, and can independently adjust the video format and quality according to the type of receiving terminal based on the collected information.

[0082] For example, a low-resolution stream for battery saving can be distributed to the first terminal (100), which is a smart glass, and multiple streams with a resolution suitable for m×n tile display can be distributed to the second terminal (100a), which is a rugged tablet.

[0083] The parameter control unit (340) can monitor the network status in real time and measure the end-to-end delay time between the first terminal (100) and the second terminal (100a).

[0084] When the end-to-end delay time is maintained at 500ms or less, the parameter adjustment unit (340) feeds back an ABR parameter adjustment command to the encoder / decoder unit (150) of the first terminal (100) so that it can be immediately utilized for tactical decision-making based on a low-latency transmission environment.

[0085] In other words, maintaining end-to-end latency at 500ms or less can provide real-time capabilities for the network to respond immediately to tactical situations.

[0086] On the other hand, if the end-to-end transmission delay exceeds 500ms or the packet loss rate exceeds a preset threshold, the parameter adjustment unit (340) can directly feed back a command to the encoder / decoder unit (150) of the first terminal (100) to lower the video resolution and frame rate.

[0087] This allows the ABR parameter adjustment feedback loop to operate continuously so that video streaming continuity can be maintained even if network quality degrades in a battlefield environment, thereby maintaining the reliability and real-time capabilities of the entire system.

[0088] The emergency blocking unit (350) can perform immediate session blocking and data deletion in the event of an emergency situation, such as the capture of combatants who are tactical subjects, the theft of equipment, or the reception of external commands.

[0089] That is, when an emergency situation occurs or a command regarding an emergency situation is received from the outside, the emergency cutoff unit (350) generates an emergency cutoff (Kill Switch) signal and simultaneously propagates it to all connected terminals.

[0090] Each of the first terminal (100) and the second terminal (100a) that received the emergency blocking signal can immediately terminate the video streaming session and delete the session video data and session key stored locally.

[0091] Additionally, the emergency blocking unit (350) can check whether the entire block is complete by collecting responses regarding the completion of the emergency block from all target devices, and if no response is received from a specific device, it can ensure that the emergency blocking command is retransmitted to that device so that a complete block can be achieved.

[0092] The watermarking unit (360) can insert protection information, i.e., a digital watermark, including a recipient-related identification element, into the image data distributed to the second terminal (100a) in an invisible form.

[0093] The protection information inserted through the watermarking unit (360) may include identification elements that can uniquely identify the recipient, such as a receiving terminal identifier and recipient account information, and may be processed so as not to affect the visual quality of the image even after the insertion process.

[0094] In addition, the watermarking unit (360) can insert a watermark into the middle frequency band of the image by applying, for example, frequency domain conversion (DCT / DWT) during the watermark insertion process, and simultaneously secure invisibility and robustness through quality verification using PSNR and SSIM indicators.

[0095] In addition, the insertion strength and insertion position pattern can be dynamically adjusted, and the processing delay is controlled to, for example, 5ms or less per frame, so that it can operate stably even in a real-time streaming environment.

[0096] In addition, the watermarking unit (360) can be used as a means to identify the leakage path of military secret images by extracting and comparing the digital watermark when the image is leaked externally, thereby allowing the image receiving device and the recipient to be tracked afterward.

[0097] Additionally, the watermarking unit (360) can request immediate session termination and data deletion (emergency blocking) in conjunction with the session management unit (310) and the emergency blocking unit (350).

[0099] The following describes a method for real-time sharing of a viewpoint image according to an embodiment of the present invention.

[0100] FIG. 4 is a flowchart illustrating a method for real-time sharing of viewpoint images according to an embodiment of the present invention.

[0101] As shown in FIGS. 1 to 4, in a real-time sharing method for a viewpoint image according to an embodiment of the present invention, when a specific trigger signal for a video sharing session start command occurs, a viewpoint image or screen data captured at a first terminal (100) is captured (S100).

[0102] In step S100, when the image acquisition unit (110) of the first terminal (100) starts capturing a viewpoint image, the processor unit (120) selects a capture type, and the captureable image may selectively include one or more of the original image, the image restoration processing image, and the HUD composite image.

[0103] Additionally, the first terminal (100) can perform encoding that adaptively adjusts the image resolution, frame rate, and bit rate according to the available network environment for the captured image data (S200).

[0104] The encoder / decoder unit (150) of the first terminal (100) performs ABR encoding that adaptively adjusts video resolution, frame rate, and bit rate according to the available network environment, and the codec is selected as, for example, one of the H.265 / HEVC low-latency profile or AV1 real-time mode, and after the codec selection, branch processing to determine whether there is sufficient bandwidth can be performed.

[0105] In addition, in normal conditions where bandwidth is sufficient, it is encoded in a high-quality mode of 1080p / 30fps or higher, and when bandwidth is insufficient, it can be automatically switched to a low-quality mode of 480p / 15fps or lower.

[0106] In addition, when the battery level is 20% or less, a low-resolution mode is automatically applied to operate in a battery-saving mode, and in smart glasses (400, 400a) with limited computing resources, power consumption can be minimized by prioritizing the use of a DSP or NPU-based hardware encoder.

[0107] Meanwhile, if the session management terminal (300) detects that the end-to-end transmission delay exceeds 500ms, an ABR parameter adjustment feedback loop may be operated to provide feedback on a downward adjustment command for the resolution and frame rate.

[0108] In addition, step S200 can generate a transmission packet by performing security authentication and encryption on the encoded video data.

[0109] That is, the video data that has been encoded can be encrypted in real time through the security module (160), for example, using an AES-256-GCM authentication encryption algorithm.

[0110] Additionally, the session key used for encryption can be securely exchanged between the first terminal (100) and the second terminal (100a) using a Diffie-Hellman key exchange method when the session is created, and can be periodically updated during the session.

[0111] The encrypted video data is packetized into RTP or SRTP packets to form a transmission packet, and is transmitted to the network layer (200) through the communication unit (140).

[0112] At this time, the session management terminal (300) can transmit to the second terminal (100a) through one or more transmission methods selected according to network availability among direct communication between terminals, a mesh network, a tactical communication network, and an edge server relay method.

[0113] While performing transmission through step S200, the session management terminal (300) can comprehensively perform and manage the creation, maintenance, termination, access control, resource management, delay-based encoding parameter adjustment, emergency blocking and session data deletion, and recipient identification-based watermarking of a video streaming session between the first terminal (100) and the second terminal (100a) (S300).

[0114] In step S300, the session management terminal (300) can sequentially perform receiving a session creation request, verifying device authentication, verifying hierarchical ACL, determining whether authorization is granted, establishing a streaming session, and allocating bandwidth priority.

[0115] If the authority is not approved, the request is rejected, and if approved, the session management terminal (300) can transmit a session request response signal to the first terminal (100) and a stream distribution signal to the second terminal (100a).

[0116] Additionally, in step S300, the session management terminal (300) can collect display size, screen resolution, and battery level information from the second terminal (100a) and independently adjust the video format and quality according to the type of receiving terminal.

[0117] Additionally, at step S300, the session management terminal (300) can insert protection information (digital watermark) including recipient-related identification elements, such as a receiving terminal identifier and recipient account information, into the video data distributed to the second terminal (100a) in an invisible form.

[0118] Through this, if the video is leaked externally, the video receiving device and the recipient can be tracked afterward by extracting protection information.

[0119] Additionally, at step S300, the session management terminal (300) collects emergency blocking completion responses from all target devices to check whether the entire block is complete, and if no response is received from a specific device, it may resend an emergency blocking command to that device.

[0120] Furthermore, the second terminal (100a) can perform real-time decoding of the transmission packet received through the network layer (200) using the AES-256-GCM algorithm and a session key, and display the video decoded by an H.265 / HEVC or AV1 decoder according to the type of the second terminal (100a) (S400).

[0121] That is, as shown in FIG. 5(a), when the first terminal (100) is smart glasses (400) and the second terminal (100a) is smart glasses 1 (400a), the restored POV of the smart glasses (400) which is the first terminal (100) can be displayed by overlaying a viewpoint image in the form of a Picture-in-Picture (PIP) on a part of the user's actual field of vision in the smart glasses 1 (400a) which is the second terminal (100a).

[0122] In addition, the full screen may be switched by the user's voice command or gaze input, or when multiple videos are received, a preview thumbnail list may be displayed at the edge of the HUD so that the user can select the video to view by gaze or voice.

[0123] In addition, as shown in FIG. 5(b), when the first terminal (100) is a smart glass (400) and the second terminal (100a) is a rugged tablet (500), a plurality of POVs can be displayed through a multi-viewpoint monitoring mode that simultaneously displays up to nine viewpoint images in a 3×3 multi-screen split form on the second terminal (100a), for example.

[0124] In addition, the participant's identification information, battery level, and network quality information can be overlaid on each screen split area, and when a specific screen area is selected among multiple screens, the participant's video is enlarged to full screen and direct instructions can be delivered through a voice instruction channel.

[0125] Additionally, if the second terminal (100a) is a rugged smartphone (600), the viewpoint image may be displayed in full screen or split screen, and identification information of the corresponding participant may be overlaid on the top of the screen.

[0126] In addition, as shown in FIG. 5(c), if the first terminal (100) is a rugged tablet (500) and the second terminal (100a) is a smart glass (400), tactical map information can be displayed on the HUD of the second terminal (100a).

[0127] In addition, as shown in FIG. 5(d), if the first terminal (100) is a rugged smartphone (600) and the second terminal (100a) is a smart glass (400), the restored POV can be displayed on the HUD in a PIP format.

[0128] In addition, if a specific device is disconnected, a warning icon may be displayed in the corresponding screen split area.

[0129] In addition, as shown in FIG. 6, in a situation where different types of terminals are mixed, some of which are smart glasses (400, 400a) and rugged smartphones (600), and some are rugged tablets (500), the session management terminal (300) can control the video quality to be automatically adjusted and shared according to the display size, resolution, and battery status of each terminal.

[0130] Additionally, the session management terminal (300) can monitor the network status in real time to check if the end-to-end delay time from step S100 to step S400 is 500ms or less, and based on this, feed back an ABR parameter adjustment command to the first terminal (100).

[0131] If the end-to-end transmission delay exceeds 500ms or the packet loss rate exceeds a preset threshold, the session management terminal (300) can directly feed back a command to lower the video resolution and frame rate to the first terminal (100).

[0132] This enables the ABR parameter adjustment feedback loop to operate continuously so that video streaming continuity can be maintained even if network quality degrades in a battlefield environment.

[0133] Meanwhile, the session management terminal (300) determines whether to continue the session (S500), and depending on the result of the determination, maintains or terminates the real-time sharing of the viewpoint video.

[0134] Additionally, the session management terminal (300) can determine whether an emergency situation occurs or an emergency block occurs when an external command is received while maintaining real-time sharing of the time-view video (S600).

[0135] As a result of the judgment in step S600, an emergency block occurs, and the session management terminal (300) generates an emergency block signal and propagates it simultaneously to all connected terminals. Upon receiving the emergency block signal, the first terminal (100) and the second terminal (100a) immediately terminate the streaming session and delete the session video data and session key stored locally, after which the real-time sharing of the viewpoint video is terminated.

[0136] Therefore, camera video or screen data from a specific device can be streamed in real time to the displays of other devices between edge devices.

[0137] In addition, video sharing between various edge devices such as smart glasses, smartphones, and tablets is possible, which can dramatically improve the quality of battlefield situational awareness and maximize command and control efficiency by enabling immediate use in tactical decision-making with a latency of 500ms or less.

[0138] Furthermore, it can provide the highest level of security required for military operational environments through real-time encryption, hierarchical access control, emergency blocking, and digital watermarking, and enables comprehensive situational awareness of the entire battlefield by allowing simultaneous monitoring of multiple viewpoints.

[0139] In addition, it is possible to share head-up display images synthesized with tactical information overlays or restored footage from night or smoke environments, enabling the immediate dissemination of enhanced information secured by the forward tactical entity to the entire unit.

[0140] Meanwhile, the real-time sharing system for viewpoint images according to one embodiment of the present invention is not limited to a specific device and can be applied to various types of edge devices capable of performing data processing and control functions.

[0141] For example, edge devices may include, but are not limited to, drones, autonomous vehicles, robots, autonomous vessels, intelligent toys, leisure mobility devices, smart factory production robots, smart home appliances, wearable devices, sensors, real-time security monitoring, medical imaging devices, etc.

[0142] In addition, it is applicable to the sports and leisure sectors, and for example, it can be applied to devices for golf convenience such as smart goggles, rangefinders, CCTV devices for video recording or control, and kiosk devices that provide user interfaces.

[0143] In addition, it is not limited to the various devices exemplified, but can be extended to all types of terminal devices capable of operating in an edge environment.

[0144] As described above, although the present invention has been explained with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the invention as described in the following claims.

[0145] In addition, the drawing numbers described in the claims of the present invention are provided for clarity and convenience of explanation only and are not limited thereto, and in the process of describing the embodiments, the thickness of lines or the size of components shown in the drawings may be exaggerated for clarity and convenience of explanation.

[0146] Furthermore, the terms described above are defined in consideration of their functions in the present invention, and since these may vary depending on the intentions or practices of the user or operator, the interpretation of these terms should be based on the content throughout this specification.

[0147] Furthermore, it is obvious that a person skilled in the art to which the present invention pertains can make various modifications including the technical concept according to the present invention from the description of the present invention, even if not explicitly shown or described, and such modifications still fall within the scope of the rights of the present invention.

[0148] Furthermore, the above embodiments described with reference to the attached drawings are described for the purpose of explaining the present invention, and the scope of the present invention is not limited to these embodiments. Explanation of the symbols

[0149] 100: 1st terminal 100a: 2nd terminal 110: Image acquisition unit 120: Processor unit 130 : Display unit 140 : Communication unit 150: Encoder / Decoder Section 160: Security Module Section 100n: nth terminal 200: Network layer 300: Session management terminal 310: Session management unit 320: Access Control Unit 330: Bandwidth Management Unit 340: Parameter adjustment unit 350: Emergency cutoff unit 360: Water marking section 400: Smart glass 400a : Smart Glasses 1 500 : Tablet 600 : Smartphone

Claims

Claim 1 A first terminal (100) that captures a captured point-of-view video or screen data, performs encoding that adaptively adjusts video quality according to the available network environment, and outputs the encoded video data after security authentication and encryption; a second terminal (100a) that receives the encoded and encrypted video data from the first terminal (100), decodes and decodes the received video data, and displays it on a video output means; and a network layer (200) that supports video data transmission between the first terminal (100) and the second terminal (100a), includes one or more of direct communication between terminals, a mesh network, a tactical communication network, and an edge server relay method, and selects a transmission method according to network availability. A real-time video sharing system comprising: a session management terminal (300) that performs the creation, maintenance, termination, access control, resource management, delay-based encoding parameter adjustment, emergency blocking and session data deletion, and recipient identification-based watermarking of a video streaming session between the first and second terminals (100, 100a); wherein the end-to-end delay time from the first terminal (100) to the second terminal (100a) is 500ms or less. Claim 2 A real-time video sharing system according to claim 1, wherein the session management terminal (300) collects the display size, resolution, and battery level of each receiving terminal and independently adjusts the video format and quality according to the type of receiving terminal. Claim 3 A real-time sharing system for viewpoint images according to claim 1, wherein when the second terminal (100a) is a multi-video output device, it provides a multi-viewpoint monitoring mode that simultaneously displays viewpoint images received from one or more first terminals (100) in an m×n multi-screen splitting format, overlays identification information, status information, and network quality information of the corresponding participant in each screen splitting area, provides a function to expand the viewpoint image of the corresponding participant to the full screen when a specific screen splitting area is selected and directly transmit instructions to the corresponding participant through an interaction channel, and provides a warning display in the corresponding screen splitting area when the connection of a specific device is disconnected. Claim 4 A real-time sharing system for point-of-view images according to claim 1, wherein the session management terminal (300) inserts protection information including recipient-related identification elements into the image data distributed to the second terminal (100a) in an invisible form, and enables tracking of the image receiving device and the recipient from the protection information when the image is leaked to the outside. Claim 5 A real-time video sharing system according to claim 1, wherein the session management terminal (300) generates and propagates an emergency block that is simultaneously transmitted to all connected terminals when an emergency situation occurs, and the first terminal (100) and the second terminal (100a) immediately terminate the video streaming session and delete the session video data and session key stored locally when they receive the emergency control command, and the session management terminal (300) collects responses regarding the completion of the emergency block from all target devices to check whether the entire block is complete. Claim 6 delete Claim 7 A real-time viewpoint image sharing system according to claim 1, characterized in that the security authentication and encryption are performed using one or more of AES-256-GCM, ChaCha20-Poly1305, or SEED algorithm. Claim 8 a) capturing a point-of-view video or screen data captured by a first terminal (100); b) the first terminal (100) performing encoding that adaptively adjusts the video resolution, frame rate, and bit rate according to the available network environment on the captured video data, and performing security authentication and encryption on the encoded video data to generate a transmission packet; c) the first terminal (100) transmitting the transmission packet to a second terminal (100a) through one or more transmission methods selected according to network availability among direct communication between terminals, a mesh network, a tactical communication network, and an edge server relay method; d) the session management terminal (300) performing creation, maintenance, termination, access control, resource management, latency-based encoding parameter adjustment, emergency blocking and session data deletion, and recipient identification-based watermarking of a video streaming session between the first terminal (100) and the second terminal (100a); and e) a step in which the second terminal (100a) decodes and decodes a received transmission packet and displays it on an image output means; wherein the end-to-end delay time from the first terminal (100) to the second terminal (100a) is 500ms or less, the method for real-time sharing of a viewpoint image. Claim 9 In claim 8, the above step d) is characterized in that the session management terminal (300) collects the display size, screen resolution, and battery level of the second terminal (100a), and independently adjusts the video format and quality according to the receiving terminal type based on the collected information, in a real-time sharing method for point-of-view video. Claim 10 In claim 8, the above step d) further comprises: a step in which the session management terminal (300) inserts protection information including a recipient-related identification element into video data distributed to the second terminal (100a) in an invisible form; and a step of enabling tracking of a video receiving device and a recipient from the protection information when the video is leaked externally; characterized in that the point-of-view video real-time sharing method. Claim 11 In claim 8, the above step d) further comprises: a step in which, when an emergency situation occurs or an external command regarding an emergency situation is received, the session management terminal (300) generates an emergency block and simultaneously propagates the generated emergency block to all connected terminals; a step in which each of the first terminal (100) and the second terminal (100a) that received the emergency block immediately terminates the video streaming session and deletes the session video data and session key stored locally; and a step in which the session management terminal (300) collects responses regarding the completion of the emergency block from all target devices to check whether the entire block has been completed; characterized in that the point-of-view video real-time sharing method further comprises Claim 12 In claim 8, the above step e) is characterized by comprising: a step of providing a multi-viewpoint monitoring mode that simultaneously displays viewpoint images received from one or more first terminals (100) in an m×n multi-screen split form when the second terminal (100a) is a multi-video output device; a step of overlaying identification information, status information, and network quality information of the corresponding participant in each screen split area; a step of expanding the viewpoint image of the corresponding participant to the full screen and directly transmitting instructions to the corresponding participant through an interaction channel when a specific screen split area is selected; and a step of providing a warning display in the corresponding screen split area when the connection of a specific device is disconnected.

Images