Predictive field of view (FOV) and queues for enforcing compliance of data capture and transmission in real-time and near real-time video

The method predicts the FOV of a video camera using a 3D sensor to prevent capture of unauthorized objects, ensuring real-time compliance with data capture and transmission protocols, addressing the limitations of existing systems in secure environments.

JP2024541875AActive Publication Date: 2024-11-13RAYTHEON CO
View PDF 14 Cites 0 Cited by

Patent Information

Application Number
JP2024523577
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-10-21
Filing Date
2022-10-19
Publication Date
2024-11-13
Estimated Expiration
2042-10-19

AI Technical Summary

Technical Problem

Existing video capture and processing systems fail to ensure real-time or near real-time compliance with data capture and transmission requirements in secure environments, leading to potential data exfiltration and violation of privacy or security protocols due to the inability to prevent unauthorized capture and transmission of sensitive information.

Method used

A method that predicts the field of view (FOV) of a video camera using a 3D sensor to generate a ground truth map, identifies allowed and disallowed objects, and controls the camera to prevent capture of unauthorized objects by adjusting its pointing direction or disabling it, ensuring compliance with alignment conditions.

Benefits of technology

Enforces real-time or near real-time compliance by preventing unauthorized objects from being captured and transmitted, thereby enhancing security and privacy in secure environments, such as augmented reality applications, without the need for post-capture editing or equipment relocation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541875000001_ABST
    Figure 2024541875000001_ABST
Patent Text Reader

Abstract

To prevent capture and transmission of excluded data, the current pose and motion of the video camera are used to predict the pose and predicted FOV of the video camera over one or more future frames. The predicted pose and predicted FOV are used to generate cues to enforce alignment conditions on permitted objects or prevent capture of disallowed objects. If the cues fail, an interrupt is generated to prevent capture of disallowed objects in the video signal and deactivate the video camera. The predicted FOV prevents the excluded data from entering the video signal and reaching circuitry or being processed downstream of the video camera. This can be implemented in real time or near real time.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Claiming priority This patent application claims the benefit of priority to U.S. Patent Application No. 17 / 507,111, filed October 21, 2021, which is incorporated by reference in its entirety herein. [Background technology]

[0002] The present invention relates to the capture and processing of video to enforce compliance of data capture and transmission in a private, restricted or secure environment in real-time or near real-time.

[0003] 2. Description of Related Art Video camera technology is becoming more prevalent throughout the world today. For example, devices such as head-mounted cameras, robot-controlled cameras, semi-autonomous or autonomous robots, mobile phones, desktop or table computers, near-eye displays, handheld gaming systems, and the like, may contain cameras and associated software that enable video capture, display, and transmission. These devices are being used to provide one-way or two-way video communication in real time or near real time. Privacy and security concerns arise when video that should not be captured, stored, displayed, or transmitted, whether intentionally or unintentionally, is so captured. The privacy of individuals, companies, or countries may be violated, possibly illegally. In certain restricted environments, such as military or corporate proprietary or secure environments, strict controls exist that govern what visual information can be captured, stored, displayed, or transmitted.

[0004] To limit unwanted video capture or transmission, some existing systems monitor the video as it is captured. These systems use human processing, artificial intelligence (AI), computational algorithms, or a combination thereof to identify problematic visual information (such as a person's face or a company's proprietary information) and then remove or conceal that information from the video file data. These systems may even shut off the recording device to prevent further capture of the problematic information. However, all of the existing systems described capture, store, and process the problematic information. Because the problematic data is stored (albeit only temporarily) and processed, the risk of data leakage still exists, which makes these systems unable to meet the requirements of certain secure or restrictive environments. The processing required to remove or conceal information from video files makes these systems incompatible with applications that require real-time or near real-time video capture and transmission.

[0005] Video capture to enforce data capture and transmission compliance in real-time or near real-time may be required for a variety of applications for individual users, companies, or countries. Such applications may include, but are not limited to, inspection / process review, supplier quality control, internal audits, equipment or system troubleshooting, factory operations, factory collaboration, verification and validation, equipment repair and upgrades, and equipment or system training. These applications may require the capture and transmission in one or two directions of video of the local scene containing the information in question or real-time or near real-time to achieve efficient and effective communication. As a special case, data capture and transmission compliance may be implemented in an augmented reality environment.

[0006] Augmented reality (AR) refers to the generation of two-dimensional or three-dimensional (3D) videographic or other media that is superimposed and registered onto surrounding objects in an environment. Artificial "markers," also known as "sources," with unique and easily identifiable characteristics, are placed on the user, objects, or scene and may be used for a variety of purposes. These markers are used to identify and locate certain objects, trigger the display of computer-generated media, or determine the user's position and pose.

[0007] A concern in certain video or AR environments, such as remote repair or inspection, primarily from customers and amplified by the video camera industry's push to maximize FOV, is that the user of the video being captured, transmitted, or viewed locally (either a field technician or expert, but primarily a field technician) may intentionally or unintentionally look away from the object of interest and capture video of another part of the scene that should not be captured or transmitted. To avoid unintentional or intentional transmission of a wide FOV, some level of data capture and transmission compliance may be required by customer requirements, industry regulations, national security, or country-specific laws. Current techniques include physically covering the object of interest with cloth or tarps so that it is not captured in the video signal, mechanically narrowing the FOV, or isolating the video before transmission and having a security-certified domain expert review and edit the video signal after capture. These are time-consuming activities. Even more common, and more costly, is to move the equipment in question to a special secure location, such as an empty garage or hangar, to ensure that no extraneous items are left on-site. In many cases, equipment removal, physical draping, or post-capture editing are not sufficient to meet compliance requirements or are impractical and costly to implement in near-real-time interactive situations. In some situations, national laws exist that prevent any kind of post-capture editing for national security or ITAR - International Traffic in Arms Regulations - reasons.

[0008] US Patent Nos. 10,403,046 and 10,679,425 entitled "Field of View (FOV) and Key Code Limited Augmented Reality to Enforce Data Capture and Transmission Compliance" disclose enforcing an alignment condition between the pointing direction of a video camera and a marker in a scene to avoid capturing excluded data in a real-time interactive situation. This can be done, for example, by determining whether the pointing direction of the video camera satisfies an alignment condition with respect to a marker in a local scene such that the FOV of the video camera is within a user-defined acceptable FOV around the marker. To satisfy the alignment condition, another sensor can be used to detect the presence of the marker within the sensor's FOV. The camera or sensor's FOV can be reduced to create a buffer zone to further ensure that the camera's FOV does not go outside the acceptable FOV. If the alignment condition is not satisfied, the video camera is controlled to exclude at least a portion of the camera FOV that is outside the user-defined acceptable FOV from being captured in the video signal. For example, this can be done by turning off the video camera or narrowing the FOV. The markers can also be used as a fail-safe to ensure that images of particularly sensitive areas of the scene are not captured or transmitted. If another sensor detects these markers, the video camera is shut down. The system may signal to the user, for example, "green" means that the alignment condition is met, "yellow" means that the technician's eyes are starting to wander, and "red" means that the alignment condition has been violated and the camera is disabled.By using additional sensors to enforce alignment conditions and detect other markers in sensitive areas, the system is specifically designed for more stringent environments or those requiring real-time or near real-time interaction where compliance requires that part of the scene or tagged object cannot be captured (detected) by the video camera itself and output in the video signal. Summary of the Invention

[0009] The following is a summary of the invention to provide a basic understanding of some aspects of the invention. This summary is not intended to identify key or critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description and claim definitions that are presented later.

[0010] The present invention provides a method for predicting the FOV of a video camera to enable compliance enforcement for real-time and near real-time video data capture and transmission.

[0011] To provide real-time or near real-time video data capture and transmission compliance enforcement, the present invention predicts the future FOV of a video camera to recognize disallowed objects and cue or control the video camera to prevent their capture in the video signal. The predicted FOV can also be used to enforce alignment requirements of the video camera relative to permitted objects.

[0012] In an embodiment, a 3D sensor, such as a 3D camera, LIDAR, or sonar, is used to generate a three-dimensional ground truth map of a local scene that includes one or more objects. One or more of the objects in the ground truth map are identified as not allowed or potentially allowed. 2D or 3D models may be used to represent the identified allowed and not allowed objects. A video camera captures a video signal at a frame rate within a camera's field of view (CFOV) in a pointing direction of the local scene. The system determines a pose, including a position and orientation of the video camera within the local scene for a current frame, and calculates a predicted pose and predicted FOV (PFOV) for one or more future frames from the pose of the current frame and measurements of the video camera's velocity and acceleration. The one or more predicted FOVs are compared to the ground truth map to recognize and identify not allowed and potentially allowed objects. If a not allowed object is recognized, the video camera is controlled to prevent capture of the not allowed object and its inclusion in one or more future frames of the video signal.

[0013] In certain embodiments, a cue may be generated to change the pointing direction of the video camera to prevent capture of the disallowed object and its inclusion in one or more future frames of the video signal. After generating the cue, the predicted FOV is updated to determine whether the updated predicted FOV contains the disallowed object. If the cue does not prevent capture of the disallowed object in the updated predicted FOV, the video camera is controlled to prevent capture of the disallowed object and its inclusion in the video signal. As a result, the disallowed object is not included in the video signal and therefore does not reach downstream circuitry, processing, or network to which the video camera is connected.

[0014] In various embodiments, objects may be defined and prohibited based on other attributes, not just content. For example, any object identified as being too close or too far from a video camera may be designated as not allowed. The movement speed (e.g., velocity and acceleration) of an object entering a local scene may be defined as an object and not allowed if its movement speed exceeds a maximum value. Additionally, the movement speed (e.g., velocity and acceleration) of a video camera may be defined as an object and not allowed if its movement speed exceeds a maximum value.

[0015] In a different embodiment, the video camera is trained on allowed objects and moved away from disallowed objects. The system determines whether the current and predicted pointing direction of the camera meets an alignment condition for one of the allowed objects. If not, the system generates a cue to change the pointing direction of the video camera to enforce the alignment condition. If the cue cannot enforce the alignment condition, the video camera is deactivated. The ground truth map can be used to verify the recognized allowed objects.

[0016] In various embodiments, the pointing direction of the video camera is coordinated or controlled by user movement (e.g., a head-mounted video camera), a user-controlled manual manipulator (e.g., a robotic arm), or a fully automated manual manipulator (e.g., an AI-controlled robotic arm, or a semi-autonomous or autonomous robot). For example, an audio, video, or vibratory cue may be presented to the user to move the user's head away from the disallowed object before capturing it, or back to the allowed object before violating an alignment condition. Similarly, a cue may be presented to the user of a robotic arm to move the camera away or back to the allowed object before capturing the disallowed object. In a fully automated system, the cue may override the system and cause the camera to point away from the disallowed object or toward the allowed object.

[0017] In different embodiments, the video camera can be controlled to prevent capture of unauthorized objects in various ways: the video camera can be controlled mechanically to change its pointing direction, optically to narrow the CFOV, to blur unauthorized objects before they reach the detector array (e.g., change the f / #), to change the local scene illumination to blind the sensor, or electrically to cut off power, to disable the electrochemical layer of the detector array, the amplifiers or A / D converters of the ROIC, to selectively turn off pixels of unauthorized objects, etc.

[0018] These and other features and advantages of the present invention will become apparent to those skilled in the art from the following detailed description of the preferred embodiments, taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0019] [Figure 1]FIG. 1 is a diagram of a system that provides predictive cues for a predicted FOV for a video camera and enables compliance enforcement for real-time and near real-time video data capture and transmission. [Diagram 2] FIG. 1 is a block diagram of an embodiment of a video capture and transmission device that enforces compliance of real-time and near real-time video data capture and transmission. [Diagram 3] FIG. 1 is a diagram of a 3D ground truth map of a local scene, including both allowed and disallowed objects, for identifying and locating objects in a predicted FOV. [Figure 4] FIG. 1 is a diagram of a sequence of captured image frames forming a video signal, using a predicted FOV to issue an interrupt before an image frame containing a disallowed object is captured into the video signal. [Diagram 5] FIG. 11 is a diagram of a reference sensor self-generated map aligned against a ground truth map to determine a 6DOF pose. [Figure 6] FIG. 1 is a diagram of an embodiment of an augmented reality (AR) system that provides predictive cues for a predicted FOV for a video camera, enforces alignment conditions on permitted objects, and prevents capture of unauthorized objects. [Figure 7] FIG. 1 is a diagram of an embodiment of an augmented reality (AR) system that provides predictive cues for a predicted FOV for a video camera, enforces alignment conditions on permitted objects, and prevents capture of unauthorized objects. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Video capture to enforce data capture and transmission compliance in real-time or near real-time may be required for a variety of applications for individual users, companies, or countries. Such applications may include, but are not limited to, inspection / process review, supplier quality control, internal audits, equipment or system troubleshooting, factory operations, factory collaboration, verification and validation, equipment repair and upgrades, and equipment or system training. These applications may require video of the local scene containing the information in question or real-time or near real-time to be captured and transmitted in one or two directions to achieve efficient and effective communication. As a special case, data capture and transmission compliance may be implemented in an augmented reality environment. The pointing direction of the video camera is engaged or controlled by the user's movements (e.g., a head-mounted or handheld video camera), a user-controlled manual manipulator (e.g., a robotic arm), or a fully automated manual manipulator (e.g., an AI-controlled robotic arm, or a semi-autonomous or autonomous robot).

[0021] The present invention is directed to these and other similar applications where some level of data capture and transmission compliance may be required by customer requirements, industry regulations, national security, or country-specific laws. In some cases, compliance may require that a part of a scene or a particular tagged object cannot be included in the video signal that the video camera outputs for display or transmission. In more stringent environments, compliance may require that a part of a scene or a tagged object cannot be stored in the camera's memory chip and also output in the video signal. The memory chip may be a memory chip only or may be a video display or video transmission chip that includes persistent memory. The level of compliance required is determined by various factors and may vary during or between the capture and display or transmission of the video signal.

[0022] The present invention predicts the FOV of a video camera to recognize a disallowed object and control the video camera to prevent the capture of the disallowed object and its inclusion in one or more future frames of the video signal. The predicted FOV can also be used to enforce alignment conditions of the video camera with respect to permitted objects. Cues can be used to prompt correction of the pointing direction of the video camera to prevent the capture of the disallowed object before it occurs or to enforce it before the alignment condition is lost. If no corrective action is taken by the cue, the video camera is controlled to prevent the capture of the disallowed object or to punish the loss of the alignment condition. As a result, the disallowed object is not included in the video signal and therefore does not reach downstream circuitry, processing, or a network to which the video camera may be connected. This can be implemented in real time or near real time, or slower if the application does not require such performance, by placing a delay line or temporary memory chip between the ROIC of the video camera and the memory chip. For example, the slowest video frame rate acceptable to most users is about 24 frames per second (fps), or about 42 milliseconds (ms). A time delay of less than 42 ms is generally acceptable for most users. High-speed video cameras are 120 fps, or about 8 ms. At these frame rates, a delay of one frame is certainly real time. The predicted FOV can be used to enforce compliance of data capture and transmission to a single image or sequence of images in the video signal.

[0023] 1, in one embodiment, a video capture, display, and transmission device 10, such as video goggles or a handheld unit (e.g., tablet or cell phone), has a pointing direction 12 that is geared to the technician's movements (e.g., whether the field technician 14 is looking at or pointing at the unit). The device 10 includes a sensor 16 (e.g., 3D camera, LIDAR, sonar) configured to capture signals sensed in the sensor's FOV 18 about the pointing direction 12, and a video camera 20 (e.g., 2D or 3D CMOS, CCD, or SWIR camera) configured to capture light within a camera FOV 22 and within the sensor FOV 18 about the pointing direction 12 to form a video signal. The sensor (16) is used to generate a 3D ground truth map 24 of the local scene, in which certain objects 26 and 28 are indicated as allowed objects and objects 30 and 32 are indicated as not allowed objects. The objects may be represented in the ground truth map as actual sensor data or as computer-generated models.

[0024] In this example, a field technician 16 may travel through a manufacturing facility to verify the presence and location of, repair or maintain, or use certain objects. These objects may be considered "authorized" objects. The technician may even be prompted or instructed to maintain a pointing direction 12 to a specified object in order to perform a particular task (e.g., verify, repair, use, etc.). The field technician 16 can capture, display, and transmit "authorized" objects. The field technician 16 cannot capture "unauthorized" objects in memory, much less display or transmit them.

[0025] The allowed or disallowed objects may depend on any specific restrictions that apply to the application, the type of object in the local scene, the authority of the field technician, supervisor, and any remote staff, and compliance with excluded data and transmissions. This information may be stored on the device. The device processes the sensed signals to detect and locate the object and determine whether it is an allowed or disallowed object. The background of the local scene and ground truth map are set as allowed or disallowed by default. When an object that is not present in the ground truth map moves into the scene, it is not properly allowed until it is identified and marked as allowed. Similarly, an unrecognized object is not properly allowed until it is identified and marked as allowed. Instead of based on just its content, an object may be defined and disallowed based on other attributes. For example, any object identified as too close (<minimum distance) or too far (>maximum distance) from the video camera may be designated as disallowed. The speed (e.g., speed and acceleration) of an object entering the video camera or local scene may be defined as an object, and if its speed of movement exceeds a maximum value, it may be defined as disallowed. The device compares recognized objects with a ground truth map to verify whether they are the same object, whether they are allowed or not, and their location, significantly improving the accuracy and reliability of object recognition.

[0026] To prevent capture and transmission of excluded data, one or more predicted camera FOVs 35 for the video cameras are utilized to recognize disallowed objects and control the video cameras to interrupt and stop transferring images to the memory chip 34 where a video signal is formed. For example, if the current camera FOV 22 and the pointing direction 12 for the one or more predicted camera FOVs 35 meet the alignment conditions for an allowed object 26 to perform some task (e.g., the pointing direction is within a few degrees of a recommended line of sight (LOS)) and do not include any disallowed objects 30 or 32, the images captured by the video cameras are transferred to the memory chip 34 where they may be formed into a video signal and displayed to a field technician or transmitted to a remote location (e.g., for storage or display to another remote user).

[0027] If both conditions are met, the device can generate a positive cue (e.g., green "Good") to reinforce the user's focus on the permitted object. If the user's pointing direction begins to move away from the permitted object or toward the disallowed object, but has not yet violated either condition, the device can generate a prompt cue (e.g., yellow "Move Left") to take corrective action. If the user's pointing direction changes to the point where an alignment condition is violated or capture of a disallowed object is imminent, the device can control the video camera to prevent capture of the disallowed object and its inclusion in the video signal, or deactivate the camera and issue an interrupt cue (e.g., red "Deactivate Video Camera").

[0028] If any of the conditions are violated, the device controls the camera to prevent capture of a video signal containing the unauthorized object or issues an "interrupt" 36 when the alignment conditions are not met. If an alignment condition is violated, the device typically turns off the video camera by cutting off power to the video camera, deactivating the detector array or the electrochemical top layer of the ROIC, or pointing the video camera in a completely different direction. In the case of a violation that involves moving to capture an unauthorized object, in addition to these options, the video camera can be controlled to optically narrow the camera FOV, selectively blur parts of the camera FOV (e.g., by changing f / #), change the local scene lighting to blind the sensor, or selectively turn off or blur pixels in the detector array that correspond to the unauthorized object.

[0029] The same method can be applied to a remote user-controlled robotic arm that points a video camera, or to a fully autonomous robot that uses a video camera as part of its vision system. In the case of a robotic arm, a "time delay" can ensure that protected data is not captured and transmitted to a remote user site or elsewhere. In the case of a fully autonomous robot, a "time delay" can ensure that protected data is not captured and used by the robot or transmitted elsewhere.

[0030] The method can be applied to applications where only allowed objects are present, as well as to local scenes where only disallowed objects are present.

[0031] 2, 3, and 4, in one embodiment, a video capture and display device 100 includes a video camera 102, a sensor 104, and an interrupt processor 106 that controls the video camera to prevent unauthorized objects 132 from being captured in the video signal and to enforce alignment conditions on permitted objects 130. The device 100 is coupled to a "platform" 107, such as a user, a robotic arm, a robot, or the like, that controls the pointing direction of the device.

[0032] The sensor 104 includes a 3D sensor 124, such as a 3D camera, LIDAR, sonar, etc., and a sensor processor 126 that captures sensed signals within a sensor FOV (SFOV) 128 along a pointing direction 110 and forms a three-dimensional ground truth map 127 in which one or more allowed or disallowed objects are identified. The identified objects may be represented by the sensed signals or computer-generated models. As described below, the sensor may also be used to determine the pose of the camera. The sensor 104 is separate from a video processor 144 or any other circuitry or network downstream of the video capture and display device 100 to which a video camera may be connected to deliver a video signal.

[0033] The video camera 102 captures light within a camera field of view (CFOV) 108 in a pointing direction 110 within a local scene. The video camera suitably includes a power source 111, optics 112 for collecting light within the CFOV, a detector array 114 for sensing and integrating the light and converting photons to electrons to form an image, a readout integrated circuit (ROIC) 116 containing amplifiers and A / D converters for reading out the image sequence at a frame rate, and a memory chip 118 for storing the image sequence to form a video signal 120.

[0034] The interrupt processor 106 determines a pose 158 including the position and orientation of the video camera within the local scene for the current frame 159. This can be done using a gyroscope 150 that measures the 6DOF pose of the video camera (e.g., x, y, z, and rotations about x, y, z axes) or by matching signals sensed at the sensor FOV 128 for the current frame against a ground truth map 127. The interrupt processor 106 calculates a predicted pose 160 and a predicted FOV (PFOV) 162 for one or more future frames 163 from the pose of the current frame and measurements of the video camera's velocity and acceleration, suitably 6DOF, obtained by the motion sensor 152. The interrupt processor 106 compares the one or more predicted FOVs 162 with the ground truth map to recognize and identify disallowed objects 132 and potentially allowed objects 130.

[0035] The predictive nature of the system allows the interrupt processor 106 to generate cues 140 to change the pointing direction of the video camera to prevent capture of the unauthorized object and its inclusion in one or more future frames of the video signal without controlling or turning off the video camera. The cues 140 are configured to block the video camera from moving towards the unauthorized object before it occurs. After generating the cues, the interrupt processor updates one or more predicted FOVs and determines if the updated predicted FOV contains the unauthorized object. If the cues do not prevent capture of the unauthorized object in the updated predicted FOV, the interrupt processor 106 issues an interrupt 136 to control the video to prevent capture of the unauthorized object and its inclusion in the video signal.

[0036] When the video camera is trained on an allowed object 130 and away from a disallowed object 132, the interrupt processor 106 determines whether the current and predicted pointing direction of the camera meets the alignment condition for one of the allowed objects. If not, the system generates a cue 140 that changes the pointing direction of the video camera to enforce the alignment condition. If the cue cannot enforce the alignment condition, the video camera becomes inactive. Loss of the alignment condition does not necessarily mean that the camera will capture a disallowed object. However, if the video camera moves away from an allowed object and the cue cannot correct the problem, turning off the video camera at least temporarily can be effective in training the platform to maintain proper alignment with the allowed object and perform the task at hand. The length of time that the video camera is turned off can be varied to more effectively train a local or remote user or robot.

[0037] In various embodiments, objects may be defined and disallowed based on other attributes rather than content alone. For example, any object identified as being too close or too far from the video camera may be designated as not allowed. The speed of movement (e.g., velocity and acceleration) of the video camera may define an object to be disallowed if its speed of movement exceeds a maximum value.

[0038] The video processor 144 processes the video signal 120 to form a video output signal 146 for display 148 or transmission. If no unauthorized objects are detected, the video output signal 146 is a normal video signal. If an interrupt is issued, the video output signal 146 may receive no video signal if power is interrupted, may receive a blank or noisy video signal if the ROIC is deactivated, or may receive a video signal with pixels corresponding to unauthorized objects removed or hidden.

[0039] 5, one way to determine the 6DOF pose of a video camera is to use a sensor 200 that shares a pointing direction 202 with a video camera 204. As shown in this example, a pair of 3D sensors 202 are placed on either side of the video camera 204 to provide a sensor FOV 206 that surrounds and is aligned with the camera FOV 208. During operation and control of the video camera 204, the sensor 202 captures 3D sensed signals and forms a self-generated map 210 in the sensor FOV 206. An interrupt processor matches the 3D sensed signals with a 3D ground truth map 212 (generated by the same sensor) to determine the pose 214 of the video camera's current frame. A simultaneous localization and mapping (SLAM) algorithm (or similar) may be used to self-localize, and then align with the ground truth map 212 with reference to the self-generated map 210 to determine the exact position and orientation.

[0040] In an AR environment, the pointing direction of the video camera is coordinated with the movements of the on-site technician (e.g., the technician's head in the case of goggles, or hands in the case of a handheld unit). A video signal is captured within the FOV of an object in a local scene at arm's length from the technician. The technician receives hand gestures for manipulation of the object from the remote expert, registers them, and superimposes them on the video signal to create an augmented reality that instructs the user to manipulate the object. In various implementations, hand gestures are provided when the expert views and responds to a video signal captured by the technician's video camera and transmitted in real time to the remote site, when the expert interacts with a replica of the object in real time, or when the expert generates "canned" instructions offline by responding to the video or interacting with a replica of the object.

[0041] A concern is that the technician may intentionally or unintentionally look away from the object of interest and capture video of another portion of the scene or object that should not be captured or transmitted. The present invention provides a system and method, under technician / customer / master ("user") control, to automatically control a video camera in such a constrained AR environment to exclude portions of the scene for data capture and transmission compliance without interfering with the AR overlay that directs the technician to manipulate the object.

[0042] 6 and 7, an embodiment of an AR system 310 includes a video capture, display, and transmission device 312, such as video goggles or a handheld unit (e.g., tablet or cell phone), whose pointing direction 314 is linked to the technician's movements (e.g., whether the field technician 316 is looking at or pointing at the unit). The field technician 316 manipulates an object 318 in a local scene 320, for example, to perform maintenance on the object or to receive instructions on how to manipulate the object. In this example, the object 318 is an access panel and the local scene includes a tractor. The access panel is an "allowed" object. For purposes of illustration, the cabin 319 is a "not allowed" object.

[0043] The device 312 includes a 3D sensor 313 (e.g., 3D camera, LIDAR, or sonar) that captures sensed signals, and a video camera 315 that captures images of objects 318 in the local on-site scene 320 and transmits a video signal 326 over a communication link 328 or network to a remote site, possibly another country. The sensor 313 and the sensed signal are separated from the communication link 328 or network. The video signal 326 is presented to an expert 330 on a computer workstation 332 that includes a device 334 for capturing hand gestures of the expert. The expert 330 manipulates objects in the video with his / her hands (or via a tool) to perform a task. The device 334 captures the hand gestures, converts them into animated hand gestures 336, and transmits them to the user 316 over the communication link 328, where they are registered and overlaid on the display. The expert 330 may provide instructions in the form of voice, text, or other media in addition to or instead of hand gestures to support the AR environment. The AR environment itself is implemented using application software 340 and configuration files 342 and 344 on the remote and local computer systems 332 , 312 and a server 346 .

[0044] In accordance with the present invention, the AR system or method determines whether the pointing direction 314 of the local video capture and display device 312 satisfies an alignment condition 350 to an allowed object 318. The alignment condition 350 relates the camera pointing direction 314 to a line of sight (LOS) 355 to the allowed object. Both must maintain a specified offset 357, given as an angle or distance, to the allowed object 318. The offset 357 is fixed, for example, plus or minus 5 degrees, and is set by the user.

[0045] One or more technician / customer, master, or expert key codes 360 may be used to identify the technician / master / expert, define permitted object pairings, and specify tolerances that define the alignment conditions 350. The key code allows the field technician / customer / master or expert to control the video camera and prevent the field technician from capturing and / or transmitting data in the field that violates customer or country policies or legal requirements. The key code allows the owner of the code to specify, for example via a GUI, objects that are permitted or not permitted, and to preload those selections for a particular local scene or task. To establish alignment conditions, the user may specify a distance of 24 inches, which creates a 24 inch radius circle around the LOS to the permitted objects. Alternatively, the user may specify a plus / minus 5 degree misalignment for the pointing direction, which may create, for example, a 12 inch radius circle for the nominal arm length working distance. Instead of or in addition to specifying a distance or angle, the user can specify and place additional markers (disallowed objects) in the scene around the allowed object that define the outer boundary or tolerance of the alignment condition. Activation of the video capture and display device 312 requires successful pairing with at least the identified allowed object. To ensure that non-compliant data is not sent or received at the remote location, a separate key code controlled by the expert may be provided to the remote field technician. This key code is enforced by the field technician's AR system. In addition to distance, the key code may also be pre-populated with a known set of allowed or disallowed shapes.

[0046] The 3D sensor 313 captures sensed signals within the sensor FOV 354 and generates a 3D ground truth map in which allowed objects 318 and disallowed objects 319 are indicated and identified. The video camera 315 captures light within the camera FOV 356, which is read out and stored as an image in a memory chip, where the image is formed into a video signal. The interrupt processor updates the video camera pose for the current frame, calculates predicted poses 359 for one or more predicted camera FOVs 361 for future frames, and compares those predicted camera FOVs to the ground truth map to recognize and identify allowed 318 or disallowed 319 objects. The interrupt processor uses detected sensed signals or gyroscope measurements, etc., to determine whether the pointing direction 314 meets an alignment condition 350 for allowed objects 318 and whether the video camera FOV 356 or predicted camera FOV 361 contains or will soon contain disallowed objects 319.

[0047] With regard to satisfaction of the alignment condition to the allowed object 318, as long as the technician is looking in the direction of the allowed object 318, the allowed object is captured in the camera's FOV 356 and the alignment condition is satisfied (green "good"), thereby allowing the camera to capture an image and form a video signal. If the technician's eyes (camera) begin to wander, the allowed object approaches the edge of the alignment condition 350 (yellow "fix left" or "fix right"), a cue (audio, video, vibration) is issued encouraging the technician to stay focused on the allowed object 318 and the task at hand. The camera remains enabled and video is captured for transmission and display. If the technician's LOS wanders too far, the predicted camera FOV 361 loses pairing with the allowed object 318, thereby violating the alignment condition 350, and the video camera is disabled and a cue (red, "deactivate camera") is issued. The length of time the video camera is disabled can be controlled to "train" the technician to keep the focus on an acceptable object.

[0048] With regard to preventing capture of unauthorized objects 319 in the video signal, if unauthorized objects 319 are captured in the predicted camera FOV 361, similar cues may be issued to issue interrupts to control the video camera so that any camera images containing unauthorized objects 319 are not transferred to the memory chip and included in the video signal, including cutting off power to the video camera, disabling the electrochemical layer on top of the detector array or disabling the ROIC, dumping or editing the time delayed images before being transferred to the memory chip. A cue such as a red "disable camera" may be issued to the technician to notify the technician of the unauthorized object violation and prompt the technician to change the LOS.

[0049] While several illustrative embodiments of the invention have been shown and described, numerous variations and alternative embodiments will occur to those skilled in the art. Such variations and alternative embodiments are contemplated and can be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. 1. A method for preventing capture and transmission of excluded data in a local scene from a video signal, the method comprising: generating a ground truth map of the local scene including one or more objects; identifying one or more of the objects in the ground truth map as not permitted; using a video camera to capture video signals at a frame rate within a camera's field of view (CFOV) in a pointing direction within the local scene; determining a pose including a position and an orientation of the video camera within the local scene for a current frame; receiving velocity and acceleration measurements in a pointing direction of the video camera; calculating one or more predicted FOVs (PFOVs) for one or more future frames from the pose and the velocity and acceleration measurements for the current frame; comparing the one or more PFOVs to the ground truth map to recognize and identify unauthorized objects; and If a disallowed object is recognized, controlling the video camera to prevent capture of the disallowed object and its inclusion in the one or more future frames of the video signal.

2. the ground truth map is three-dimensional; The method of claim 1 , further comprising generating the ground truth map in three dimensions of the local scene using a sensor.

3. The method of claim 2 , wherein the sensor is one of a 3D video camera, a LIDAR, or a sonar.

4. The method of claim 2 , wherein the ground truth map includes a computer-generated 2D or 3D model of at least one of the disallowed objects.

5. using the sensor to capture sensed signals within a three-dimensional sensor FOV (SFOV) along the pointing direction; The method of claim 2 , further comprising: comparing signals sensed within the three-dimensional sensor FOV to the three-dimensional ground truth map to determine the pose for the current frame.

6. distributing said video signal over a network; and The method of claim 2 , further comprising isolating the sensor from the network.

7. Measuring the speed of movement of an object entering the local scene or the speed of movement of the video camera relative to the local scene; and The method of claim 1 , further comprising: deactivating the video camera if the measured speed of movement exceeds a maximum value.

8. The method of claim 1 , wherein a gyroscope of the video camera determines the pose for the current frame.

9. If a disallowed object is identified, generating cues to change the pointing direction of the video camera to prevent capture of unauthorized objects and their inclusion in the one or more future frames of the video signal; updating the predicted FOV after generating the queue and determining whether the updated predicted FOV includes the disallowed object; 2. The method of claim 1, further comprising controlling the video camera to prevent capture of the unauthorized object and its inclusion in the video signal if the cue does not prevent capture of the unauthorized object within the updated predicted FOV.

10. The method of claim 9 , wherein one of a user, a user-controlled robot, or an autonomous robot responds to the cue by changing the pointing direction of the video camera.

11. One or more of the objects in the ground truth map are permitted; determining whether the current and predicted pointing directions satisfy an alignment condition for a permitted object; generating cues to change the pointing direction of the video camera to enforce the alignment condition if the current and predicted pointing directions do not satisfy the alignment condition; updating the current and predicted pointing directions after generating the cues to determine whether they satisfy the alignment condition to the permitted object; and The method of claim 1 , further comprising deactivating the video camera if the cues fail to enforce the alignment condition.

12. the local scene is an augmented reality (AR) environment; receiving instructions from a remote location in the form of audio, text, or video media related to said permitted object; and and registering the instruction in the video signal and displaying an augmented reality to a user to instruct the user. The pointing direction of the video camera is linked to a user's movement; The method of claim 11 , wherein the cue prompts the user to change the user's pointing direction to enforce the alignment condition.

13. The video camera includes a detector array having an electrochemical layer that senses and integrates light to form an image, a read out integrated circuit (ROIC) having an amplifier and an A / D converter that reads out an image sequence at a frame rate, and a memory chip that stores the image sequence to form the video signal, and the step of controlling the video camera to prevent capture of unauthorized objects includes: mechanically controlling the video camera to change the pointing direction; Optically controlling the video camera to narrow the CFOV or blur the disallowed objects; and 10. The method of claim 1, comprising one or more of electrically controlling the video camera to cut off power, disabling the electrochemical layer, amplifiers, or A / D converters, or selectively turning off pixels of the disallowed objects.

14. 10. The method of claim 1, wherein the video camera includes a detector array having an electrochemical layer that senses and integrates light to form an image, and controlling the video camera to prevent capture of unauthorized objects includes disabling the electrochemical layer.

15. The method of claim 1 , wherein controlling the video camera to prevent capture of the unauthorized objects comprises selectively turning off camera pixels of the unauthorized objects.

16. 1. A method for preventing capture and transmission of local scene excluded data from a video signal, the method comprising: generating a ground truth map of the local scene including one or more objects; identifying one or more of the objects in the ground truth map as not permitted; using a video camera to capture video signals at a frame rate within a camera's field of view (CFOV) in a pointing direction within the local scene; determining a pose including a position and an orientation of the video camera within the local scene for a current frame; receiving velocity and acceleration measurements in a pointing direction of the video camera; calculating one or more predicted pointing directions and predicted FOVs (PFOVs) for one or more future frames from the pose and the velocity and acceleration measurements for the current frame; comparing the PFOV to the ground truth map to recognize and identify unauthorized objects; and generating a first cue when a disallowed object is recognized that alters a pointing direction of the video camera to prevent capture of the disallowed object and its inclusion in the one or more future frames of the video signal.

17. the ground truth map includes one or more allowed objects; updating the predicted FOV after generating the queue and determining whether the updated predicted FOV includes the disallowed object; 17. The method of claim 16, further comprising: if the first cue does not prevent capture of the unauthorized object within the updated predicted FOV, controlling the video camera to prevent capture of the unauthorized object and its inclusion in the video signal.

18. the ground truth map includes one or more allowed objects; determining whether the current and predicted pointing directions satisfy an alignment condition for a permitted object; generating a second cue to change the pointing direction of the video camera to enforce the alignment condition if the current and predicted pointing directions do not satisfy the alignment condition; updating the current and predicted pointing directions and predicted FOV after generating the first or second cue to determine whether they satisfy the alignment condition for the permitted object and whether the updated predicted FOV includes the disallowed object; and 17. The method of claim 16, further comprising deactivating the video camera if either the first cue or the second cue fails to enforce the alignment condition or fails to prevent capture of the unauthorized object.

19. 1. A method for preventing capture and transmission of excluded data in a local scene from a video signal, the method comprising: generating a ground truth map of the local scene including one or more objects; identifying one or more of the objects in the ground truth map as permissible; using a video camera to capture video signals at a frame rate within a camera's field of view (CFOV) in a pointing direction within the local scene; determining a pose including a position and an orientation of the video camera within the local scene for a current frame; receiving velocity and acceleration measurements in a pointing direction of the video camera; calculating one or more predicted pointing directions for one or more future frames from the pose and the velocity and acceleration measurements for the current frame; determining whether the current and predicted pointing directions satisfy an alignment condition for a permitted object; and generating a first cue for changing a pointing direction of the video camera to enforce the alignment condition if the current and predicted pointing directions do not satisfy the alignment condition.

20. updating the current and predicted pointing directions after generating the first cue to determine whether they satisfy the alignment condition to the permitted object; and 20. The method of claim 19, further comprising deactivating the video camera if the first cue fails to enforce the alignment condition.

Citation Information

Patent Citations

  • Monitor camera

    JP2004179971A

  • Object tracking method and object tracking apparatus

    JP2005057743A

  • Automatic imaging apparatus

    JP2009147647A

  • Video management device, video management method and program

    JP2017118168A

  • System and method used for playback of panoramic video content

    JP2017528947A