Augmenting one or more objects from set of objects in video stream

Abstract

To make it easy to use selection of an object to be augmented by leveraging a PTZ command having familiarity of users.SOLUTION: A camera receives a signal to activate a second mode, focused on object augmentation rather than standard PTZ configurations, at S602. Then, a spatial distance from the camera to each object is determined using indicating a spatial coordinate of the object, at S604. A zoom parameter is determined from a PTZ command, at S606. The zoom parameter determines a range of spatial distances, at S608. Objects within this range are selected at S610 and subsequently augmented within a video feed at S612, which enhances an informational value of a video stream.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to video stream enhancement techniques, and in particular to methods, devices and software for enhancing one or more objects from a set of objects in a video stream captured by a camera. [Background technology]

[0002] In recent years, the integration of real-time data overlays with video streams to improve situational awareness and operational efficiency has become increasingly common in various industries. This technology allows users to view dynamic data superimposed directly on a live video feed, thereby facilitating immediate, informed decision-making. Common applications include surveillance, navigation, and interactive broadcasting, where real-time data augmentation provides enhanced visual insight into the monitored environment. One common implementation of this technology involves the display of identifiers or tags in the video feed that correspond to specific objects or entities in the field of view. These identifiers are often linked to databases or data streams that provide real-time parameters, such as location, speed, or status updates. Basic overlays typically contain minimal data to maintain a clutter-free field of view and provide only the most critical information at a glance.

[0003] If more detailed data about an entity is required, the user generally needs to perform an additional action, such as clicking on the entity's identifier in the video feed, which should ideally trigger a query to retrieve expanded information and display it, for example, in a separate detailed panel, thereby improving the user's understanding of the situation.

[0004] However, standardizing these interactions across different platforms and devices presents significant challenges. There is no widely adopted way to transmit user interaction events, such as clicks, back from the display interface to the video processing client, which is typically implemented in the camera that captures the video feed. Instead, each system requires custom development to support interactive features, which can increase the complexity and cost of deployment.

[0005] Therefore, there is a need for improvement in this context.

[0006] KR2021 / 0067107A (KOREA E NAVI INFORMATION TECH CO LTD [KR]) discloses an augmented reality (AR)-based digital telescope system for ships, in which navigation information about the ship captured by a PTZ camera is displayed using AR.

[0007] U.S. Patent Application Publication No. 2021 / 0185238(A1) (SEIKE YASUYUKI [JP] et al.) discloses a system for displaying information about objects moving on the water around a ship using augmented reality (AR). The system displays markers corresponding to the objects moving on the water in an AR image. When a marker is selected by a user, information about the objects moving on the water corresponding to the selected marker is displayed in a predetermined location in the AR image. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] KR2021 / 0067107A [Patent Document 2] U.S. Patent Application Publication No. 2021 / 0185238(A1) Summary of the Invention

[0009] In view of the above, it would be advantageous to overcome or at least mitigate one or some of the above-described disadvantages as set out in the attached independent patent claims.

[0010] According to a first aspect of the present invention, there is provided a method implemented in a camera for augmenting one or more objects from a set of objects in a video stream captured by the camera, the video stream depicting a scene and each object being associated with first data indicating the spatial coordinates of the object, the camera implementing a first mode in which pan-zoom-tilt (PTZ) commands control the PTZ configuration of the camera, and a second mode in which PTZ commands control the object augmentation.

[0011] The method includes receiving a signal indicating that a second mode is activated, and for each object in the set of objects, determining a spatial distance from the camera to the object in the scene using first data associated with the object.

[0012] The method further includes receiving a first PTZ command, determining a zoom parameter from the first PTZ command, determining a range of spatial distances using the zoom parameter, selecting one or more objects from the set of objects having a spatial distance that is included in the range of spatial distances, and expanding the one or more objects in the video stream.

[0013] The inventors have realized that most video clients support PTZ controls, which are commonly used to adjust the camera view. Advantageously, as described herein, these standard PTZ commands can be reused to enable object selection for augmentation in scenarios where such functionality did not previously exist. This adds a layer of functionality without requiring additional hardware or controls, thereby simplifying the integration of object selection and augmentation in existing camera systems. For example, previous approaches often required implementing additional control mechanisms on the camera side to process such click commands, in addition to developing a clickable interface on the operator's side. These solutions not only required significant software development, but also introduced complexity in terms of both hardware and user interaction. By adapting PTZ controls / commands for object selection, advanced functionality can be seamlessly incorporated directly into existing camera systems with low complexity. Such adaptations can reduce barriers to implementation and maintenance by leveraging existing infrastructure and user familiarity with PTZ interfaces.

[0014] To this end, the camera implements a first mode and a second mode. In the first mode, PTZ commands are used to control the camera's pan, tilt, and zoom settings in their traditional role. In the second mode, PTZ commands are repurposed to control target augmentation. This mode is activated via a signal, thereby shifting the function of the PTZ controls from adjusting the camera's view to selecting and augmenting targets based on their spatial characteristics.

[0015] In particular, after receiving a signal indicating reuse of PTZ commands from normal use of PTZ commands (i.e., transitioning from the first mode to the second mode), the zoom parameters are used to select items based on their respective spatial distances, for example, by using the zoom parameters to determine a range of spatial distances and expanding all objects with spatial distances within that range. In other words, in the second mode, the zoom parameters are used to define a range of spatial distances. Objects within this specified range are selected for expansion.

[0016] As used herein, object enhancement includes enhancing or altering the appearance of a selected object in a video stream by adding information about the selected object in the video stream. Enhancements may include adding visual markers, highlighting the object, overlaying additional information, or other visual enhancements that make some objects stand out. For example, enhancements may include presenting the name or type of the selected object, showing a bounding box of the selected object, or including other visual enhancements or information about the selected object in the video stream.

[0017] As used herein, "first data" refers to an initial set of information associated with objects detected in a scene that is used to determine the spatial location of the objects relative to the camera. This data can vary in type and include GPS coordinates, radar data, or video data, depending on the detection and tracking technology used.

[0018] In some examples, determining the range of spatial distances includes mapping the zoom parameter to a range of spatial distances using a predefined mapping table, with each possible value of the zoom parameter mapped to a predefined range of spatial distances. For example, lower zoom levels may correspond to a first range, such as 0 to 100 meters, and higher zoom levels may target a second range, such as 100 to 200 meters. The mapping may vary, with some zoom levels corresponding to larger or smaller increments depending on desired accuracy and operational requirements. Advantageously, using a mapping table to link zoom request parameters to ranges of spatial distances may provide a practical, efficient, and user-friendly way to improve subject selection and enhancement in a video feed.

[0019] In some examples, determining the range of spatial distances includes determining a total range of spatial distances between multiple objects, dividing the total range into multiple subranges, and mapping zoom parameters to subranges within the subranges. In this example, a total range of spatial distances between objects detected in a scene is calculated (using first data associated with the objects). This total range represents the minimum and maximum distances from the camera at which the objects are located. This total range is then divided into several subranges. The division may be uniform, resulting in equal-length intervals, or dynamic, such that the length of each subrange varies based on certain factors, such as the zoom level or the density of objects in different depth fields. For example, areas with a dense distribution of objects may be segmented into shorter subranges to enable more granular control and expansion, while coarser areas may be covered by longer subranges to simplify the interface. Depending on the zoom level adjusted via the PTZ control, the system associates a corresponding subrange with the zoom level. Advantageously, by dividing the spatial range into sub-ranges as described in this example, the system may more precisely target and expand objects. The operator can select a zoom level corresponding to the sub-range that best suits the operator's immediate needs. For example, varying the length of each sub-range may be beneficial in environments with varying object densities across the scene. The length may depend on the density of objects in an area of ​​the scene. Such an embodiment may prove advantageous in scenarios in which most, if not all, objects are clustered within a narrower spatial range, as opposed to scenarios in which they are uniformly spread across the entire spectrum of distances captured by the video stream.

[0020] In some examples, the one or more objects include multiple objects, and the method further includes selecting a first object among the multiple objects and further expanding the first object, the first object being selected using one or more pan or tilt parameters of additional received PTZ command(s). In these examples, after the range of objects is selected using the zoom parameters, the operator can further refine their focus by selecting a single object to expand on in greater detail. This selection is achieved using pan and / or tilt parameters from additional PTZ commands received after the initial zoom-based selection. Through efficient use of PTZ controls, these examples minimize the need for manual input or additional hardware to achieve detailed expansion. The operator can leverage existing controls to achieve detailed views and insights, thereby reducing operating costs. The zoom command initially selects a subset of objects based on their spatial distance, thereby effectively grouping them for further interaction. Once this subset is defined, pan and / or tilt commands are reused (again, in the second mode) to step through these pre-selected objects one at a time. For example, tilt commands may enable vertical selection, while the operator may use pan commands to navigate each object horizontally, providing a comprehensive method for cycling through and focusing on individual objects within a determined range, which may add additional information in the video stream. This methodological use of PTZ commands may increase the interactivity and focus of the method for enhancing one or more objects from a set of objects in a captured video stream, thereby enabling detailed examination and enhancement of specific objects in a targeted and efficient manner.

[0021] In some examples, the method further includes ordering the plurality of targets, and the step of selecting the first target includes: for each received further PTZ command, determining a pan direction of a pan parameter of the further received PTZ command, where the pan direction is one of a negative pan direction and a positive pan direction; and using the pan direction to change from the currently selected first target to a new selected first target so that the ordered plurality of targets can be cycled through in a direction corresponding to the pan direction.

[0022] Sorting of objects selected using a zoom command can be achieved using spatial attributes indicated by the first data, such as their relative location in the scene, their azimuth angle relative to the camera, or GPS coordinates such as longitude and latitude. Organizing objects according to one of these criteria establishes a structured, logical sequence, thereby enabling intuitive navigation. For example, if objects are arranged from left to right as they appear in the camera's view, using the PTZ controls to pan right or left correspondingly cycles through these objects in a predictable manner. When the operator issues a pan command, the system first determines the pan direction based on the received pan parameters. This can be a negative (e.g., left) or positive (e.g., right) direction. The direction specified in the command defines how the system transitions from the currently selected object to a new object in the ordered list. Depending on the setup, the pan direction can be determined using various modes, such as absolute mode (directly specifying an angle or position and determining the pan direction based on this), relative mode (adjusting from the current position and the pan direction is related to the adjustment direction), or continuous mode (adjustment continues until the command is changed), allowing flexibility and precision in how objects are cycled through and viewed.

[0023] In some examples, the method further includes dividing the plurality of objects into two or more subsets of objects according to their respective spatial distances to the camera, ordering the two or more subsets, and ordering the objects within each subset, wherein selecting the first object includes, for each received further PTZ command, determining whether the further PTZ command corresponds to a tilt command and / or a pan command.

[0024] When the further PTZ command corresponds to a tilt command, the method includes determining a tilt direction of a tilt parameter of the further received PTZ command, where the tilt direction is one of a negative tilt direction and a positive tilt direction, and using the tilt direction to change from a currently selected first object included in a first subset of the two or more subsets to a new selected first object included in a second subset of the two or more subsets, such that the ordered subsets may be cycled through in a direction corresponding to the tilt direction.

[0025] When the further PTZ command corresponds to a pan command, the method includes determining a pan direction of a pan parameter of the further received PTZ command, where the pan direction is one of a negative pan direction and a positive pan direction, and using the pan direction to change from a currently selected first object included in a first subset of the two or more subsets to a new selected first object included in the first subset, such that the ordered objects in the first subset can be cycled through in a direction corresponding to the pan direction.

[0026] In addition to the functionality described above, in which the pan command is used to cycle through selected objects horizontally, the tilt command can also be employed to navigate through a subset of selected objects vertically based on their spatial distance. In particular, objects selected using the zoom command can be further organized into subgroups, each located within a distinct subrange of the initially selected spatial distance range. Navigation methods applicable to the pan command, such as absolute, continuous, and relative modes, can similarly be implemented with the tilt command to ensure seamless vertical cycling through these subsets. When a new subset is selected via the tilt command, an initial object within that subset—potentially the horizontally middle object or the object closest in the horizontal plane to a previously selected object in the previous subset—is automatically selected, and this newly selected initial object is then further expanded (i.e., more detailed information about this object is added to the video stream). This initial selection facilitates a smooth transition between subsets, thereby maintaining spatial coherence. Within each vertically segmented subset, objects can be cycled through using pan commands, as previously described, thereby enabling comprehensive, systematic exploration and expansion of a scene. Note that in some implementations, the functions of the pan and tilt commands can be reversed. In particular, pan commands can be used to select vertical subsets of objects, while tilt commands can be utilized to navigate objects horizontally within those subsets.

[0027] As used herein, the term "cycling through" refers to the process of sequentially moving from one object to another within a predefined set or order. This is generally done by activating a control (such as pan or tilt), as described above. When the method "cycles through" objects, it is iterating in a controlled manner across the objects. For example, if the objects are organized based on their spatial arrangement (left to right, near to far, etc.), cycling through the objects using a pan control would involve stepping through each object, such as from left to right or vice versa. Similarly, when using a tilt control, the method may move from an object at the top end of a range of spatial distances (determined by a zoom command) to an object at the bottom end, or vice versa. In some embodiments, the function of cycling through objects is implemented in a loop or circular manner. This means that upon reaching the end of the set in one direction, such as the left or bottom end, the next step will automatically loop back to the starting position at the right or top end, respectively.

[0028] In some examples, the further enhancement includes enhancing the first object in the video stream with data (such as information) including one or more of the object's name, object type, object speed, or object location.

[0029] As described herein, a two-tiered approach to augmentation may be applied. First, basic augmentation is applied to selected objects (selected using a zoom command), which may include displaying a simple bounding box or the name of each object to identify the object in the video feed. Then, a more detailed additional augmentation is reserved for a specific object selected from the selected objects (using a pan / tilt command, as described above). This first selected object may be enhanced with additional data / information, such as its name, type, speed, or exact location. This approach allows for a layered presentation of information. That is, general identification helps distinguish between multiple objects at a glance and provides basic information about the multiple objects, while detailed augmentation provides detailed information about the specific object of interest, thereby improving the utility of the surveillance or monitoring system by meeting the needs for both general and specific information.

[0030] In some examples, the signal indicating that the second mode has been activated includes multiple PTZ commands with parameters according to a predetermined pattern. Such pattern-based activation can be designed to distinguish mode switching from normal PTZ operation without requiring additional hardware or interfaces. A pattern that can be used to activate the second mode can involve a sequence of directional inputs (PTZ commands) that are unlikely to occur during standard camera operation. For example, an operator can perform a circular motion with a joystick controlling a PTZ setting (i.e., corresponding to moving it up, then right, then down, then left in quick succession, possibly multiple times), and this particular motion pattern would signal the system to switch to the extended mode. In another example, an operator can move the joystick right, then left, then right, then left in quick succession to input a zigzag pattern with the joystick. Such intentional and distinctive patterns can be recognized by the system as commands to transition to the second mode, thereby enabling increased functionality without interfering with the primary PTZ controls used for camera adjustment. These patterns can be predefined and programmed into the camera (or configurable by the user) to ensure that mode activation is intentional and seamless, thereby optimizing the interface for intuitive and efficient use. Note that there are other means of providing a signal indicating that the second mode can be implemented, such as using a selection button on a joystick or keyboard. Furthermore, input devices other than a joystick can be employed both to signal the switch to the second mode and to issue PTZ commands to the camera. For example, a keyboard can effectively serve both purposes, thereby providing a flexible and accessible way to manage camera functions and mode transitions. Moreover, the same pattern or signal used to activate the second mode can also be employed to return the camera to the first mode.Alternatively, a different pattern or signal may be designated for this purpose.

[0031] In some examples, the first data associated with the object includes one of GPS coordinates associated with the object, radar data detecting the object, or video data depicting the object. The "first data" associated with the object in the video feed may vary in format depending on the detection technology used and the requirements of the application. This data may include GPS coordinates if the object is equipped with a GPS device that provides precise latitude and longitude measurements to give a precise geographic location. Alternatively, if radar technology is employed, the first data may consist of radar data that detects the object by emitting radio waves and analyzing the returned echoes. This method is effective for determining the distance of the object from the radar source. In another example, the first data may also be video data from the camera itself, where the object is visually identified in the video feed.

[0032] In some examples, the first data associated with the object includes video data depicting the object, and determining the spatial distance from the camera to the object in the scene includes identifying the object using the video data, determining the object's physical dimensions using the identification, determining the object's depicted dimensions from the video data, and using the depicted size and actual size to determine the spatial distance. In scenarios where the first data consists of video data, determining the object's spatial distance from the camera can be performed by identifying the object in the video stream based on visual characteristics such as shape and color, or using additional data such as GPS coordinates. Once identified, the object's physical dimensions, such as height and width, are obtained. These dimensions may be known from previous data or estimated from an external source that correlates visual features or other identifiers with dimensional data. After establishing the physical size, the depicted dimensions from the video are measured. The spatial distance is then calculated by comparing these actual dimensions with the depicted dimensions, thereby enabling an accurate assessment of how far the object is from the camera.

[0033] The techniques outlined herein are versatile and can be applied across a variety of settings. In some examples, the object corresponds to one of a vessel, an aircraft, or a person equipped with a body-worn camera. If the object corresponds to a vessel, the method may further include receiving GPS data of the objects from an external Automatic Identification System (AIS) connected to the camera. This AIS data provides precise location information about the vessel, and such location information can either be used in real time to dynamically augment the video feed with the latest positioning or received intermittently and stored in the camera's memory for later use.

[0034] According to a second aspect of the present invention, the above object is achieved by a non-transitory computer-readable storage medium storing instructions for performing the method according to the first aspect when executed on a camera having processing capabilities.

[0035] According to a third aspect of the present invention, the above object is achieved by a camera for augmenting one or more objects from a set of objects in a video stream captured by the camera, the video stream depicting a scene and each object associated with first data indicating the spatial coordinates of the object, the camera implementing a first mode in which pan-zoom-tilt (PTZ) commands control a PTZ configuration of the camera and a second mode in which the PTZ commands control the object augmentation, the camera being configured to: receive a signal indicating that the second mode is activated; determine, for each object in the set of objects, a spatial distance from the camera to the object in the scene using the first data associated with the object; receive the first PTZ command; determine zoom parameters from the first PTZ command; determine a range of spatial distances using the zoom parameters; select one or more objects in the set of objects having a spatial distance that is included in the range of spatial distances; and augment the one or more objects in the video stream.

[0036] The second and third aspects may generally have the same features and advantages as the first aspect. Furthermore, it should be noted that the present disclosure relates to all possible combinations of features unless otherwise specified.

[0037] The above, as well as additional objects, features, and advantages of the present invention will be better understood from the following illustrative, non-limiting detailed description of embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which like reference numerals are used for similar elements, and in which: [Brief explanation of the drawings]

[0038] [Figure 1] FIG. 10 illustrates an image frame containing a set of objects, some of which are selected and expanded based on the objects' respective spatial distances to the camera, according to an embodiment. [Figure 2] FIG. 10 illustrates further expansion of a first object among the selected objects according to an embodiment. [Figure 3] FIG. 10 illustrates using pan parameters to cycle through a selected object, according to an embodiment. [Figure 4] FIG. 10 illustrates cycling through a selected object using pan and tilt parameters, according to an embodiment. [Figure 5] FIG. 1 illustrates a system including a camera that implements the video stream enhancement techniques described herein, according to an embodiment. [Figure 6] FIG. 1 illustrates a flowchart of a method for enhancing one or more objects from a set of objects in a video stream, according to an embodiment. [Figure 7] FIG. 10 shows a flowchart of a method for determining a first object among selected objects and further expanding the first object. [Figure 8] FIG. 10 shows a flowchart of a method for determining a first object among selected objects and further expanding the first object. DETAILED DESCRIPTION OF THE INVENTION

[0039] DETAILED DESCRIPTION OF THE INVENTION The present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the invention are shown. The systems and devices disclosed herein will be described in operation.

[0040] The techniques described herein revolve around improved methods for enhancing objects in a video feed using pan-tilt-zoom (PTZ) controls traditionally employed to adjust camera views. These techniques reuse standard PTZ controls to select and enhance objects based on the object's spatial distance, determined via the zoom parameter of a PTZ command, and are optionally further enhanced with pan and tilt parameters of additional PTZ commands for precise navigation of the selected object for further enhancement. Advantageously, these techniques are implemented to take advantage of the ubiquity of PTZ functionality across video clients, thereby avoiding the need for additional hardware or complex software modifications. By using existing PTZ controls in novel ways, the techniques simplify the integration and operation process, thereby enabling more dynamic and detailed interaction with objects in the video stream. The techniques provide a cost-effective solution for enhancing objects in video streams within a monitored environment.

[0041] Next, an embodiment for enhancing one or more objects from a set of objects in a video stream captured by a camera will be described with reference to Figures 1 to 5 and further to the method steps of the flowcharts of Figures 6 to 8.

[0042] 5 illustrates, by way of example, a system 500 including a camera 504 that performs a method 600 for enhancing one or more objects from a set of objects in a video stream 514 captured by the camera 504. The camera 504 includes a processing module 506. The processing module 504 includes one or more processors and one or more non-transitory computer-readable media that store instructions executable by the one or more processors, which, when executed, cause the camera to perform the methods described herein. Suitable processors for executing a program of instructions include, by way of example, both general-purpose and special-purpose microprocessors, and a single processor or one of multiple processors or cores of any kind of processing module. The processor may be supplemented by or incorporated in an ASIC (application-specific integrated circuit).

[0043] The camera 504 utilizes pan, tilt, and zoom (PTZ) functionality to provide both wide area coverage and detailed views of the scene 502 using a single device. These PTZ functions may be controlled remotely, allowing a remote operator to adjust the PTZ parameters of the camera 504 via an input device 516, such as a keyboard 516 or a joystick (not shown in FIG. 1 ). Thus, the camera 504 is configured to receive PTZ commands from a remote entity.

[0044] The camera 504 is configured to capture a video stream depicting a scene 502 including a set of objects. In the examples described herein, the objects in the scene 502 are generally illustrated as boats or watercraft. However, this is provided by way of example only, and the objects may, in other embodiments, be other types of objects, such as aircraft, people equipped with body-worn cameras, land vehicles, surveillance drones, etc.

[0045] The camera 504 implements a first mode in which pan-zoom-tilt (PTZ) commands control the PTZ configuration of the camera 504, and a second mode in which the PTZ commands control the focus extension.

[0046] The camera 504 is configured to receive (S602) a signal 512 indicating that a second mode has been activated. The signal 512 may be provided using an input device 516. In some embodiments, the signal 512 indicating that a second mode has been activated includes multiple PTZ commands with parameters according to a predetermined pattern. The processing module 506 may then be configured to recognize the pattern, for example, by analyzing the sequence and timing of the PTZ inputs in the signal 512 to match them with known configurations stored in the memory of the processing module 506. This pattern recognition allows the camera 504 to seamlessly switch between operational modes using an existing interface for receiving PTZ commands. Two example patterns that may be used to activate a second mode in the camera 504 via a PTZ command are a circular motion and a zigzag pattern. For a circular motion, the operator may move the joystick in an intentional clockwise or counterclockwise direction, which the processing module 506 recognizes as a cue to switch modes. Alternatively, the zigzag pattern may involve alternating left-right or up-down movements of the joystick in quick succession. Such a pattern would be distinctive and clearly signal an intentional command to change modes, thereby minimizing the possibility of accidental activation.

[0047] In a second mode, the camera 504 (e.g., the processing module 506) is configured to augment the video stream, and in particular to add information about a subset of the set of objects captured in the video stream. To that end, the camera 504 is configured to select one or more objects from the set of objects, and these one or more objects are augmented in the video stream. The selection is made by first determining (S604) a spatial distance from the camera 504 to each of the objects in the scene 502 using first data associated with each object and indicating the spatial coordinates of the object.

[0048] The first data may be, for example, radar data detecting the object received by a radar sensor associated with the camera 504. Using such data to determine the spatial distance (S604) is essentially done by calculating the time delay between the radar signal emission and its return after reflecting off the object, thereby enabling the camera 504 to pinpoint the location of the object relative to the position of the camera 504.

[0049] The first data may also be video data depicting the object, in other words, a video stream capturing a scene. Using such data to determine spatial distance may be done by analyzing the size of the object in the video relative to the object's known physical dimensions. This method involves comparing the object's actual size to how it appears in the video stream. Techniques such as perspective analysis, or using standard visual references within the video stream, may further be used to calculate distance based on how the size of the object in the video stream changes with the object's position relative to the camera 504.

[0050] The first data may also be GPS data associated with the object. Such data may be received from any suitable source, such as a GPS tracking device installed on the object. This GPS data may be communicated to the camera using wireless communication technology. Common methods include using Wi-Fi, a cellular network (such as 4G or 5G), or satellite communication. Furthermore, as illustrated in FIG. 1 , the GPS data 508 may be sourced from an external GPS tracking service 510 that maintains a real-time location database for various assets. One example of such a GPS tracking service is an external automatic identification system (AIS) 510 connected to the camera 504.

[0051] The camera is configured to receive PTZ commands 513 from an input device 516. Zoom parameters within these commands 513 are determined (S606) and utilized to define a spatial distance range (S608) that aids in selecting specific objects (S610) for augmentation (S612). Details of this selection and augmentation process are further described below in conjunction with FIGS. 1-4 and 6-7. Once the objects are augmented (S612), the thus-enhanced video stream 514, including these augmented objects, is then transmitted to a display 518. This allows the enhanced video stream 514 to be presented to an operator as a graphical interface 520, thereby providing a comprehensive view that integrates both real-time imagery and augmented data, for example, for improved situational awareness and decision-making.

[0052] 1 schematically illustrates an image frame 100 containing a set of captured objects 102 in a video stream. These objects 102 are mapped to specific spatial distances 105 from the camera that captured the image frame 100. The totality of these spatial distances 105 within the image frame 100 may be segmented into various ranges 106 of spatial distances. Zoom parameters from the PTZ command may then be determined (S606) and utilized to select (S608) one of these ranges 106a-c, thereby facilitating the selection (S610) of a particular object 102a-g from the set of objects 102 for expansion (S612). In the depicted example of FIG. 1, three distinct ranges 106a-c are identified.

[0053] These ranges 106 may be predefined, e.g., each representing a set interval of spatial distance. The interval assigned to each range may be consistent across all ranges 106, or the interval may vary between them. For example, each range 106a-c may represent a distance interval of X meters, where X may be a suitable measurement, such as 50 meters, 100 meters, 200 meters, etc., depending on specific requirements for precision and granularity in the augmentation process. In other examples, closer objects may be grouped within shorter distance intervals (e.g., every 50 meters) to allow for more detailed augmentation because they are more prominent and distinct in the video stream. Conversely, more distant objects may be grouped into wider intervals (e.g., every 200 meters) because fine details may be less discernible at greater distances. In some examples, maximum and minimum distances 105 from the camera at which objects are located may be determined. Following this, the resulting total range may be divided into multiple subranges of the same or different lengths.

[0054] To determine the range that the value of the zoom parameter can represent (S606), each possible value of the zoom parameter may be mapped to one of the possible ranges 106a-c. In some examples, the initial range is selected based on the current zoom parameter when the PTZ command is first received. Subsequent adjustments to the zoom level then dynamically shift the selected range of spatial distance closer to or farther away from the camera, depending on whether the new zoom parameter is larger or smaller than the previous setting. For example, zooming out (decreasing the zoom parameter) generally shifts the selected range to cover a closer spatial range, while zooming in (increasing the zoom parameter) selects more distant objects for extension.

[0055] 1, an intermediate range 106b of spatial distances is determined (S608) based on the zoom parameters determined (S606) from the received PTZ command. Objects 102c-e having spatial distances that fall within the selected range 106b are then selected (S610) for expansion (S612).

[0056] Augmenting the selected objects 102c-e (S612) may include adding any type of information 104 to the video stream. In the example of FIG. 1, augmenting the objects 102c-e (S612) includes adding information 104 including the names of the objects 102c-e, lines between the objects 102c-e, and names to clarify the object to which the names belong. Another example would be marking the objects in some way, for example, by including a bounding box of the object in the video stream. Another example of augmentation would be the display of an icon or symbol near each object. For example, in the case of a ship, the icon could indicate the type of ship (cargo ship, tanker, passenger ship) or its operational status (such as an anchor symbol for an anchored ship).

[0057] As shown in FIG. 1, to avoid visual clutter in the augmented video stream, it may be advantageous to keep the added information 104 about the selected (S610) multiple objects 102c-e small and unobtrusive. By using minimal enhancements 104, such as simple lines, names, or small icons for the initial augmentation (S612) of (potentially) some objects, the overall clarity of the scene is preserved, thereby making it easier for the viewer to maintain an overall understanding of the situation. To provide more comprehensive information without cluttering the view, one of the initially selected (S610) augmented objects 102c-e may be further augmented (S614) with additional, more detailed information. This selective augmentation approach allows for deeper focus on a specific object of interest when needed, while keeping the broader view less crowded. Such a technique is now described in conjunction with FIG. 2.

[0058] For ease of explanation, FIG. 2 shows only three selected objects 102c-e in the spatial distance range 106b from FIG. 1. Of these, one object 102c ("first object") is chosen for further enhancement (S612). This process involves adding more detailed information 202, 204 specific to the first object 102c. The additional enhancement (S614) can include adding various types of information 202, 204 to the video stream that improves the operator's understanding of the first object 102c. For example, an additional information panel 204 can be incorporated into the video stream. This panel can be consistently positioned in a specific area of ​​each image frame, such as near the bottom or top edge, depending on design needs.

[0059] Such an information panel 204 may display details, such as the name, type, speed, or location of the first object 102c, tailored to the context and requirements of the monitoring system. Furthermore, a bounding box 202 may be added around the object 102c, or an existing bounding box may be enhanced, to highlight the relationship between the information panel 204 and the first object 102c. Such visual cues ensure clear identification of the first object 102c among the selected objects 102c-e, thereby clarifying the object to which the information panel 204 pertains. This method of selectively enhancing one object with additional information helps maintain overall visual clarity while providing detailed insight where needed.

[0060] As explained above, in some embodiments, to achieve additional enhancements (S614), other components of the PTZ command, in addition to the zoom command, may be utilized to provide an interface for navigating among selected objects 102. Such techniques will now be illustrated with reference to the flowcharts of Figures 7-8 in conjunction with Figures 3-4.

[0061] 3, the objects selected (using the zoom parameters as described above) are collectively referred to as references 302. These objects 302 may be ordered (S702) in a particular order, such as horizontally, to facilitate navigation via PTZ commands. This ordering helps to systematically select each object for potential further enhancement.

[0062] Initially, when a new range of spatial distances is determined from the PTZ commands, resulting in a new set of one or more selected objects 302, the first object (shown in FIG. 2 using dashed rectangle 202a) may be selected in a different manner. For example, the horizontally middle one of the one or more selected objects 302 may be selected as the first object for further expansion. In another example, the object closest in the horizontal plane to the previously selected first object (i.e., with respect to the previous range of spatial distances) may be selected as the first object once the new range of spatial distances is determined. In FIG. 2, the object located in the middle of the horizontal arrangement of selected objects 302 is initially chosen as the first object for further detailed expansion.

[0063] Following the initial selection of the first object, further PTZ command inputs facilitate subsequent navigation through the ordered objects. In an example, a pan direction from the pan parameter of the received PTZ command is determined (S704). The pan direction is classified as either negative or positive, corresponding to, for example, a movement to the left or a movement to the right, respectively. Based on this direction, the currently highlighted (202a) first object is changed to a new highlighted (202b) first object (S706). This transition uses the established pan command to coherently cycle through the ordered objects 302, moving either left or right along the ordered succession of selected objects 302.

[0064] In some scenarios, the number of objects within the distance determined by the zoom command may be large, or the selected objects 302 may be positioned in a way that makes navigating them complicated. In such cases, it may be advantageous to allow selection of a first object to occur in two directions, i.e., both horizontally and vertically. Such a technique will now be illustrated with reference to the flowchart of FIG. 8, in conjunction with FIG. 4.

[0065] 4, the step of selecting a first object among a plurality of objects 302 is efficiently implemented to manage dense or complex object arrangements captured in a video stream. First, the plurality of selected objects 302 are divided into two or more subsets 402 (S802). Dividing the plurality of objects into subsets is achieved by categorizing the objects into groups based on their proximity to the camera. For example, one subset may include all objects located between 75 meters and 100 meters away, another subset may include objects located between 100 meters and 125 meters away, and so on.

[0066] These subsets 402 are ordered according to their respective spatial distance from the camera, and the objects within each subset are ordered as described above to facilitate navigation.

[0067] First, a first object from the lower (eg, closer to the camera) subset 402c is selected (highlighted 202a), eg, as described above.

[0068] The process for selecting a first target from these subsets dynamically responds to the type of PTZ command received. When a further PTZ command is issued, the camera first determines whether the command is a tilt command or a pan command (S804).

[0069] If the PTZ command is identified as a pan command, the pan direction is determined to be either negative (left) or positive (right) (S806). Based on this direction, the selection of the first object is shifted within the same subset 402c from the currently highlighted first object (highlighted 202a) to another object (highlighted 202b) in the same subset 402c (S808). This action enables horizontal cycling through of objects, thereby enabling the user to use PTZ commands to navigate laterally across objects that are spatially closer or farther away but on the same plane (within the same subset 402c).

[0070] Conversely, if the command corresponds to a tilt command, the camera determines a tilt direction (S810), which can be either negative (up) or positive (down). Using this tilt direction, the system facilitates a change (S812) from the currently selected first object (highlighted (202b)) in one subset 402c to a new first object (highlighted (202c)) in another subset 402b. This allows the user to cycle vertically through the ordered subsets 402 in alignment with the tilt direction while moving from one spatial layer of objects to another.

[0071] This bidirectional approach, using tilt for vertical navigation and pan for horizontal movement, or vice versa, may improve an operator's ability to efficiently manage and interact with multiple objects, allowing for quick adjustment of focus, either in depth or laterally, thereby ensuring that an operator can accurately and easily access and extend any object within a complex scene.

[0072] The above-described embodiments should be understood as illustrative examples of the present invention. Additional applications of the extension techniques are possible and anticipated, such as in production facility settings. The methods described herein can be effectively utilized to track objects as they move through various stages of a production flow. It should be understood that any feature described with respect to any one embodiment may be used alone or in combination with other described features, and in combination with one or more features of any other embodiment, or in any combination of any other embodiment. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the present invention, as defined in the appended claims.

Claims

1. A method (600) implemented in a camera (504) for augmenting one or more objects (102c-e) from a set of objects (102) depicted in image frames in a video stream (514) captured by the camera, the video stream depicting a scene (502), each object associated with first data (508) indicating spatial coordinates of the object, the camera implementing a first mode in which pan-zoom-tilt (PTZ) commands control a PTZ configuration of the camera, and a second mode in which PTZ commands select one or more objects in the set of objects for augmentation, the method comprising: receiving (S602) a signal (512) indicating that the second mode has been activated; For each object in the set of objects, determining (S604) a spatial distance (105) from the camera to the object in the scene using the first data associated with the object; receiving a first PTZ command (513); Determining zoom parameters from the first PTZ command (S606); determining (S608) a range of spatial distances (106) using the zoom parameters; selecting (S610) the one or more objects in the set of objects having the spatial distance that falls within the range of spatial distances; augmenting the one or more objects in the video stream (S612); A method comprising:

2. 2. The method of claim 1, wherein determining the range of spatial distances comprises mapping the zoom parameter to a range of spatial distances using a predefined mapping table, wherein each possible value of the zoom parameter is mapped to a predefined range of spatial distances.

3. determining the range of spatial distances determining a range of the spatial distances between the plurality of objects; dividing the total range into a plurality of subranges; mapping said zoom parameters to a sub-range within said sub-ranges; The method of claim 1 , comprising:

4. wherein the one or more subjects include a plurality of subjects, and the method further comprises: selecting a first object (102f) from the plurality of objects; Further expanding the first target (S614); further comprising the first target is selected using one or more of the pan or tilt parameters of the further received PTZ command(s); 4. The method according to any one of claims 1 to 3.

5. further comprising ordering the plurality of objects (S702); The step of selecting the first target comprises, for each received further PTZ command: determining a pan direction of a pan parameter of the further received PTZ command (S704), wherein the pan direction is one of a negative pan direction and a positive pan direction; changing (S706) from a currently selected first object to a new selected first object using the pan direction so that the ordered plurality of objects can be cycled through in a direction corresponding to the pan direction; The method of claim 4, comprising:

6. Dividing the plurality of objects into two or more subsets (402) of objects according to their respective spatial distances to the camera (S802); ordering the two or more subsets; and ordering the objects within each subset. further comprising The step of selecting the first target comprises, for each received further PTZ command: Determining whether the further PTZ command corresponds to a tilt command and / or a pan command (S804); When the further PTZ command corresponds to a tilt command, determining a tilt direction of the tilt parameter of the further received PTZ command (S810), wherein the tilt direction is one of a negative tilt direction and a positive tilt direction; using the tilt direction to change from a currently selected first object contained in a first subset of the two or more subsets to a new selected first object contained in a second subset of the two or more subsets (S812), such that the ordered subsets can be cycled through in a direction corresponding to the tilt direction; When the further PTZ command corresponds to a pan command, determining a pan direction of the pan parameter of the further received PTZ command (S806), wherein the pan direction is one of a negative pan direction and a positive pan direction; changing (S808) from a currently selected first object contained in a first subset of the two or more subsets to a new selected first object contained in the first subset using the pan direction so that the ordered objects in the first subset can be cycled through in a direction corresponding to the pan direction; The method of claim 4, comprising:

7. 7. The method of claim 4, wherein the further enhancement comprises enhancing the first object in the video stream with data including one or more of a name of the object, a type of the object, a speed of the object, or a location of the object.

8. 8. The method of claim 1, wherein the signal indicating that the second mode is activated comprises a plurality of PTZ commands with parameters according to a predetermined pattern.

9. the first data associated with the object, GPS coordinates associated with the object; radar data detecting said object; or video data depicting said object 9. The method of claim 1, wherein the method comprises one of the following steps:

10. wherein the first data associated with the object includes video data depicting the object, and determining a spatial distance from the camera to the object in a scene comprises: identifying the object using the video data; and determining a physical dimension of the object using said identification; determining depicted dimensions of the object from the video data; and determining said clearance distance using the depicted size and the actual size; 10. The method of claim 9, comprising:

11. The object is Vessels, aircraft, or people equipped with body-worn cameras 11. The method according to claim 1, wherein the method corresponds to one of the following:

12. the object comprises a vessel, and the method comprises: receiving GPS data (508) of the plurality of objects from an external Automatic Identification System (AIS) (510) connected to the camera; The method of claim 11 further comprising:

13. A non-transitory computer-readable storage medium having stored thereon instructions for performing the method of any one of claims 1 to 12 when executed on a camera having processing capabilities.

14. 1. A camera for augmenting one or more objects from a set of objects depicted in image frames in a video stream captured by the camera, the video stream depicting a scene, each object associated with first data indicating spatial coordinates of the object, the camera implementing a first mode in which pan-zoom-tilt (PTZ) commands control a PTZ configuration of the camera, and a second mode in which PTZ commands select one or more objects in the set of objects for augmentation, the camera comprising: receiving a signal indicating that the second mode has been activated; for each object in the set of objects, determining a spatial distance from the camera to the object in the scene using the first data associated with the object; receiving a first PTZ command; determining a zoom parameter from the first PTZ command; determining a range of spatial distances using the zoom parameters; selecting one or more objects in said set of objects having a spatial distance that falls within said range of spatial distances; augmenting the one or more objects in the video stream; and A camera configured to:

Images