Adaptive neural network selection method and system

Abstract

The present disclosure relates to adaptive neural network selection methods and systems, the method comprising: obtaining one or more images from a camera; obtaining a zoom factor of the camera, the zoom factor representing a measure of a current zoom level applied by the camera when capturing the one or more images; selecting a neural network from a plurality of neural networks configured to segment the one or more images and / or detect one or more objects in the one or more images based on the zoom factor, the plurality of neural networks being configured to operate at different image resolutions, a higher zoom factor corresponding to selecting a neural network configured to operate at a lower image resolution, and a lower zoom factor corresponding to selecting a neural network configured to operate at a higher image resolution; downscaling the one or more images to an image resolution required by the selected neural network; applying the selected neural network to segment the downscaled one or more images and / or detect the one or more objects in the down scaled one or more images.

Description

Technical Field

[0001] This disclosure relates to the processing of images captured by a camera, and more particularly to methods and systems for selecting neural networks based on zoom levels in a camera equipped with zoom capabilities. Background Technology

[0002] Artificial intelligence (AI) and neural networks have become central to modern imaging systems, providing powerful capabilities for tasks such as object detection, segmentation, and scene understanding. These technologies leverage deep learning to process images in ways previously impossible, enabling automated recognition and analysis of complex visual data. In particular, neural networks have been optimized for imaging applications to suit various fields such as security, surveillance, and autonomous driving. Their integration with imaging systems significantly enhances the camera's ability to perform real-time analysis and context-sensitive adjustments based on the captured scene.

[0003] One factor affecting the performance of neural networks in these applications is the resolution at which these networks operate. High-resolution neural networks increase the detection range, enabling the system to identify and analyze objects located at greater distances. This expanded range is particularly advantageous for applications requiring early detection of distant objects. However, high-resolution processing also incurs computational costs, resulting in longer inference times. This can be problematic in time-sensitive environments. For cameras that continuously receive new image frames at high speeds, schemes that do not allow new frames to be received before the neural network has completed processing are not feasible.

[0004] Pan-tilt-zoom (PTZ) cameras, commonly used in surveillance and security, are particularly affected by this challenge. PTZ cameras may sometimes use relatively high frames per second (FPS) to ensure smooth and accurate tracking of moving objects or to provide clear and detailed real-time feeds in dynamic environments. The detection range and field of view can change dynamically as these cameras move to track targets or switch to new angles. PTZ cameras often face challenges due to the inherent processing lag of high-resolution neural networks. For example, if a camera system is configured to mask objects virtually in real time, situations involving motion in the scene can present challenges in accurately placing the mask. PTZ cameras can even make estimating mask placement more difficult because not only can the objects in the scene change, but the camera itself may also change. One approach to this problem is to freeze the displayed image before fully analyzing the object and accurately positioning the mask. However, this approach may require reducing the FPS to allow sufficient processing time, which may not be a good solution for PTZ cameras.

[0005] Therefore, the purpose of this disclosure is to achieve efficient processing while maintaining optimal detection accuracy, thereby addressing the challenges associated with the complexity of using neural networks in tasks such as object detection and segmentation. Summary of the Invention

[0006] This disclosure relates to a computer-implemented method for segmentation and / or object detection in images captured by a camera with zoom capability, the method comprising: Acquire one or more images from the camera; Get the camera's zoom level, where the zoom level represents a measure of the current zoom level applied by the camera when capturing the one or more images; Based on zoom magnification, a neural network is selected from a plurality of neural networks configured to segment the one or more images and / or detect one or more objects in the one or more images, wherein the plurality of neural networks are configured to operate at different image resolutions, wherein a higher zoom magnification corresponds to the selection of a neural network configured to operate at a lower image resolution, and a lower zoom magnification corresponds to the selection of a neural network configured to operate at a higher image resolution. Reduce one or more images to the image resolution required by the selected neural network; The selected neural network is applied to segment one or more scaled-down images and / or to detect one or more objects in one or more scaled-down images.

[0007] This method enables the camera system to dynamically adjust image processing based on the camera's current zoom level, thereby improving efficiency by reducing unnecessary detail processing overhead at high zoom levels. When zooming in, a neural network configured to operate at a higher image resolution is selected. This allows for capturing a wider range of scene details. Zooming in causes the system to select a neural network operating at a lower image resolution, which minimizes the computational load for a narrower field of view. By associating a specific resolution neural network with each zoom range, the system can adapt to changing imaging needs virtually in real time. In one embodiment, the zoom magnification can be associated with a suitable model based on, for example, a predetermined table.

[0008] By employing a structured configuration of a neural network with defined image resolution capabilities, this method optimizes processing speed, resource allocation, and detection reliability in environments using real-time segmentation and object detection. This method can be considered to achieve accurate detection across varying distances and zoom magnifications without incurring the latency typically associated with high-resolution neural network processing.

[0009] The method further includes downscaling the captured images to match the resolution of the selected neural network, ensuring that each image is processed appropriately without exceeding the necessary detail for accurate segmentation or object detection. Once downscaling is complete, the selected neural network is applied to the images to perform segmentation or object detection within the adjusted resolution frame. This design allows the system to adapt to variable zoom levels, maintaining effective detection performance while efficiently managing processing resources, especially in time-sensitive environments with frequent zoom magnification fluctuations.

[0010] This method may include dynamically reselecting a neural network from multiple neural networks as the zoom level changes. This ensures that the selected neural network always corresponds to the camera's current zoom level, allowing for continuous adjustment during segmentation and detection. This dynamic reselection supports smooth zoom transitions and is particularly suitable for PTZ cameras where zoom levels may be frequently adjusted.

[0011] More specifically, the method may include: reselecting a neural network configured to operate at a lower image resolution when the zoom magnification exceeds a predetermined upper zoom magnification threshold, and reselecting a neural network configured to operate at a higher image resolution when the zoom magnification is below a predetermined lower zoom magnification threshold. These thresholds create a framework for balancing processing speed and detection accuracy as the zoom level crosses certain boundaries.

[0012] This disclosure further relates to a camera system comprising: a camera for capturing images, the camera having a zoom function; and processing circuitry configured to: acquire one or more images from the camera; acquire a zoom factor of the camera, wherein the zoom factor represents a measure of the current zoom level applied by the camera when capturing the one or more images; select a neural network from a plurality of neural networks configured to segment the one or more images and / or detect one or more objects in the one or more images based on the zoom factor, wherein the plurality of neural networks are configured to operate at different image resolutions, wherein a higher zoom factor corresponds to selecting a neural network configured to operate at a lower image resolution, and a lower zoom factor corresponds to selecting a neural network configured to operate at a higher image resolution; reduce the one or more images to the image resolution required by the selected neural network; and apply the selected neural network to segment the reduced one or more images and / or detect one or more objects in the reduced one or more images. Attached Figure Description

[0013] The accompanying drawings are exemplary and intended to illustrate a portion of the features of the currently disclosed computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability, and should not be construed as limiting the invention currently disclosed.

[0014] Figure 1 A flowchart is shown of an embodiment of a currently disclosed computer-implemented method for segmentation and / or object detection in images captured by a camera with zoom capabilities.

[0015] Figure 2 An embodiment of a camera system for performing segmentation and / or object detection is shown.

[0016] Figure 3A and Figure 3B An example of dynamically reselecting a neural network as the zoom level changes is shown.

[0017] Figure 4 The illustration shows a currently disclosed configuration of a camera system for segmentation and / or object detection with maximum depth of field of view.

[0018] Figure 5 An example of the variation in maximum depth is shown when the camera scans the environment. Detailed Implementation

[0019] This disclosure relates to a computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability.

[0020] A "zoom camera" refers to a camera equipped with a mechanism that allows the camera to change its field of view and the magnification of the scene being photographed by adjusting its zoom level. This function allows the camera to bring distant objects closer to the field of view, or to zoom out to cover a wider scene. Zoom functionality is achieved through optical zoom, digital zoom, or a combination of both methods.

[0021] Neural networks typically have input resolution and output resolution. The image resolution mentioned in this disclosure refers to the input resolution. This is the resolution or size of the image fed into the neural network. It defines the size of the input layer of the neural network, meaning that the neural network requires the image to be resized or reduced to that specific resolution if necessary before processing begins.

[0022] "Downscaling" refers to reducing the resolution of an image, which reduces the number of pixels and thus all detail. Downscaling results in a lower resolution while preserving the original content of the image. Therefore, downscaling can be a uniform reduction. Unlike cropping, downscaling retains the entire scene, but the level of detail is reduced.

[0023] Segmentation and object detection are processes commonly known to those skilled in the art of computer vision and image processing. In the context of this disclosure, neural networks are used to perform segmentation and object detection in order to identify and classify different regions or objects within images captured by a camera. Nevertheless, these concepts will be described in more detail below.

[0024] Segmentation is the process of dividing an image into different regions based on specific criteria (such as color, texture, or the presence of specific features). The goal of segmentation is to separate meaningful regions within an image, group pixels that share specific attributes, and distinguish these pixels from other regions. This process can be used to identify objects or specific parts of objects within an image.

[0025] Those skilled in the art are generally familiar with existing segmentation techniques and the implementation of segmentation algorithms.

[0026] As an example, segmentation can be achieved using convolutional neural networks (CNNs) or fully convolutional networks (FCNs). Unlike traditional CNNs, FCNs do not include fully connected layers. Instead, FCNs apply convolutional layers across the entire input image, thus preserving the spatial information needed for pixel-by-pixel prediction. This architecture allows the model to assign a class label to each pixel, resulting in a segmentation map of the output image. To preserve spatial detail, FCNs typically incorporate techniques such as up-sampling or deconvolutional layers, which restore spatial resolution after down-sampling operations within the neural network.

[0027] Some segmentation models incorporate multi-scale feature extraction to improve accuracy across various environments. For example, pyramid pooling modules or feature pyramids enable models to analyze input images at multiple scales. By processing images simultaneously at different resolutions, models can better handle variations in object size and improve segmentation accuracy, especially in complex scenes containing both large and small objects.

[0028] Segmentation models are typically trained on large datasets of labeled images, where each pixel is correctly labeled with its category. Training involves optimizing the model to minimize a loss function, such as cross-entropy loss or the Dice coefficient, which quantifies the difference between the predicted segmentation map and the ground truth. Data augmentation techniques, such as random cropping, scaling, and rotation, are often used to improve the model's robustness and performance on unseen data.

[0029] Once training is complete, the segmentation model generates a map that labels every pixel in the input image, enabling detailed scene analysis. This map can be used in a variety of applications. The output segmentation map can be used as the basis for further analysis, including object detection, tracking, and instance segmentation, where instances of the same object category are distinguished.

[0030] Object detection involves identifying and locating specific targets within an image, typically achieved by drawing bounding boxes around the detected objects. Object detection is not limited to simple classification, as it can also provide information about the location and size of each object detected within a scene. This process is commonly used in applications such as surveillance.

[0031] Those skilled in the art are generally familiar with existing object detection technologies and the implementation methods of object detection algorithms.

[0032] One approach to object detection is the convolutional neural network (CNN), which runs through an image to analyze spatial features and identify patterns associated with different object categories. Object detection models are generally divided into two categories: single-stage detectors and two-stage detectors. Single-stage detectors (such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector)) directly predict bounding boxes and class labels through a single forward propagation of the network, making them efficient in real-time applications. Two-stage detectors (such as Faster R-CNN (Region-based Convolutional Neural Network)) separate the detection process into two steps: generating region proposals and then classifying these proposals. This typically results in higher accuracy but slower processing speed.

[0033] YOLO is a commonly used single-stage detector known for its speed and efficiency. It divides an image into a grid and predicts both the bounding box and class probability for each grid cell simultaneously.

[0034] To implement object detection in practice, images are first processed by a CNN that generates feature maps. For single-stage models like YOLO, the neural network directly outputs bounding box coordinates, class labels, and confidence scores for each detected object. For two-stage models like Faster R-CNN, the RPN generates initial bounding boxes, which are further processed by a classification network to improve predictions. The final output is a set of bounding boxes with associated class labels and confidence scores, indicating the detected objects and their locations within the image.

[0035] Training an object detection model requires a large dataset in which each object is labeled with a bounding box and a class label. The model learns to minimize a multipart loss function, which combines a classification loss (for correct labeling) and a localization loss (for accurate bounding box prediction). Data augmentation techniques (such as scaling, cropping, and flipping) are typically used during training to improve the model's robustness and generalization ability.

[0036] Figure 1 A flowchart is shown illustrating an embodiment of a currently disclosed computer-implemented method 100 for segmentation and / or object detection in images captured by a camera with zoom capabilities. Figure 1 In a specific example, the computer implementation method 100 includes the following steps: Acquire one or more images from the camera (101); Obtain the zoom level of the camera, where the zoom level represents a measure of the current zoom level applied by the camera when capturing the one or more images (102). Based on the zoom magnification, a neural network is selected from a plurality of neural networks configured to segment the one or more images and / or detect one or more objects in the one or more images, wherein the plurality of neural networks are configured to operate at different image resolutions (103), wherein a higher zoom magnification corresponds to selecting a neural network configured to operate at a lower image resolution, and a lower zoom magnification corresponds to selecting a neural network configured to operate at a higher image resolution. Reduce the one or more images to the image resolution (104) required by the selected neural network; Apply the selected neural network to segment one or more scaled-down images and / or detect one or more objects in one or more scaled-down images (105).

[0037] In the context of this method, "obtaining zoom magnification" refers to the process of determining the current zoom level applied by the camera when capturing one or more images. Zoom magnification is a quantitative measure that represents the degree of magnification or reduction of the field of view achieved by adjusting the camera's zoom setting. Zoom magnification can be expressed in common terms (such as 1X, 2X, or 5X) to indicate the applied relative zoom level, but other measures may also be used.

[0038] Depending on the type of zoom the camera implements, zoom magnification can be obtained in several ways. For cameras with optical zoom, the zoom magnification can be read directly from the camera's hardware settings, which are typically associated with the physical position of the lens elements. For example, modern pan-tilt-zoom (PTZ) cameras usually have an internal sensor that provides feedback on the lens's current zoom position, enabling the system to obtain an accurate zoom magnification value at any given time.

[0039] A neural network is selected from multiple neural networks based on zoom magnification. The multiple neural networks are configured to operate at different image resolutions, where a higher zoom magnification corresponds to selecting a neural network configured to operate at a lower image resolution, and a lower zoom magnification corresponds to selecting a neural network configured to operate at a higher image resolution.

[0040] The method may further include a step of downscaling one or more images to the image resolution required by the selected neural network. This step may refer to adjusting the resolution of the captured images to match the input requirements of the selected neural network. Each neural network in the system is configured to operate at a specific image resolution, so the input image may need to be resized to match that resolution before processing.

[0041] Downsizing is the process of reducing the number of pixels in an image, which lowers the image's resolution. This can be achieved by applying image resampling techniques, such as bilinear interpolation or bicubic interpolation, to retain as much detail as possible within the new, smaller resolution while reducing the image size. For example, if a high-resolution image is captured at 1080p, but the chosen neural network is configured to operate at 720p, the image will be downsized from 1080p to 720p.

[0042] The selected neural network can then be used to segment one or more scaled-down images and / or detect one or more objects within those images.

[0043] In one embodiment of this disclosure, a neural network is dynamically reselected from multiple neural networks as the zoom level changes. This feature allows the system to adapt to adjustments in the camera's zoom level in real time. By dynamically reselecting the neural network based on the updated zoom level, the system can always operate using the neural network model best suited for the current zoom setting. This dynamic selection can be particularly useful in applications utilizing PTZ cameras, where the zoom level may change frequently to track objects or capture specific regions of interest.

[0044] The reselection process can be implemented using a control algorithm that continuously monitors the zoom level and triggers the selection of an appropriate neural network from memory whenever a change in zoom level is detected. The system can store multiple neural networks in memory, each operating at a different image resolution. When the zoom level changes, the control algorithm can select a neural network suitable for the new zoom level. This means that a neural network can be selected to ensure reliable detection of the object of interest. The neural network typically outputs a confidence score or confidence probability for each detected object, indicating the likelihood that the object was correctly identified. These probabilities can be used to determine constraints when selecting a neural network for each new zoom level.

[0045] In one embodiment of this disclosure, the method includes the step of: reselecting a neural network configured to operate at a lower image resolution when the zoom magnification exceeds a predetermined zoom magnification threshold.

[0046] Using a zoom threshold can reduce the processing load and the time required to perform segmentation and / or object detection. Such savings can initially be used to enable these tasks to be performed continuously in near real-time. Alternatively, the saved processing time can be used to include more features. For example, as the camera zooms in, more detailed analysis of the object can be valuable, such as pose estimation or detection of attributes worn by the object (such as a hat or items carried by the object). A combination of these two options is also feasible. This approach can leverage the architecture of a typical detection model, which typically includes a backbone network and one or more head modules. The backbone network can be used to extract basic features from the image. The head modules can then use input from the backbone network to perform specific tasks. Each head can be dedicated to a specific analysis task. In real-time scenarios where some additional processing time is available for each frame, the system can therefore incorporate additional head modules to extend the task scope. For example, in addition to standard detection tasks, additional heads can be used for, for example, identifying sub-objects, analyzing object behavior (e.g., detecting motion or interaction), or providing richer object attribute classifications. Accordingly, in one embodiment, the computer-implemented method further includes the steps of: performing additional analysis and / or detecting other sub-objects associated with an object detected in an image, wherein the additional analysis and / or sub-object detection is performed if sufficient processing time is available before acquiring subsequent images, as described below.

[0047] In one embodiment of this disclosure, the method further includes the step of reselecting a neural network configured to operate at a higher image resolution when the zoom magnification is below a predetermined lower zoom magnification threshold. This feature complements the use of an upper threshold by enabling the system to increase resolution when the camera zooms out to cover a wider field of view. Therefore, the lower zoom magnification threshold sets a point where the system determines that the background of the scene is expanded, thus requiring a more detailed view. When the zoom magnification drops below this threshold, the system can reselect a neural network configured to operate at a higher image resolution.

[0048] In some implementations, the zoom threshold can be configured to align with the specific operational requirements of the camera environment. For example, in traffic monitoring applications, the threshold can be set to activate when the camera zooms out sufficiently to cover the entire intersection, ensuring enough detail is available to detect distant vehicles and pedestrians.

[0049] Figure 3A An example of dynamically reselecting a neural network as the zoom magnification changes is shown. This example illustrates the change in zoom magnification over time. The zoom magnification is initially low. In this mode, a neural network configured to operate at high image resolution, i.e., a relatively large and slow neural network, can be used. At t1, the zoom magnification exceeds a first zoom magnification threshold 210. Accordingly, the method changes to a neural network configured to operate at a lower image resolution, in which case it could be a medium-sized neural network. At t2, the zoom magnification exceeds a second zoom magnification threshold 209. Accordingly, the method changes to a neural network configured to operate at even lower image resolutions, in which case it could be a relatively small and fast neural network. It should be noted that the terms "small," "medium," and "large" are all relative terms. The actual size of the neural network depends on the objective environment. At t3, the zoom magnification falls below the second zoom magnification threshold 209. Accordingly, the method changes to a medium-sized neural network. At t4, the zoom magnification falls below the first zoom magnification threshold 210. Accordingly, the method was adapted for large neural networks.

[0050] This method may include the steps of loading at least two or at least three neural networks from a plurality of neural networks into memory (preferably a cache), and dynamically reselecting between the at least two neural networks in memory when the zoom magnification changes. This feature allows the system to store multiple neural networks in accessible memory, thereby reducing the time required to switch between models when adjusting the zoom magnification. By preloading two or more networks, the system avoids the latency associated with loading models from slower storage devices such as hard drives or external storage. Caching is particularly advantageous in this implementation because it allows fast access to neural networks, thereby supporting the necessary real-time adjustments in applications requiring continuous zoom and fast response times.

[0051] The selection of preloaded neural networks can be managed by a control algorithm that monitors the zoom level and dynamically switches to the appropriate neural network stored in memory. For example, the system can preload a high-resolution neural network optimized for a wide field of view and a low-resolution neural network suitable for close-ups. When the zoom level changes, the control algorithm can immediately reselect the relevant neural network from memory, achieving a seamless transition with no processing latency.

[0052] Loading multiple networks into memory also provides flexibility in optimizing for different zoom ranges, as additional neural networks can be preloaded if hardware resources allow. For example, in systems with more advanced memory capabilities, three or more neural networks, each covering a different resolution range, can be preloaded. This configuration allows for finer-grained selection of neural networks as the zoom level changes, further optimizing detection and segmentation performance.

[0053] Furthermore, the currently disclosed method may include the following steps: when the zoom magnification increases to exceed a predetermined loading threshold (which is lower than a predetermined zoom magnification threshold), preloading is performed on a neural network configured to operate at a lower image resolution. This feature allows the system to anticipate the need for a lower-resolution neural network as the zoom magnification approaches a higher level, thereby preloading in advance to ensure a smooth transition. By setting a predetermined loading threshold lower than the zoom magnification threshold used for actual neural network reselection, the system can prepare in advance without prematurely triggering network switching.

[0054] Figure 3B An example of dynamically reselecting a neural network as the zoom magnification changes is shown, where the neural network is preloaded when the zoom magnification approaches a corresponding zoom magnification threshold. The selection of the neural network and... Figure 3AThe same applies to the above. Besides the choice of neural network, the figure also illustrates how to preload the neural network into memory at the correct time. The zoom magnification is initially low. As the zoom magnification increases and exceeds the first loading threshold 212 at t1, a medium-sized neural network is preloaded. At t2, the method changes to a medium-sized neural network. As the zoom magnification increases further and exceeds another first loading threshold 212 at t3, a small neural network is preloaded. At t4, the method changes to a small neural network. Similarly, neural networks can be preloaded as the zoom magnification decreases. As the zoom magnification decreases and falls below the second loading threshold 213 at t5, a medium-sized neural network is preloaded. At t6, the method changes to a medium-sized neural network. As the zoom magnification decreases and falls below another second loading threshold 213 at t7, a large neural network is preloaded. At t8, the method changes to a large neural network.

[0055] The loading threshold can be configured based on application-specific needs, thus reserving buffer time for the system to load the appropriate neural network before it is needed. For example, if the zoom level threshold for switching to a lower resolution neural network is set to 15X, the loading threshold can be set to 13X. When the zoom level increases beyond 13X, the system preloads the lower resolution neural network into the cache, preparing for the transition. Different loading thresholds can exist for different zoom level thresholds.

[0056] Similarly, the currently disclosed method may include the following steps: when the zoom magnification decreases to below a predetermined loading threshold (which is above a predetermined zoom magnification threshold), a neural network configured to operate at a higher image resolution is preloaded. This feature allows the system to prepare for a wider field of view by preloading a higher-resolution neural network before it is actually needed. The loading threshold, set above the actual zoom magnification threshold, acts as a prediction trigger, allowing the system to load the high-resolution neural network into memory in advance. This preloading mechanism ensures that the higher-resolution neural network is immediately available once the zoom magnification decreases to a point where more detail is required.

[0057] The loading threshold can be configured to provide an appropriate buffer to adapt to varying zoom speeds and usage scenarios. For example, if the zoom threshold for switching to a higher resolution neural network is set to 5X, the loading threshold can be set to 6X. When the zoom magnification drops below 6X, the system preloads the high-resolution neural network into the cache, ensuring that the neural network is ready for use once the zoom magnification drops to 5X or below. Different loading thresholds can exist for different zoom magnification thresholds.

[0058] In one embodiment of this disclosure, the step of selecting one neural network from multiple neural networks is performed based on a table that associates each neural network with a predetermined zoom range of the camera. Preferably, the table includes the association between each neural network and its corresponding zoom range, enabling the system to make a direct and efficient selection based on the current zoom range. By referring to this table, the system can quickly identify which neural network to activate when the zoom range changes, thereby optimizing detection accuracy and computational efficiency. This table-based approach allows for the customization of predetermined configurations to meet different operational needs. Using the table to manage neural network selection also facilitates easy modification and updates. The table can be adjusted based on empirical data, evolving application requirements, or changes in the environment, allowing the system to recalibrate its network associations as necessary. For example, the table can be reconfigured to adapt to new zoom ranges as hardware capabilities or application requirements change without altering the underlying system architecture.

[0059] Based on specific and non-limiting examples, the table may include three different neural networks: a large (L) neural network, a medium (M) neural network, and a small (S) neural network. These neural networks can be used in conjunction with the following non-limiting zoom ranges. Table 1 shows examples of feasible configurations. Table 1

[0060] In one embodiment of this disclosure, the predetermined zoom range in the table is selected such that for an object of a given physical size located at the maximum distance from the camera in the scene, at least a minimum predetermined pixel density is always provided to the selected neural network. This feature ensures that the system maintains an acceptable level of image detail even at the maximum distance for accurate object detection and segmentation. The predetermined pixel density ensures that the selected neural network provides sufficient resolution to accurately identify and classify objects within the field of view. In a practical example, "minimum predetermined pixel density for an object of a given physical size" can be interpreted as: a given object (e.g., a car or a person) of a specific size in the captured image, when at a certain distance from the camera, has at least a minimum number of pixels depicted in the scaled-down image. In other words, the zoom range and its association with the neural network can be formulated to take these factors into account, thereby ensuring that the combination of zoom and neural network correctly detects the object of interest. In one embodiment of the currently disclosed method, the neural network is selected such that an object of a given physical size located at the maximum distance from the camera in the scene is depicted by at least a predetermined minimum number of pixels in the scaled-down image.

[0061] The term "object detection" should be interpreted broadly. The term itself can be broadly regarded as determining whether a particular category of objects exists in the field of view, but it can also be understood to include more specific detections, such as the identification of a specific individual.

[0062] The term "pixel density" refers to the concentration of pixels used to represent an object in an image, in relation to the object's physical size in the real world. Essentially, the term describes the resolution at which a physical object is depicted in an image. For example, if an object that is 2 meters in one direction at real-world dimensions is represented by 200 pixels in that direction, then the pixel density is 100 pixels per meter. Those skilled in the art will understand that pixel density can also be expressed in two dimensions: pixels per unit area in the real world.

[0063] Different detection and recognition tasks are known to require different minimum pixel densities. For example, facial recognition might require 125 pixels per meter. Other factors, such as light direction and optical quality, can also influence this requirement.

[0064] The concept of creating the table can be explained through an example. For each image acquired by the camera at a specific zoom magnification, the required neural network width and height can be calculated. The neural network width and height refer to the dimensions of the image that serves as input to the neural network. These parameters can be calculated as follows: Required network width = Network width at minimum zoom / (Zoom magnification × constant), and Required network height = Network height at minimum zoom / (Zoom magnification × constant). Once these calculations are complete, the available neural networks can be evaluated. A neural network that best approximates the calculated required network width and height can be selected. Therefore, the constant can be chosen such that objects in the image at the selected resolution are depicted with at least the desired minimum pixel density. This desired minimum pixel density can correspond to the minimum number of pixels required to identify the object. For example, if, for accurate detection, the object of interest needs to be represented by at least 50 pixels, the table is configured to allocate the zoom range to each neural network in a way that the object will still be detected even if it is at its maximum distance from the camera in the current field of view.

[0065] In one embodiment of this disclosure, the plurality of neural networks includes at least three neural networks. By including at least three neural networks, the system achieves finer control over the overall image processing at different zoom levels, thereby ensuring that the selected neural network is highly consistent with the camera's current field of view and the required level of detail. Each of the three neural networks can be configured for different resolution ranges. For example, the first neural network can be configured to operate at high resolution and is selected when the camera zooms out to cover a large field of view. The second neural network operates at medium resolution and is optimized for a medium zoom level range, balancing detail and efficiency when the camera zooms in to a specific area. The third neural network can be configured for low resolution, suitable for close-up views at high zoom magnifications, where the field of view is narrow and less image detail is required. The methods and systems disclosed herein are not limited to two or three neural networks. In some embodiments, even more neural networks may be included.

[0066] In one embodiment of this disclosure, the computer-implemented method includes the steps of: acquiring and / or extracting the maximum depth of one of the fields of view in an image, and selecting a neural network in such a manner that objects of a given physical size located within the maximum depth are depicted at least with a predetermined pixel density required by the neural network. "Predetermined pixel density required by the neural network" may refer to selecting the neural network in such a manner that a given object (e.g., a car or person) of a specific size in the captured image, when located at a certain distance from the camera, is depicted in at least a minimum number of pixels in the scaled-down image used by the neural network, enabling the neural network to perform a task with a certain probability of success. For example, the task may be segmentation or detection of a specific category of objects. This feature allows the system to utilize information about the maximum depth in the scene to optimize the selection of the neural network. When a maximum depth is identified (e.g., when the camera's field of view includes buildings or other background structures), the system can be configured to assess that targets of interest (such as people and vehicles) are unlikely to be located outside that maximum depth. This information allows the system to reasonably adjust the resolution requirements of the selected neural network for a given task, as the system can limit its focus to the range where the object is likely to actually be present.

[0067] For example, if the camera is facing a building 30 meters away, that building serves as the maximum depth background for that particular field of view. Knowing this maximum depth, the system can ensure that objects in front of the building (such as pedestrians or vehicles) are depicted using a pixel density sufficient for accurate detection and classification. Since any object detected in front of the building must be within that 30-meter range, the system can select a neural network optimized for that specific depth, rather than one suitable for a wider range, and can scale the image down to the appropriate neural network.

[0068] Figure 4 This illustration shows a currently disclosed configuration of a camera system for segmentation and / or object detection with a maximum depth 208 for a field of view 205b. In this example, camera 201 initially has a first field of view 205a. As the camera zooms in to the field of view 205b, it identifies building 207 as an occluding background object and sets a maximum depth 208 for the field of view 205b. Within this field of view 205b, objects of a specific type (e.g., person 206a) are known to have a specific size and a specific minimum pixel density because the camera cannot see behind the building. Figure 4 In the example, the camera cannot see object 206b. Since the camera cannot see object 206b, the method does not need to consider any parameters related to the detection of object 206b. In this image, object 206a has been identified and is surrounded by a bounding box.

[0069] Figure 4 The configuration can be illustrated by the following non-limiting example. As a first step, the maximum depth of the field of view (beyond which objects cannot be observed) is calculated or extracted. In this example, building 207 is therefore set to a maximum depth 208 for the current field of view 205b. As an example, the maximum depth can be set to 30 meters. In the second step, the pixel density at the maximum depth can be calculated for the current zoom level. As an example, if a person 2 meters tall is 30 meters away from the camera and the camera has a given zoom level, the person will be depicted with a certain number of pixels. In the third step, this pixel density can now be compared to the pixel density required for the neural network to perform a specific task (such as object detection). If the calculated pixel density at the maximum depth is greater than the required pixel density, it means the image can be scaled down while still meeting the pixel density required for the neural network to perform the task. Based on this information, the method can then select a neural network. For example, the method can select a neural network with the minimum resolution capable of performing the task. This example illustrates how the maximum depth can be considered to balance processing efficiency while maintaining sufficient accuracy for the task.

[0070] The currently disclosed computer implementation method may further include the following steps: scanning the environment (such as a 360° field of view) using a camera, and processing one or more images to generate a depth map, further including: considering the depth map to set a maximum depth of the field of view for each image. This feature enables the system to acquire a comprehensive characterization of the environment by scanning across a wide range (potentially up to a 360° full field of view).

[0071] The depth map generated from the scanned image can provide depth information for each region of the environment, allowing the system to establish a maximum depth for each field of view. This depth map can be used as a reference, enabling the system to dynamically adjust its neural network selection based on the maximum depth for each field of view being processed.

[0072] When the camera moves or sweeps across a scene from one side to the other, the scan can be (but does not necessarily have to be) a horizontal scan. If a PTZ camera performs such a scan, it can be called a panning scan or a panoramic scan. In this mode, the camera rotates horizontally, sweeping across the scene from one side to the other to capture a wide field of view.

[0073] Figure 5 An example of the variation of the maximum depth 208 is shown when the camera scans the environment. Translation in the horizontal angle is shown on the x-axis, while the maximum depth 208 in the horizontal direction is shown on the y-axis.

[0074] In one embodiment, the computer-implemented method further includes the steps of performing additional analysis and / or detecting other sub-objects associated with an object detected in an image, wherein the additional analysis and / or sub-object detection is performed if sufficient processing time remains before subsequent images are acquired. This feature utilizes any reserved processing time to perform supplementary tasks. When the system selects a smaller, less computationally intensive neural network, processing resources can be freed up within frame rate intervals, allowing time for further analysis without interrupting real-time processing of incoming frames.

[0075] Typically, a system operates within a predetermined frame rate, meaning each frame needs to be processed within a specific time interval to maintain real-time performance. By selecting a smaller neural network at a specific zoom level or under specific conditions, the system can complete the main object detection and segmentation tasks faster, thus allowing additional processing time before the next frame must be processed. This extra time can be used to perform more detailed analysis of the detected objects, such as identifying sub-objects (e.g., attributes of items like hats, bags, or devices carried by a person), or performing fine-grained analysis (such as pose estimation or behavior analysis) to gather more contextual information.

[0076] The ability to perform additional analyses based on available processing time provides flexibility and enhances the system's adaptability. For example, in security applications, after detecting a person, the system can use the additional processing time to perform sub-object detection to identify objects carried by that individual (such as backpacks) or items in their hands. Alternatively, the system can use the additional time to refine the classification of detected objects to improve the confidence level of the detection, or to identify specific attributes (such as clothing color or other distinguishing features). This supplementary information can be valuable for applications that require more detailed object recognition or context-aware analysis.

[0077] This disclosure further relates to a computer program having instructions that, when executed by a computing device or computing system, cause the computing device or computing system to perform a method for segmentation and / or object detection in images captured by a camera with zoom capability, according to the described embodiments. The computer program may be stored on a computer-readable medium such as non-volatile memory, optical disc, or magnetic disk, or may be transmitted as a data signal via a communication network or other transmission medium. The computer program may take the form of a standalone application, library, or embedded firmware configured to run on a range of devices, including but not limited to servers, desktop computers, mobile devices, and dedicated hardware systems (such as cameras) with built-in processing capabilities.

[0078] This disclosure further relates to a camera system comprising: a camera for capturing images, the camera having a zoom function; and processing circuitry configured to: acquire one or more images from the camera; acquire a zoom factor of the camera, wherein the zoom factor represents a measure of the current zoom level applied by the camera when capturing the one or more images; select a neural network based on the zoom factor from a plurality of neural networks configured to segment the one or more images and / or detect one or more objects in the one or more images, wherein the plurality of neural networks are configured to operate at different image resolutions, wherein a higher zoom factor corresponds to selecting a neural network configured to operate at a lower image resolution, and a lower zoom factor corresponds to selecting a neural network configured to operate at a higher image resolution; reduce the one or more images to the image resolution required by the selected neural network; and apply the selected neural network to segment the reduced one or more images and / or detect one or more objects in the reduced one or more images.

[0079] Those skilled in the art will understand that the camera system can be configured to perform computer-based methods for segmentation and / or object detection in an image, as currently disclosed in any embodiment, and vice versa.

[0080] Figure 2An embodiment of a camera system 200 for performing segmentation and / or object detection is illustrated. The camera system 200 includes a camera 201 and processing circuitry 202 configured to perform steps of a currently disclosed computer-implemented method for performing segmentation and / or object detection in an image captured by the camera 201. The camera system 200 further includes memory 203. In this example, three different neural networks 204a, 204b, and 204c are loaded in memory 203.

[0081] The processing circuitry may include one or more processors, such as a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other dedicated hardware components designed to support image processing, neural network inference, and real-time data processing. The processing circuitry may be directly integrated into the camera system.

[0082] The processing circuitry can be designed to support parallel processing, hardware acceleration, and optimized data processing techniques. For example, the circuitry can utilize GPUs or dedicated neural network processing units (NPUs) to accelerate neural network computations and achieve real-time segmentation and object detection even in the context of high-resolution imaging. The circuitry can further support instruction pipelines, vector processing, and data caching to improve efficiency and processing speed. Furthermore, the processing circuitry may include components for dynamically allocating processing resources based on zoom levels and associated neural network requirements.

[0083] The processing circuitry can be configured to access and execute instructions from non-volatile memory.

[0084] List of components in the attached diagram 100 – Computer-based implementation methods for segmentation and / or object detection 101 - Acquire one or more images from the camera 102 - Get the camera's zoom level 103 – Selecting a neural network from multiple neural networks configured to segment one or more images and / or detect one or more objects in one or more images based on zoom magnification. 104 – Resize one or more images to the image resolution required for the selected neural network. 105 – Apply a selected neural network to segment one or more scaled-down images and / or detect one or more objects within one or more scaled-down images. 200 - Camera System 201 - Camera 202 - Processing Circuit 203 - Memory 204 - Neural Networks 205 - Field of View 206 - Object 207 - Background Object 208 - Maximum Depth 209 - Second zoom threshold / Upper zoom threshold 210 - First zoom level threshold / Lower zoom level threshold 211 - Bounding Box 212 - First Loading Threshold 213 - Second Loading Threshold

Claims

1. A computer-implemented method for segmentation and / or object detection in images captured by a camera with zoom capability, the method comprising: Acquire one or more images from the camera; Obtain the zoom level of the camera, wherein the zoom level represents a measure of the current zoom level applied by the camera when capturing the one or more images; Based on the zoom magnification, a neural network is selected from a plurality of neural networks configured to segment the one or more images and / or detect one or more objects in the one or more images, wherein the plurality of neural networks are configured to operate at different image resolutions, wherein a higher zoom magnification corresponds to selecting a neural network configured to operate at a lower image resolution, and a lower zoom magnification corresponds to selecting a neural network configured to operate at a higher image resolution. The one or more images are reduced to the image resolution required by the selected neural network; The selected neural network is applied to segment the scaled-down images and / or to detect the one or more objects in the scaled-down images.

2. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 1, wherein, When the zoom level changes, the neural network is dynamically reselected from the plurality of neural networks.

3. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 2, comprising the following steps: When the zoom factor exceeds a predetermined upper threshold, the neural network configured to operate at a lower image resolution is reselected.

4. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 2, comprising the following steps: When the zoom factor is lower than a predetermined zoom factor threshold, the neural network configured to operate at a higher image resolution is reselected.

5. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 1, comprising the following steps: At least two or at least three of the plurality of neural networks are loaded into memory, and when the zoom magnification changes, the at least two neural networks in the memory are dynamically reselected. The memory is preferably a cache.

6. A computer-implemented method for segmentation and / or object detection in images captured by a camera with zoom capability, according to any one of claims 3 to 5, comprising the following steps: When the zoom magnification increases to exceed a predetermined upper loading threshold, a neural network is preloaded to operate at a lower image resolution, and / or when the zoom magnification decreases to below a predetermined lower loading threshold, a neural network is preloaded to operate at a higher image resolution, wherein the upper loading threshold is lower than the predetermined upper zoom magnification threshold, and the lower loading threshold is higher than the predetermined lower zoom magnification threshold.

7. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 1, wherein, The step of selecting one neural network from the plurality of neural networks is performed based on a table that associates each neural network with a predetermined zoom range of the camera.

8. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 7, wherein, The predetermined zoom range in the table is selected such that for an object of a given physical size located at the maximum distance from the camera, the selected neural network is always provided with the minimum predetermined pixel density.

9. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 1, wherein, The neural network is selected in such a way that an object of a given physical size located at the maximum distance from the camera is depicted at least at the predetermined pixel density required by the neural network after being scaled down.

10. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 1, wherein, The plurality of neural networks includes at least three neural networks.

11. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 1, comprising the following steps: Obtain and / or extract the maximum depth of the field of view of one of the images, and select a neural network in such a way that objects of a given physical size located within the maximum depth are depicted at least at a predetermined pixel density required by the neural network after being scaled down.

12. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 1, comprising the following steps: The camera is used to scan the environment with a field of view such as 360°, and the one or more images are processed to generate a depth map. This further includes: taking into account the depth map to set the maximum depth of the field of view for each image.

13. The computer implementation method for segmentation and / or object detection in images captured by a camera with zoom capability according to claim 1, further comprising the following steps: Additional analysis and / or detection of other sub-objects associated with an object are performed based on an object detected in the image, wherein the additional analysis and / or sub-object detection are performed if sufficient processing time is available before subsequent images are acquired.

14. A computer program having instructions that, when executed by a computing device or computing system, cause the computing device or computing system to perform a method for segmentation and / or object detection in an image captured by a camera with zoom capability according to any one of claims 1 to 13.

15. A camera system comprising: A camera for taking images, the camera having a zoom function; The processing circuit is configured as follows: Acquire one or more images from the camera; Obtain the zoom level of the camera, wherein the zoom level represents a measure of the current zoom level applied by the camera when capturing the one or more images; Based on the zoom magnification, a neural network is selected from a plurality of neural networks configured to segment the one or more images and / or detect one or more objects in the one or more images, wherein the plurality of neural networks are configured to operate at different image resolutions, wherein a higher zoom magnification corresponds to selecting a neural network configured to operate at a lower image resolution, and a lower zoom magnification corresponds to selecting a neural network configured to operate at a higher image resolution. The one or more images are reduced to the image resolution required by the selected neural network; The selected neural network is applied to segment the scaled-down images and / or to detect the one or more objects in the scaled-down images.

Images