Maintaining the target object within the frame at a constant size.

Abstract

Techniques are provided for processing one or more frames. For example, a region of interest may be determined in a first frame of a sequence of frames. The region of interest in the first frame includes an object having a size in the first frame. A portion of a second frame of the sequence of frames (which follows the first frame in the sequence of frames) may be cropped and scaled so that the object in the second frame has the same size (and possibly the same position) as the object in the first frame.

Description

[Technical Field]

[0001] This disclosure relates to video analysis in general, and more specifically to techniques and systems for maintaining a stable (e.g., constant or nearly constant) size of a target object in one or more frames (e.g., in video analysis of recorded video, among various applications). [Background technology]

[0002] Many devices and systems enable scene capture by generating images (or frames) and / or video data (including multiple frames) of the scene. For example, a camera or a computing device containing cameras (e.g., a mobile device such as a cell phone or smartphone containing one or more cameras) can capture a sequence of frames of a scene. Another example is the Internet Protocol camera (IP camera), a type of digital video camera that can be used for surveillance or other purposes. Unlike analog closed-circuit television (CCTV) cameras, IP cameras can transmit or receive data over computer networks and the internet.

[0003] Image and / or video data may be captured and processed by such devices and systems (e.g., mobile devices, IP cameras, etc.) and output for consumption (e.g., displayed on the device and / or other devices). In some cases, image and / or video data may be captured by such devices and systems and output for processing and / or consumption by other devices. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," CoRR, abs / 1409.1556, 2014. [Non-Patent Document 2] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," arXiv preprint arXiv:1506.02640, 2015. [Overview of the project] [Means for solving the problem]

[0005] In some examples, techniques and systems are described for processing one or more frames of image or video data so as to maintain a target object (also called an object of interest) in one or more frames at a constant size. According to at least one example for description, a method for one or more frames is provided. The method includes the steps of: determining a region of interest in a first frame of a sequence of frames, wherein the region of interest in the first frame includes an object having a certain size in the first frame; cropping a portion of a second frame of the sequence of frames, wherein the second frame follows the first frame in the sequence of frames; and scaling that portion of the second frame based on the size of that object in the first frame.

[0006] In another example, a device for processing one or more frames is provided, comprising a memory configured to store at least one frame, and one or more processors implemented by circuitry and coupled to the memory. The one or more processors are configured and can perform: determining a region of interest in a first frame of a sequence of frames, wherein the region of interest in the first frame includes an object of a certain size in the first frame; cutting out a portion of a second frame of a sequence of frames, wherein the second frame is after the first frame in the sequence of frames; and scaling that portion of the second frame based on the size of that object in the first frame.

[0007] In another example, a non-temporary computer-readable medium is provided which, when executed by one or more processors, causes one or more processors to determine a region of interest in a first frame of a sequence of frames, wherein the region of interest in the first frame includes an object of a certain size in the first frame; to cut out a portion of a second frame of a sequence of frames, wherein the second frame is after the first frame in the sequence of frames; and to scale that portion of the second frame based on the size of the object in the first frame.

[0008] In another example, an apparatus for processing one or more frames is provided. The apparatus includes means for determining a region of interest in a first frame of a sequence of frames, wherein the region of interest in the first frame includes an object having a certain size in the first frame; means for cropping a portion of a second frame of a sequence of frames, wherein the second frame follows the first frame in the sequence of frames; and means for scaling the portion of the second frame based on the size of the object in the first frame.

[0009] In some aspects, the methods, apparatuses, and computer-readable media described above further comprise receiving a user input corresponding to a selection of an object in a first frame and determining a region of interest in the first frame based on the received user input. In some aspects, the user input includes a touch input provided using a touch interface of the device.

[0010] In some aspects, the methods, apparatuses, and computer-readable media described above further comprise determining a point of an object region determined for an object in a second frame and cropping and scaling a portion of the second frame such that the point of the object region is at the center of the cropped and scaled portion.

[0011] In some aspects, the point of the object region is the center point of the object region. In some cases, the object region is a bounding box (or other bounding region). The center point can be the center point of the bounding box (or other region), the center point of the object (e.g., the centroid or center point of the object).

[0012] In some aspects, by scaling the portion of the second frame based on the size of the object in the first frame, the object in the second frame is sized to be the same as the object in the first frame.

[0013] In some aspects, the methods, apparatuses, and computer-readable media described above further comprise determining a first length associated with an object in a first frame, determining a second length associated with the object in a second frame, determining a scaling factor based on a comparison of the first length and the second length, and scaling the portion of the second frame based on the scaling factor.

[0014] In some embodiments, the first length is the length of a first object region determined for an object in a first frame, and the second length is the length of a second object region determined for an object in a second frame. In some embodiments, the first object region is a first bounding box, the first length is the diagonal length of the first bounding box, the second object region is a second bounding box, and the second length is the diagonal length of the second bounding box.

[0015] In some embodiments, by scaling that portion of the second frame based on a scaling factor, the second object region in the cropped and scaled portion will have the same size as the first object region in the first frame.

[0016] In some embodiments, the methods, apparatus, and computer-readable media described above further comprise determining points of a first object region generated for an object in a first frame, determining points of a second object region generated for an object in a second frame, determining a motion coefficient for the object based on a smoothing function using the points of the first and second object regions, wherein the smoothing function controls changes in the object's location across multiple frames in a sequence of frames, and cropping the portion of the second frame based on the motion coefficient.

[0017] In some embodiments, a point in the first object region is the center point of the first object region, and a point in the second object region is the center point of the second object region.

[0018] In some embodiments, the smoothing function includes a translation function, which is used to determine the position of points in each of multiple frames of a sequence of frames, based on a statistical measure of the object's motion.

[0019] In some embodiments, the methods, apparatus, and computer-readable media described above further comprise determining a first length relating to an object in a first frame, determining a second length relating to an object in a second frame, determining a scaling factor for the object based on a comparison of the first and second lengths and on a smoothing function using the first and second lengths, wherein the smoothing function controls the change in size of the object in multiple frames of a sequence of frames, and scaling that portion of the second frame based on the scaling factor.

[0020] In some embodiments, the smoothing function includes a translation function, which is used to determine the length of the object in each of multiple frames of a sequence of frames, based on a statistical measure of the object's size.

[0021] In some embodiments, the first length is the length of the first bounding box generated for an object in the first frame, and the second length is the length of the second bounding box generated for an object in the second frame.

[0022] In some embodiments, the first length is the diagonal length of the first bounding box, and the second length is the diagonal length of the second bounding box.

[0023] In some embodiments, by scaling that portion of the second frame based on a scaling factor, the second bounding box in the cropped and scaled portion will have the same size as the first bounding box in the first frame.

[0024] In some embodiments, cropping and scaling of a portion of the second frame maintains the object at the center of the second frame.

[0025] In some embodiments, the methods, apparatus, and computer-readable media described above further include detecting and tracking objects in one or more frames of a sequence of frames.

[0026] In some embodiments, the apparatus comprises a camera (e.g., an IP camera), a mobile device (e.g., a mobile phone or a so-called “smartphone” or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other devices. In some embodiments, the apparatus includes a camera or a number of cameras for capturing one or more images. In some embodiments, the apparatus further includes a display for displaying one or more images, notifications, and / or other viewable data.

[0027] This summary does not identify the primary or essential features of the claimed subject matter, nor is it to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the entire specification of this patent, any or all of the drawings, and the appropriate parts of each claim.

[0028] The above, along with other features and embodiments, will become clearer with reference to the following specification, claims, and accompanying drawings.

[0029] Exemplary embodiments of this application are described in detail below with reference to the following drawings. [Brief explanation of the drawing]

[0030] [Figure 1] This block diagram shows exemplary architectures of image acquisition and processing systems, with several examples. [Figure 2] This block diagram shows examples of systems including video sources and video analysis systems, with several examples. [Figure 3] This is an example of video analysis that processes video frames, using several examples. [Figure 4] This is a block diagram showing examples of blob detection systems, with several examples. [Figure 5] This block contains examples of object tracking systems, with several examples. [Figure 6A] Here is another diagram illustrating examples of machine learning-based object detection and tracking systems, with several examples. [Figure 6B] This figure shows examples of upsampling components for machine learning-based object detection and tracking systems, with several examples. [Figure 6C] This figure shows examples of backbone architectures for machine learning-based tracking systems, with several examples. [Figure 7] This figure shows examples of machine learning-based object classification systems, illustrated with several examples. [Figure 8A] This figure shows an example of a system that includes a frame cropping and scaling system, with several examples. [Figure 8B] This figure shows examples of frame cropping and scaling systems, with several examples. [Figure 8C] This figure shows examples of the frame cropping and scaling process, with several examples. [Figure 9A] This flowchart illustrates another example of the frame cropping and scaling process, using several examples. [Figure 9B] This flowchart illustrates another example of the frame cropping and scaling process, using several examples. [Figure 10A] This figure shows examples of the initial frames of a video, with several examples. [Figure 10B] This figure shows examples of subsequent frames in a video following the initial frame in Figure 10A, using several examples. [Figure 11] This figure shows examples of various motion models, illustrated by several examples. [Figure 12] This flowchart illustrates an example of the process for performing image stabilization, using several examples. [Figure 13A] This figure shows examples of the process for executing the automatic zoom function, with several examples. [Figure 13B] This figure shows an example of the process for performing additional aspects of the auto-zoom function, with several examples. [Figure 13C] This figure shows another example of the process for performing an automatic zoom function, with several examples. [Figure 13D] This figure shows an example of the process for performing additional aspects of the auto-zoom function, with several examples. [Figure 14] This graph shows examples of Gaussian filter smoothing functions, illustrated by several examples. [Figure 15] This graph shows examples of Fibonacci filter smoothing functions, using several examples. [Figure 16] This figure shows examples of the zoom process in a camera pipeline, with several examples. [Figure 17] This figure shows examples of zoom latency for camera pipelines, with several examples. [Figure 18] This flowchart illustrates an example of a process for processing one or more frames, using several examples. [Figure 19] These images illustrate simulations using the cropping and scaling techniques described herein, using several examples. [Figure 20] These images illustrate simulations using the cropping and scaling techniques described herein, using several examples. [Figure 21] These images illustrate simulations using the cropping and scaling techniques described herein, using several examples. [Figure 22]These images illustrate simulations using the cropping and scaling techniques described herein, using several examples. [Figure 23] These images illustrate simulations using the cropping and scaling techniques described herein, using several examples. [Figure 24] This figure shows examples of machine learning-based object detection and tracking systems, illustrated by several examples. [Figure 25] This flowchart illustrates an example of a camera lens switching pipeline, using several examples. [Figure 26] This flowchart illustrates an example of the camera lens switching process, using several examples. [Figure 27] This figure shows examples of using the camera lens switching technique described herein, with several examples. [Figure 28] This figure shows examples of using the camera lens switching technique described herein, with several examples. [Figure 29] This figure shows examples of using the camera lens switching technique described herein, with several examples. [Figure 30] This figure shows examples of using the camera lens switching technique described herein, with several examples. [Figure 31] This figure shows examples of using the camera lens switching technique described herein, with several examples. [Figure 32] This figure shows examples of using the camera lens switching technique described herein, with several examples. [Figure 33] This figure shows examples of using the camera lens switching technique described herein, with several examples. [Figure 34] This figure shows examples of using the camera lens switching technique described herein, with several examples. [Figure 35]This figure shows examples of using the camera lens switching technique described herein, with several examples. [Figure 36] This figure shows examples of using the camera lens switching technique described herein, with several examples. [Figure 37] These images illustrate simulations using the camera lens switching technique described herein, with several examples. [Figure 38] These images illustrate simulations using the camera lens switching technique described herein, with several examples. [Figure 39] These images illustrate simulations using the camera lens switching technique described herein, with several examples. [Figure 40] These images illustrate simulations using the camera lens switching technique described herein, with several examples. [Figure 41] These images illustrate simulations using the camera lens switching technique described herein, with several examples. [Figure 42] This block contains examples of deep learning networks, with several examples. [Figure 43] This block diagram shows examples of convolutional neural networks, illustrated with several examples. [Figure 44] This figure shows examples of Cifar-10 neural networks, illustrated by several examples. [Figure 45A] This figure shows examples of single-shot object detectors, illustrated with several examples. [Figure 45B] This figure shows examples of single-shot object detectors, illustrated with several examples. [Figure 45C] This figure shows examples of single-shot object detectors, illustrated with several examples. [Figure 46A] This figure shows examples of You Only Look Once (YOLO) detectors, illustrated with several examples. [Figure 46B] This figure shows examples of YOLO detectors, illustrated by several examples. [Figure 46C] This figure shows examples of YOLO detectors, illustrated by several examples. [Figure 47] This figure shows an example of a system for implementing some of the embodiments described herein. [Modes for carrying out the invention]

[0031] Several aspects and embodiments of this disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. Specific details are provided below for illustrative purposes to provide a complete understanding of the embodiments of this application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting.

[0032] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of exemplary embodiments provides a description that enables the implementation of the exemplary embodiments to those skilled in the art. It should be understood that various modifications can be made to the function and configuration of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0033] An image acquisition device (e.g., a camera or a device containing a camera) is a device that uses an image sensor to receive light and capture image frames, such as still images or video frames. The terms “image,” “image frame,” and “frame” are used interchangeably herein. The camera of an image acquisition device may be configured with various image acquisition and image processing settings. Different settings result in images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f / stop, shutter speed, focus, and gain, are determined and applied before or during the acquisition of one or more image frames. For example, settings or parameters may be applied to the image sensor for capturing one or more image frames. Other camera settings, such as contrast, brightness, saturation, sharpness, levels, curves, or color changes, can constitute post-processing of one or more image frames. For example, settings or parameters may be applied to a processor (e.g., an image signal processor or ISP) for processing one or more image frames captured by the image sensor.

[0034] A camera may include, or communicate with, a processor such as an ISP that can receive one or more image frames from an image sensor and process one or more image frames. For example, raw image frames captured by a camera sensor may be processed by an ISP to produce a final image. In some examples, the ISP may process the image frames using a number of filters or processing blocks applied to the captured image frames, among other things, such as demosaicing, gain adjustment, white balance adjustment, color balance or color correction, gamma compression, tone mapping or tone adjustment, noise reduction or noise filtering, edge enhancement, contrast adjustment, intensity adjustment (such as darkening or brightening). In some examples, the ISP may include a machine learning system (e.g., one or more neural networks and / or other machine learning components) that can process the image frames and output a processed image frame.

[0035] In various scenarios (e.g., mobile imaging and video analysis, among other use cases), it may be desirable to maintain the size of the region of interest and / or object of interest (or target object) across frames in a sequence of frames, even if the region of interest and / or object moves relative to one or more cameras capturing the sequence of frames (e.g., video). For example, when imaging a person playing soccer in a video capture scenario, it may be desirable to maintain a constant size of the person throughout the video, even if the person moves relative to the camera (e.g., towards the camera, and laterally relative to the camera). In another example, with respect to video analysis, it may be desirable to maintain the size of a tracked object (e.g., a delivery person) across video clips captured by one or more Internet Protocol (IP) camera systems.

[0036] Image acquisition devices are offering increasingly greater effective zoom ranges. For example, multi-camera systems can be designed to allow for zoom ranges greater than the digital zoom range of a single camera. However, when a user is trying to record video of a moving object (e.g., a person playing soccer) and has already adjusted the camera zoom so that the object is the desired size in the frame, the object's size ratio (the object's size relative to the frame, also known as the object size-to-frame ratio) changes dynamically as the object moves. Maintaining the desired object size in the frame sequence (the object's size in the original frame when video acquisition first begins) can be difficult as the object moves for one or more cameras capturing a sequence of frames. For example, manually changing the object size-to-frame ratio during video acquisition can be cumbersome for the user. Tracking a subject during video recording (e.g., automatically tracking) can also be difficult.

[0037] Systems, apparatus, processes (also called methods), and computer-readable media (collectively referred to as "systems and techniques") for maintaining a constant size of a target object in a sequence of frames (referred to as "target size fixing function"). A sequence of frames may be a video, a group of sequentially captured images, or other sequences of frames. For example, the systems and techniques described herein can determine a region of interest in the first frame (or initial frame). In some cases, the user can select the first frame. For example, in some examples, the user can select any frame from the video as the starting point. In some examples, the systems and techniques can determine the region of interest based on a user selection of the region of interest or on objects within the region of interest. In some cases, the user selection may be based on user input provided using a user interface (e.g., a device touchscreen, an electronic drawing tool, a gesture-based user interface, a voice input-based user interface, or other user interfaces). In some examples, the systems and techniques can determine the region of interest automatically based on object detection and / or recognition techniques. For example, the system and techniques can detect and / or recognize a person in the frame and define a region of interest around that person.

[0038] In some cases, the system and techniques can determine the size of an object and / or region of interest in the first (or initial) frame when the region of interest is determined (e.g., when user input is provided to identify an object or a region of interest containing an object). In some cases, the user can provide input to define a desired size for the object or region of interest (e.g., zoom by providing a pinch input), or keep the size of the object the same as it was in the first / initial frame. In some cases, the user can provide input to cause the device to adjust the size of the region of interest and / or the object to define a preferred size for the object in a sequence of frames. When the region of interest is determined (e.g., at the time of user selection of an object), the system and techniques can crop and scale (e.g., upsample) one or more subsequent frames (after the first or initial frame) in a sequence of frames to maintain the size of the object in each subsequent frame to match the size of the object in the first frame. In some cases, systems and techniques can perform cropping and scaling so that a selected object is maintained at the same size as it was in the first frame, and so that it is maintained at a specific location in each frame (e.g., the center of each frame, the location in the frame where the object was in the first frame, or another location). In some examples, systems and techniques can utilize object detection and tracking techniques to ensure that the object's location and / or size does not change throughout the sequence of frames.

[0039] In some examples, the system and technique may apply one or more smoothing functions to an object or a bounding box (or other type of bounding region) related to the object or the region of interest containing the object. The one or more smoothing functions may be such that cropping and scaling are performed gradually to minimize the movement and size changes of the object between frames in a sequence of frames. The application of smoothing functions can prevent the object from appearing to move unnaturally (e.g., bouncing) in a sequence of frames due to cropping and scaling performed to maintain the object at a specific size and / or position in each frame. In some implementations, the smoothing functions may account for displacement (movement within a frame) and / or changes in the size of the bounding box (the object's size changing independently of its center point). In some cases, the displacement may be relative to a point on the object (e.g., the center point) or to a point within the bounding box related to the region of interest containing the object (e.g., the center point). In some cases, changes in the size of the bounding box may include changes relative to distances related to the object (e.g., the distance between a first part of the object and a second part of the object) or to distances related to the bounding box corresponding to the region of interest containing the object (e.g., diagonal distances of the bounding boxes).

[0040] In some cases, the system and techniques may be applied for video playback. In other cases, the system and techniques may be applied for other use cases. For example, the system and techniques can produce video results in which the size of a target object is stable (e.g., constant or nearly constant, so that the user watching the video does not perceive any change in size) at a specific point in the frame of the video sequence (e.g., the center point). Multiple video resources may be supported.

[0041] In some cases, a device can implement one or more dual-camera mode features. For example, a dual-camera mode feature can be implemented using two camera lenses of a device simultaneously, such as a primary camera lens (e.g., a telephoto lens) and a secondary camera lens (e.g., a zoom lens such as a wide-angle lens). An example of a dual-camera mode feature is a "dual-camera video recording" feature, where the two camera lenses record two videos simultaneously. The two videos can then be displayed, stored, sent to another device, and / or used in other ways. Using a dual-camera mode feature (e.g., dual-camera video recording), a device can display two viewpoints of a scene (e.g., split-screen video) on a display at the same time. The advantages of a dual-camera mode feature may include, among other things, allowing the device to capture a wide-angle view of a scene (e.g., with more background and surrounding objects in the scene), and allowing the device to capture a large event or a complete view of a scene.

[0042] In video (or another sequence of frames or images) captured using a single camera, various problems can arise regarding maintaining a constant size of the target object within the sequence of frames. For example, when the target object moves toward the device's camera, the device may not be able to perform a zoom-out effect due to the limits of the field of view from the original video frame. In another example, when the target object moves away from the device's camera, the zoomed-in image generated based on the original video frame may be blurry, contain one or more visual artifacts, and / or lack clarity. Devices implementing dual-camera mode functionality do not incorporate any artificial intelligence technology. Such systems require the end user to use video editing tools or software applications to manually edit the images.

[0043] Systems and techniques for switching lenses or cameras in devices capable of implementing one or more of the dual-camera mode functions described above are also described herein. For example, systems and techniques can be used in a dual-camera system to maintain a constant size of a target object in a sequence of video frames from the dual-camera system by implementing a camera lens switching algorithm. In some cases, systems and techniques can perform dual-camera zoom. In some cases, systems and techniques can provide a more detailed object zoom effect. In some examples, systems and techniques can be applied to systems or devices having two or more cameras used to capture video or other sequences of frames.

[0044] Using such systems and techniques, a video can be generated or recorded in which the size of a target object is stable (e.g., constant or nearly constant, so that the user viewing the video does not perceive any change in size) at a specific point (e.g., the center point) within the frame of the video sequence. This zoom-based system and technique can be applied to real-time video recording, still image (e.g., photograph) capture, and / or other use cases. In some cases, the user can select the object of interest, or the system can automatically determine the prominent object (object of interest). Support for multi-camera systems is also provided, as described above.

[0045] The techniques described herein can be applied by any type of image acquisition device, such as a mobile device containing one or more cameras, an IP camera, a digital camera, and / or other image acquisition devices. The systems and techniques can be applied to any type of content, such as pre-recorded video content, raw video content (e.g., unrecorded video), or other content, including sequences of frames or images.

[0046] Various aspects of the systems and techniques described herein are discussed below with reference to the drawings. Figure 1 is a block diagram showing the architecture of the image acquisition and processing system 100. The image acquisition and processing system 100 includes various components used to acquire and process images in a sequence (for example, images of scene 110). The image acquisition and processing system 100 can acquire standalone images (or photographs) and / or video containing multiple images (or video frames) in a particular sequence. The lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0047] One or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or from the image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components. For example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may also include additional control mechanisms other than those shown, such as control mechanisms for analog gain, flash, HDR, depth of field, and / or other image capture characteristics.

[0048] The focus control mechanism 125B of the control mechanism 120 can acquire the focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B can adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B can adjust the focus by moving the lens 115 closer to or further away from the image sensor 130 by acting a motor or servo. In some cases, the system 100 may include additional lenses, such as one or more microlenses on each photodiode of the image sensor 130, each of which bends the light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting can be determined via contrast-detection autofocus (CDAF), phase-detection autofocus (PDAF), or any combination thereof. The focus setting can be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. Focus settings may also be called image capture settings and / or image processing settings.

[0049] The exposure control mechanism 125A of the control mechanism 120 can acquire the exposure setting. In some cases, the exposure control mechanism 125A stores the exposure setting in a memory register. Based on this exposure setting, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or f / stop), the length of time the aperture is open (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure setting may be called the image acquisition setting and / or image processing setting.

[0050] The zoom control mechanism 125C of the control mechanism 120 can acquire the zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C can control the focal length of the lens assembly, which includes lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by acting one or more motors or servos to move one or more of the lenses relative to each other. The zoom setting may be called the image acquisition setting and / or image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable-focus zoom lens. In some examples, the lens assembly may include a focusing lens (which may be lens 115) that initially receives light from the scene 110, and the light then passes through an infinite-focus zoom system between the focusing lens (e.g., lens 115) and the image sensor 130 before the light reaches the image sensor 130. In some cases, an infinity zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference) and a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control mechanism 125C moves one or more of the lenses in the infinity zoom system, such as the negative lens and one or both of the positive lenses.

[0051] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image produced by the image sensor 130. In some cases, different photodiodes may be covered by different color filters of a color filter array, and thus may measure light that matches the color of the color filter covering the photodiode. Various color filter arrays can be used, including Bayer color filter arrays, quad color filter arrays (also called quad Bayer filters), and / or other color filter arrays. For example, a Bayer color filter array includes a red color filter, a blue color filter, and a green color filter, and each pixel in the image is produced based on red light data from at least one photodiode covered by the red color filter, blue light data from at least one photodiode covered by the blue color filter, and green light data from at least one photodiode covered by the green color filter. Other types of color filter arrays may use yellow, magenta, and / or blue-green (also called "emerald") color filters instead of, or in addition to, red, blue, and / or green color filters. Some image sensors may lack color filters entirely and instead use different photodiodes (sometimes stacked vertically) across the entire pixel array. Different photodiodes across the entire pixel array may have different spectral sensitivity curves and thus correspond to different wavelengths of light. Monochrome image sensors may also lack color filters and therefore lack color depth.

[0052] In some cases, the image sensor 130 may, as an alternative or addition, include an opaque and / or reflective mask that prevents light from reaching some or some of the photodiodes at a given time and / or from a given angle, which may be used for phase-detection autofocus (PDAF). The image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output by the photodiodes and / or analog-to-digital converter (ADC) and converting the analog signal output of the photodiodes (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, instead or in addition, some components or functions discussed with respect to one or more of the control mechanism 120 may be included in the image sensor 130. The image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplier CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide-semiconductor (CMOS) sensor, an N-type metal-oxide-semiconductor (NMOS) sensor, a hybrid CCD / CMOS sensor (e.g., sCMOS), or any other combination thereof.

[0053] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 4710 discussed in relation to the computing system 4700. The host processor 152 may be a digital signal processor (DSP) and / or other types of processors. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 / 4720, read-only memory (ROM) 145 / 4725, cache 4712, system memory 4715, another storage device 4730, or any combination thereof.

[0054] In some implementations, the image processor 150 is a single integrated circuit or chip (referred to as a system-on-a-chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth®, Global Positioning System (GPS), etc.), any combination thereof, and / or other components. I / O port 156 may include any suitable input / output port or interface in accordance with one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a Serial General Purpose Input / Output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as MIPI CSI-2), a physical (PHY) layer port or interface, an Advanced High-performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an example for one explanation, the host processor 152 can communicate with the image sensor 130 using the I2C port, and the ISP 154 can communicate with the image sensor 130 using the MIPI port.

[0055] The host processor 152 of the image processor 150 can configure the image sensor 130 using parameter settings (for example, via an external control interface such as I2C, I3C, SPI, GPIO, and / or other interfaces). In an example for one explanation, the host processor 152 can update the exposure settings used by the image sensor 130 based on the results of internal processing of the exposure control algorithm from past image frames. The host processor 152 can also dynamically configure the parameter settings of the internal pipeline or modules of the ISP 154 to match the settings of one or more input image frames from the image sensor 130, so that the image data is processed correctly by the ISP 154. The processing (or pipeline) blocks or modules of the ISP 154 may include, among other things, modules for lens / sensor noise correction, demosaicing, color conversion, correction or enhancement / suppression of image attributes, noise reduction filters, and sharpening filters. For example, a processing block or module of ISP154 can perform several tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging of image frames to form an HDR image, image recognition, object recognition, feature recognition, input reception, output management, memory management, or any combination thereof. The configuration of different modules of ISP154 can be configured by the host processor 152.

[0056] The image processing device 105B may include various input / output (I / O) devices 160 connected to the image processor 150. The I / O devices 160 may include a display screen, keyboard, keypad, touchscreen, trackpad, touch-sensitive surface, printer, any other output device 4735, any other input device 4745, or any combination thereof. In some cases, captions may be input to the image processing device 105B through the physical keyboard or keypad of the I / O device 160, or through the virtual keyboard or keypad of the touchscreen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that enable wired connections between the system 100 and one or more peripheral devices, through which the system 100 may receive data from and / or send data to one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that enable wireless connectivity between the system 100 and one or more peripheral devices, through which the system 100 may receive data from and / or transmit data to one or more peripheral devices. The peripheral devices may include any of the previously discussed types of I / O devices 160, and when coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector, they themselves may be considered I / O devices 160.

[0057] In some cases, the image acquisition and processing system 100 may be a single device. In some cases, the image acquisition and processing system 100 may be two or more separate devices, including an image acquisition device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image acquisition device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly via one or more wireless transceivers. In some implementations, the image acquisition device 105A and the image processing device 105B may be disconnected from each other.

[0058] As shown in Figure 1, the vertical dashed line divides the image acquisition and processing system 100 in Figure 1 into two parts, representing the image acquisition device 105A and the image processing device 105B, respectively. The image acquisition device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, some components shown in the image acquisition device 105A, such as the ISP 154 and / or the host processor 152, may be included in the image acquisition device 105A.

[0059] The image acquisition and processing system 100 may include, or be part of, an electronic device such as a mobile or fixed telephone handset (e.g., a smartphone, mobile phone), an Internet Protocol (IP) camera, a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, or any other suitable electronic device. In some examples, the image acquisition and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or any combination thereof. In some implementations, the image acquisition device 105A and the image processing device 105B may be different devices. For example, the image acquisition device 105A may include a camera device, and the image processing device 105B may include a computing device such as a mobile handset, a desktop computer, or other computing device.

[0060] While the image acquisition and processing system 100 is shown as comprising several components, those skilled in the art will understand that the image acquisition and processing system 100 may comprise more components than those shown in Figure 1. The components of the image acquisition and processing system 100 may comprise one or more combinations of software, hardware, or software and hardware. For example, in some implementations, the components of the image acquisition and processing system 100 may comprise one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), electronic circuits, or other electronic hardware, and / or may be implemented using them, and / or may comprise computer software, firmware, or any combination thereof, to perform the various operations described herein, and / or may be implemented using them. The software and / or firmware may comprise one or more instructions, stored on a computer-readable storage medium and executable by one or more processors of the electronic devices implementing the image acquisition and processing system 100.

[0061] In some examples, the image acquisition and processing system 100 may be implemented as part of a system that can be used to perform object detection and / or tracking of objects from video frames. An example of such a system is a video analysis system. Object detection and tracking are important components in a wide range of computer vision applications, including, among others, surveillance cameras and human-computer interaction. Assuming an initial state (e.g., location and size) of a target object (or object of interest) in a video frame, the goal of tracking is to estimate the state of the target in subsequent frames. An object detection and tracking system (e.g., a video analysis system) has the ability to output patches (e.g., bounding boxes) as detection and tracking results for each frame of video. Based on those patches, blob or object classification techniques (e.g., neural network-based classification) may be applied to determine whether an object should be classified as a certain type of object (e.g., a car or a person). One task of object detection, recognition, and tracking is to analyze the movement and behavior of objects in video. The advantage of such tasks is that the video analysis system may be able to obtain high-resolution (e.g., 1080p, 4K, or 8K) video frames to gain further details about the object being tracked.

[0062] Generally, a video analysis system can acquire a sequence of video frames from a video source and process the video sequence to perform various tasks. Examples of video sources may include IP cameras or other video capture devices. IP cameras are a type of digital video camera that can be used for surveillance, home security, and / or other suitable applications. Unlike analog closed-circuit television (CCTV) cameras, IP cameras can transmit or receive data over computer networks and the internet. In some cases, one or more IP cameras may be placed in a scene or environment and remain stationary while capturing a video sequence of the scene or environment.

[0063] In some cases, IP camera systems can be used for two-way communication. For example, data (e.g., audio, video, metadata, etc.) can be transmitted by the IP camera using one or more network cables or wireless networks, allowing the user to communicate with what they are seeing. In one example for explanation, a gas station attendant could use video data provided by an IP camera to assist a customer with how to use a pump with a payment machine (e.g., by seeing the customer's behavior at the payment machine). Commands for pan, tilt, and zoom (PTZ) cameras can also be transmitted over a single or multiple networks. Furthermore, IP camera systems offer flexibility and wireless capabilities. For example, IP cameras enable easy network connectivity, adjustable camera positioning, and remote access to services over the internet. IP camera systems also enable distributed intelligence. For example, IP cameras allow video analytics to be placed within the camera itself. IP cameras also easily provide encryption and authentication. For example, IP cameras result in secure data transmission through predefined encryption and authentication methods for IP-based applications. Moreover, IP cameras improve the efficiency of labor costs. For example, video analysis can generate an alert for a particular event, which reduces the labor cost of monitoring all cameras in the system (based on the alert).

[0064] Video analysis offers a wide range of tasks, from the immediate detection of events of interest to analyzing previously recorded video for the purpose of extracting events over a long period, and many other tasks. Various studies and real-world experience have shown that, in surveillance systems, even when monitoring footage from a single camera, a human operator typically cannot maintain vigilance and attention for longer than 20 minutes. When there are two or more cameras to monitor, or when the time exceeds a certain length (e.g., 20 minutes), the operator's ability to monitor the video and respond effectively to events decreases significantly. Video analysis can automatically analyze video sequences from cameras and issue alerts for events of interest. In this way, a human operator can passively monitor one or more scenes. Furthermore, video analysis can analyze large amounts of recorded video and extract specific video segments containing events of interest.

[0065] Video analytics also offers a variety of other features. For example, video analytics can act as an intelligent video motion detector by detecting and tracking moving objects. In some cases, video analytics can generate and display bounding boxes around valid objects. Video analytics can also act as an intrusion detector, video counter (e.g., by counting people, objects, vehicles, etc.), camera tampering detector, object retention detector, object / asset movement detector, asset protector, loitering detector, and / or slip-and-fall detector. Video analytics can further be used to perform various types of recognition functions, such as face detection and recognition, license plate recognition, object recognition (e.g., bags, logos, scars, etc.), or other recognition functions. In some cases, video analytics can be trained to recognize certain objects. Another function that video analytics can perform includes providing statistics on customer metrics (e.g., customer count, gender, age, time spent, and other appropriate measures). Video analytics can also perform video retrieval (e.g., extracting basic activity for a given area) and video summarization (e.g., extracting significant motion). In some cases, event detection may be performed by video analysis, including the detection of fire, smoke, combat, crowd formation, or any other appropriate event that video analysis is programmed to detect or learns to detect. The detector can trigger the detection of an event of interest and can send a warning or alarm to a central control room to alert the user of the event of interest.

[0066] In some cases, as will be described in more detail herein, a video analysis system may generate and detect foreground blobs, which can be used to perform a variety of actions, such as object tracking (also called blob tracking) and / or other actions described above. An object tracker (sometimes also called a blob tracker) may be used to track one or more objects (or blobs representing objects) in a video sequence using one or more bounding regions. A bounding region may include a bounding box, a bounding circle, a bounding ellipse, or any other suitable shape representing an object and / or region of interest. Details of exemplary video analysis systems for blob detection and object tracking are described below with reference to Figures 2 to 5.

[0067] Figure 2 is a block diagram showing an example of a video analysis system 200. The video analysis system 200 receives video frames 202 from a video source 230. Video frames 202 may also be referred to herein as a sequence of frames. Each frame may also be referred to as a video picture or picture. Video frames 202 may be part of one or more video sequences. The video source 230 may include an image acquisition device (e.g., an image acquisition and processing system 100, a video camera, a camera phone, a video phone, or other suitable acquisition device), a video storage device, a video archive containing stored video, a video server or content provider providing video data, a video feed interface receiving video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or other sources of video content. In one example, the video source 230 may include an IP camera or a group of IP cameras. In an example for one explanation, a group of IP cameras may be located throughout the environment and can provide video frames 202 to the video analysis system 200. For example, IP cameras can be positioned in various fields of view within the environment so that surveillance can be performed based on captured video frames 202 of the environment.

[0068] In some embodiments, the video analysis system 200 and the video source 230 may be part of the same computing device. In some embodiments, the video analysis system 200 and the video source 230 may be part of separate computing devices. In some examples, the computing device (or device) may include one or more wireless transceivers for wireless communication. The computing device (or device) may include electronic devices such as cameras (e.g., IP cameras or other video cameras, camera phones, video phones, or other suitable capture devices), mobile or fixed telephone handsets (e.g., smartphones, mobile phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, display devices, digital media players, video gaming consoles, video streaming devices, or any other suitable electronic devices.

[0069] The video analysis system 200 includes a blob detection system 204 and an object tracking system 206. Object detection and tracking enable the video analysis system 200 to provide various end-to-end functions, such as the video analysis functions described above. For example, intelligent motion detection and tracking, intrusion detection, and other functions can generate end-to-end events by directly using the results from object detection and tracking. Other functions, such as counting and classifying people, vehicles, or other objects, can be greatly simplified based on the results of object detection and tracking. The blob detection system 204 can detect one or more blobs in a video frame of a video sequence (e.g., video frame 202), and the object tracking system 206 can track one or more blobs across frames of a video sequence. The object tracking system 206 may be based on any type of object tracking algorithm, such as cost-based tracking or machine learning-based tracking, among other things.

[0070] As used herein, a blob refers to foreground pixels of at least a portion of an object in a video frame (for example, a portion of an object or the entire object). For example, a blob may include a contiguous group of pixels that constitute at least a portion of a foreground object in a video frame. In another example, a blob may refer to a contiguous group of pixels that constitute at least a portion of a background object in a frame of image data. A blob may also be called an object, a portion of an object, a pixel blotch, a pixel patch, a pixel cluster, a pixel blot, a pixel spot, a pixel cluster, or any other term that refers to a group or portion of pixels of an object. In some examples, a bounding region may be associated with a blob. In some examples, a tracker may also be represented by a tracker bounding region. The bounding region of a blob or tracker may include a bounding box, a bounding circle, a bounding ellipse, or any other suitable shape of region that represents the tracker and / or blob. While examples of the use of bounding boxes are described herein for illustrative purposes, the techniques and systems described herein may also be adapted to use bounding areas of other suitable shapes. Bounding boxes associated with trackers and / or blobs may have rectangular, square, or other suitable shapes. In tracking layers, the terms blob and bounding box may be used interchangeably when it is not necessary to know how the blobs are organized within the bounding boxes.

[0071] As will be explained in more detail below, blobs can be tracked using blob trackers. Blob trackers may be associated with a tracker bounding box and may be assigned a tracker identifier (ID). In some examples, the bounding box for a blob tracker in the current frame may be the bounding box of a previous blob in a previous frame to which the blob tracker was associated. For example, when a blob tracker is updated in a previous frame (after being associated with a previous blob in a previous frame), the updated information for the blob tracker may include tracking information from the previous frame and a prediction of the blob tracker's position in the next frame (which in this example is the current frame). The prediction of the blob tracker's position in the current frame may be based on the blob's position in the previous frame. As will be explained in more detail below, a history and motion model may be maintained for the blob tracker, including a history of various states, velocity history, and position history for consecutive frames.

[0072] In some examples, a motion model for a blob tracker can determine and maintain two positions for the blob tracker for each frame. For example, a first position for the blob tracker for the current frame can include a predicted position in the current frame. The first position is referred to as the predicted position herein. The predicted position of the blob tracker in the current frame includes the position of the blob associated with the blob tracker in the previous frame. Thus, the position of the blob associated with the blob tracker in the previous frame can be used as the predicted position of the blob tracker in the current frame. A second position for the blob tracker for the current frame can include the position of the blob in the current frame that the tracker is associated with in the current frame. The second position is referred to as the actual position herein. Thus, the position of the blob in the current frame that is associated with the blob tracker is used as the actual position of the blob tracker in the current frame. The actual position of the blob tracker in the current frame can be used as the predicted position of the blob tracker in the next frame. The position of the blob can include the position of the bounding box of the blob.

[0073] The speed of the blob tracker can include the displacement of the blob tracker between consecutive frames. For example, the displacement between the centers (or centroids) of two bounding boxes for the blob tracker in two consecutive frames can be determined. In one example for illustration, the speed of the blob tracker is defined as V t =C t -C t-1 where C t -C t-1 =(C tx -C t-1x ,C ty -C t-1y ). The term C t (C tx ,C ty ) indicates the center position of the bounding box of the tracker in the current frame, C tx is the x coordinate of the bounding box, and Cty This is the y-coordinate of the bounding box. Term C t-1 (C t-1x ,C t-1y ) represents the center position (x and y) of the tracker's bounding box in the previous frame. In some implementations, it is also possible to use four parameters to estimate x, y, width, and height simultaneously. In some cases, the timing of video frame data is constant, or at least does not vary significantly over time (following the frame rate, such as 30 frames per second, 60 frames per second, 120 frames per second, or other appropriate frame rate), so a time variable may not be necessary in velocity calculations. In some cases, a time constant can be used (according to the instantaneous frame rate), and / or a timestamp can be used.

[0074] Using the blob detection system 204 and the object tracking system 206, the video analysis system 200 can perform blob generation and detection for each frame or picture in a video sequence. For example, the blob detection system 204 can perform background subtraction on a frame and then detect foreground pixels in the frame. Foreground blobs are generated from foreground pixels using morphological operations and spatial analysis. Furthermore, blob trackers from previous frames must be associated with the foreground blob in the current frame and also updated. Both the association of the tracker data with the blob and the tracker update may depend on cost function calculation. For example, when a blob is detected from the current input video frame, blob trackers from previous frames may be associated with the detected blob according to the cost calculation. The tracker is then updated according to the data association, which includes updating the tracker's state and position so that it can perform object tracking in the current frame. Further details regarding the blob detection system 204 and the object tracking system 206 are described with reference to Figures 4 and 5.

[0075] Figure 3 shows an example of a video analysis system (e.g., video analysis system 200) that processes video frames over time t. As shown in Figure 3, video frame A302A is received by blob detection system 304A. Blob detection system 304A generates a foreground blob 308A for the current frame A302A. After blob detection is performed, the foreground blob 308A may be used for temporary tracking by object tracking system 306A. The cost between the blob tracker and the blob (e.g., cost including distance, weighted distance, or other costs) may be calculated by object tracking system 306A. Using the calculated cost (e.g., using a cost matrix or other appropriate association technique), object tracking system 306A can perform data association to associate or match blob 308A with block trackers (e.g., blob trackers generated or updated based on previous frames or newly generated blob trackers). The blob tracker may be updated according to data associations, including information about the tracker's location, to generate an updated blob tracker 310A. For example, the state and location of the blob tracker for video frame A302A may be calculated and updated. The location of the blob tracker in the next video frame N302N may also be predicted from the current video frame A302A. For example, the predicted location of the blob tracker for the next video frame N302N may include the location of the blob tracker (and its associated blob) in the current video frame A302A. Tracking of the blob in the current frame A302A may be performed once the updated blob tracker 310A is generated.

[0076] When the next video frame N302N is received, the blob detection system 304N generates a foreground blob 308N for frame N302N. The object tracking system 306N can then perform a temporary tracking of the blob 308N. For example, the object tracking system 306N obtains an updated blob tracker 310A based on the previous video frame A302A. The object tracking system 306N can then calculate the cost and use the newly calculated cost to associate the blob tracker 310A with the blob 308N. The blob tracker 310A can be updated according to the data association and an updated blob tracker 310N can be generated.

[0077] Figure 4 is a block diagram showing an example of a blob detection system 204. Blob detection is used to separate moving objects from the global background in a scene. The blob detection system 204 includes a background subtraction engine 412 that receives video frames 402. The background subtraction engine 412 can perform background subtraction to detect foreground pixels in one or more video frames 402. For example, background subtraction may be used to separate moving objects from the global background in a video sequence and to generate a foreground-background binary mask (referred to herein as a foreground mask). In some examples, background subtraction can perform a subtraction of the current frame or picture with a background model that includes the background portion of the scene (e.g., a static or nearly static portion of the scene). Based on the results of background subtraction, the morphology engine 414 and the concatenated component analysis engine 416 can perform foreground pixel processing to group foreground pixels into foreground blobs for tracking purposes. For example, after background subtraction, morphological operations may be applied to remove noisy pixels and to smooth the foreground mask. Next, concatenated component analysis may be applied to generate blobs. Then, blob processing can be performed, which may include further removing some blobs, integrating some blobs together, and providing a bounding box as input for tracking.

[0078] The background subtraction engine 412 can model the background of a scene (for example, captured in a video sequence) using any suitable background subtraction technique (also called background extraction). One example of a background subtraction method used by the background subtraction engine 412 involves modeling the background of a scene as a statistical model based on relatively static pixels in previous frames that are not considered to belong to any moving region. For example, the background subtraction engine 412 can use a Gaussian distribution model for each pixel position, with parameters called mean and variance to model each pixel position in a frame of a video sequence. All the values ​​of previous pixels at a particular pixel position are used to calculate the mean and variance of the target Gaussian model for that pixel position. When a pixel at a given position in a new video frame is processed, its value is evaluated by the current Gaussian distribution for this pixel position. Classifying a pixel as either a foreground pixel or a background pixel is done by comparing the difference between the pixel value and the mean of a given Gaussian model. In an example for explanation, a pixel is classified as a background pixel if the distance between the pixel value and the Gaussian mean is less than three times the variance. Otherwise, in the example for this explanation, the pixel is classified as a foreground pixel. Simultaneously, the Gaussian model for pixel locations is updated by taking the current pixel value into account.

[0079] The background subtraction engine 412 can also perform background subtraction using Gaussian mixtures (also known as Gaussian mixture models (GMMs)). A GMM models each pixel as a Gaussian mixture and updates the model using an online learning algorithm. Each Gaussian model is represented by its mean, standard deviation (or covariance matrix if the pixel has multiple channels), and weights. The weights represent the probability that the Gaussian occurred in the past history.

[0080]

number

[0081] The formula for the GMM model is given in equation (1), where there are K Gaussian models. Each Gaussian model has a distribution with mean μ and variance Σ, and weights ω, where i is the index of the Gaussian model and t is the time instance. As shown by the formula, the parameters of the GMM change with time after one frame (at time t) has been processed. In GMM or any other learning-based background subtraction, the current pixel influences the entire model of pixel positions based on the learning rate, which may be constant or typically the same for at least each pixel position. Background subtraction methods based on GMM (or other learning-based background subtraction) adapt to local changes for each pixel. Thus, when a moving object comes to a stop, for each pixel position of the object, the same pixel value continues to contribute significantly to its associated background model, and the region associated with the object becomes the background.

[0082] The background subtraction techniques mentioned above are based on the assumption that the camera is fixed and mounted, and a new background model needs to be computed whenever the camera moves or changes orientation. There are also background subtraction methods that can handle foreground subtraction based on a moving background, including techniques such as keypoint tracking, optical flow, saliency, and other motion estimation-based methods.

[0083] The background subtraction engine 412 can generate a foreground mask with foreground pixels based on the background subtraction results. For example, the foreground mask may include a binary image containing pixels that make up foreground objects in the scene (e.g., moving objects) and background pixels. In some examples, the background (background pixels) of the foreground mask may be a solid color, such as a solid white background, a solid black background, or another solid color. In such examples, the foreground pixels of the foreground mask may be a different color from those used for the background pixels, such as a solid black, a solid white, or another solid color. In an example for one explanation, the background pixels may be black (e.g., the color value of the pixels in 8-bit grayscale is 0, or another appropriate value), and the foreground pixels may be white (e.g., the color value of the pixels in 8-bit grayscale is 255, or another appropriate value). In an example for another explanation, the background pixels may be white, and the foreground pixels may be black.

[0084] Using a foreground mask generated from background subtraction, the morphology engine 414 can perform morphological functions to filter foreground pixels. Morphological functions may include shrinking and dilating functions. For example, a shrinking function may be applied, followed by a series of one or more dilating functions. A shrinking function may be applied to remove pixels on object boundaries. For example, the morphology engine 414 may apply a shrinking function (e.g., FilterErode3x3) to a 3x3 filter window of the central pixel currently being processed. The 3x3 window may be applied to each foreground pixel (as a central pixel) in the foreground mask. Those skilled in the art will understand that other window sizes other than 3x3 may be used. A shrinking function may include a shrinking operation that sets the current foreground pixel (acting as a central pixel) in the foreground mask as a background pixel if one or more adjacent pixels in the 3x3 window are background pixels. Such a shrinking operation may be called a strong shrinking operation or a single-neighbor shrinking operation. Here, the neighboring pixels of the current central pixel include 8 pixels within a 3x3 window, and the 9th pixel is the current central pixel.

[0085] Dilatation operations can be used to enhance the boundaries of foreground objects. For example, the morphology engine 414 may apply a dilatation function (e.g., FilterDilate3x3) to a 3x3 filter window of central pixels. A 3x3 dilatation window may be applied to each background pixel (as a central pixel) in the foreground mask. Those skilled in the art will understand that other window sizes other than 3x3 may be used. The dilatation function may include a dilatation operation that sets the current background pixel (acting as a central pixel) in the foreground mask as a foreground pixel if one or more adjacent pixels in the 3x3 window are foreground pixels. The neighboring pixels of the current central pixel include eight pixels in the 3x3 window, with the ninth pixel being the current central pixel. In some examples, multiple dilatation operations may be applied after a deflation function has been applied. In an example for one explanation, three function calls of a 3x3 window size dilatation may be applied to the foreground mask before the foreground mask is sent to the connected component analysis engine 416. In some examples, a condensation function can be applied first to remove noisy pixels, and then a series of dilation functions can be applied to improve the foreground pixels. In one example for explanation, one condensation function with a window size of 3x3 is called first, and then three function calls of dilation with a window size of 3x3 are applied to the foreground mask before the foreground mask is sent to the connected component analysis engine 416. Further details on content adaptive morphology operations are described below.

[0086] After the morphological operations are performed, the connected component analysis engine 416 can apply connected component analysis to concatenate neighboring foreground pixels and organize the connected components and blobs. In some implementations of connected component analysis, a set of bounding boxes is returned such that each bounding box contains one component of the concatenated pixels. An example of connected component analysis performed by the connected component analysis engine 416 is carried out as follows:

[0087] For each pixel of the foreground mask { - If it is a foreground pixel and has not been processed, the following steps apply: - Apply the FloodFill function to concatenate this pixel with other foreground elements to generate a concatenated component. - Insert this connected component into the list of connected components. - Marks pixels in the connected component as being processed.

[0088] The Floodfill (seedfill) function is an algorithm that determines the area to be concatenated to a seed node in a multidimensional array (for example, a 2-D image in this case). The Floodfill function first obtains the color or intensity value at the seed location (e.g., foreground pixel) in the source foreground mask, and then finds all neighboring pixels with the same (or similar) value based on 4-concatenation or 8-concatenation. For example, in the case of 4-concatenation, the neighbors of the current pixel are defined as pixels with coordinates (x+d,y) or (x,y+d), where d is equal to 1 or -1, and (x,y) is the current pixel. Those skilled in the art will understand that other numbers of concatenation can be used. Some objects are separated into different concatenation components, while others are grouped into the same concatenation component (e.g., neighboring pixels with the same or similar values). Additional processing may be applied to further process the concatenation components for grouping. Finally, a blob 408 is generated containing the neighboring foreground pixels according to the concatenation components. In one example, the blob may consist of one concatenation component. In another example, a blob can contain multiple linked components (for example, when two or more blobs are merged together).

[0089] The blob processing engine 418 can perform additional processing to further process the blobs generated by the concatenated component analysis engine 416. In some cases, the blob processing engine 418 can generate bounding boxes to represent the detected blobs and blob trackers. In some cases, the blob bounding boxes may be output from the blob detection system 204. In some cases, there may be a filtering process on the concatenated components (bounding boxes). For example, the blob processing engine 418 can perform content-based filtering of some blobs. In some cases, a machine learning method can determine that the current blob contains noise (e.g., leaves in the scene). Using the machine learning information, the blob processing engine 418 can determine that the current blob is a noisy blob and remove it from the resulting blobs provided to the object tracking system 206. In some cases, the blob processing engine 418 can remove one or more small blobs that are below a certain size threshold (e.g., the area of ​​the bounding box surrounding the blob is less than the area threshold). In some examples, there may be a consolidation process to merge some connected components (represented as bounding boxes) into a larger bounding box. For example, the blob processing engine 418 may consolidate nearby blobs into one larger blob to eliminate the risk of too many small blobs potentially belonging to a single object. In some cases, two or more bounding boxes may be consolidated together based on some rule, even when the foreground pixels of the two bounding boxes are not connected at all. In some embodiments, the blob detection system 204 does not include the blob processing engine 418, or in some cases, does not use the blob processing engine 418. For example, the blobs generated by the connected component analysis engine 416 may be input to the object tracking system 206 to perform blob and / or object tracking without further processing.

[0090] In some implementations, density-based blob area trimming may be performed by the blob processing engine 418. For example, density-based blob area trimming may be applied if all blobs are organized after post-filtering and before the blobs are fed into the tracking layer. A similar process is applied vertically and horizontally. For example, density-based blob area trimming may be performed first vertically and then horizontally, or vice versa. The purpose of density-based blob area trimming is to remove columns (in vertical processing) and / or rows (in horizontal processing) of the bounding box when a column or row contains only a small number of foreground pixels.

[0091] Vertical processing involves calculating the number of foreground pixels in each column of the bounding box and expressing the number of foreground pixels as a column density. Columns are then processed one by one, starting from the leftmost column. The column density of each current column (the column currently being processed) is compared to the maximum column density (the column density of all columns combined). If the column density of the current column is less than a threshold (e.g., a percentage of the maximum column density, such as 10%, 20%, 30%, 50%, or another appropriate percentage), that column is removed from the bounding box, and the next column is processed. However, if the current column has a column density that is not less than the threshold, such processing ends, and the remaining columns are not processed further. A similar process can then be applied starting from the rightmost column. Those skilled in the art will understand that vertical processing can begin with a different column than the leftmost column, such as the rightmost column or any other appropriate column in the bounding box.

[0092] Horizontal density-based blob area trimming is similar to vertical trimming, except that rows, rather than columns, of the bounding box are processed. For example, the number of foreground pixels in each row of the bounding box is calculated and denoted as the row density. Rows are then processed one by one, starting from the top row. For each current row (the row currently being processed), the row density is compared to the maximum row density (the row density of all rows). If the row density of the current row is less than a threshold (e.g., a percentage of the maximum row density, such as 10%, 20%, 30%, 50%, or another appropriate percentage), the row is removed from the bounding box and the next row is processed. However, if the current row has a row density that is not less than the threshold, such processing ends, and the remaining rows are not processed further. A similar process can then be applied starting from the bottom row. Those skilled in the art will understand that horizontal processing can start processing rows from a different row than the top row, such as the bottom row or any other appropriate row in the bounding box.

[0093] One purpose of density-based blob area trimming is shadow removal. For example, density-based blob area trimming may be applied when a person is detected in a single blob (bounding box) along with a long, thin shadow of that person. Such shadow areas can be removed after applying density-based blob area trimming because the column density of the shadow area is relatively low. Unlike morphology, which changes the thickness of the blob but generally preserves the shape of the bounding box (apart from filtering out some isolated foreground pixels by the blob's arrangement), such density-based blob area trimming methods can significantly alter the shape of the bounding box.

[0094] Once a blob is detected and processed, object tracking (also known as blob tracking) may be performed to track the detected blob. In some examples, tracking may be performed using cost-based techniques, as illustrated with respect to Figure 5. In some examples, tracking may be performed using one or more machine learning systems (for example, one or more neural network-based systems), as further explained below.

[0095] Figure 5 is a block diagram showing an example of an object tracking system 206. The input to blob / object tracking is a list of blobs 508 (e.g., bounding boxes of blobs) generated by the blob detection system 204. In some cases, trackers are assigned unique IDs and a history of bounding boxes is maintained. Object tracking in video sequences can be used for many applications, including surveillance, among a variety of other uses. For example, the ability to detect and track multiple objects in the same scene is of great interest in many security applications. When a blob (which constitutes at least part of an object) is detected from an input video frame, blob trackers from previous video frames must be associated with the blob in the input video frame according to cost calculations. Blob trackers may be updated based on the associated foreground blobs. In some cases, the object tracking steps may be performed in a sequential manner.

[0096] The cost determination engine 512 of the object tracking system 206 can obtain the blob 508 of the current video frame from the blob detection system 204. The cost determination engine 512 can also obtain the updated blob tracker 510A from a previous video frame (e.g., video frame A302A). A cost function can then be used to calculate the cost between the blob tracker 510A and the blob 508. Any suitable cost function can be used to calculate the cost. In some examples, the cost determination engine 512 can measure the cost between the blob tracker and the blob by calculating the Euclidean distance between the centroid of the tracker (e.g., the bounding box of the tracker) and the centroid of the bounding box of the foreground blob. In an example for one explanation using a 2-D video sequence, this type of cost function is calculated as follows:

[0097]

number

[0098] term(t x ,t y ) and (b x ,b y) are the center positions of the blob tracker and the blob bounding box, respectively. As described herein, in some examples, the bounding box of the blob tracker may be the bounding box of the blob related to the blob tracker in the previous frame. In some examples, other cost function techniques may be employed, which calculate the cost using the minimum distance in the x or y direction. Such techniques can be good for some controlled scenarios, such as transportation on well-aligned lanes. In some examples, the cost function may also be based on the distance between the blob tracker and the blob, in which case the boundaries of their bounding boxes are considered rather than using the center positions of the blob's bounding box and the tracker's bounding box to calculate the distance, so that a negative distance is introduced when the two bounding boxes geometrically overlap. In addition, such distance values ​​are further adjusted according to the size ratio of the two related bounding boxes. For example, the cost can be weighted based on the ratio of the area of ​​the blob tracker bounding box to the area of ​​the blob bounding box (e.g., by multiplying the determined distance by the ratio).

[0099] In some embodiments, a cost is determined for each tracker-blob pair between each tracker and each blob. For example, if there are three trackers, including tracker A, tracker B, and tracker C, and three blobs, including blob A, blob B, and blob C, then separate costs can be determined between tracker A and each of blobs A, B, and C, as well as between trackers B and C and each of blobs A, B, and C. In some examples, the costs can be arranged in a cost matrix, which can be used for data association. For example, the cost matrix may be a two-dimensional matrix, where one dimension is blob tracker 510A and the second dimension is blob 508. Each tracker-blob pair or combination between tracker 510A and blob 508 contains a cost included in the cost matrix. The best match between tracker 510A and blob 508 can be determined by identifying the tracker-blob pair with the lowest cost in the matrix. For example, the lowest cost between tracker A and blobs A, B, and C is used to determine which blob tracker A should be associated with.

[0100] The association of data between tracker 510A and blob 508, as well as the updating of tracker 510A, can be based on the determined cost. The data association engine 514 associates or assigns trackers (or tracker bounding boxes) to their corresponding blobs (or blob bounding boxes), and vice versa. For example, as previously described, the tracker-blob pair with the lowest cost may be used by the data association engine 514 to associate blob tracker 510A with blob 508. Another technique for associating blob trackers with blobs involves the Hungarian algorithm, a combinatorial optimization algorithm that solves such assignment problems in polynomial time and foreshadows the later primal-dual algorithm. For example, the Hungarian algorithm can optimize the global cost with blob 508 across all blob trackers 510A to minimize the global cost. The blob tracker-blob combination in the cost matrix that minimizes the global cost is determined and can be used as the association.

[0101] In addition to the Hungarian method, other robust methods may be used to perform data association between blobs and blob trackers. For example, the association problem can be solved while linking as many trackers and blobs as possible, along with additional constraints to make the solution more robust to noise. Regardless of the association technique used, the data association engine 514 may depend on the distance between the blobs and trackers.

[0102] Once the association between blob tracker 510A and blob 508 is complete, the blob tracker update engine 516 can update the status (or state) of tracker 510A for the current frame using the information of the associated blob, as well as the temporary status of the tracker. After updating tracker 510A, the blob tracker update engine 516 can perform object tracking using the updated tracker 510N and can also provide the updated tracker 510N for use when processing the next frame.

[0103] The status or state of a blob tracker may include the identified (or actual) location of the tracker in the current frame and the predicted location of the tracker in the next frame. The location of the foreground blob is identified by the blob detection system 204. However, as will be explained in more detail below, the location of the blob tracker in the current frame may need to be predicted based on information from the previous frame (for example, using the location of the blob associated with the blob tracker in the previous frame). After the data association has been performed for the current frame, the location of the tracker in the current frame may be identified as the location of the associated blob in the current frame. The tracker's location may further be used to update the tracker's motion model and predict the tracker's location in the next frame. Furthermore, in some cases, there may be a temporarily lost state of trackers (for example, when the blob that the tracker was tracking is no longer detected), in which case the location of such a tracker also needs to be predicted (for example, by a Kalman filter). Such trackers are temporarily not shown to the system. Predicting the location of bounding boxes not only helps maintain a certain level of tracking for lost and / or merged bounding boxes, but also provides a more accurate estimate of the tracker's initial position, thereby allowing for a more precise association between bounding boxes and trackers.

[0104] As mentioned above, the position of a blob tracker in the current frame can be predicted based on information from the previous frame. One way to perform a tracker position update is to use a Kalman filter. A Kalman filter is a framework that involves two operations: the first is to predict the state of the tracker, and the second is to correct or update the state using the measurement results. In this case, the tracker from the last frame (using the blob tracker update engine 516) predicts its position in the current frame, and when the current frame is received, the tracker first corrects its position state using the measurement results of the blob (e.g., blob bounding box), and then predicts its position in the next frame. For example, a blob tracker can use a Kalman filter to measure its trajectory and predict its future position. The Kalman filter relies on the measurement results of the relevant blob to correct the motion model for the blob tracker and predict the position of the object tracker in the next frame. In some examples, if the blob tracker is related to a blob in the current frame, the blob's position is used directly to correct the motion model of the blob tracker in the Kalman filter. In some cases, if the blob tracker is not associated with any blobs in the current frame, the blob tracker's position in the current frame is determined as the predicted position of the tracker from the previous frame, which means that the motion model for the blob tracker is not corrected and the prediction propagates along with the last model of the blob tracker (from the previous frame).

[0105] In addition to the tracker's location, the tracker's state or status may also include, or alternatively, the tracker's transient state or status. A tracker's transient state may include whether the tracker is a new tracker that did not exist before the current frame; a normal state for a tracker that has been valid for a certain length of time and will be output to the video analysis system as a specific tracker-blob pair; a lost state for a tracker that is not associated with or cannot be associated with any foreground blob in the current frame; a dead state for a tracker that has failed to associate with any blob during a certain number of consecutive frames (e.g., two or more frames, a threshold time length, etc.); and / or other appropriate transient statuses. Another transient state that may be maintained for a blob tracker is the tracker's time length. The time length of a blob tracker includes the number of frames (or other temporal measurement such as time) in which the tracker has been associated with one or more blobs.

[0106] There may be other state or status information needed to update the tracker, which may require a state machine for object tracking. Assuming associated blob information and the tracker's own status history table, the status also needs to be updated. The state machine collects all the necessary information and updates the status accordingly. Various statuses of a tracker may be updated. For example, in addition to the tracker's life status (e.g., new, lost, dead, or other appropriate life status), the tracker's association confidence and relationships with other trackers may also be updated. Using tracker relationships as an example, when two objects (e.g., people, vehicles, or other objects of interest) overlap, the two trackers associated with the two objects are merged together for a given frame, and a merge or occlusion status needs to be recorded for high-level video analysis.

[0107] Regardless of the tracking method used, a new tracker begins associating with a blob in a given frame, and as it progresses, that new tracker may be associated with potentially moving blobs across multiple frames. If a tracker is continuously associated with a blob for a certain period of time (a threshold time), the tracker may be promoted to a normal tracker. For example, the threshold time is the length of time a new blob tracker must be continuously associated with one or more blobs before it is converted to a normal tracker (transitions to the normal state). Normal trackers are output as identified tracker-blob pairs. For example, a tracker-blob pair is output at the system level as an event when the tracker is promoted to a normal tracker (e.g., presented on the display as a tracked object, output as a warning, and / or other appropriate event). In some implementations, a normal tracker (including some status data for the normal tracker, a motion model for the normal tracker, or other information about the normal tracker) may be output as part of the object metadata. Metadata, including normal trackers, may be output from the video analysis system (e.g., the IP camera running the video analysis system) to a server or other system storage. The metadata may then be analyzed for event detection (e.g., by a rule interpreter). Trackers that do not rise to the status of normal trackers may be removed (or killed), after which they can be considered dead.

[0108] As mentioned above, in some implementations, blob or object tracking can be performed using one or more machine learning systems (for example, using one or more neural networks). In some cases, using machine learning systems for blob / object tracking can provide online operability and high speed.

[0109] Figure 6A shows an example of a machine learning-based object detection and tracking system 600, which includes a fully convolutional deep neural network. The system 600 can perform object detection, object tracking, and object classification. As shown in Figure 6A, the input to the object detection and tracking system 600 includes one or more reference object images (referring to 255 × 255 images with three color channels such as red, green, and blue, shown in Figure 6A as a 255 × 255 × 3 image, referred to as "references") and one or more query image frames (shown in Figure 6A as a 127 × 127 × 3 image, referred to as "search patches"). For example, multiple search patches from the references may be input to the system 600 to detect, track, and classify one or more objects in the references.

[0110] The object detection and tracking system 600 includes a ResNet-50 neural network (up to the last convolutional layer of the fourth stage) as the backbone of the system 600's neural network. To obtain higher spatial resolution in deeper layers, the output stride is reduced to 8 by using a convolution with a stride of 1. The receptive field is increased by using an expanded convolution. For example, in the 3x3 convolutional layer of conv4_1 (the top conv4 layer in Figure 6A), the stride can be set to 1 and the expansion rate can be set to 2. As shown in Figure 6A, the top conv4_1 layer has a feature map size of 15x15x1024, and the bottom conv4_2 layer has a feature map size of 31x31x1024. Unlike the original ResNet-50 architecture, there is no downsampling in the conv4_1 or conv4_2 layers.

[0111] One or more adjustment layers (labeled “adjust” in Figure 6A) are added to the backbone. In some cases, each adjustment layer may contain a 1×1 convolutional layer with 256 output channels. Two adjustment layers can perform depth-wise correlation to generate a feature map of a specific size (a 17×17 size is shown in Figure 6A). For example, the output features of the adjustment layers are correlated depth-wise, resulting in a 17×17 size feature map (with 256 channels). The purpose of the adjustment layers is to find a target object from a lower layer of the network (for example, in a 17×17 image size). For example, the adjustment layers may be used to extract feature maps from a reference object image (example) and a query image frame (search patch). The Row of Wires (RoW) in the last layer of the second row of system 600 represents the window candidate response, where the window candidate is the target object region from the query image frame input to system 600. The example and search patches share network parameters from conv_1 to conv4_x, but not the parameters of the tuning layer.

[0112] An improved module-type structure can be used, which combines the feature maps of the backbone and performs upsampling to obtain finer results. For example, the layers in the top row of system 600 perform deconvolution followed by upsampling (shown as upsampling components U2, U3, and U4), which aims to reconstruct the position of the target object at a higher level (e.g., to an image size of 127 × 127). An example of the U3 component is shown in Figure 6B. The U2 and U4 components have a similar structure and operation to the U3 component. The last convolutional layer before the sigmoid operation (labeled "conv.3*3,1") is used to reduce the dimension of the feature map from 127 × 127 × 4 to 127 × 127 × 1. The sigmoid function is used to binarize the output of the object mask, which is the result of object segmentation. The object mask may contain a binary mask with each pixel having a value of either 0 or 1. The purpose of generating the object mask is to obtain an accurate object bounding box. The bounding box can contain a rectangle in any direction. In some cases, the object bounding box is close to the object's center point or centroid (e.g., the center is determined relative to the object). In some cases, a scoring branch may be included in system 600 to generate a scoring matrix based on the object mask. In such cases, the scoring matrix can be used for precise object localization. As mentioned above, the first four stages of the ResNet-50 network share parameters, and the outputs are concatenated into a 1x1 convolution of the shared parameters to adjust the channels *d for cross-correlation at each depth. Further details regarding the backbone architecture in Figure 6A are shown in Figure 6C.

[0113] In some implementations, a classification system may be used to classify objects detected and tracked in one or more video frames of a video sequence. Different types of object classification applications can be used. In a first exemplary classification application, a relatively low-resolution input image is used to provide a classification of the entire input image along with its class and confidence level. In such an application, classification is performed on the entire image. In a second exemplary classification system, a relatively high-resolution input image is used, and multiple objects in the image are output, each object having its own bounding box (or ROI) and classified object type. The first exemplary classification application is referred to herein as “image-based classification,” and the second exemplary classification application is referred to herein as “blob-based classification.” The accuracy of classification in both applications can be increased when neural network (e.g., deep learning) based solutions are utilized.

[0114] Figure 7 shows an example of a machine learning-based classification system. As shown, machine learning-based classification (which may also be called region-based classification) first extracts region proposals (e.g., blobs) from an image. The extracted region proposals, which may contain blobs, are fed into a deep learning network for classification. A deep learning classification network generally starts with an input layer (image or blob), followed by a series of convolutional and pooling layers (among various other layers), and ends with a fully connected layer. After the convolutional layers, there may be a layer with a normalized linear unit (ReLU) activation function. The convolutional, pooling, and ReLU layers act as learnable feature extractors, and the fully connected layer acts as a classifier.

[0115] In some cases, when blobs are fed into a deep learning classification network, one or more shallow layers in the network may learn simple geometric objects, such as lines and / or other objects, that represent the object to be classified. Deeper layers learn much more abstract and detailed features about the object, such as sets of lines that define shape or other detailed features, and ultimately, sets of shapes from earlier layers that constitute the shape of the object being classified (e.g., a person, a car, an animal, or any other object). Further details of the structure and function of the neural network are described below with respect to Figures 42 to 46C.

[0116] Blob-based classification requires far less computational complexity and less memory bandwidth (for example, the memory needed to maintain the network structure), and it can be used directly.

[0117] Various deep learning-based detectors can be used to classify or detect objects in video frames. For example, a Cifar-10 network-based detector can be used to classify blobs by performing blob-based classification. In some cases, a Cifar-10 detector can be trained to classify only people and passenger cars. A Cifar-10 network-based detector can take blobs as input and classify them with a certain confidence score as one of several predetermined classes. Further details of the Cifar-10 detector are described below with reference to Figure 21.

[0118] Another deep learning-based detector is the Single Shot Detector (SSD), a fast single-shot object detector that can be applied to multiple object categories. A key feature of the SSD model is the use of multiscale convolutional bounding box outputs attached to multiple feature maps at the top level of the neural network. Such representations enable SSDs to efficiently model a variety of box shapes. Under the same VGG-16-based architecture, SSDs have been demonstrated to be comparable to their modern object detector counterparts in both accuracy and speed. The SSD deep learning detector is described in more detail in K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," CoRR, abs / 1409.1556, 2014, which is incorporated herein by reference in its entirety for all purposes. Further details of the SSD detector are described below with respect to Figures 25A to 25C.

[0119] Another example of a deep learning-based detector that can be used to detect and classify objects in video frames is the You Only Look Once (YOLO) detector. When run on a Titan X, the YOLO detector processes images at 40–90 frames per second (fps) with an mAP of 78.6% (based on VOC 2007). The SSD300 model runs at 59 fps on an Nvidia Titan X, which is typically faster than the current YOLO 1. YOLO 1 has also recently been superseded by its successor, YOLO 2. The YOLO deep learning detector is described in more detail in J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," arXiv preprint arXiv:1506.02640, 2015, which is incorporated herein by reference in its entirety for all purposes. Further details of the YOLO detector are described below with respect to Figures 46A to 46C. While SSD and YOLO detectors are described as examples to illustrate deep learning-based object detectors, those skilled in the art will understand that any other suitable neural network may be used to perform object classification.

[0120] As mentioned above, it is desirable in many scenarios to maintain the size of the region of interest and / or object of interest across frames in a sequence of frames as the region of interest and / or object of interest moves relative to one or more cameras capturing the sequence of frames. An example of such a scenario may include when a user provides an input to a device that captures video of an event containing an object of interest. For example, the device may record video of a person performing a dance routine, in which case the person moves relative to the camera (in depth and lateral directions) as the video is captured. The user may want the person to maintain a constant size (and in some cases a stable position in the captured frames) throughout the video as the person moves relative to the camera. Another example of such a scenario is in video analysis when an IP camera is capturing video of a scene. For example, an IP camera may capture video of a user's living room, in which case it may be desirable to maintain the size (and in some cases a stable position in the captured frames) of one or more people in the room, even as one or more people move away from the camera (in depth directions).

[0121] As a device captures a sequence of frames of an object (for example, a video of a person performing a dance routine), the object may move relative to one or more cameras capturing the sequence of frames. As a result, it can be difficult for the device to maintain the desired size of the object (for example, the size of the object in the original frame when the video capture first begins) as the object moves during the capture of the sequence of frames. For example, a user might adjust the camera zoom so that the object has the desired size in the frame. However, the object's size ratio (the object's size relative to the frame, also called the object size-to-frame ratio) changes dynamically as the object moves. Manually changing the object size-to-frame ratio during video capture can be cumbersome for the user. Automatically tracking a subject during video recording can also be difficult.

[0122] As described above, systems and techniques for maintaining a constant size of a target object in a sequence of frames are described herein. In an example for a particular explanation, the initial frame of a video or other sequences of frames may be captured and displayed. In some cases, the user can provide user input indicating the object of interest in the initial frame (e.g., by drawing a bounding box around the object, selecting the object, or zooming in on the object). In some cases, the object may be detected automatically without user input. In some cases, the size of the object in the initial frame may be determined and used as the reference size of the object in subsequent frames of the video after the initial frame. In some cases, a bounding box may be set relative to the object in the initial frame. In some examples, the center point coordinates of the bounding box (or other points related to the bounding box or object) and the diagonal length (or other length related to the bounding box or object) may be determined and used as a reference for subsequent frames of the video.

[0123] Object detection and tracking may be initialized and performed to detect and track objects in subsequent frames of a video. For each subsequent video frame, the center point coordinates of the object bounding box (or other points related to the bounding box or object) and the diagonal length of the bounding box (or other lengths related to the bounding box or object) may be determined or recorded. Once the set of coordinates and diagonal lengths (or other lengths) of the bounding box center point (or other points) are obtained for the video frames, a smoothing function may be applied to smooth the amount of change in the diagonal length (and therefore size) of the bounding box in each video frame. In some cases, a smoothing function may also be applied to smooth the movement trajectory of the bounding box center point in the video frames. As described herein, a scaling factor may be calculated for each frame by comparing the diagonal length of the bounding box in the initial video frame (called the reference frame) with the current frame being processed. A scaling factor may be used to scale or resize each frame. Cropping and scaling may be performed on each video frame based on the coordinates of the center point and the scaling factor. In some cases, video stabilization may be applied after cropping and scaling. The output video may then be provided with objects maintained at their reference size and, in some cases, at their common positions within the video frame (e.g., at the center of each frame).

[0124] Figure 8A shows an example of a system for capturing and processing frames or images. The system in Figure 8A includes an image sensor 801, one or more image processing engines 803, a video processing engine 805, a display processing engine 807, an encoding engine 809, an image analysis engine 811, a sensor image metadata engine 813, and a frame cropping and scaling system 815. An exemplary frame cropping and scaling system 800 is described below with respect to Figure 8B.

[0125] The system in Figure 8A may include, or be part of, electronic devices such as mobile or fixed telephone handsets (e.g., smartphones, mobile phones), IP cameras, desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, or any other suitable electronic devices. In some examples, the system may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or any combination thereof. In some implementations, the frame clipping and scaling system 800 may be implemented as part of the image acquisition and processing system 100 shown in Figure 1.

[0126] While the system in Figure 8A is shown to include several components, those skilled in the art will understand that the system may include a greater number of components than those shown in Figure 8A. The components of the system may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), electronic circuits or other electronic hardware, and / or may be implemented using them, and / or may include computer software, firmware, or any combination thereof to perform the various operations described herein, and / or may be implemented using them. The software and / or firmware may include one or more instructions, which are stored on a computer-readable storage medium and are executable by one or more processors of the electronic devices implementing the system in Figure 8A.

[0127] Image sensor 801 can perform operations similar to those of image sensor 130 described above with respect to Figure 1. For example, image sensor 801 may include one or more arrays of photodiodes or other photosensitive elements. Each photodiode can measure the amount of light corresponding to a particular pixel in the image produced by image sensor 130. In some examples, one or more image processing engines 803 may include a camera serial interface decoder module, an image front-end, a Bayer processing segment (for example, which may be used for a snapshot or preview image), an image processing engine, any combination thereof, and / or other components.

[0128] The video processing engine 805 can perform video encoding and / or video decoding operations. In some cases, the video processing engine 805 includes a synthetic video encoder-decoder (also called a “codec”). The video processing engine 805 can perform any type of video coding technique to encode video data and / or decode encoded video data. Examples of video coding techniques or standards include, among others, General Purpose Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Moving Picture Experts Group (MPEG)-2 Part 2 coding, VP9, ​​and Alliance of Open Media (AOMedia) Video 1 (AV1). Using video coding techniques, the video processing engine 805 can perform one or more prediction methods (e.g., interpretation, intrapretation, etc.) that take advantage of redundancy present in a video image or sequence. The goal of video coding is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. The goal of video decoding is to decompress the video data and obtain any other information in the encoded video bitstream that can be used to decode and / or play back the video data. The video output by the video processing engine 805 may be stored in memory 817 (for example, a decoded picture buffer (DPB), random access memory (RAM), one or more cache memories, any combination thereof, and / or other memory) and / or output for display. For example, the decoded video data may be stored in memory 817 for use when decoding other video frames and / or displayed on display 819.

[0129] The display processing engine 807 may be used for preview images. For example, the display processing engine 807 may process, manipulate, and / or output a preview image that has the same (or in some cases similar) aspect ratio as the camera output image, but with a lower image resolution. The preview image may be displayed (as a "preview") on the display of the system or the device containing the system before the actual output image is generated.

[0130] The image coding engine 809 can perform image encoding (compression) and / or image decoding (decompression) operations. In some cases, the image coding engine 809 includes a synthetic image encoder-decoder (or codec). The image coding engine 809 can perform any type of image coding technique to encode image data and / or decode compressed image data. Examples of image coding techniques or standards include, among others, the Joint Photographic Experts Group (JPEG) and the Tagged Image File Format (TIFF). Using image coding techniques, the image coding engine 809 can leverage the visual perception and statistical properties of image data to compress images with minimal loss of fidelity or quality.

[0131] The frame analysis engine 811 can perform frame or image analysis on preview frames acquired or received from the display processing engine 807. For example, the frame analysis engine 811 can acquire or receive a copy of a preview image (with a lower image resolution compared to the camera output image) from the display processing engine 807. The frame analysis engine 811 can perform object detection and / or tracking operations on the preview image to detect and / or track one or more objects in the image (e.g., a target object). The frame analysis engine 811 can determine and output size information, position information, and center point (or other point) information of the bounding boxes of one or more tracked objects (e.g., a tracked target object). Information about the bounding boxes of one or more tracked objects can be output to the frame cropping and scaling system 815.

[0132] The sensor frame metadata engine 813 generates and outputs the final output image. The sensor frame (or image) metadata represents the output image information and has the same image resolution as the output image.

[0133] Figure 8B shows an example of a frame cropping and scaling system 800 that can process one or more frames to maintain a constant size (and in some cases a constant position) of objects in one or more frames. In some cases, the frame cropping and scaling system 800 is an example of the frame cropping and scaling system 815 of the system shown in Figure 8A. In some cases, the frame cropping and scaling system 800 may be different from the system shown in Figure 8A. The frame cropping and scaling system 800 includes a region of interest (ROI) determination engine 804, an object detection and tracking system 806, a frame cropping engine 808, a frame scaling engine 810, and a smoothing engine 812. Examples of the operation of the cropping and scaling system 800 are described below with respect to Figures 8C to 41. In some examples, process 820 in Figure 8C, process 930 in Figure 9A, process 935 in Figure 9B, and / or other processes described herein may be performed based on user-selected operation. For example, the device may receive user input from a user (e.g., touch input via the device's touchscreen, voice input via the device's microphone, gesture input using one or more of the device's cameras) that instructs the device to capture video and maintain objects in the video at a constant size. Based on the user input, the device may perform process 820, process 930 in Figure 9, and / or other processes described herein.

[0134] The frame clipping and scaling system 800 may include, or be part of, an electronic device such as a mobile or fixed telephone handset (e.g., a smartphone, mobile phone), an IP camera, a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, or any other suitable electronic device. In some cases, the frame clipping and scaling system 800 may be part of the same device as the system in Figure 8A. In some examples, the frame clipping and scaling system 800 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or any combination thereof. In some implementations, the frame clipping and scaling system 800 may be implemented as part of the image acquisition and processing system 100 shown in Figure 1.

[0135] While the frame clipping and scaling system 800 is shown as comprising several components, those skilled in the art will understand that the frame clipping and scaling system 800 may comprise more components than those shown in Figure 8B. The components of the frame clipping and scaling system 800 may comprise one or more combinations of software, hardware, or software and hardware. For example, in some implementations, the components of the frame clipping and scaling system 800 may comprise one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), electronic circuits, or other electronic hardware, and / or may be implemented using them, and / or may comprise computer software, firmware, or any combination thereof, to perform the various operations described herein, and / or may be implemented using them. The software and / or firmware may comprise one or more instructions, stored on a computer-readable storage medium and executable by one or more processors in the electronic devices implementing the frame clipping and scaling system 800.

[0136] The sequence of frames 802 is input to the frame cropping and scaling system 800. Frame 802 may be part of a sequence of frames. A sequence of frames may be a video, a group of images captured sequentially, or another sequence of frames. The ROI determination engine 804 can determine an initial region of interest (ROI) in a particular frame based on user input and / or automatically. For example, in block 822 of Figure 8C, process 820 may perform object selection in the initial frame of a sequence of frames (e.g., in a video). The ROI may be represented by a bounding box or other bounding region. In some implementations, the bounding region is visible in the frame when output to a display device. In some implementations, the bounding region may not be visible (e.g., to the viewer, such as a user) when the frame is output to a display device. The frame in which the initial ROI is determined is called the initial frame (or reference frame) of the sequence of frames.

[0137] In some examples, the ROI determination engine 804 (or other components of the frame cropping and scaling system 800) may determine which video frame from a sequence of frames 802 is to be used as the initial frame. In some cases, the first frame in a sequence of frames 802 may be selected as the initial frame. In an example for one explanation, such as in real-time video recording, the initial frame may also be the first frame of the video after the end user has provided an input indicating the desired size of an object in the frame (e.g., a pinch gesture to zoom in to an ideal camera zoom ratio), after which video recording can begin. In an example for another explanation, such as in video playback (e.g., of previously recorded video) or an optional post-processing-based auto-zoom function, the end user may select any frame of the video and provide an input with respect to the frame to indicate the desired size of an object (e.g., a pinch input to zoom), thereby setting that frame as the initial frame.

[0138] In some examples, the ROI may be determined based on user selection of a portion of the initial frame, such as an object depicted in the initial frame. User input may be received using any input interface of the device, including the frame cropping and scaling system 800 or other devices. For example, input interfaces may include a touchscreen, an electronic drawing tool, a gesture-based user interface (e.g., one or more image sensors used to detect gesture input), a voice-based user interface (e.g., a speaker and voice recognition tool used to identify voice input), and / or other user interfaces. In some examples, object selection may include tapping an object displayed in the initial frame (e.g., single tap, double tap, etc.), the user drawing a bounding box around an object, the user providing input on a touchscreen interface to zoom in on an object (e.g., pinching, including bringing two fingers closer together or further apart), or other types of object selection. In some cases, guidance may be provided to the end user on how to utilize the ability to keep the size of a target object constant throughout the rest of the video or frame. For example, the user may be prompted to select an object that should be kept constant throughout the video. For a video, the user can select an object of interest by tapping on it (for example, on a touchscreen) or by drawing a bounding box around an object in the initial frame of the video. Based on the selected portion of the initial frame, the ROI determination engine 804 can define an ROI around the selected portion (for example, around the selected object). The ROI indicates the size (for example, the ideal size) that the object should remain throughout other sequences of the video or frame. For example, the user can zoom in on an object to indicate the size of the object that the user wants to maintain throughout other sequences of the video or frame, and the ROI determination engine 804 can define an ROI around the object using the indicated size.

[0139] In some cases, objects in the initial frame may be automatically detected in the initial frame (e.g., using object detection and / or recognition), and the ROI determination engine 804 can define an ROI around the detected objects. Objects may be detected using object detection and / or recognition techniques (e.g., face detection and / or recognition algorithms, feature detection and / or recognition algorithms, edge detection algorithms, boundary tracking functions, any combination thereof, and / or other object detection and / or recognition techniques). Any of the detection and tracking techniques described above may be used to automatically detect objects in the initial frame. In some cases, feature detection may be used to detect (or locate) the features of an object from the initial frame. Based on features, object detection and / or recognition may detect an object, and in some cases, recognize the detected object and classify it into an object category or type. For example, feature recognition may identify the number of edges and corners in an area of ​​the scene. Object detection may detect that all detected edges and corners in the area belong to a single object. If face detection is performed, face detection may identify that an object is a human face. Object recognition and / or facial recognition can further identify the identity of the person whose face corresponds to it.

[0140] In some implementations, object detection and / or recognition algorithms may be based on machine learning models trained using machine learning algorithms on images of the same type of objects and / or features, which may extract features from images and detect and / or classify objects possessing those features based on the model's training by the algorithm. For example, machine learning algorithms can be neural networks (NNs), such as convolutional neural networks (CNNs), time-delayed neural networks (TDNNs), deep feedforward neural networks (DFFNNs), recurrent neural networks (RNNs), autoencoders (AEs), variation AEs (VAEs), denoising AEs (DAEs), sparse AEs (SAEs), Markov chains (MCs), perceptrons, or any combination thereof. Machine learning algorithms can be supervised learning algorithms, unsupervised learning algorithms, semi-supervised learning algorithms, generative adversarial network (GAN)-based learning algorithms, any combination thereof, or other learning techniques.

[0141] In some implementations, computer vision-based feature detection and / or recognition techniques may be used. Different types of computer vision-based object detection algorithms may be used. In an example for one explanation, a template matching-based technique may be used to detect one or more hands in an image. Various types of template matching algorithms may be used. An example of a template matching algorithm can perform Haar or Haar-like feature extraction, integral image generation, Adaboost training, and cascade classifier. Such object detection techniques perform detection by applying a sliding window (e.g., having a rectangle, circle, triangle, or other shape) across the image. The integral image may be computed from the image to be an image representation that evaluates specific regional features, such as rectangular or circular features. For each current window, the Haar features of the current window may be computed from the integral image described above, and this may be computed before computed the Haar features.

[0142] Haar features can be calculated by calculating the sum of image pixels within a specific feature region of an object image, such as a feature region of an integrated image. In a face, for example, the area containing the eyes is usually darker than the area containing the bridge of the nose or the cheeks. Haar features may also be selected by a learning algorithm (e.g., the Adaboost learning algorithm) that selects the best features and / or trains a classifier that uses them, and may be used with a cascaded classifier to substantially classify windows of faces (or other objects) or windows of non-faces. A cascaded classifier consists of multiple classifiers combined in a cascaded form, which allows background areas of an image to be quickly discarded while more computation is performed on areas like objects. Using a face as an example of a body part of an external observer, a cascaded classifier can classify the current window into the category of faces or non-faces. If a classifier classifies the window as a non-face category, the window is discarded. Instead, if a classifier classifies the window as a face category, the next classifier in the cascaded configuration is used to test again. A window is labeled as a candidate for a hand (or other object) until all classifiers determine that the current window is a face (or other object). After all windows have been detected, a non-max suppression algorithm may be used to group the windows around each face to generate a final result of one or more detected faces.

[0143] Returning to Figure 8B, the ROI determination engine 804 can define an ROI based on a selected portion of the initial image (e.g., a selected object) or based on detected objects in the initial image. As mentioned above, an ROI can be represented by a bounding box or other type of bounding region. In some cases, the ROI determination engine 804 can generate a bounding box for the ROI that fits the boundaries of objects in the ROI. For example, the maximum x-coordinate (horizontal), minimum x-coordinate, maximum y-coordinate (vertical), and minimum y-coordinate may be determined for an object, and an ROI having the maximum x-coordinate, minimum x-coordinate, maximum y-coordinate, and minimum y-coordinate may be defined. In some cases, the bounding box for the ROI may be defined around an object and not limited to the boundaries of objects in the ROI.

[0144] The ROI determination engine 804 can determine the size of an object and / or the size of the region of interest containing the object in the initial frame. The size of the object can be used as a reference size to determine how much to crop and scale subsequent frames in the sequence of frames 802. In some cases, the user can adjust the region of interest and / or the size of the object to define a preferred size of the object in the sequence of frames. For example, the first frame may be displayed (e.g., as a preview image), and the user can adjust the zoom level of the image to make the object larger (by zooming in) or smaller (by zooming out). In such an example, once the user has finished zooming and the final size of the object has been determined relative to the initial frame, the size of the object and / or the region of interest containing the object is determined and used as a reference size. This reference size can then be used to determine how much to crop and scale subsequent frames in the sequence of frames 802.

[0145] Subsequently, frames following the sequence of frames 802 (captured after the initial frame) can be input to the frame clipping and scaling system 800. The operation of the object detection and tracking system 806, the frame clipping engine 808, the frame scaling engine 810, and the smoothing engine 812 is described with respect to a specific subsequent frame after the initial frame (for example, the first subsequent frame after the initial frame). However, the same or similar operation may be performed for some or all subsequent frames following the initial frame in the sequence of frames 802.

[0146] The object detection and tracking system 806 can detect and track objects in subsequent frames of a sequence of frames. For example, in block 824 of Figure 8C, process 820 can perform object detection and tracking to detect and track objects in a sequence of frames. In some examples, objects may be detected and tracked using techniques performed by the video analysis system 200 described above with respect to Figures 2 to 7.

[0147] The frame clipping engine 808 can clip subsequent frames, and the frame scaling engine 810 can scale subsequent frames so that the size of an object is maintained in subsequent frames to the same size as determined in the initial frame. For example, in block 826 of Figure 8C, process 820 can perform video frame clipping and scaling of subsequent frames. In some cases, clipping and scaling may be performed to maintain an object to the same size as determined in the initial frame, and to maintain an object at a specific position within each frame. For example, clipping and scaling may be performed to maintain an object at the center of each subsequent frame, at the position in each subsequent frame where the object was initially located in the initial frame, at a user-defined position, or at any other position within the subsequent frames. As will be explained in more detail below, the frame scaling engine 810 can calculate a scaling factor for each subsequent frame in the sequence of frames 802. In an example for illustrative purposes using the diagonal length of a bounding box, the scaling factor may be determined by comparing the diagonal length of the bounding box in the initial frame with the diagonal length of the bounding box in the current frame being processed. The ratio of these diagonal lengths can be used as the scaling factor. The scaling factor can be used to scale each subsequent frame so that the object in the current frame is the same size as the object in the initial frame. Details of cropping and scaling are described below.

[0148] The smoothing engine 812 can apply one or more smoothing functions so that the cropping and scaling of subsequent frames is performed gradually, minimizing the inter-frame movement and resizing of objects in a sequence of frames. For example, the initial cropping and scaling outputs from the frame cropping engine 808 and the frame scaling engine 810 may indicate that subsequent frames will be cropped and scaled by a certain amount. The smoothing engine 812 can determine the amount of cropping and scaling to be modified in order to reduce the amount that subsequent frames are modified. The smoothing functions can prevent objects from appearing to move unnaturally (e.g., bouncing) in a sequence of frames 802 due to the amount of cropping and scaling determined by the frame cropping engine 808 and the frame scaling engine 810.

[0149] In some cases, cropping, scaling, and smoothing may be based on a point on the object (e.g., the center point) or a point within the bounding box related to the ROI containing the object (e.g., the center point), and / or on a distance related to the object (e.g., the distance between a first part of the object and a second part of the object) or a distance related to the bounding box representing the ROI containing the object (e.g., the diagonal distance of the bounding box). For example, a certain amount of cropping to be performed to move or shift an object in subsequent frames may be performed with respect to a point on the object or a point within the bounding box. In another example, the amount of scaling to be performed to make an object larger or smaller may be based on a distance related to the object (e.g., between different parts of the object) or a distance related to the bounding box (e.g., the diagonal distance of the bounding box).

[0150] Frame cropping and scaling may proceed along with the actual change in the size of the target object. The smoothing engine 812 can output a final output frame 814 (e.g., output video) which has the effect of the object having a constant size (based on a reference size determined for the object in the initial frame) and, in some cases, being maintained in the same position throughout the entire sequence of frames. For example, in block 828 of Figure 8C, process 820 can generate an output video that has the effect of a constant size and position for the object, based on the target size fixing function described above.

[0151] Figures 9A and 9B are flowcharts illustrating other examples of processes 930 and 935 that may be performed by the frame cropping and scaling system 800 for video. In some examples, processes 930 and / or 935 may be performed based on actions selected by the user. For example, the device may receive user input from the user (e.g., touch input via the device's touchscreen, voice input via the device's microphone, gesture input using one or more of the device's cameras) instructing the device to capture video and maintain objects in the video at a constant size. Based on the user input, the device may perform processes 930 and / or 935.

[0152] Processes 930 and 935 are described as being performed on pre-recorded video (in which case all frames of the video are available for processing). However, in some cases, processes 930 and 935 may be modified to process raw video. In some examples, process 930 may be performed before process 935. For example, process 930 may be performed to select an initial video frame from a sequence of frames and to set the center point (or other point) and the diagonal length (or other length) of the object bounding box as reference points. Process 935 may be performed to crop and scale subsequent video frames so as to maintain the size and / or position of the object throughout the sequence of frames.

[0153] As shown in Figure 9A, in block 931, process 930 includes acquiring a sequence of frames. The sequence of frames may be a video, a group of images captured sequentially, or any other sequence of frames. In block 932, process 930 includes selecting or determining a video frame from the sequence of frames to use as the initial frame (or reference frame). In one example, the first frame in the sequence of frames 802 may be selected as the initial frame. As mentioned above, the initial frame may be used as the frame for determining the initial ROI.

[0154] In block 933, process 930 includes selecting a target object having a given size (e.g., an ideal size). As described above, the target object (or ROI) may be selected based on user input or detected automatically. For example, the object or ROI may be determined based on user input indicating a selection of a portion of an initial frame, such as an object depicted in the initial frame. In an example for explanation, the user may perform a pinch-to-zoom (e.g., using a pinch gesture on a touchscreen interface) or provide another input to zoom in on the target object on the display. In some cases, process 930 may include generating a bounding box for the target object or ROI in the initial frame of a sequence of frames (e.g., a video). For example, the ROI may be determined for an object, and the bounding box may be generated to represent the ROI.

[0155] In block 934, process 930 includes setting the center point and diagonal length of a bounding box as a reference to be used for subsequent frames in a sequence of frames (for example, to perform process 935 on a subsequent frame). The center point and diagonal length of the bounding box are used herein for illustrative purposes only, but other points and lengths may be used to perform the cropping, scaling, smoothing, and / or other operations described herein. In some examples, a different point on the bounding box, such as the top-left point of the bounding box, may be used as the reference point instead of the center point of the bounding box. In other examples, a point on an object in the bounding box may be used as the reference point, such as the center point of the object. In some examples, the length between two points on an object in the bounding box may be used to determine the size of the object in the current subsequent frame, instead of the diagonal length of the bounding box. For example, if the object is a person, the length between the top of the person's head and the soles of the person's feet may be used as the length.

[0156] As shown in Figure 9B, in block 937, process 935, similar to block 824 of process 820, includes performing object detection and tracking on each subsequent frame (or subset of subsequent frames) of the sequence of frames following the initial frame. Object detection and tracking may be performed to track an object across each frame of the video. In some examples, process 935 may perform a coordinate transformation to match each subsequent frame to the initial frame. For example, a coordinate transformation may be performed to make each subsequent frame the same size as the initial frame. In an example for one explanation, the coordinate transformation may be an upscaling operation. In an example for another explanation, the coordinate transformation may be a downscaling operation.

[0157] In block 938, process 935 includes determining the bounding box center point and diagonal length of bounding boxes throughout the entire sequence of video frames. For example, bounding box information may be obtained for each frame of the video based on object detection and tracking performed by the object detection and tracking system 806. The position of the center point and diagonal length of each bounding box in each video frame may be determined and used as an indicator of the trajectory of the object's motion and the change in the object's size throughout the video.

[0158] For example, every frame in a sequence of video frames may be processed to determine the center point and diagonal length of each bounding box in each frame. The center point and diagonal length of the bounding box in each frame may be used by the frame cropping engine 808, the frame scaling engine 810, and / or the smoothing engine 812 to perform cropping, scaling, and smoothing of subsequent frames of the video, respectively. For example, the center point of the bounding box may be used as a reference for determining the position of an object in a frame, and the diagonal length may be used to determine the size of an object in the current subsequent frame relative to the size of an object in the initial frame. While the center point and diagonal length of the bounding box are used herein for illustrative purposes, in some implementations other points and lengths may be used to perform cropping, scaling, and smoothing. In some examples, a different point on the bounding box, such as the top-left point of the bounding box, may be used as the reference point instead of the center point of the bounding box. In another example, a point on an object within the bounding box may be used as the reference point, such as the center point of the object. In some examples, instead of the diagonal length of the bounding box, the length between two points on an object within the bounding box may be used to determine the size of the object in the current subsequent frame. For example, if the object is a person, the length between the top of the person's head and the soles of their feet may be used as the length.

[0159] Block 939 represents a smoothing operation that can be performed by the smoothing engine 812. In block 940, process 935 includes performing trajectory smoothing of the center point of the bounding box. The smoothing engine 812 can perform trajectory smoothing of the center point of the bounding box based on any suitable smoothing algorithm. An example of a smoothing algorithm can be based on a moving average algorithm. The moving average technique can be applied to smooth changes in the position and diagonal length of the center point of the bounding box over subsequent frames. In general, moving averages are used to analyze time series data (such as video) by calculating the average of different subsets of the complete dataset (e.g., different frames of a video). Based on moving averages, data can be smoothed so that changes occurring between consecutive parts of the data are more gradual.

[0160] A moving average can be calculated based on a sliding window used to average over a set number of periods (for example, a certain number of video frames). For example, the number of periods can be based on the time between consecutive frames of a video (33 ms for a video with 30 frames per second). The moving average can be an equally weighted average of the previous n data points. For example, define a column of n values ​​as follows: x1, x2, ..., x n

[0161] In this case, the equally weighted moving average of n data points is essentially the average of the previous M data points, where M is the size of the sliding window.

[0162]

number

[0163] To calculate the subsequent moving average, the new value can be added to the sum, and the values ​​from previous periods can be omitted. Since the average of previous periods is available, previous periods can be omitted, and in this case, a full summation of each time period is not required. The subsequent moving average can be calculated as follows:

[0164]

number

[0165] For the current frame of the video being processed by system 800 according to process 935, the (x,y) coordinate positions of the bounding box centers of a certain number of M video frames may be processed using a moving average formula. For example, in block 940, the moving average of the coordinates of the bounding box centers of M video frames

[0166]

number

[0167] This can then be determined. Next, the moving average

[0168]

number

[0169] This can be used as the position of the center point relative to the bounding box in the current video frame.

[0170] In block 941, process 935 includes performing smoothing of the diagonal length changes of the bounding box. The smoothing engine 812 can smooth the diagonal length changes of the bounding box based on any suitable smoothing algorithm. In some cases, the smoothing engine 812 can use the moving average algorithm described above. For example, for the current frame of the video being processed, the smoothing engine 812 can use a moving average formula to process the diagonal length of the bounding box from a certain number of M video frames of the video. For example, in block 942, process 935 performs a moving average of the diagonal length of the bounding box of M video frames.

[0171]

number

[0172] This can be determined. Process 935 is moving average

[0173]

number

[0174] This can be used as the diagonal length of the bounding box in the current video frame.

[0175] In some cases, object detection and tracking may not be accurate for the current frame of the video being processed. For example, the (x,y) coordinate position of the center point of the detected object's bounding box (or other points of the bounding box or object) may be incorrect. Calculated moving (or rolling) average of the coordinates of the center point of the bounding box for the current frame.

[0176]

number

[0177] This can minimize false alarms (by minimizing the location of the bounding box of an object that is incorrectly detected / tracked) and keep the object on the correct moving or tracking trajectory for the most part. For example, the calculated moving (or rolling) average of the center point of the bounding box (or other point in the bounding box or the object).

[0178]

number

[0179] This can be more accurate than the actual detected center point. Moving averages can also minimize false alarms regarding object size (for example, by minimizing the diagonal length of an object bounding box that is misdetected / tracked, or the length that is misdetected / tracked between parts or sections of an object). For example, a calculated moving (or rolling) average of the diagonal length of the bounding box.

[0180]

number

[0181] This may be more accurate than the actual detected diagonal length in a given frame.

[0182] In block 942, process 935 includes calculating a frame scaling factor based on the diagonal length of the initial frame and the smoothed diagonal lengths of other frames in the video. For example, instead of using the actual diagonal length of the bounding box in the current frame, the scaling factor for the current frame (other than the initial frame) may be determined using the smoothed diagonal length (e.g., the average diagonal length) determined for the current frame by the smoothing engine 812. In some cases, the scaling factor may be a scaling ratio. The frame scaling engine 810 may determine a scaling factor for the current frame by comparing the smoothed diagonal length of the bounding box in the current frame with the diagonal length of the bounding box in the initial frame.

[0183] In some examples, process 935 may include determining whether a change in video resources has occurred. For example, a frame cropping and scaling system 815 may support multiple video resources, and an end user may import multiple videos to perform an auto-zoom (cropping and scaling) operation. To determine whether a change in video resources has occurred, process 935 may determine whether the current video is still playing. If the current video is playing, process 935 may proceed to block 943. If it is determined that another video source has started, the update operation cannot be performed, and in this case, the system may restart from the beginning of the process (for example, the beginning of block 931 of process 930).

[0184] In block 943, process 935 includes cropping and scaling each frame in the video based on the position of the center point of the smoothed object bounding box (for example, the average center point position determined for each frame) and a frame scaling factor determined for each frame in block 939. Based on the cropping and scaling, the cropped and scaled subsequent frames are produced such that the object in the subsequent frame has the same size and position as the object in the initial frame.

[0185] Examples relating to Figures 10A and 10B are explained. Figure 10A shows an example of the initial frame 1002 of a video. The user has selected a person as the object of interest. A bounding box 1004 is generated to represent the region of interest for that person. A bounding box 1004 with height h and width w is shown. The position of the center point 1006 of the bounding box 1004 (e.g., (x,y) coordinate position) and the diagonal length 1008 of the bounding box 1004 are determined and used as a basis for cropping and scaling subsequent frames of the video to maintain the person at a constant size and position in subsequent frames.

[0186] Figure 10B shows an example of a subsequent frame 1012 following an initial frame 1002 in a video. Based on object detection and tracking, a bounding box 1014 is generated around the person in the subsequent frame 1012. The bounding box 1014 has a width wn and a height hm. The width wn of the bounding box 1014 is shorter than the width w of the bounding box 1004 in the initial frame 1002, and the height hm of the bounding box 1014 is shorter than the height h of the bounding box 1004 in the initial frame 1002. The center point 1016 and the position (e.g., (x,y) coordinate position) of the bounding box 1004 are determined.

[0187] In some examples, the frame clipping engine 808 can clip a subsequent frame 1012 such that the person depicted in the subsequent frame 1012 remains centered within frame 1012. For example, the frame clipping engine 808 can clip a subsequent frame 1012 to produce a clipped region 1022 such that the center point 1016 of the bounding box 1014 is at the center of the clipped region 1022. In some examples, the frame clipping engine 808 can clip a subsequent frame 1012 such that the person depicted in frame 1012 remains in the same relative position as the person was in the initial frame 1002. For example, the frame clipping engine 808 can determine the position of the center point 1006 of the bounding box 1004 relative to a point in the initial frame 1002 that is common to all frames of the video. For example, a point common to all frames could be the top-left point in the video frame (for example, the top-left point 1007 in the initial frame 1002). The relative distance 1009 from the center point 1006 of the bounding box 1004 in the initial frame 1002 to the upper left point 1007 is shown in Figure 10A. The frame clipping engine 808 can clip the subsequent frame 1012 to generate the clipped region 1022 such that the center point 1016 is at the same relative position and distance 1029 to the upper left point 1017 of the clipped region 1022 as the position and distance of the center point 1006 to the upper left point 1007 in the initial frame 1002.

[0188] The frame scaling engine 810 can determine a scaling factor (e.g., a scaling ratio) for scaling the cropped region 1022 by comparing the smoothed diagonal length of the bounding box 1014 in the subsequent frame 1012 with the diagonal length 1008 of the bounding box 1004 in the initial frame 1002. The smoothed diagonal length of the bounding box 1014 may be determined by the smoothing engine 812 as described above. For example, if the actual diagonal length 1018 of the bounding box 1014 is a value of 1.5, the smoothed diagonal length of the bounding box 1014 may be determined as a value of 1.2 (based on the moving average determined as described above). The diagonal length 1008 of the bounding box 1004 in the initial frame 1002 may be a value of 3. The scaling factor is the scaling ratio.

[0189]

number

[0190] It can be determined as follows: Using such a formula based on diagonal lengths of 1008 and 1018,

[0191]

number

[0192] A scaling ratio of 2.5 can be determined. Based on this scaling ratio, the cropped region 1022 can be increased by a coefficient of 2.5 (become 2.5 times larger).

[0193] As a result of cropping and scaling, a cropped and scaled subsequent frame 1032 is generated. Since the diagonal length 1038 of the bounding box 1034 is the same as the diagonal length 1008 of the bounding box 1004, the person depicted in the cropped and scaled subsequent frame 1032 is the same size as the person depicted in the initial frame 1002. In some examples, the center point 1036 of the bounding box 1034 is at the center of the cropped and scaled subsequent frame 1032. In some examples, the position and distance 1039 of the center point 1036 relative to the top-left point 1037 of frame 1032 is the same as the position and distance 1009 of the center point 1006 relative to the top-left point 1007 in the initial frame 1002, and the person is maintained in the cropped and scaled subsequent frame 1032 in the same location where the person was located in the initial frame 1002. Therefore, the person depicted in the subsequent frame 1032, which has been cropped and scaled, is the same size as the person depicted in the initial frame 1002 and is maintained in a consistent position throughout the video compared to other frames.

[0194] Returning to Figure 9B, process 935 in block 944 involves performing video stabilization. Any suitable video stabilization technique may be used to stabilize the video frame. Generally, video stabilization techniques are used to avoid a decrease in visual quality by reducing unwanted vibrations and jitter of the device (e.g., mobile devices, handheld cameras, head-mounted displays, etc.) during video acquisition. Video stabilization reduces vibrations and jitter without affecting moving objects or intentional camera panning. Video stabilization can be useful for handheld imaging devices (e.g., mobile phones) that are more susceptible to vibrations due to their small size. Unstable images are usually caused by unwanted camera shake and intentional camera panning, and unwanted changes in the camera's orientation result in an unstable image sequence. By using video stabilization techniques, it is possible to ensure that stable video footage with high visual quality is acquired even under suboptimal conditions.

[0195] One example of a video stabilization technique that can be implemented is a fast and robust two-dimensional motion model of the Euclidean transform, which can be used by motion models to solve video optimization problems. In the Euclidean motion model, a square in an image can be transformed into any other square with different positions, sizes, and / or rotations for motion stabilization (since camera movement between consecutive frames of video is usually small). Figure 11 shows an example of an applied motion model, including the original square and the various transformations applied to the original square. Transformations include translational transformations, Euclidean transformations, affine transformations, and projective transformations.

[0196] Figure 12 is a flowchart showing an example of a process 1200 for performing image stabilization. The image stabilization process involves tracking one or more feature points between two consecutive frames. The tracked features allow the system to estimate and compensate for motion between frames. An input frame sequence 1202, containing a sequence of frames, is provided as input to process 1200. The input frame sequence 1202 may include an output frame 814. In block 1204, process 1200 includes performing saliency point detection using optical flow. Saliency detection is performed to determine feature points in the current frame. In block 1204, any suitable type of optical flow technique or algorithm may be used. In some cases, optical flow motion estimation may be performed pixel by pixel. For example, for each pixel in the current frame y, motion estimation f defines the position of the corresponding pixel in the previous frame x. Motion estimation f for each pixel may include an optical flow vector indicating the motion of the pixel between frames. In some cases, the optical flow vector for a pixel may be a displacement vector indicating the movement of the pixel from the first frame to the second frame (for example, showing horizontal and vertical displacements such as x-displacement and y-displacement).

[0197] In some examples, an optical flow map (also called a motion vector map) may be generated based on the calculation of optical flow vectors between frames. Each optical flow map may contain a 2D vector field, where each vector is a displacement vector indicating the movement of a point from the first frame to the second frame (e.g., showing horizontal and vertical displacements such as x-displacement and y-displacement). The optical flow map may also contain an optical flow vector for each pixel in the frame, where each vector indicates the movement of the pixel between frames. For example, dense optical flow can be calculated between adjacent frames to generate an optical flow vector for each pixel in a frame, and the optical flow vectors may be contained in a dense optical flow map. In some cases, the optical flow map may contain vectors for fewer pixels than all pixels in the frame, such as pixels that belong only to one or more parts of the external observer being tracked (e.g., the external observer's eyes, one or more hands of the external observer, and / or other parts). In some examples, the Lucas-Kanade optical flow can be computed between adjacent frames to generate an optical flow vector for some or all pixels in a frame, and this optical flow vector may be included in an optical flow map.

[0198] As mentioned above, an optical flow vector or optical flow map is used to represent the flow between adjacent frames in a sequence of frames (for example, adjacent frames x t and x t-1 It can be calculated between sets of frames. Two adjacent frames can be two directly adjacent frames that are captured consecutively, or two frames that are separated by a certain distance in the sequence of frames (for example, within two frames of each other, within three frames of each other, or another appropriate distance). Frame x t-1 From frame x t The optical flow to Ox t-1, x t=dof(x t-1 ,x t ) may be given by and dof is dense optical flow. Any suitable optical flow process can be used to generate the optical flow map. In an example for one explanation, frame x t-1 The pixel I(x,y,t) inside is the next frame x t Then it can move by a distance (Δx, Δy). The pixels are the same, and the intensity is frame x. t-1 and the next frame x t Assuming that it does not change between [values], we can assume the following equation. I(x,y,t)=I(x+Δx,y+Δy,t+Δt)

[0199] By taking the Taylor series approximation on the right side of the above equation, removing the general term, and dividing by Δt, we can derive the equation for optical flow. f x u+f y v+f t =0

[0200] Here,

[0201]

number

[0202] That is the case.

[0203] Using the optical flow equation above, the image gradient f x and f y However, the gradient along time (f tIt can be found together with (denoted as ). The terms u and v are the x and y components of the velocity or optical flow of I(x,y,t), and are unknown. In some cases where it is not possible to solve the equation for optical flow with two unknown variables, estimation techniques may be required. Any suitable estimation technique can be used to estimate the optical flow. Examples of such estimation techniques include finite difference methods (e.g., Lucas-Kanade estimation, Horn-Schunck estimation, Buxton-Buxton estimation, or other suitable finite difference methods), phase correlation, block-based methods, or other suitable estimation techniques. For example, Lucas-Kanade assumes that the optical flow (displacement of image pixels) is small and approximately constant in the local neighborhood of pixel I, and uses the least squares method to solve the basic optical flow equation for all pixels in that neighborhood.

[0204] In block 1206, process 1200 includes selecting correspondences between saliency points in a sequence of images. In block 1208, process 1200 performs transform estimation from noisy correspondences. In block 1210, process 1200 includes applying transform approximation and smoothing to generate an output frame sequence 1212 containing a sequence of output frames. For example, significant feature points can be detected from previous and current image frames, and then these feature points with one-to-one correspondences are used. Based on the locations of the feature points used, region-based transforms may be applied to map from previous frames to image content on the current image frame.

[0205] In some cases, video frame extraction and merging are applied before and after the entire process 935 in Figure 9B. For example, in some cases, the inputs and outputs of system 800 may contain image frames (and not video), in which case video frame extraction and merging are required before and after the entire process.

[0206] In some cases, the inherent zoom ratio and camera lens switching capabilities of a device (e.g., a mobile phone or smartphone) may be used to perform one or more of the techniques described herein. For example, the system may output a single video with the target object size-fixing effect described herein. Such a solution may, in some cases, be used as a real-time function (for raw video) and may automatically adjust the camera's zoom ratio during video recording.

[0207] In some examples, automatic zoom operation may be performed using one or more of the techniques described above and / or other techniques. Figure 13A shows an example of process 1300 for performing an automatic zoom mode. For example, process 1300 may determine or set as a reference point (e.g., center point) and distance (e.g., diagonal length) of the bounding box of the object and / or region of interest in the first frame (or initial frame). In some examples, process 1300 may be initiated based on an automatic zoom operation selected by the user. For example, the device may receive user input from the user (e.g., touch input via the device's touchscreen, voice input via the device's microphone, gesture input using one or more of the device's cameras) to instruct the device to enter automatic zoom mode. Based on the user input, the device may perform process 1300. In some examples, once an automatic zoom operation is selected, the device may begin using an object detection and tracking system (e.g., object detection and tracking system 806 in Figure 8B) to perform object detection and tracking of any region of interest or object of interest.

[0208] In block 1302, process 1300 includes acquiring the first frame (or initial frame) of a sequence of frames (for example, the first video frame of a video in which the user identifies an object and / or region of interest). In block 1304, process 1300 includes determining the target object of interest in the initial frame. For example, as described above, the region of interest (ROI) determination engine 804 in Figure 8B can determine the ROI in the initial frame based on user input and / or automatically. The ROI may correspond to a target object (or object of interest). The ROI and / or target object may be represented by a bounding box or other bounding region. In some examples, the bounding box is visible in the frame when output to a display device. In some examples, the bounding box may not be visible when the frame is output to a display device. The frame in which the initial ROI is determined is called the initial frame (or reference frame) of the sequence of frames.

[0209] As described above, in some examples, the ROI may be determined based on user selection of a portion of the initial frame, such as an object depicted in the initial frame. For example, the user may select a target object to be used in an auto-zoom process to maintain a constant size (e.g., the size of the object in the initial frame) across multiple frames in a sequence of frames. User input may be received using any input interface of the device, such as a touchscreen, electronic drawing tools, gesture-based user interfaces (e.g., one or more image sensors used to detect gesture inputs), voice-based user interfaces (e.g., a speaker and voice recognition tool used to identify voice inputs), and / or other user interfaces. Any of the inputs described above with respect to Figures 8C and 9, as well as / or other inputs, may be provided by the user. For example, object selection may be performed based on tapping an object displayed in the initial frame (e.g., single tap, double tap, etc.), the user drawing a bounding box around an object, or other types of object selection. In some cases, the end user may be provided with guidance on how to utilize the ability to keep the size of a target object constant across other sequences of the video or frames. For example, the user may be prompted to select objects that should remain constant throughout the video. For the video, the user can select objects of interest by tapping them (e.g., on a touchscreen) or by drawing bounding boxes around objects in the initial frames of the video. Based on the selected portion of the initial frames, the ROI determination engine 804 can define an ROI around the selected portion (e.g., around the selected objects).

[0210] In some examples, objects in the initial frame may be automatically detected in the initial frame (e.g., using object detection and / or recognition), and the ROI determination engine 804 can define an ROI around the detected objects. Objects may be detected using object detection and / or recognition techniques (e.g., face detection and / or recognition algorithms, feature detection and / or recognition algorithms, edge detection algorithms, boundary tracking functions, any combination thereof, and / or other object detection and / or recognition techniques).

[0211] In block 1306, process 1300 includes determining or setting points and distances relative to an object bounding box. In one example for illustrative purposes, a point may include the center point of the bounding box. In some cases, other points of the bounding box may also be used, such as the top-left point or corner of the bounding box. In another example, a point on the object within the bounding box may be used as a reference point, such as the center point of the object. In another example for illustrative purposes, the distance may be the diagonal length of the bounding box (for example, the length from the bottom-left point of the bounding box to the top-right point of the bounding box, or the length from the bottom-right point of the bounding box to the top-left point of the bounding box). In some examples, the distance may include the length between two points on the object within the bounding box. For example, if the object is a person, the length between the top of the person's head and the soles of the person's feet may be used as the length.

[0212] By setting the center point (or other point) and diagonal length (or other distance) of the bounding box of the object, process 1300 can initialize target object information, including the coordinates of the object's center point, the diagonal length of the object's bounding box, and the current zoom ratio relative to the object.

[0213] Figure 13B shows an example of process 1310 for performing an additional mode of auto-zoom for one or more subsequent frames (for example, those that are after the initial frame in a sequence of frames) captured after the initial frame. In block 1312, process 1310 includes acquiring one or more subsequent frames. In some cases, process 1310 may be executed once for one frame at some point in time from one or more subsequent frames. In some cases, process 1310 may be executed once for multiple frames at some point in time from one or more subsequent frames. The subsequent frame being processed by process 1310 is called the current subsequent frame.

[0214] In block 1314, process 1310 includes obtaining a frame from the display processing engine 807. The frame may be called an analysis frame or a preview frame. As described above with respect to Figure 8A, a preview (or analysis) frame may have the same aspect ratio as the output frame but with a lower resolution (smaller size). For example, a preview frame may be a lower-resolution version of the current subsequent frame compared to the full output version of the current subsequent frame. A frame clipping and scaling system (e.g., frame clipping and scaling system 815 and / or frame clipping and scaling system 800) may use the preview frame for object detection and tracking processing. For example, in block 1316, process 1310 performs object detection and / or tracking to detect and / or track a target object (determined from the initial frame) in the preview frame (a lower-resolution version of the current subsequent frame being processed by process 1310). As described above, the frame analysis engine 811 may perform object detection and / or tracking on the analysis (preview) frame.

[0215] In block 1318, process 1310 performs a coordinate transformation on the preview (analysis) frame. For example, since the preview frame and the sensor frame metadata (corresponding to the full output frame) have the same image content but different image resolutions, a coordinate transformation may be performed to make the preview frame and the full output frame the same size. In an example for illustrative purposes, the coordinate transformation may be an upscaling operation. For example, process 1310 may upscale the preview frame so that the preview frame has the same resolution as the full output frame corresponding to the sensor frame metadata.

[0216] In block 1320, process 1310 determines a point (e.g., center point or other point) and a scaling ratio for the target object in the current subsequent frame, based on the tracked target object information. The tracked target object information includes information related to the detected and tracked target object from the current subsequent frame. The tracked object information may include the detected object bounding box for the target object, the position of the bounding box, and the center point (or other point) of the bounding box. The point determined for the target object may be the same point as the point determined for the target object in the initial frame. For example, if the point determined in block 1306 for the target object in the initial frame is the center point of the object or ROI, the point determined in block 1320 for the target object in the current subsequent frame may also include the center point of the object or ROI.

[0217] In block 1322, process 1310 includes determining or calculating a step value for a point of an object (e.g., the center point) determined in block 1320 and a step value for a scaling ratio determined in block 1320. In an example for illustration, the step value for the x coordinate of the point can be determined as diff_x = (curr_x - prev_x) / frame_count, which is a linear step function. The term frame_count may be a constant integer and can be defined as any suitable value (e.g., 1, 2, 3, or other suitable integer values). Using the linear step function, the step count can be determined as the difference between the x coordinate of the center point of the target object in the current subsequent frame and the x coordinate of the center point of the target object in the previous frame (e.g., the immediate previous frame before the current subsequent frame of the video), divided by the frame count. In another example for illustration, the step value for the y coordinate of the point can be determined as diff_y = (curr_y - prev_y) / frame_count, similar to what is used for the x coordinate. In another example for illustration, the step value for the scaling ratio can be determined as diff_zoom = (curr_ratio - prev_ratio) / frame_count. For example, the step count for the scaling ratio can be determined by dividing the difference between the scaling ratio of the target object in the current subsequent frame and the scaling ratio of the target object in the previous frame (e.g., the immediate previous frame before the current subsequent frame of the video) by the frame count.

[0218] In block 1324, process 1310 includes obtaining sensor frame metadata from the sensor frame metadata engine 813. As described above, the sensor frame metadata can represent output image information and has the same image resolution as the output image. The image metadata frame has the same aspect ratio as the preview frame but a higher resolution.

[0219] In block 1326, process 1310 includes determining an updated scaling ratio and an updated point (e.g., a center point) based on a step value (determined using, for example, the linear step function described above). The step value is calculated from a linear step, and the parameter progresses from a start value to a stop value using the number of steps in a linearly spaced sequence. The number of steps to be executed always becomes the parameter entered in the steps number field.

[0220] In block 1328, process 1310 includes performing scaling ratio smoothing and / or bounding box point trajectory smoothing operations based on the output from block 1320 (the object scaling ratio and point determined for an object in the current subsequent frame) and the output from block 1326 (the updated object scaling ratio and point). For example, the smoothing engine 812 can determine a smoothed value for the scaling ratio by performing smoothing of the scaling ratio. In another example, the smoothing engine 812 can determine a smoothed value for the center point of the ROI or object by performing trajectory smoothing of the center point of the bounding box. As described above, the scaling ratio smoothing operation smooths the change in the size of the bounding box (e.g., the change in size of the diagonal length), enabling the size of the target object in the image to gradually change between frames. The bounding box point trajectory smoothing operation enables the object to gradually move between frames (e.g., based on the center point of the object).

[0221] In some examples, the smoothing engine 812 can perform scaling ratio smoothing and / or bounding box point trajectory smoothing operations using the moving average algorithm described above. In some examples, the smoothing engine 812 can use a Gaussian filter function for scaling ratio smoothing. Figure 14 is a graph 1400 showing an example of a Gaussian filter smoothing function. For example, a Gaussian filter function with a window size of N can be used, where N represents an empirical threshold that can be set to any appropriate value such as N=31 or other values. An example for illustrating the Gaussian filter smoothing function is shown below (window size N is shown as window_size). function f = gaussian(window_size) sigma = double(window_size) / 5; h = exp(-((1:window_size) - ceil(window _size / 2)) .^2 / (2* sigma ^2)); f = h(:) / sum(h) end

[0222] In some examples, the smoothing engine 812 can use a median filter function with a window size M for scaling ratio smoothing. In some examples, the smoothing engine 812 can use a Fibonacci sequence filter function with a window size M for scaling ratio smoothing. M represents an empirical threshold that can be set to any appropriate value, such as M=31 or other values. Figure 15 is a graph 1500 showing the Fibonacci filter smoothing function. An example illustrating the Fibonacci filter smoothing function is shown below. M=window size F0=0, F1=1 F M =F M-1 +F M-2

[0223] In block 1330, process 1310 includes updating the region of the current subsequent frame for zooming. For example, process 1310 may send the region as zoom information (e.g., a zoom rectangle for upscaling or upsampling as the final output frame) to a camera pipeline such as an image acquisition device 105A including image sensor 801, image sensor 130, and one or more zoom control mechanisms 125C. In one example, one or more zoom control mechanisms 125C of the image acquisition device 105A can use the zoom information (region for zooming) to crop and scale the captured frame so that the object has a desired zoom level. An example illustrating the information is given below. curr_ratio += diff_zoom curr_x += diff_x curr_y += diff_y

[0224] Here, curr_ratio is the zoom ratio value of the previous frame, and curr_x and curr_y are the x and y coordinates relative to the position of the center point of the previous frame, respectively. The symbols diff_zoom, diff_x, and diff_y are the camera zoom ratio and the step value of the position of the center point of the current frame.

[0225] In block 1332, process 1300 outputs a frame that has been cropped and scaled from the original captured frame so that the target object is maintained at the same size as the target object in the initial frame.

[0226] In some cases, automatic zoom operation may be performed based on audio analysis, in addition to or instead of using one or more of the techniques described above. For example, by analyzing audio data associated with the video, the system may automatically focus on a prominent or target object that is emitting sound. In some cases, the audio source may be automatically amplified and concentrated on the prominent object along with the camera zoom. In some cases, background noise can be removed. For example, if a user is recording video of a person during a performance, zooming in on that person may enhance their voice (e.g., by increasing the volume, removing background noise, etc.). Such techniques may be used to produce or record video in which the size of the target object is stable at a specific point (e.g., the center point) in one or more video frames. Such techniques may be applied in real-time video recording and / or other use cases.

[0227] Figures 13C and 13D show examples of processes 1340 and 1350 for performing an auto-zooming behavior based on audio analysis. Referring to Figure 13C, in block 1342, process 1340 includes acquiring the initial (or first) audio-video source. The initial audio-video source may include video frame and audio information related to the video frame.

[0228] In block 1344, process 1340 performs visual processing, processing video data from a first audio-video source to detect one or more target object candidates. For example, one or more target objects may be detected in a given frame. Visual processing may include detecting one or more prominent objects (e.g., object of interest candidates) from the video frame. In block 1346, process 1340 performs audio processing, processing audio data from a first audio-video source to detect sounds associated with target objects. Audio processing may include audio recognition and / or classification to recognize and / or classify audio associated with the video frame. In an example for explanation, the visual and audio processing may be performed using a deep learning neural network (e.g., deep learning network 4200 in Figure 42, convolutional neural network 4300 in Figure 43, or other deep neural networks). In such an example, the input is video (audio-video source), and the neural network output is an image highlighted with at least one object making a sound.

[0229] In block 1347, process 1340 includes determining, based on audio processing, whether the detected object candidate (detected based on visual processing) is making a sound. If it is determined that the object candidate is making a sound, process 1340 in block 1348 may include determining or setting points and distances on the object bounding box, similar to block 1306 of process 1300 in Figure 13A. In an example for explanation, the points may include the center point of the bounding box, and the distances may be diagonal lengths of the bounding box (for example, the length from the bottom left point of the bounding box to the top right point of the bounding box, or the length from the bottom right point of the bounding box to the top left point of the bounding box). Other points and / or distances may be used as described above. In block 1347, if it is determined that the target object candidate is not making a sound, the following target object candidates may be analyzed in relation to whether the object is making any sound. Similar to what is described with respect to Figure 13A, by setting the bounding box points (e.g., the center point of the object) and distances (e.g., the diagonal lengths), process 1340 can initialize target object information, including the coordinates of the object's center point, the diagonal lengths of the object's bounding box, and the current zoom ratio relative to the object.

[0230] Referring to Figure 13D, process 1350 is similar to process 1310 in Figure 13B and further includes audio processing operations (in blocks 1352, 1354, 1356, 1358, and 1360). Blocks in Figure 13D with the same numbering as in Figure 13B are described above with respect to Figure 13B. In block 1352, process 1350 includes performing audio three-dimensional (3D) localization. 3D sound source localization refers to acoustic techniques used to locate sound sources in 3D space. The position of the sound source may be determined by the orientation of the incoming sound waves (e.g., horizontal and vertical angles) and the distance between the sound source and the sensor. Once the 3D localization of the audio is performed, process 1350 proceeds to block 1332, which outputs a cropped and scaled frame as described above with respect to Figure 13B.

[0231] In block 1354, process 1300 includes acquiring subsequent audio, which may be audio related to one or more subsequent frames acquired in block 1312. In block 1356, process 1300 includes updating by zooming in on an audio source and amplifying its sound.

[0232] In block 1358, process 1300 includes performing background noise reduction. Audio background noise, such as crumpling paper, typing on a keyboard, the sound of a ventilation fan, dog barking, and other noises, impairs the auditory perception of audio signals. Removing audio background noise can help eliminate distracting noise, which in turn creates a better audio experience. In block 1360, process 1300 includes outputting audio related to the frame output in block 1332.

[0233] Figure 16 shows the zoom process in a camera pipeline (e.g., an image acquisition device 105A including an image sensor 801, an image sensor 130, and one or more zoom control mechanisms 125C). As shown, the image acquisition device can stream output frames (e.g., by default) at a zoom ratio of 1.0x (referring to zoom 0 or no zoom). A zoom region of interest (ROI) 1604 (also called a cropped rectangle or zoom rectangle) is shown in frame 1602 with a zoom ratio of 1.0x. For example, as described above, the ROI determination engine 804 in Figure 8B can determine an initial region of interest (ROI) in a particular frame based on and / or automatically on user input. In an example for one explanation, the user can provide user input that defines the zooming ROI 1604, including the position and size of the rectangle. As shown, the zooming ROI 1604 is cropped from frame 1602. When cropped from frame 1602, the zooming ROI 1604 is upscaled or upsampled for the output stream (shown as the upscaled frame 1606).

[0234] Figure 17 shows the zoom latency for a camera pipeline with a 7-frame latency for a zoom request. The example shown in Figure 17 represents a 7-frame latency, in which case a zoom request made in a given frame is applied 7 frames later. For example, for a request 1702 for a 1.1x zoom based on frame 1 in Figure 17, the corresponding zoom adjustment is applied 7 frames later in frame 8. As shown, frame 8 has a zoom amount of 1.1. The zoom increment can be adjusted frame by frame. For example, a request 1704 for a 1.2x zoom may be made based on frame 2, and the corresponding zoom adjustment is applied 7 frames later in frame 9 (shown as having a zoom amount of 1.2). A request 1706 for a 1.8x zoom may be made based on frame 4, and the corresponding zoom adjustment is applied 7 frames later in frame 11 (shown as having a zoom amount of 1.8).

[0235] Several advantages are achieved by using the frame cropping and scaling techniques described above. For example, cropping and scaling techniques enable the feature of providing a fixed size for target objects in a video system (e.g., mobile devices, video analysis systems, etc.). Systems implementing cropping and scaling techniques can achieve good performance and can be deployed in any type of device, such as mobile devices and IP cameras, etc.

[0236] FIG. 18 is a flowchart showing an example of a process 1800 for processing one or more frames using the techniques described herein. At block 1802, process 1800 includes determining a region of interest in a first frame of a sequence of frames. The region of interest in the first frame includes an object having a certain size in the first frame. As described above, the region of interest may be determined based on user input or may be determined automatically. In some examples, process 1800 includes receiving user input corresponding to the selection of an object in the first frame and determining the region of interest in the first frame based on the received user input. In some aspects, the user input includes touch input (e.g., selecting an object, drawing a shape around an object, etc.) provided using a touch interface of the device. As described herein, the user input may include other types of user input.

[0237] At block 1804, process 1800 includes cropping a portion of a second frame of the sequence of frames, where the second frame is after the first frame in the sequence of frames. At block 1806, process 1800 includes scaling that portion of the second frame based on the size of the object in the first frame. For example, by scaling that portion of the second frame based on the size of the object in the first frame, the object in the second frame will have the same size as the object in the first frame. In some examples, cropping and scaling the portion of the second frame maintains the object at the center of the second frame. In some cases, process 1800 includes detecting and tracking an object in one or more frames of the sequence of frames.

[0238] In some examples, process 1800 includes determining a point in the object region determined for an object in a second frame, and cropping and scaling that portion of the second frame so that the point in the object region is the center of the cropped and scaled portion. In some cases, the point in the object region is the center point of the object region. In some cases, the object region is a bounding box (or other bounding region). In some cases, the center point is the center point of the bounding box (or other region). In some cases, the center point is the center point of the object (e.g., the centroid or center point of the object). The center point can be found by performing object segmentation (e.g., using system 600 shown in Figure 6A).

[0239] In some embodiments, process 1800 includes determining a first length relating to an object in a first frame and determining a second length relating to an object in a second frame. Process 1800 may also include determining a scaling factor based on a comparison of the first and second lengths and scaling that portion of the second frame based on the scaling factor. In some cases, scaling that portion of the second frame based on the scaling factor results in the second object region in the cropped and scaled portion having the same size as the first object region in the first frame. In some examples, the first length is the length of the first object region determined for the object in the first frame, and the second length is the length of the second object region determined for the object in the second frame. In some cases, the first object region is a first bounding box (or other bounding region), and the second object region is a second bounding box (or other bounding region). The first length may be the diagonal length (or other length) of the first bounding box, and the second length may be the diagonal length (or other length) of the second bounding box. In some cases, the first length may be the distance between points of an object in the first frame, and the second length may be the distance between points of an object in the second frame.

[0240] In some embodiments, process 1800 includes determining points of a first object region to be generated for an object in a first frame, and determining points of a second object region to be generated for an object in a second frame. In some implementations, the points of the first object region are the center points of the first object region (the center points of the object in the first frame or the center points of the first bounding box), and the points of the second object region are the center points of the second object region (e.g., the center points of the object in the second frame or the center points of the second bounding box). Process 1800 may include determining motion coefficients for the object based on a smoothing function using the points of the first and second object regions. The smoothing function can control changes in the object's location over multiple frames in a sequence of frames. For example, the smoothing function can control changes in the object's location so that it changes gradually over multiple frames in a sequence of frames (e.g., so that the change does not exceed a threshold change in location, such as a change of 5 pixels, 10 pixels, or another threshold change in location). In some examples, the smoothing function includes a moving function (e.g., a moving average function or other moving function) used to determine the position of points in each of multiple frames of a sequence of frames, based on a statistical measure of the object's motion (e.g., average, mean, standard deviation, variance, or other statistical measure). In an example for one explanation, the smoothing function includes a moving average function used to determine the average position of points in each of multiple frames of the object. For example, as described above, a moving average can reduce or eliminate false alarms (e.g., by minimizing the position of an object bounding box that is falsely detected / tracked). Process 1800 may include cropping that portion of a second frame based on a motion coefficient.

[0241] In some examples, process 1800 includes determining a first length related to an object in a first frame and a second length related to an object in a second frame. In some examples, the first length is the length of a first bounding box generated for the object in the first frame, and the second length is the length of a second bounding box generated for the object in the second frame. In some cases, the first length is the diagonal length of the first bounding box, and the second length is the diagonal length of the second bounding box. Process 1800 may include determining a scaling factor for the object based on a comparison of the first and second lengths and on a smoothing function using the first and second lengths. The smoothing function can control the change in the size of the object across multiple frames in a sequence of frames. For example, a smoothing function can control the change in the size of an object so that its size changes gradually across multiple frames in a sequence of frames (for example, so that the change does not exceed a threshold change in size, such as a change greater than 5%, 10%, 20%, or other threshold changes in size). In some cases, the smoothing function includes a moving function (e.g., a moving average function or other moving function) used to determine the length associated with the object in each of multiple frames in a sequence of frames based on a statistical measure of the object's size (e.g., average, mean, standard deviation, variance, or other statistical measure). In an example for one explanation, the smoothing function includes a moving average function used to determine the average length associated with the object in each of multiple frames. For example, as described above, a moving average can reduce or eliminate false alarms (e.g., by minimizing the diagonal length of a falsely detected / tracked object bounding box, or the falsely detected / tracked length between parts of an object). Process 1800 may include scaling that part of a second frame based on a scaling factor.In some embodiments, by scaling that portion of the second frame based on a scaling factor, the second bounding box in the cropped and scaled portion will have the same size as the first bounding box in the first frame.

[0242] Figures 19, 20, 21, 22, and 23 illustrate the simulations performed on four video clips. All video clips include 720p and 1080p resolutions and are captured at 30 frames per second (fps). Each of the examples in Figures 19 through 23 is an example to illustrate the zoom-in effect (frames are cropped and upsampled or upscaled, similar to the examples in Figures 10A and 10B). As shown in Figure 19, “Original Frame 0” is the first frame from the video, and “Original Frame X” is the current frame during the video recording. To achieve the zoom-in effect, the frame cropping and scaling system 800 crops a region from the original frame X and then upsamples the region to the original frame size.

[0243] As stated above, a device may include multiple cameras and / or lenses (e.g., two cameras in a dual-camera lens system) to perform one or more dual-camera mode functions. For example, the dual-camera lenses of a device (e.g., a mobile phone or smartphone, including a rear dual-camera lens or other dual-camera lenses) may be used to record multiple videos (e.g., two videos) simultaneously, which may be called a “dual-camera video recording” function. In some cases, the primary camera lens of the device’s dual-camera lens (e.g., a telephoto lens) may capture (and / or record) the first video, and the secondary camera lens of the dual-camera lens (e.g., a zoom lens such as a wide-angle lens) may capture (and / or record) the second video. In some cases, the second video may be used to perform the frame cropping and scaling techniques described above, so as to keep the size of the target object constant between videos. In some cases, such a solution may be used as a video post-processing function (e.g., processing the output image by the ISP before it is displayed or stored).

[0244] In some cases, the dual-camera mode feature can be implemented using two camera lenses of a device simultaneously, such as the device's primary camera lens (e.g., a telephoto lens) and secondary camera lens (e.g., a zoom lens). The dual-camera video recording feature described above allows two camera lenses to record two videos simultaneously. For example, a device can record separate videos using a wide-angle lens and a standard lens. In some cases, a device can record videos simultaneously using three, four, or more camera lenses. The videos can then be displayed (e.g., simultaneously displayed), stored, sent to another device, and / or used in other ways. For example, using the dual-camera mode feature (e.g., dual-camera video recording), a device can display two viewpoints of a scene (e.g., split-screen video) on a display at the same time. Among other things, the dual-camera mode feature offers various advantages, such as allowing a device to capture a wide-angle view of a scene (e.g., with more background and surrounding objects in the scene), or allowing a device to capture a large event or a complete view of a scene.

[0245] In some cases, multiple camera modules and lenses may be used to perform the zoom function. For example, a secondary camera lens can be set to a higher zoom level (e.g., a 2.0x camera zoom ratio) compared to the primary camera and / or lens (which may have a 1.0x camera zoom ratio).

[0246] Various problems can arise regarding maintaining a constant size of a target object within a sequence of frames. For example, when a target object moves toward the device's camera, the device may be unable to perform a zoom-out effect. Such problems can stem from limitations in the field of view from the original video frame. For instance, there may not be enough space in the frame to zoom out sufficiently while maintaining the target object's size (e.g., black space appears around the scaled frame). In another example, when a target object moves away from the device's camera, the zoomed-in image generated based on the original video frame may be of low quality, such as being blurry, containing one or more visual artifacts, or lacking clarity. Furthermore, devices implementing dual-camera mode functionality do not incorporate any artificial intelligence technology. Such systems require the end user to use video editing tools or software applications to manually edit the images.

[0247] As described above, systems and techniques for switching cameras or lenses of devices capable of implementing one or more of the dual-camera mode functions described above are described herein. While systems and techniques are described herein in relation to dual-camera systems or two-camera systems, systems and techniques may be applied to systems having more than two cameras (for example, three cameras, four cameras, or any other number of cameras used to capture images or video). In some cases, the systems and techniques described herein can be used in a dual-camera system to maintain a constant size of a target object in a sequence of video frames captured using the dual-camera system. In some examples, systems and techniques can perform dual-camera zoom, which may be used to produce a more detailed object zoom effect.

[0248] As described above, an object detection and tracking system (e.g., object detection and tracking system 806) can detect and / or track objects in one or more frames. The object detection and tracking system can use any suitable object detection and tracking technique for the multi-camera (e.g., dual-camera) implementations described herein, such as those described above. In some cases, as described above, a region of interest (ROI) or target object may be identified based on user input or automatically.

[0249] In some examples, object detection and tracking systems can detect and / or track objects in a frame by performing object matching for dual-camera video analysis using machine learning object detection and tracking systems. For example, an object detection and tracking system can extract points of interest from one or more input frames. Points of interest may include stable two-dimensional (2D) locations in a frame that are reproducible under different lighting conditions and viewpoints. Points of interest may also be called keypoints or landmarks (e.g., facial landmarks on a person's face). An example of a machine learning system is a convolutional neural network (CNN). In some cases, CNNs can outperform human-processed representations for various tasks that use frames or images as input. For example, CNNs may be used to predict 2D keypoints or landmarks for various tasks such as object detection and / or tracking.

[0250] Figure 24 shows an example of a machine learning-based object detection and tracking system 2400. In some cases, the system 2400 is self-managed using self-training (rather than using human intervention to define points of interest in real training images). Object tracking is then performed by point mapping using point feature matching. In some cases, a large dataset of pseudo-ground truth locations of points of interest in real images or frames is used, which may be pre-configured or pre-set by the system 2400 itself, rather than requiring extensive human annotation effort.

[0251] System 2400 comprises a fully convolutional neural network architecture. In some examples, System 2400 takes one or more full-size images as input and operates on them. For example, a pair of images, including images 2402 and 2404, as shown in Figure 24, may be input to the system (e.g., during training and / or during inference). In some cases, System 2400 (using full-size images as input) produces point-of-interest detection along with fixed-length descriptors in a single forward pass. The neural network model of System 2400 includes a single shared encoder 2406 (shown as having four convolutional layers, but may contain more or fewer layers) to handle and reduce the dimensionality of the input images. After the encoder, the neural network architecture splits into two decoder heads, which learn task-specific weights. For example, the first decoder head 2408 is trained for point-of-interest detection, and the second decoder head 2410 is trained to generate descriptions of points of interest (called descriptors). The task of finding points of interest may involve detection and description (performed, for example, by decoder heads 2408 and 2410, respectively). Detection is the positioning of points of interest within an image or frame, while description is the description of each detected point (for example, using vectors). The overall goal of system 2400 is to effectively and efficiently find distinctive and stable visual features.

[0252] In some cases, system 2400 can distort each region of pixels in the input image (e.g., each 8x8 pixel region). After distortion, this region can be treated as a single pixel, in which case each region of the pixel may be represented by a specific pixel in a feature map with 64 channels followed by one dustbin channel. If no points of interest (e.g., keypoints) are detected in a particular 8x8 region, the dustbin may have high activation. If keypoints are detected in an 8x8 region, the other 64 channels can be routed through a softmax architecture to find the keypoints in the 8x8 region. In some cases, system 2400 can compute 2D points and descriptors of points of interest in a single forward pass, which can be performed at 70 frames per second (fps) for a 480x640 image using a Titan X graphics processing unit (GPU).

[0253] Figure 25 is a flowchart illustrating an example of a camera lens switching pipeline 2500. Pipeline 2500 is an example of dual camera lens switching logic. The first lens, as referenced in Figure 25, is the lens and / or camera that the device uses (for example, based on user input) as the primary lens for capturing video. In some cases, the first lens may be a telephoto lens, and the second lens, as referenced in Figure 25, may be a wide-angle lens. Any other type of lens may be used for the first and second lenses.

[0254] In block 2502, the target size lock function can be started with a first lens (for example, the telephoto lens if the user selects a telephoto lens as the primary lens for recording video). When certain conditions are met (as described below), blocks 2504 and 2508 of pipeline 2500 can switch the primary lens for performing the target size lock function from the first lens (e.g., the telephoto lens) to a second lens (e.g., the wide-angle lens). In such a case, the second lens may be used to capture one or more major video frames. When certain conditions are met (as described below), blocks 2506 and 2510 of pipeline 2500 can switch the primary lens for performing the target size lock function back from the second lens (e.g., the wide-angle lens) to the first lens (e.g., the telephoto lens). In such a case, the first lens may be used to capture any major video frames.

[0255] An example of an algorithm (called Algorithm 1A) that can be used to perform camera lens switching is as follows (using a telephoto lens as an example of the first lens and a wide-angle lens as an example of the second lens): Initialize disp_xy based on the x and y displacements of the center of the target object's bounding box from the center point of the first or initial frame. Initialize done_by_tele as true and tele_lens_ratio as 1.0. Initialize the camera's zooming_ratio value for telephoto and wide-angle lenses.

[0256] When the term done_by_tele is true (for example, it can be assigned a value of 1), a telephoto lens is used to perform the target size fixing function. zooming_ratio is the scaling (or zoom) ratio described above and is used to determine how much to scale the ROI or object in the input frame.

[0257] In some cases, the above camera lens switching algorithm can be continued as follows (called Algorithm 1B): For each iteration of the input video frame #Option 1) Start with the first video frame, from the telephoto camera lens. #Option 2) Continue using the telephoto lens #Option 3) Switch from a wide-angle lens to a telephoto lens. if tele_zooming_ratio 1.0 The width and height of the object bounding box are changed based on the tele_zooming_ratio. Move the position of an object if the width or height of the object's bounding box is outside the image. #Switching to a wide-angle lens done_by_tele==False skip Processes cropping and resizing of video frames. Update disp_xy_displacement done_by_tele=True Set wide_lens_tirnes_ratio=1.0 else done_by_tele=False #Option 1) Continue using a wide-angle lens. #Option 2) Switch from a telephoto lens to a wide-angle lens. if done_by_tele==False if previous iterations were performed with a telephoto lens Updating the wide-angle lens time ratio. if wide_lens_times_ratio*wide_zooming_ratio 1.0 if disp_xy != 0 Update disp_xy_displacement Processes cropping and resizing of video frames. else Preserve the original video frames without cropping or resizing.

[0258] Figure 26 is a flowchart illustrating an example of a camera lens switching process 2600. In block 2602, for a primary video containing frames captured using the first camera lens, process 2600 can perform video frame selection from the video captured using the first lens (e.g., a telephoto camera lens). For example, a user can select a video frame as the first frame to be used as a starting point for performing a target size fixing function. For example, as described above, the first frame may be used to define the size, point (e.g., the center point of the bounding box), and distance (e.g., the diagonal length of the bounding box) of the ROI and / or target object. In block 2603, process 2600 can determine or locate a corresponding video frame from the video captured or recorded using the second camera lens (e.g., a wide-angle camera lens). In an example for explanation, the video frames from the first and second camera lenses may have reference numbers that correspond to the output time for those frames. Process 2600 (in block 2603) can determine the corresponding video frame using the frame reference number. The first camera lens is shown in Figure 26 as a telephoto lens (also referred to herein as a "tele lens"), and the second lens is shown in Figure 26 as a wide-angle camera lens. However, those skilled in the art will understand that the first and second lenses could be any other suitable lenses.

[0259] In block 2604, process 2600 includes selecting and / or drawing a bounding box (or other bounding region) for a target object in a first video frame. For example, from a first video frame, a user can select a target object (e.g., a single object or multiple objects in some cases) by providing user input (e.g., tapping the target object in the frame displayed on a touchscreen display, drawing a bounding box around the target object, or providing any other appropriate type of input). In another example, the target object may be determined automatically using the techniques described above (e.g., by the object detection and tracking system 806).

[0260] In block 2605, process 2600 determines or finds the same target object in the corresponding video frame of the video captured using the second lens determined in block 2603. In some cases, in order to find the same object from the video captured using the second lens, process 2600 can determine the approximate location of the target object from the video captured using the first lens (for example, using the object detection and tracking system 806). Process 2600 can then apply an object matching algorithm (for example, using system 2400 in Figure 24) to locate the target object in the video captured using the second lens, and the target object may be associated with a bounding box and information.

[0261] In block 2606, process 2600 can perform object detection and tracking. In some cases, object detection and tracking may be similar to the object detection and tracking described above with respect to Figures 8B to 13B. For example, the object detection and tracking system 806 can automatically detect and track an object in parallel in two videos (a video captured by a first lens and a video captured by a second lens). In block 2608, process 2600 determines or captures the coordinates (e.g., center point) and distances (e.g., diagonal lengths) of points (e.g., center point) and distances (e.g., diagonal lengths) of the bounding box determined for the target object (e.g., by the object detection and tracking system 806) across the frames of the two videos. In some cases, the points (e.g., center point) and distances (e.g., diagonal lengths) may be stored for later use by process 2600.

[0262] In block 2610, process 2600 applies a smoothing function. For example, as described above, the smoothing engine 812 may apply a smoothing function to smooth the frame scaling ratio (or resize ratio). The scaling ratio or resize ratio may be calculated by comparing the diagonal length (or other distance) of the target object bounding box in a first selected video frame with the diagonal length (or other distance) of the target object bounding box in the current frame. As described above, the smoothing function may include a moving average function in some cases. For example, the smoothing function may be used to determine the average length related to an object in each of multiple frames in a sequence of frames.

[0263] In block 2612, process 2600 can decide whether to perform a camera lens switch. For example, process 2600 can decide whether to use video frames from a first lens (e.g., a telephoto lens) or from a second lens (e.g., a wide-angle lens) using the camera lens switch algorithms described above (e.g., algorithm 1A and / or algorithm 1B). In block 2614, process 2600 can perform frame cropping and scaling (or zooming). For example, the frame scaling engine 810 can upsample (or upscale) the ROI (e.g., bounding box) of a target object based on the coordinates of points in the object bounding box (e.g., the coordinates of the center point) and the scaling ratio or resize ratio. In block 2616, process 2600 performs video stabilization, such as using the image stabilization techniques described above with respect to Figure 12. In block 2618, process 2600 outputs a frame that has been cropped and scaled from the original captured frame so that the target object is maintained at the same size as the target object in the initial frame or first frame.

[0264] In some cases, as stated above, the camera lens switching systems and techniques described herein may be applied to or extended to other multi-camera systems that record multiple images and / or videos simultaneously (for example, camera systems including three, four, or five cameras).

[0265] In some examples, a move-step algorithm may be used to achieve a smoothed effect. In some cases, the techniques described may be used with respect to move-step values ​​(for example, as described with respect to Figure 13B). An example illustrating the move-step algorithm is given below. (1) Operation 1: Initialize the target object coordinates center_xy in the output frame as (w / 2, h / 2), where w is the width of the output frame and h is the height of the output frame. (2) Operation 2: Update center_xy when the lens is maintained as a telephoto lens (for example, as shown in Figures 32 and 33 described below) or when the lens is maintained as a wide-angle lens (for example, as shown in Figures 34, 35, and 36 described below). (3) Operation 3: When the example in Operation 2 changes (switching from a telephoto lens to a wide-angle lens, or from a wide-angle lens to a telephoto lens), update the target object coordinates to (center_xy[1]±moving_step, center_xy[2]±moving_step) and apply these target object coordinates to the output frame. (4) Operation 4: From Operation 3, update center_xy using moving_step to bring it closer to (w / 2, h / 2). (5) Repeat operations 3 and 4 until operation 5: center_xy=(w / 2, h / 2)

[0266] Figures 27 to 36 illustrate examples of using the camera lens switching technique described above. The examples in Figures 27 to 36 are illustrated using a telephoto camera lens (shown as “Telephoto Frame”) as an example of a lens selected (for example, by the user) as the primary lens, and a wide-angle lens (shown as “Wide-Angle Frame”) as an example of a secondary lens.

[0267] Figure 27 shows an example of lens selection. For example, when the size of the target object in the current telephoto frame 2704 (shown as telephoto frame N) is smaller than the size of the target object in the reference telephoto frame 2702, the device or system may decide to use the telephoto lens frame 2704 to produce the result of the output frame (for example, in block 2612 in Figure 26). Figure 28 is another diagram showing an example of lens selection. For example, when the size of the target object in the current telephoto frame 2804 (shown as telephoto frame M) is larger than the size of the object in the reference telephoto frame 2802, the device or system may decide to use the wide-angle lens frame 2806 to produce the result of the output frame (shown as wide-angle frame M to indicate that wide-angle frame M and telephoto frame M are captured from the same angle and at the same time with respect to the camera). Figure 29 is another diagram showing an example of lens selection. For example, if the size of the target object in the current wide-angle frame 2904 (shown as wide-angle frame P) is larger than the size of the target object in the reference wide-angle frame 2902, the device or system may decide to use the current wide-angle frame 2904 to generate the result of the output frame.

[0268] Figure 30 shows an example of switching from a telephoto lens to a wide-angle lens. For example, if output frame N is generated by telephoto frame N, and the size of the target object in the current telephoto frame 3004 (shown as telephoto frame N+1) is larger than the size of the object in reference telephoto frame 3002, the device or system can switch to wide-angle frame 3006 (shown as wide-angle frame N+1) to generate output frame 3008.

[0269] Figure 31 is another diagram illustrating an example of switching from a telephoto lens to a wide-angle lens. For example, if output frame N is generated by telephoto frame N, and the target's position in the current telephoto frame 3104 (shown as telephoto frame N+1) is close to the frame boundary (for example, in this case the object is not centered in the frame after scaling or zooming), the device or system may switch from the telephoto frame to the wide-angle frame to generate output frame 3108. In some cases, the device or system may determine if an object is close to the frame boundary by determining whether a point of the target object (for example, the center point of the object's bounding box) is within a threshold distance, such as within 10 pixels, 20 pixels, or other distances from the boundary. Even if the size of the target object in the current telephoto frame 3104 is smaller than the size of the target object in the reference telephoto frame 3102, if the object is close to the boundary, a switch from the current telephoto frame 3104 (captured using the telephoto lens) to the wide-angle frame 3106 (captured using the wide-angle lens) may be performed.

[0270] Figure 32 shows an example of switching from a wide-angle lens to a telephoto lens. For example, if output frame N is generated by wide-angle frame N, and the size of the target object in the current wide-angle frame 3206 (shown as wide-angle frame N+1) is smaller than the size of the object in the reference telephoto frame 3202, and the position of the target object is within the image boundary after zooming in, the device or system can switch from the current wide-angle frame 3206 to the current telephoto frame 3204 (shown as telephoto frame N+1) to generate output frame 3208.

[0271] Referring again to Figure 32, an example is given in which the use of a telephoto lens is maintained. For example, if the output frame N is generated by the wide-angle frame N, and the size of the target object in the current telephoto frame 3204 (telephoto frame N+1) is smaller than the size of the target object in the reference telephoto frame 3202, and the position of the target object is within the image boundary after zooming in, the device or system can continue to use the telephoto frame 3204 to generate the output frame 3208.

[0272] Figure 33 shows another example of maintaining the use of a telephoto lens. For example, starting from the current telephoto frame 3304 (shown as telephoto frame N), if the size of the target object in the current telephoto frame 3304 (telephoto frame N) is smaller than the size of the target object in the reference telephoto frame 3302, the object's position is close to the frame boundary (e.g., the center point or bounding box of the target object is within a threshold distance from the boundary), and no camera lens switch occurred within a threshold period (e.g., no camera lens switch occurred within a certain number of frames, a certain length of time, and / or other periods), the device or system can continue to use telephoto frame 3304 to generate output frame 3308.

[0273] Figure 34 shows an example of maintaining the use of a wide-angle lens. For example, if the size of the target object in the current wide-angle frame 3408 (shown as wide-angle frame N) is smaller than the size of the target object in the reference wide-angle frame 3404, and the size of the object in the current telephoto frame 3406 (shown as telephoto frame N) is larger than the size of the object in the reference telephoto frame 3402, the device or system can continue to use the current wide-angle frame 3408 to generate the output frame 3410.

[0274] Figure 35 shows another example of maintaining the use of a wide-angle lens. For example, if the size of the target object in the current wide-angle frame 3506 (shown as wide-angle frame M) is larger than the size of the target object in the reference wide-angle frame 3502, the device or system may continue to use the current wide-angle frame 3506 to generate the output frame 3510.

[0275] Figure 36 shows another example of maintaining the use of a wide-angle lens. For example, if the output frame N is generated by a wide-angle frame N, and the position of the target object in the current telephoto frame 3604 (shown as telephoto frame N+1) is close to the frame boundary (e.g., the center point or bounding box of the target object is within a threshold distance from the boundary), the frame may not be able to be scaled or zoomed to obtain the output frame, and the device or system may continue to use the current wide-angle lens frame 3606 (shown as wide-angle lens N+1) to generate the output frame 3608. The device or system may also continue to use the current wide-angle lens frame 3606 (shown as wide-angle lens N+1) to generate the output frame 3608 if the size of the target object in the current telephoto frame 3604 is smaller than the size of the target object in the reference telephoto frame 3602, as long as the object is close to the boundary.

[0276] Figures 37 to 41 are images illustrating simulations using the camera lens switching system and techniques described herein. For example, two rear cameras of a mobile phone (e.g., a smartphone), including a telephoto camera lens and a wide-angle camera lens, are used to simultaneously simulate dual-camera video recording. The start and end points of this dual video recording are manually aligned from the dual camera lenses. The test sample video used in the simulation results has a resolution of 1080P at 30 frames per second (fps). As mentioned above, the end user can select a target object from the telephoto lens video frame (e.g., using a touchscreen that displays the video frame).

[0277] Figure 37 shows the first or initial video frame from the telephoto lens (left side of Figure 37) and the first or initial video frame from the wide-angle lens (right side of Figure 37). Figure 38 shows the last video frame from the telephoto lens (left side of Figure 38) and the last video frame from the wide-angle lens (right side of Figure 38). Figure 39 shows the target size lock function applied to the telephoto lens video frame at time point n (left side of Figure 39), and the target size lock function applied to the wide-angle lens video frame at time point n+1 (right side of Figure 39) after switching from the telephoto lens to the wide-angle lens. Figure 40 shows the target size lock function applied to the wide-angle lens video frame at time point m (left side of Figure 40), and the target size lock function applied to the telephoto lens video frame at time point m+1 (right side of Figure 40) after switching from the wide-angle lens to the telephoto lens. Figure 41 shows the target size lock function applied to the telephoto lens video frame at time point p (left side of Figure 41), and the target size lock function applied to the wide-angle lens video frame at time point p+1 (right side of Figure 41) after switching from the telephoto lens to the wide-angle lens.

[0278] The lens switching systems and techniques described herein offer various advantages. For example, they enable the use of the target size fixing feature described above in multiple video recording scenarios (for example, in dual video recording using two camera lenses) while achieving high-quality results.

[0279] In some examples, the processes described herein (for example, processes 820, 930, 1200, 1300, 1310, 1800, 2500, 2600, and / or other processes described herein) may be performed by a computing device or apparatus. In one example, one or more processes may be performed by the image acquisition and processing system 100 of Figure 1. In another example, one or more processes may be performed by the frame cropping and scaling system 800 of Figure 8B. In yet another example, one or more processes may be performed by the computing system 4700 shown in Figure 47. For example, a computing device with the computing system 4700 shown in Figure 47 may include components of a frame clipping and scaling system 800, and can perform the operations of process 820 in Figure 8C, process 930 in Figure 9A, process 935 in Figure 9B, process 1300 in Figure 13A, process 1310 in Figure 13B, process 1800 in Figure 18, and / or other processes described herein.

[0280] Computing devices may include any suitable device, such as a mobile device (e.g., a cell phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device for an autonomous vehicle, a robotic device, a television, and / or any other computing device having the resource capacity to perform the processes described herein, including processes 820, 930, 935, 1800, and / or other processes described herein. In some cases, a computing device or apparatus may include a variety of components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0281] Components of a computing device may be implemented in circuits. For example, components may include and / or be implemented using one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), as well as computer software, firmware, or any combination thereof, to perform the various operations described herein, and / or be implemented using them.

[0282] Processes 820, 930, 1200, 1300, 1310, 1800, 2500, and 2600 are presented as logical flow diagrams, and their operations represent sequences of operations that can be performed by hardware, computer instructions, or combinations thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored in one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which operations are described is not intended to be interpreted as limiting, and any number of operations described may be combined in any order and / or in parallel to implement a process.

[0283] In addition, processes 820, 930, 1200, 1300, 1310, 1800, 2500, 2600, and / or other processes described herein may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors by hardware or a combination thereof. As stated above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-temporary.

[0284] As stated above, various aspects of this disclosure can utilize machine learning systems, particularly for object tracking and object classification. Figure 42 is an example for illustrating a deep learning neural network 4200 that may be used to perform the machine learning-based object tracking and / or classification described above. The input layer 4220 contains input data. In an example for one illustration, the input layer 4220 may contain data representing pixels of an input video frame. The neural network 4200 includes several hidden layers 4222a, 4222b through 4222n. The hidden layers 4222a, 4222b through 4222n contain "n" hidden layers, where "n" is an integer greater than or equal to 1. The number of hidden layers may be made to include as many layers as required for a given application. The neural network 4200 further includes an output layer 4224 that provides the output resulting from the processing performed by the hidden layers 4222a, 4222b through 4222n. In one example for explanation, output layer 4224 can provide classification of objects in the input video frame. The classification may include a class that identifies the type of object (e.g., person, dog, cat, or other object).

[0285] The neural network 4200 is a multilayer neural network of interconnected nodes. Each node can represent information. The information associated with a node is shared between different layers, and each layer holds the information as it is processed. In some cases, the neural network 4200 can include a feedforward network, in which case there are no feedback connections such that the output of the network is fed back to itself. In some cases, the neural network 4200 can include a recurrent neural network, which may have loops that allow information to be carried across nodes while an input is being read.

[0286] Information can be exchanged between nodes through the interconnection of nodes in various layers. Nodes in input layer 4220 can activate a set of nodes in the first hidden layer 4222a. For example, as shown, each input node in input layer 4220 is connected to each node in the first hidden layer 4222a. Nodes in the first hidden layer 4222a can transform the information of each input node by applying an activation function to the input node information. The information derived from the transformation can then be passed to the nodes of the next hidden layer 4222b, which can activate it, allowing the next hidden layer 4222b to perform a specific designated function. Illustrative functions include convolution, upsampling, data transformation, and / or any other suitable function. The output of hidden layer 4222b can then activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 4222n can activate one or more nodes in output layer 4224, from which the output is provided. In some cases, a node in the neural network 4200 (for example, node 4226) is shown as having multiple output lines, while a node has a single output, and all lines shown as outputs from the node represent the same output value.

[0287] In some cases, each node or interconnection between nodes may have weights, which are a set of parameters derived from training the neural network 4200. Once the neural network 4200 is trained, it can be called a trained neural network and can be used to classify one or more objects. For example, interconnections between nodes can represent information learned about the interconnected nodes. Interconnections may have adjustable numerical weights that can be adjusted (for example, based on the training dataset), allowing the neural network 4200 to be adaptive to inputs and to learn as more data is processed.

[0288] The neural network 4200 is pre-trained to process features from the data in the input layer 4220, using different hidden layers 4222a, 4222b through 4222n to provide an output through the output layer 4224. In an example where the neural network 4200 is used to identify objects in an image, the neural network 4200 may be trained using training data that includes both images and labels. For example, training images can be input to the network, and each training image has a label that indicates the class of one or more objects in each image (essentially telling the network what the object is and what features the object has). In an example for one explanation, the training images may include an image of the number 2, in which case the label of the image may be [0 0 1 0 0 0 0 0 0 0].

[0289] In some cases, the neural network 4200 can adjust the node weights using a training process called backpropagation. Backpropagation may include a forward pass, loss function, backward pass, and weight updates. The forward pass, loss function, backward pass, and parameter updates are performed for each training iteration. This process can be repeated a certain number of times for each set of training images until the neural network 4200 is sufficiently trained so that the layer weights are precisely adjusted.

[0290] In an example of identifying objects in an image, the forward pass might involve passing the training image through the neural network 4200. Before the neural network 4200 is trained, the weights are first randomized. The image might contain, for example, an array of numbers representing the pixels of the image. Each number in the array might contain a value from 0 to 255 that describes the pixel intensity at that location in the array. In one example, the array might contain a 28x28x3 array of numbers with 28 rows and 28 columns of pixels and three color components (red, green, and blue, or lumens and two chroma components, for example).

[0291] In the initial training iterations for neural network 4200, the output is likely to contain values ​​that do not indicate a preference for any particular class, due to the random selection of weights during initialization. For example, if the output is a vector with probabilities of objects belonging to different classes, the probability values ​​for each of the different classes may be equal, or at least very similar (for example, for 10 possible classes, each class may have a probability value of 0.1). Using the initial weights, neural network 4200 cannot make an accurate determination of what the object's classification might be, as it is unable to determine low-level features. A loss function can be used to analyze the error in the output. Any appropriate definition of a loss function can be used. An example of a loss function is the mean squared error (MSE). MSE is,

[0292]

number

[0293] This is defined as the sum of half of the squares obtained by subtracting the predicted (output) answer from the actual answer. The loss is E total It can be set to be equal to the value of .

[0294] Since the actual values ​​differ significantly from the predicted output, the loss (or error) is large in the first training image. The goal of training is to minimize the amount of loss so that the predicted output matches the training label. The neural network 4200 can perform a backward pass by determining which inputs (weights) contributed the most to the network's loss, and can adjust the weights so that the loss decreases and is eventually minimized.

[0295] To determine which weight contributed most to the network loss, we can calculate the derivative of the loss with respect to the weight (denoted as dL / dW, where W is the weight in a particular layer). After the derivative is calculated, we can perform a weight update by updating all the weights in the filter. For example, we can update the weights so that they change in the opposite direction of the gradient.

[0296]

number

[0297] It can be written as, where w indicates the weight, and w i η represents the initial weights, and η represents the learning rate. The learning rate can be set to any appropriate value; a higher learning rate indicates updates to larger weights, while a lower value indicates updates to smaller weights.

[0298] Neural Network 4200 can include any suitable deep network. One example is a convolutional neural network (CNN), which has input and output layers, with multiple hidden layers between the input and output layers. The hidden layers of a CNN include a series of convolutional layers, nonlinear layers, pooling (for downsampling) layers, and fully connected layers. Neural Network 4200 can include any other deep network besides CNNs, such as autoencoders, deep belief networks (DBNs), and recurrent neural networks (RNNs), among others.

[0299] Figure 43 is an example illustrating a convolutional neural network (CNN) 4300. The input layer 4320 of the CNN 4300 contains data representing an image. For example, the data may contain an array of digits representing pixels in the image, where each digit in the array contains a value from 0 to 255 describing the pixel intensity at that location in the array. Using the previous example above, the array may contain a 28 × 28 × 3 array of digits with 28 rows and 28 columns of pixels and three color components (red, green, and blue, or luma and two chroma components, for example). The image can be passed through a convolutional hidden layer 4322a, an optional nonlinear activation layer, a pooling hidden layer 4322b, and a fully connected hidden layer 4322c to obtain the output in the output layer 4324. Although only one of each hidden layer is shown in Figure 43, those skilled in the art will understand that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in the CNN 4300. As previously explained, the output may indicate a single class of objects, or it may include the probability of the class that best describes the objects in the image.

[0300] The first layer of CNN4300 is the convolutional hidden layer 4322a. The convolutional hidden layer 4322a analyzes the image data of the input layer 4320. Each node in the convolutional hidden layer 4322a is connected to a region of nodes (pixels) in the input image called the receptive field. The convolutional hidden layer 4322a can be thought of as one or more filters (each filter corresponding to a different activation or feature map), and each convolutional iteration of the filter is a node or neuron in the convolutional hidden layer 4322a. For example, the region of the input image that the filter is responsible for in each convolutional iteration is the receptive field of the filter. In an example for explanation, if the input image contains a 28x28 array and each filter (and its corresponding receptive field) is a 5x5 array, then there are 24x24 nodes in the convolutional hidden layer 4322a. Each connection between a node and its receptive field learns weights, and possibly an overall bias, so that each node learns to analyze a specific local receptive field in the input image. Each node in the hidden layer 4322a has the same weights and biases (called shared weights and shared biases). For example, a filter has an array of weights (numbers) and the same depth as the input. The filter has a depth of 3 (according to the three color components of the input image) in the example of a video frame. An exemplary size for describing the filter array is 5 × 5 × 3, which corresponds to the size of the node's receptive field.

[0301] The convolutional properties of the convolutional hidden layer 4322a stem from the fact that each node in the convolutional layer is applied to its corresponding receptive field. For example, a filter in the convolutional hidden layer 4322a can start at the top-left corner of the input image array and convolve around the input image. As mentioned above, each iteration of the convolution of the filter can be thought of as a node or neuron in the convolutional hidden layer 4322a. In each iteration of the convolution, the filter value is multiplied by a corresponding number of original pixel values ​​in the image (for example, a 5x5 filter array is multiplied by a 5x5 array of input pixel values ​​at the top-left corner of the input image array). The multiplications from each iteration of the convolution can be added together to obtain a sum for that iteration or node. This process then continues at the next position in the input image, according to the receptive field of the next node in the convolutional hidden layer 4322a. For example, the filter can be moved to the next receptive field by a certain step amount (called a stride). The stride can be set to 1 or any other appropriate amount. For example, if the stride is set to 1, the filter is moved one pixel to the right in each iteration of the convolution. Processing the filter at each unique position of the input volume yields a number representing the filtered result for that position, thereby determining the total value for each node of the convolutional hidden layer 4322a.

[0302] The mapping from the input layer to the convolutional hidden layer 4322a is called an activation map (or feature map). The activation map contains the values ​​of each node representing the filter result at each position in the input volume. The activation map may contain an array of various aggregate values ​​resulting from each iteration of the filter on the input volume. For example, if a 5x5 filter is applied to each pixel of a 28x28 input image (stride of 1), the activation map will contain a 24x24 array. The convolutional hidden layer 4322a may contain several activation maps to identify multiple features in the image. The example shown in Figure 43 includes three activation maps. Using the three activation maps, the convolutional hidden layer 4322a can detect three different types of features, each of which is detectable throughout the image.

[0303] In some cases, a nonlinear hidden layer can be applied after the convolutional hidden layer 4322a. A nonlinear layer can be used to introduce nonlinearity into a system that was previously computing linear operations. An example for one explanation of a nonlinear layer is the Normalized Linear Unit (ReLU) layer. A ReLU layer can apply the function fx=max(0,x) to all values ​​in the input volume, which turns all negative activations to 0. Thus, ReLU can enhance the nonlinear nature of network 4300 without affecting the receptive field of the convolutional hidden layer 4322a.

[0304] A pooling hidden layer 4322b can be applied after the convolutional hidden layer 4322a (and after the nonlinear hidden layer when used). The pooling hidden layer 4322b is used to simplify the information in the output from the convolutional hidden layer 4322a. For example, the pooling hidden layer 4322b can take each activation map output from the convolutional hidden layer 4322a and generate a compressed activation map (or feature map) using a pooling function. Maximizing pooling (max-pooling) is one example of a function performed by the pooling hidden layer. Other forms of pooling functions, such as average pooling, L2-norm pooling, or other suitable pooling functions, may be used by the pooling hidden layer 4322a. The pooling function (e.g., a maximizing pooling filter, an L2-norm filter, or other suitable pooling filter) is applied to each activation map contained in the convolutional hidden layer 4322a. In the example shown in Figure 43, three pooling filters are used for three activation maps in the convolutional hidden layer 4322a.

[0305] In some examples, maximizing pooling can be used by applying a maximizing pooling filter (for example, having a size of 2x2) to an activation map output from a convolutional hidden layer 4322a with a certain stride (for example, a stride of 2, equal to the dimension of the filter). The output from the maximizing pooling filter contains the largest number in any subregion that the filter convolves around. Using a 2x2 filter as an example, each unit in the pooling layer can summarize a region of 2x2 nodes (each node being a value in the activation map) in the previous layer. For example, four values ​​(nodes) in the activation map are analyzed by the 2x2 maximizing pooling filter in each iteration of the filter, and the maximum value of the four values ​​is output as the "maximum" value. If such a maximizing pooling filter is applied to an activation filter from a convolutional hidden layer 4322a with a dimension of 24x24 nodes, the output from the pooling hidden layer 4322b is an array of 12x12 nodes.

[0306] In some examples, L2 norm pooling filters may also be used. An L2 norm pooling filter involves calculating the square root of the sum of the squares of the values ​​in a 2x2 region (or other suitable region) of the activation map (rather than calculating the maximum value as is done in maximizing pooling), and using the calculated value as the output.

[0307] Intuitively, a pooling function (e.g., maximization pooling, L2 norm pooling, or other pooling functions) determines whether a given feature is found somewhere in the image region and discards the exact location information. This can be done without affecting the feature detection results because, once a feature is found, its exact location is less important than its approximate location relative to other features. Maximization pooling (and other pooling methods) benefits from having more features pooled less, thus reducing the number of parameters required in later layers of CNN4300.

[0308] The final layer of connectivity in the network is a fully connected layer that connects each node from the pooling hidden layer 4322b to each output node in the output layer 4324. Using the example above, the input layer contains 28x28 nodes encoding the pixel intensity of the input image, the convolutional hidden layer 4322a contains 3x24x24 hidden feature nodes based on the application of 5x5 local receptive fields (for filtering) to three activation maps, and the pooling hidden layer 4322b contains a layer of 3x12x12 hidden feature nodes based on the application of a maximizing pooling filter to 2x2 regions across each of the three feature maps. Extending this example, the output layer 4324 could contain 10 output nodes. In such an example, each node of the 3x12x12 pooling hidden layer 4322b is connected to each node of the output layer 4324.

[0309] The fully connected layer 4322c can take the output of the preceding pooling hidden layer 4322b (which should represent the activation map of high-level features) and determine the features that are most correlated to a particular class. For example, the fully connected layer 4322c can determine the high-level features that are most strongly correlated to a particular class and can include weights (nodes) for these high-level features. The product of the weights in the fully connected layer 4322c and the weights in the pooling hidden layer 4322b can be calculated to obtain probabilities for different classes. For example, if CNN4300 is used to predict that an object in a video frame is a person, there will be high values ​​in the activation map representing high-level features of a person (e.g., having two legs, having a face on top of the object, having two eyes on the upper left and upper right of the face, having a nose in the middle of the face, having a mouth at the bottom of the face, and / or other features common to people).

[0310] In some examples, the output from output layer 4324 may contain an M-dimensional vector (M=10 in the previous example), where M can represent the number of classes from which the program must select when classifying objects in an image. Other exemplary outputs may also be provided. Each number in the N-dimensional vector can represent the probability that an object belongs to a particular class. In an example for one explanation, if the 10-dimensional output vector represents that the 10 distinct classes of objects are [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates that there is a 5% probability that the image belongs to the third class of objects (e.g., a dog), an 80% probability that the image belongs to the fourth class of objects (e.g., a person), and a 15% probability that the image belongs to the sixth class of objects (e.g., a kangaroo). The probability of a class can be thought of as the confidence level of an object being part of that class.

[0311] Various object detectors can be used to perform object detection and / or classification. One example involves a Cifar-10 neural network-based detector. Figure 44 shows an example of a Cifar-10 neural network 4400. In some cases, a Cifar-10 neural network can be trained to classify only people and passenger cars. As shown, the Cifar-10 neural network 4400 includes various convolutional layers (Conv1 layer 4402, Conv2 / Relu2 layer 4408, and Conv3 / Relu3 layer 4414), numerous pooling layers (Pool1 / Relu1 layer 4404, Pool2 layer 4410, and Pool3 layer 4416), and normalized linear unit layers mixed in with them. Normalization layers Norm1 4406 and Norm2 4412 are also provided. The last layer is the ip1 layer 4418.

[0312] Another deep learning-based detector that can be used to detect and / or classify objects in an image is the SSD detector, which is a fast single-shot object detector that can be applied to categories or classes of multiple objects. The SSD model uses multiscale convolutional bounding box outputs attached to multiple feature maps at the top level of a neural network. Such representations enable SSD to efficiently model a variety of box shapes. Figure 45A contains an image, and Figures 45B and 45C contain diagrams showing how the SSD detector (with a VGG deep network-based model) works. For example, SSD matches objects to default boxes of different aspect ratios (shown as dashed rectangles in Figures 45B and 45C). Each element of a feature map has several default boxes associated with it. Any default box with some intersection-over-union with a ground truth box exceeding a threshold (e.g., 0.4, 0.5, 0.6, or other appropriate threshold) is considered a match for that object. For example, two of the 8x8 boxes (shown in blue in Figure 45B) match cats, and one of the 4x4 boxes (shown in red in Figure 45C) matches dogs. The SSD has multiple feature maps, each responsible for objects of different sizes, enabling the SSD to identify objects across a wide range of sizes. For example, the boxes in the 8x8 feature map in Figure 45B are smaller than the boxes in the 4x4 feature map in Figure 45C. In an example for explanation, the SSD detector could have a total of six feature maps.

[0313] For each default box within each cell, the SSD neural network outputs a probability vector of length c, where c is the number of classes and represents the probability that the box contains an object of each class. In some cases, a background class is included, indicating that the box contains no object. The SSD network also outputs an offset vector (for each default box within each cell) with four entries containing the predicted offsets needed to match the default box with the bounding box of the object behind it. The vector is given in the format (cx,cy,w,h), where cx represents the center x, cy represents the center y, w represents the width offset, and h represents the height offset. These vectors are only meaningful if there is actually an object contained within the default box. In the image shown in Figure 45A, all probability labels indicate the background class, with the exception of three matching boxes (two cats and one dog).

[0314] Another deep learning-based detector that can be used to detect and / or classify objects in an image includes the You Only Look Once (YOLO) detector, which is an alternative to SSD object detection systems. Figure 46A includes an image, and Figures 46B and 46C include diagrams illustrating how the YOLO detector works. The YOLO detector can apply a single neural network to the entire image. As shown, the YOLO network divides the image into regions and predicts bounding boxes and probabilities for each region. These bounding boxes are weighted by the predicted probabilities. For example, as shown in Figure 46A, the YOLO detector divides the image into a 13x13 cell grid. Each cell is responsible for predicting five bounding boxes. A confidence score is provided, indicating how certain the predicted bounding boxes are that they actually enclose an object. This score does not include the classification of possible objects within the box, but indicates whether the shape of the box is appropriate. The predicted bounding boxes are shown in Figure 46B. Boxes with higher confidence scores have thicker borders.

[0315] Each cell also predicts the class of each bounding box. For example, a probability distribution is provided across all possible classes. Any number of classes can be detected, such as bicycle, dog, cat, person, car, or other appropriate object classes. The confidence scores of the bounding boxes and the confidence scores of the class predictions are combined to form a final score indicating the probability that a bounding box contains a particular type of object. For example, the yellow box with the thick border on the left side of the image in Figure 46B contains the object class "dog" with an 85% probability. There are 169 grid cells (13x13), and each cell predicts 5 bounding boxes, resulting in a total of 4645 bounding boxes. Many of the bounding boxes have very low scores, in which case only boxes whose final score exceeds a threshold (e.g., a 30% probability, a 40% probability, a 50% probability, or another appropriate threshold) are retained. Figure 46C shows the final predicted bounding boxes with the classes containing dog, bicycle, and car. As shown, out of a total of 4645 bounding boxes generated, only the three bounding boxes shown in Figure 46C were retained because they had the best final scores.

[0316] Figure 47 shows an example of a system for implementing several aspects of this technique. Specifically, Figure 47 shows an example of a computing system 4700, which may be, for example, an internal computing system, a remote computing system, a camera, or any computing device that constitutes any of these components, and the components of the system communicate with each other using connections 4705. Connections 4705 may be a physical connection using a bus, or a direct connection to a processor 4710 in a chipset architecture, for example. Connections 4705 may also be a virtual connection, a network connection, or a logical connection.

[0317] In some embodiments, the computing system 4700 is a distributed system in which the functions described herein can be distributed across a data center, multiple data centers, a peer network, and so on. In some embodiments, one or more of the described system components represent many components, each of which performs some or all of the functions described herein. In some embodiments, the components may be physical or virtual devices.

[0318] An exemplary system 4700 includes at least one processing unit (CPU or processor) 4710 and connection 4705 that connects various system components to the processor 4710, including system memory 4715 such as read-only memory (ROM) 4720 and random access memory (RAM) 4725. The computing system 4700 may include a high-speed memory cache 4712 that is directly connected to, adjacent to, or integrated as part of the processor 4710.

[0319] The processor 4710 may include any general-purpose processors and hardware or software services, such as services 4732, 4734, and 4736, stored in memory device 4730, which are configured to control the processor 4710 and dedicated processors such that software instructions are incorporated into actual processor designs. The processor 4710 may essentially be a fully self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. The multicore processor may be symmetrical or asymmetrical.

[0320] To enable user interaction, the computing system 4700 includes an input device 4745, which may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, or speech. The computing system 4700 may also include an output device 4735, which may be one or more of a number of output mechanisms. In some cases, a multimodal system may allow a user to communicate with the computing system 4700 by providing multiple types of input / output. The computing system 4700 may include a communication interface 4740, which can generally control and manage user input and system output.Communication interfaces include audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple® Lightning® ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, Bluetooth® wireless signal transmission, Bluetooth® low energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, Wireless Local Area Network (WLAN) signal transmission, Visible Light Communication (VLC), and Worldwide Interoperability for Microwave Access. The communication interface 4740 may perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including (WiMAX), infrared (IR) communications wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof. The communication interface 4740 may also include one or more GNSS receivers or transceivers used to determine the position of the computing system 4700 based on the reception of one or more signals from one or more satellites associated with one or more Global Navigation Satellite Systems (GNSS) systems. GNSS systems include, but are not limited to, the US Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's Beidou Navigation Satellite System (BDS), and Europe's Galileo GNSS.Since there are no restrictions on which specific hardware configuration it can operate on, this fundamental characteristic can easily be superseded by improved hardware or firmware configurations as development progresses.

[0321] The storage device 4730 may be a non-volatile and / or non-temporary and / or computer-readable memory device, a hard disk, or a magnetic cassette, flash memory card, solid-state memory device, digital multipurpose disk, cartridge, floppy disk, flexible disk, hard disk, magnetic tape, magnetic strip / stripe, any other magnetic storage medium, flash memory, memory stick memory, any other solid-state memory, compact disc read-only memory (CD-ROM) optical disc, rewritable compact disc (CD) optical disc, digital video disc (DVD) optical disc, Blu-ray disc (BDD) optical disc, holographic optical disc, another optical medium, Secure Digital (SD) card, microSecure Digital (microSD) card, Memory Stick® card, smart card This may also include other types of computer-readable media capable of storing computer-accessible data, such as code chips, EMV chips, subscriber identification module (SIM) cards, mini / micro / nano / pico SIM cards, other integrated circuit (IC) chips / cards, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase-shift memory (PCM), spin-transfer torque RAM (STT-RAM), other memory chips or cartridges, and / or combinations thereof.

[0322] The storage device 4730 may include software services, servers, services, etc., which cause the system to perform functions when code defining such software is executed by the processor 4710. In some embodiments, a hardware service that performs a particular function may include software components stored on a computer-readable medium, which are connected to the necessary hardware components, such as the processor 4710, connection 4705, and output device 4735, in order to perform the function.

[0323] As used herein, the term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instructions and / or data. Computer-readable medium may include non-transient media in which data can be stored, and which do not contain carrier waves and / or transient electronic signals that propagate wirelessly or via wired connections. Examples of non-transient media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as compact discs (CDs) or digital multipurpose discs (DVDs), flash memory, memory, or memory devices. Computer-readable medium may store code and / or machine-executable instructions that may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted using any appropriate means, including memory sharing, message passing, token passing, network transmission, etc.

[0324] In some embodiments, computer-readable storage devices, media, and memory may include cables or wireless signals, such as bitstreams. However, non-temporary computer-readable storage media, as referred to, explicitly exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0325] Specific details are given in the above description to provide a complete understanding of the embodiments and examples given herein. However, it will be understood by those skilled in the art that these embodiments may be practiced without these specific details. For clarity of description, in some cases the art may be presented as including individual functional blocks, which include functional blocks that include steps or routines in a method embodied in a device, device components, software, or a combination of hardware and software. Additional components other than those shown in the drawings and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details so as not to obscure the embodiments.

[0326] Individual embodiments may be described above as processes or methods shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts may describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operation is complete, but it may have additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or main function.

[0327] The processes and methods described above may be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions may include instructions and data that cause a general-purpose computer, a dedicated computer, or a processing device to perform a particular function or group of functions, or otherwise configure a general-purpose computer, a dedicated computer, or a processing device to perform such functions. The portion of computer resources used may be accessible over a network. Computer-executable instructions may be binary or intermediate format instructions, such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods described above include magnetic or optical disks, flash memory, USB devices with non-volatile memory, and network-connected storage devices.

[0328] Devices implementing the processes and methods described herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the required tasks may be stored in computer-readable or machine-readable media. A processor may perform the required tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mount devices, and standalone devices. The functions described herein may also be embodied in peripheral devices or add-in cards. Such functions may, as a further example, be implemented on different processes running on circuit boards in different chips or in a single device.

[0329] Instructions, a medium for transmitting such instructions, computing resources for executing the instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described herein.

[0330] In the above description, aspects of this application are described with reference to their specific embodiments, but those skilled in the art will recognize that this application is not limited thereto. Therefore, while exemplary embodiments of this application are described in detail herein, the concepts of the present invention may be embodied or utilized in various other ways, and it should be understood that, apart from the limitations of the prior art, the appended claims are intended to be interpreted as including such variations. The various features and aspects of the applications described above may be used individually or together. Furthermore, embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, this specification and the drawings should be considered illustrative, not restrictive. For illustrative purposes, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than described.

[0331] Those skilled in the art will understand that the symbols or terms less than ("<") and greater than (">") used herein may be replaced by the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively, without departing from the scope of this description.

[0332] When components are described as being "configured to" perform certain operations, such configurations can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuit) to perform the operations, or by any combination thereof.

[0333] The phrase "combined" refers to any component that is physically connected to another component, either directly or indirectly, and / or communicates, either directly or indirectly, with another component (for example, connected to another component via a wired or wireless connection and / or other appropriate communication interface).

[0334] The wording of a claim that includes “at least one of” a set and / or “one or more” of a set indicates that one element of a set or multiple elements of a set (in any combination) satisfy the claim. For example, the wording of a claim that includes “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, the wording of a claim that includes “at least one of A, B, and C” or “at least one of A, B, or C” means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The wording “at least one of” a set and / or “one or more” of a set does not limit the set to items enumerated in the set. For example, the wording of a claim that includes “at least one of A and B” or “at least one of A or B” could mean A, B, or A and B, and could also include items not enumerated in the set A and B.

[0335] Various exemplary logic blocks, modules, circuits, and algorithmic steps described in relation to the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly demonstrate this hardware- and software compatibility, various exemplary components, blocks, modules, circuits, and steps are described above in general terms of their function. Whether such functions are implemented as hardware or software will depend on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functions in various ways for each specific application, but such decisions on implementation forms should not be construed as causing a departure from the scope of this application.

[0336] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices having multiple applications, including applications in wireless communication device handsets and other devices. Any feature described as a module or component may be implemented together in an integrated logic device, or separately as individual but interoperable logic devices. When implemented in software, the technique may be at least partially implemented by a computer-readable data storage medium comprising program code that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media such as random access memory (RAM), for example synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electro-erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. The technique may, as an addition or alternative, be at least partially implemented by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures, such as propagating signals or waves, and which can be accessed, read, and / or executed by a computer.

[0337] The program code may be executed by a processor which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit configurations. Such processors may be configured to perform any of the techniques described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working with a DSP core, or any other such configuration. Accordingly, the term “processor” as used herein may refer to any of the above structures, any combination thereof, or any other structure or device suitable for implementing the techniques described herein.

[0338] The embodiments for explaining this disclosure include the following:

[0339] Embodiment 1: A method for processing one or more frames, comprising the steps of: determining a region of interest in a first frame of a sequence of frames, wherein the region of interest in the first frame includes an object having a certain size in the first frame; cropping a portion of a second frame of a sequence of frames, wherein the second frame follows the first frame in the sequence of frames; and scaling the portion of the second frame based on the size of the object in the first frame.

[0340] Embodiment 2: The method of Embodiment 1, further comprising the steps of receiving user input corresponding to the selection of an object in a first frame, and determining a region of interest in the first frame based on the received user input.

[0341] Embodiment 3: The method of Embodiment 2, wherein the user input includes touch input provided using the device's touch interface.

[0342] Embodiment 4: Any one of embodiments 1 to 3, further comprising the steps of determining a point in an object region determined for an object in a second frame, and cropping and scaling that portion of the second frame so that the point in the object region is at the center of the cropped and scaled portion.

[0343] Embodiment 5: The method of Embodiment 4, wherein the point in the object region is the center point of the object region.

[0344] Embodiment 6: Any one of embodiments 1 to 5, wherein the object in the second frame is made to be the same size as the object in the first frame by scaling the portion of the second frame based on the size of the object in the first frame.

[0345] Embodiment 7: Any one of embodiments 1 to 6, further comprising the steps of determining a first length relating to an object in a first frame, determining a second length relating to an object in a second frame, determining a scaling factor based on a comparison of the first length and the second length, and scaling that portion of the second frame based on the scaling factor.

[0346] Embodiment 8: The method of Embodiment 7, wherein the first length is the length of a first object region determined with respect to an object in a first frame, and the second length is the length of a second object region determined with respect to an object in a second frame.

[0347] Embodiment 9: The method of Embodiment 8, wherein the first object region is a first bounding box, the first length is the diagonal length of the first bounding box, and the second object region is a second bounding box, the second length is the diagonal length of the second bounding box.

[0348] Embodiment 10: A method of any one of Embodiments 8 or 9, wherein the portion of the second frame is scaled based on a scaling factor so that the region of the second object in the cropped and scaled portion has the same size as the region of the first object in the first frame.

[0349] Embodiment 11: Any one of embodiments 1 to 10, further comprising: determining points of a first object region generated for an object in a first frame; determining points of a second object region generated for an object in a second frame; determining motion coefficients for an object based on a smoothing function using the points of the first and second object regions, wherein the smoothing function controls changes in the object's location across multiple frames in a sequence of frames; and cropping that portion of the second frame based on the motion coefficients.

[0350] Embodiment 12: The method of Embodiment 11, wherein a point in the first object region is the center point of the first object region, and a point in the second object region is the center point of the second object region.

[0351] Embodiment 13: Any one of Embodiments 11 or 12, wherein the smoothing function includes a translation function, the translation function being used to determine the position of a point in each of multiple frames of a sequence of frames, based on a statistical measure of the motion of the object.

[0352] Embodiment 14: Any one of embodiments 1 to 13, further comprising: determining a first length relating to an object in a first frame; determining a second length relating to an object in a second frame; determining a scaling factor for an object based on a comparison of the first and second lengths and on a smoothing function using the first and second lengths, wherein the smoothing function controls the change in size of the object in multiple frames of a sequence of frames; and scaling that portion of the second frame based on the scaling factor.

[0353] Embodiment 15: The method of Embodiment 14, wherein the smoothing function includes a moving function, the moving function being used to determine the length of an object in each of a plurality of frames of a sequence of frames based on a statistical measure of the size of the object.

[0354] Embodiment 16: Any one of Embodiments 14 or 15, wherein the first length is the length of a first bounding box generated for an object in a first frame, and the second length is the length of a second bounding box generated for an object in a second frame.

[0355] Embodiment 17: The method of Embodiment 16, wherein the first length is the diagonal length of the first bounding box and the second length is the diagonal length of the second bounding box.

[0356] Embodiment 18: A method of any one of Embodiments 16 or 17, wherein the portion of the second frame is scaled based on a scaling factor so that the second bounding box in the cropped and scaled portion has the same size as the first bounding box in the first frame.

[0357] Embodiment 19: Any one of Embodiments 1 to 18, wherein the steps of cutting out and scaling a portion of the second frame maintain the object at the center of the second frame.

[0358] Embodiment 20: Any one of embodiments 1 to 19, further comprising the step of detecting and tracking an object in one or more frames of a sequence of frames.

[0359] Embodiment 21: An apparatus for processing one or more frames, comprising: a memory configured to store at least one frame; and a processor, implemented by circuitry, configured to determine a region of interest in a first frame of a sequence of frames, wherein the region of interest in the first frame includes an object having a certain size in the first frame; to cut off a portion of a second frame of a sequence of frames, wherein the second frame is after the first frame in the sequence of frames; and to scale that portion of the second frame to maintain the size of the object in the second frame.

[0360] Embodiment 22: The apparatus of Embodiment 21, wherein the processor is configured to receive user input corresponding to the selection of an object in a first frame and to determine a region of interest in the first frame based on the received user input.

[0361] Embodiment 23: The apparatus of Embodiment 22, wherein user input includes touch input provided using the device's touch interface.

[0362] Embodiment 24: Any one of embodiments 21 to 23, wherein the processor is configured to determine a point in an object region determined for an object in a second frame, and to crop and scale that portion of the second frame so that the point in the object region is at the center of the cropped and scaled portion.

[0363] Embodiment 25: The apparatus of Embodiment 24, wherein a point in the object region is the center point of the object region.

[0364] Embodiment 26: Any one of embodiments 21 to 25, wherein the object in the second frame becomes the same size as the object in the first frame by scaling the portion of the second frame based on the size of the object in the first frame.

[0365] Embodiment 27: Any one of Embodiments 21 to 26, wherein the processor is configured to determine a first length relating to an object in a first frame, a second length relating to an object in a second frame, a scaling factor based on a comparison of the first length and the second length, and scale that portion of the second frame based on the scaling factor.

[0366] Embodiment 28: The apparatus of Embodiment 27, wherein the first length is the length of a first object region determined with respect to an object in a first frame, and the second length is the length of a second object region determined with respect to an object in a second frame.

[0367] Embodiment 29: The apparatus of Embodiment 28, wherein the first object region is a first bounding box, the first length is the diagonal length of the first bounding box, and the second object region is a second bounding box, the second length is the diagonal length of the second bounding box.

[0368] Embodiment 30: An apparatus in which, by scaling that portion of the second frame based on a scaling factor, the second object region in the cropped and scaled portion has the same size as the first object region in the first frame, as is the case in any one of Embodiments 28 or 29.

[0369] Embodiment 31: Any one of Embodiments 21 to 30, wherein the processor determines points of a first object region to be generated for an object in a first frame, determines points of a second object region to be generated for an object in a second frame, and determines a motion coefficient for the object based on a smoothing function using the points of the first and second object regions, wherein the smoothing function controls and determines changes in the position of the object in multiple frames of a sequence of frames, and cuts out that portion of the second frame based on the motion coefficient.

[0370] Embodiment 32: The apparatus of Embodiment 31, wherein a point in the first object region is the center point of the first object region, and a point in the second object region is the center point of the second object region.

[0371] Embodiment 33: An apparatus in any one of Embodiments 31 or 32, wherein the smoothing function includes a moving average function, and the moving average function is used to determine the average position of points in each of a plurality of frames of a sequence of frames.

[0372] Embodiment 34: Any one of Embodiments 21 to 33, wherein the processor determines a first length relating to an object in a first frame, a second length relating to an object in a second frame, and a scaling factor for the object based on a comparison of the first and second lengths and on a smoothing function using the first and second lengths, wherein the smoothing function determines that the size of the object gradually changes over multiple frames in a sequence of frames, and scales that portion of the second frame based on the scaling factor.

[0373] Embodiment 35: The apparatus of Embodiment 34, wherein the smoothing function includes a moving average function, and the moving average function is used to determine the average length of an object in each of a plurality of frames of a sequence of frames.

[0374] Embodiment 36: The apparatus of any one embodiment 34 or 35, wherein the first length is the length of a first bounding box generated for an object in a first frame, and the second length is the length of a second bounding box generated for an object in a second frame.

[0375] Embodiment 37: The apparatus of Embodiment 36, wherein the first length is the diagonal length of the first bounding box and the second length is the diagonal length of the second bounding box.

[0376] Embodiment 38: Any one of embodiments 34 to 37, wherein the second bounding box in the cut-out and scaled portion is the same size as the first bounding box in the first frame by scaling that portion of the second frame based on a scaling factor.

[0377] Embodiment 39: An apparatus for cutting and scaling a portion of a second frame, wherein the object is maintained at the center of the second frame, as per any one of Embodiments 21 to 38.

[0378] Embodiment 40: Any one of embodiments 21 to 39, wherein the processor is configured to detect and track objects in one or more frames of a sequence of frames.

[0379] Embodiment 41: Any one of embodiments 21 to 40, comprising a mobile device with a camera for capturing at least one frame.

[0380] Embodiment 42: Any one of embodiments 21 to 41, further comprising a display for displaying one or more images.

[0381] Aspect 43: A computer-readable medium storing instructions that, when executed by a processor, perform any of the operations of aspects 1 to 40.

[0382] Embodiment 44: An apparatus comprising means for performing any of the operations of Embodiments 1 to 40. [Explanation of symbols]

[0383] 100 Image acquisition and processing systems 105A Image Acquisition Device 105B Image Processing Device 110 scenes 115 Lens 120 Control mechanism 125A Exposure control mechanism 125B Focus Control Mechanism 125C Zoom Control Mechanism 130 Image Sensor 140 Random Access Memory (RAM) 145 Read-only memory (ROM) 150 Image Processors 152 host processors 154 Image Signal Processor (ISP) 156 Input / Output (I / O) Ports 160 Input / Output (I / O) Devices 200 Video Analysis Systems 202 video frames 204 Blob Detection System 206 Object Tracking System 230 video sources 302A Video Frame A 304A Blob Detection System 306A Object Tracking System 308A Foreground blob 310A Blob Tracker 402 video frames 408 Blob 412 Background difference engine 414 Morphology Engine 416 Linked Component Analysis Engine 418 Blob Processing Engine 508 Blob 510A Blob Tracker 512 Cost Determination Engine 514 Data Association Engine 516 Blob Tracker Update Engine 800 frame cropping and scaling system 801 Image Sensor 802 frames 803 Image Processing Engine 804 Region of Interest (ROI) Determination Engine 805 Video Processing Engine 806 Object Detection and Tracking System 807 Display processing engine 808 Frame Cutting Engine 809 encoding engine, image coding engine 810 Frame Scaling Engine 811 Image analysis engine, frame analysis engine 812 Smoothing Engine 813 Sensor image metadata engine, sensor frame metadata engine 814 output frames 815 Frame cropping and scaling system 817 memory 819 displays 1002 Initial Frame 1004 Bounding Box 1006 center point 1007 Top left dot 1008 Diagonal length 1009 distance 1012 Subsequent frames 1014 Bounding Box 1016 center point 1017 Top left dot 1018 Diagonal length 1022 Cut-off area 1029 distance 1032 Subsequent frames that have been cropped and scaled. 1034 Bounding Box 1036 center point 1037 Top left dot 1038 Diagonal length 1039 distance 1602 Frame 1604 Zoom Region of Interest (ROI), Zooming ROI 1606 upscaled frame 2402 images 2404 images 2406 encoder 2408 First Decoder Head 2410 Second Decoder Head 2702 Standard telephoto frame 2704 Telephoto Frame 2802 Standard telephoto frame 2804 Telephoto Frame 2806 Wide-angle lens frame 2902 Standard wide-angle frame 2904 Wide-angle frame 3002 Standard telephoto frame 3004 Telephoto Frame 3006 Wide-angle frame 3008 Output Frame 3102 Standard telephoto frame 3104 Telephoto Frame 3106 Wide-angle frame 3108 Output Frames 3202 Standard telephoto frame 3204 Telephoto Frame 3206 Wide-angle frame 3208 output frames 3302 Standard telephoto frame 3304 Telephoto Frame 3308 Output Frames 3402 Standard telephoto frame 3404 Standard wide-angle frame 3406 Telephoto Frame 3408 Wide-angle frame 3410 Output Frame 3502 Standard wide-angle frame 3506 Wide-angle frame 3510 Output Frame 3602 Standard telephoto frame 3604 Telephoto Frame 3606 Wide-angle lens frame 3608 output frames 4200 Deep Learning Networks, Deep Learning Neural Networks 4220 Input Layer 4222a Hidden layer 4224 Output Layer 4226 nodes 4300 Convolutional Neural Networks (CNNs) 4320 Input Layer 4322a Convolutional Hidden Layer 4322b Pooling Hidden Layer 4322c Fully Connected Hidden Layer 4324 Output Layer 4402 Conv1 layer 4404 Pool1 / Relu1 layer 4406 Normalization layer Norm1 4408 Conv2 / Relu2 layer 4410 Pool 2 layers 4412 Normalization layer Norm2 4414 Conv3 / Relu3 layer 4416 Pool 3 layers 4418 ip1 layer 4700 Computing Systems 4705 Connection 4710 Processor, Processing Unit 4712 cache 4715 System Memory 4720 Random Access Memory (RAM), Read-Only Memory (ROM) 4725 Read-only memory (ROM), Random access memory (RAM) 4730 Storage Devices 4732 Service 4734 Service 4735 Output device 4736 Services 4740 Communication Interface 4745 Input Devices

Claims

1. A method for processing one or more frames, A step of determining a region of interest in a first frame of a sequence of frames, wherein the region of interest in the first frame includes an object having a certain size in the first frame; A step of determining a first object region generated for the object in the first frame, A step of determining a second object region to be generated for the object in a second frame, wherein the second frame is after the first frame in the sequence of frames; A step of determining a motion coefficient for an object based on a smoothing function using the first object region and the second object region, wherein the smoothing function controls the change in the object's location in multiple frames of the sequence of frames so that the change does not exceed a threshold change in location in multiple frames of the sequence of frames. A step of cutting out a portion of the second frame of the sequence of frames based on the motion coefficient, The process includes the step of scaling the portion of the second frame based on the size of the object in the first frame, A method wherein the smoothing function includes a translation function, the translation function being used to determine the position of a point in each of the plurality of frames of the sequence of frames, based on a statistical measure of the motion of the object.

2. A method for processing one or more frames, A step of determining a region of interest in a first frame of a sequence of frames, wherein the region of interest in the first frame includes an object having a certain size in the first frame; A step of determining a first object region generated for the object in the first frame, A step of determining a second object region to be generated for the object in a second frame, wherein the second frame is after the first frame in the sequence of frames; A step of determining a motion coefficient for an object based on a smoothing function using the first object region and the second object region, wherein the smoothing function controls the change in the object's location in multiple frames of the sequence of frames so that the change does not exceed a threshold change in location in multiple frames of the sequence of frames. A step of cutting out a portion of the second frame of the sequence of frames based on the motion coefficient, The process includes the step of scaling the portion of the second frame based on the size of the object in the first frame, The method described above is A step of determining a first length relating to the object in the first frame, A step of determining a second length relating to the object in the second frame, A step of determining a scaling factor for the object based on a comparison of the first length and the second length, and based on a smoothing function using the first length and the second length, wherein the smoothing function controls the change in the size of the object in a plurality of frames of the sequence of frames, The step further includes scaling the portion of the second frame based on the scaling factor, A method wherein the smoothing function includes a movement function, the movement function being used to determine the length of the object in each of the plurality of frames of the sequence of frames based on a statistical measure of the size of the object.

3. The steps include receiving user input corresponding to the selection of the object in the first frame, The method according to claim 1 or 2, further comprising the step of determining the region of interest in the first frame based on the user input received.

4. The method according to claim 3, wherein the user input includes touch input provided using the device's touch interface.

5. A step of determining the points of the object region determined for the object in the second frame, The method according to claim 1 or 2, further comprising the step of cropping and scaling the portion of the second frame such that the point in the object region is at the center of the cropped and scaled portion.

6. The method according to claim 5, wherein the point in the object region is the center point of the object region.

7. The method according to claim 1 or 2, wherein the portion of the second frame is scaled based on the size of the object in the first frame so that the object in the second frame is the same size as the object in the first frame.

8. A step of determining a first length relating to the object in the first frame, A step of determining a second length relating to the object in the second frame, A step of determining a scaling factor based on a comparison of the first length and the second length, The method according to claim 1, further comprising the step of scaling the portion of the second frame based on the scaling factor.

9. The method according to claim 8, wherein the first length is the length of a first object region determined with respect to the object in the first frame, and the second length is the length of a second object region determined with respect to the object in the second frame.

10. The method according to claim 9, wherein the first object region is a first bounding box, the first length is the diagonal length of the first bounding box, and the second object region is a second bounding box, and the second length is the diagonal length of the second bounding box.

11. The method according to claim 9, wherein the portion of the second frame is scaled based on the scaling factor so that the second object region in the cropped and scaled portion has the same size as the first object region in the first frame.

12. The steps include determining a point in the first object region, The process further includes the step of determining a point in the second object region, The method according to claim 1 or 2, wherein the point in the first object region is the center point of the first object region, and the point in the second object region is the center point of the second object region.

13. A device for processing one or more frames, A memory configured to store at least one frame, The system comprises a processor implemented in a circuit, and the processor is Determining a region of interest in a first frame of a sequence of frames, wherein the region of interest in the first frame includes an object having a certain size in the first frame. Determining a first object region generated for the object in the first frame, Determining a second object region to be generated for the object in the second frame, wherein the second frame follows the first frame in the sequence of frames, Determining a motion coefficient for the object based on a smoothing function using the first object region and the second object region, wherein the smoothing function controls the change in the object's location in multiple frames of the sequence of frames so that the change does not exceed a threshold change in location in the multiple frames of the sequence of frames. Based on the motion coefficient, a portion of the second frame of the sequence of frames is cut out, It is configured to scale the portion of the second frame based on the size of the object in the first frame, A device in which the smoothing function includes a translation function, the translation function being used to determine the position of a point in each of the plurality of frames of the sequence of frames, based on a statistical measure of the motion of the object.

14. An apparatus for processing one or more frames, A memory configured to store at least one frame, The system comprises a processor implemented in a circuit, and the processor is Determining a region of interest in a first frame of a sequence of frames, wherein the region of interest in the first frame includes an object having a certain size in the first frame. Determining a first object region generated for the object in the first frame, Determining a second object region to be generated for the object in the second frame, wherein the second frame follows the first frame in the sequence of frames, Determining a motion coefficient for the object based on a smoothing function using the first object region and the second object region, wherein the smoothing function controls the change in the object's location in multiple frames of the sequence of frames so that the change does not exceed a threshold change in location in the multiple frames of the sequence of frames. Based on the motion coefficient, a portion of the second frame of the sequence of frames is cut out, It is configured to scale the portion of the second frame based on the size of the object in the first frame, The aforementioned processor, Determining a first length related to the object in the first frame, Determining a second length related to the object in the second frame, Determining a scaling factor for the object based on a comparison of the first length and the second length, and based on a smoothing function using the first length and the second length, wherein the smoothing function controls the change in the size of the object in multiple frames of the sequence of frames, The system is further configured to scale the portion of the second frame based on the scaling coefficient, An apparatus in which the smoothing function includes a movement function, the movement function being used to determine the length of the object in each of the plurality of frames of the sequence of frames based on a statistical measure of the size of the object.

15. A computer program comprising instructions, wherein when the computer program is executed by a computing device, the computing device causes the computing device to perform the method according to any one of claims 1 to 12.

Images