Heatmap visualization of object detection

Abstract

Electronic devices and methods for heatmap visualization of object detection are provided. The electronic device inputs an image frame including an object of interest (OIO) into a deep neural network (DNN) model and extracts object detection results associated with the OIO as the output of the DNN model for the input image frame. The object detection results include bounding box coordinates for the OIO. The electronic device segments the input image frame into multiple image portions based on the bounding box coordinates and determines a weight value for each of the multiple image portions, indicating the probability that the corresponding image portion belongs to the OIO. The electronic device generates a visualization framework based on the determined weight values, which includes a heatmap visualization of the region included in the image frame and defined by the bounding box coordinates.

Description

[0001] Cross-reference to related applications / incorporation by reference

[0002] none. Technical Field

[0003] Various embodiments of this disclosure relate to computer vision, object detection, and explainable artificial intelligence (XAI). More specifically, various embodiments of this disclosure relate to electronic devices and methods for heatmap visualization of object detection from images. Background Technology

[0004] Advances in computer vision and explainable artificial intelligence have led to the development of various visualization techniques that render the results of deep learning models in a way that human experts can understand. Traditionally, deep learning models, acting as object classifiers, produce classification results that simply select class labels for objects in an input image. In some cases, techniques relying on such classification results may produce inaccurate visualizations of the focused regions intended to include the classified objects.

[0005] By comparing the described system with some aspects of this disclosure, the further limitations and disadvantages of conventional and traditional methods will become clear to those skilled in the art, as set forth in the remainder of this application and with reference to the accompanying drawings. Summary of the Invention

[0006] As set forth more fully in the claims, and substantially as illustrated in at least one figure and / or described in combination with at least one figure, an electronic device and method for thermal image visualization of object detection in an image are provided.

[0007] These and other features and advantages of this disclosure can be understood by reading the following detailed description of the disclosure and the accompanying drawings, in which the same reference numerals always denote the same parts. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating an exemplary network environment for heatmap visualization of object detection according to embodiments of the present disclosure.

[0009] Figure 2 This is a block diagram illustrating an exemplary electronic device for thermal image visualization of object detection according to an embodiment of the present disclosure.

[0010] Figure 3A and 3B These are diagrams illustrating exemplary operations for thermal mapping visualization of object detection according to embodiments of the present disclosure.

[0011] Figure 3CThis is a diagram illustrating an exemplary operation for calculating heatmap metrics for video according to an embodiment of the present disclosure.

[0012] Figure 4 The illustration shows the use of an embodiment according to this disclosure. Figure 2 A block diagram illustrating an exemplary operation of an end-to-end artificial intelligence (XAI) solution for electronic devices.

[0013] Figure 5 This is a diagram illustrating an exemplary user interface of an electronic device for thermal image visualization of object detection according to an embodiment of the present disclosure.

[0014] Figure 6 This is a diagram illustrating an exemplary scene for generating a first perturbed image according to an embodiment of the present disclosure.

[0015] Figure 7 This is a diagram illustrating an exemplary relationship between a dynamic window segmentation function according to an embodiment of the present disclosure and heatmaps of objects with different window sizes.

[0016] Figure 8 This is a diagram illustrating exemplary operations for selecting a single object from multiple object detections and for heatmap visualization based on such selection, according to embodiments of the present disclosure.

[0017] Figure 9 This is a diagram illustrating an exemplary scenario for thermal visualization of two partially overlapping objects in an image frame according to an embodiment of the present disclosure.

[0018] Figure 10 This is a diagram illustrating an exemplary scenario for performing heatmap visualization of an object located within a region of another object in an image frame, according to an embodiment of the present disclosure.

[0019] Figure 11 This is a diagram illustrating exemplary operations for thermal visualization of one or more objects detected in a series of frames of a video feed, according to embodiments of the present disclosure.

[0020] Figure 12 This is a diagram illustrating an exemplary scenario of controlling an unmanned aerial vehicle to autonomously monitor a farm according to an embodiment of the present disclosure.

[0021] Figure 13A and 13B These are diagrams illustrating exemplary operations for thermal mapping visualization of object detection according to embodiments of the present disclosure.

[0022] Figure 14 This is a flowchart illustrating an exemplary method for thermal image visualization of object detection according to an embodiment of the present disclosure. Detailed Implementation

[0023] The embodiments described below can be found in the disclosed electronic devices and methods for heatmap visualization of object detection. An exemplary aspect of this disclosure provides an electronic device for implementing a deep neural network (DNN) model for detecting single or multiple objects in a single static image (and in a video feed) and under various time-varying and static conditions. Such conditions may include, for example, one object partially or completely overlaps or occludes another object, one or more images are missing an object, two separate objects are clustered together in successive frames, a smaller object is included within a larger object, and so on.

[0024] As part of an interpretable artificial intelligence (XAI) workflow, the electronic device also implements visualization tools that output heatmap visualizations of single or multiple object detections from a single still image or video feed. Heatmap visualizations use color intensity to indicate each segment of the image, with the color intensity depending on the weight or contribution of the corresponding segment to the object detection results of the DNN model. This visualization can help developers or end-users view the results of the DNN model and can help them derive meaningful insights from the images. Furthermore, end-users or developers can be able to use appropriate visual metrics to evaluate the accuracy of the heatmap visualizations and improve the detection performance of the DNN model.

[0025] In conventional methods, heatmap visualizations are coarse and generalized, and they may not be able to handle multiple objects. Furthermore, they require knowledge experts to understand the heatmaps. Experts perform visual analysis of the heatmaps and provide inferences as feedback to the application. This introduces a time lag.

[0026] The disclosed electronic device receives input data from a sensing / imaging device and generates thermal visualizations and metric data to output inference results for one or more objects of interest. For each user selection (one or more objects of interest), a thermal map can be generated for one or more detected objects (single / multiple). This disclosure provides more refined thermal visualizations generated for each bounding box, making it suitable for multi-object detection. Knowledge experts are not required to understand the thermal visualizations. The disclosed electronic device can be configured to generate appropriate inference results from the thermal visualizations and can generate interpretations describing the inference results.

[0027] Heatmap visualizations can be used for visual analysis during application audits to gain a deeper understanding of the input data. Heatmap visualizations and visual metrics can be integrated to provide better solutions for applications. Furthermore, using heatmap visualizations and metrics, the input (such as input image frames or the imaging device acquiring such images) can be adjusted, and the heatmap visualization can then be regenerated without retraining the DNN model.

[0028] The disclosed electronic device implements a method applicable to any object detection DNN model and capable of generating more accurate and refined heatmap visualizations compared to conventional methods. The disclosed method provides metrics to quantify the performance of the DNN model on focused regions. While some conventional methods can be applied only to object detection, the disclosed method can generate heatmap visualizations of object detection beyond object classification. The heatmap visualization can be based on focused regions within bounding boxes. The disclosed method can also be applied to video to generate heatmap visualizations of one or more objects in the video based on tracking the continuity and motion of changes associated with those objects(s) in the video.

[0029] Figure 1 This is a block diagram illustrating an exemplary network environment for heatmap visualization of object detection according to embodiments of the present disclosure. Reference Figure 1 A network environment 100 is illustrated. The network environment 100 may include an electronic device 102, a display device 104, and a deep neural network (DNN) model 106 implemented on the electronic device 102. The electronic device 102 may be coupled to the display device 104 via a communication network 108. A user 110, who may be associated with the electronic device 102 and / or the display device 104, is also illustrated. Figure 1 In this disclosure, electronic device 102 and display device 104 are shown as two separate devices; however, in some embodiments, the full functionality of display device 104 may be incorporated into electronic device 102 without departing from the scope of this disclosure.

[0030] Electronic device 102 may include suitable logic, circuitry, code, and / or interfaces that can be configured to detect one or more objects in one or more input image frames using DNN model 106, and generate a visualization framework that includes heatmap visualizations of one or more detected objects based on the object detection results of DNN model 106. Examples of electronic device 102 may include, but are not limited to, image / video editing machines, servers, laptops, computer workstations, mainframes, gaming devices, virtual reality (VR) / augmented reality (AR) / mixed reality (MR) devices, smartphones, mobile phones, and / or any consumer electronics (CE) devices.

[0031] Display device 104 may include suitable logic, circuitry, and / or interfaces that can be configured to display a visualization framework including thermal visualizations of one or more detected objects. In one embodiment, display device 104 may be a touch-enabled device that allows user 110 to provide user input via display device 104. Display device 104 may include display units that can be implemented using a number of known technologies, such as, but not limited to, liquid crystal display (LCD) displays, light-emitting diode (LED) displays, plasma displays, or organic LED (OLED) display technologies or at least one of other display devices. According to embodiments, the display units of display device 104 may refer to a display screen of a head-mounted device (HMD), a smart glasses device, a see-through display, a projection-based display, an electro-color display, or a transparent display.

[0032] DNN model 106 can be an object detector model, which can be trained for an object detection task to detect objects in a series of image frames or a single image frame from a video feed. DNN model 106 can be defined by its hyperparameters, such as activation function(s), number of weights, cost function, regularization function, input size, number of layers, etc. DNN model 106 can be referred to as a system or computational network of artificial neurons (also called nodes). The nodes of DNN model 106 can be arranged in multiple layers, as defined in the neural network topology of DNN model 106. The multiple layers of DNN model 106 can include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers can include one or more nodes (or artificial neurons, e.g., represented by circles). The outputs of all nodes in the input layer can be coupled to at least one node in one or more hidden layers. Similarly, the input of each hidden layer can be coupled to the output of at least one node in the other layers of DNN model 106. The output of each hidden layer can be coupled to the input of at least one node in the other layers of DNN model 106. The nodes in the last layer can receive input from at least one hidden layer to output a result. The number of layers and the number of nodes in each layer can be determined based on the hyperparameters of the DNN model 106. These hyperparameters can be set before or simultaneously with training the DNN model 106 on the training dataset.

[0033] Each node of the DNN model 106 can correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) with a set of parameters that can be tunable during network training. The set of parameters may include, for example, weight parameters, regularization parameters, etc. Each node can use the mathematical function to compute an output based on one or more inputs from nodes in one or more other layers (e.g., one or more preceding layers) of the DNN model 106. All or some nodes of the DNN model 106 can correspond to the same or different mathematical functions.

[0034] In training DNN model 106, one or more parameters of each node of DNN model 106 can be determined based on whether the output of the last layer of a given input (from the training dataset) matches the correct result based on the loss function applied to DNN model 106. This process can be repeated for the same or different inputs until the minimum of the loss function is reached and the training error is minimized. Several methods for training are known in the art, such as gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, metaheuristics, etc.

[0035] In an embodiment, the DNN model 106 may include electronic data, which may be implemented as a software component of an application executable, for example, on electronic device 102. The DNN model 106 may rely on libraries, external scripts, or other logic / instructions to be executed by a processing device such as electronic device 102. The DNN model 106 may include computer-executable code or routines to enable a computing device such as electronic device 102 to perform one or more operations to detect objects in an input image frame. Additionally or alternatively, the DNN model 106 may be implemented using hardware including a processor, a microprocessor (e.g., to perform or control the execution of one or more operations), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). For example, an inference accelerator chip may be included in electronic device 102 to accelerate the computation of the DNN model 106 for object detection tasks. In some embodiments, the DNN model 106 may be implemented using a combination of hardware and software.

[0036] Examples of DNN models 106 may include, but are not limited to, artificial neural networks (ANN), convolutional neural networks (CNN), regions with CNNs (R-CNN), fast R-CNN, faster R-CNN, You Only See Once (YOLO) network, residual neural networks (Res-Net), feature pyramid network (FPN), retina network (Retina-Net), single-shot detector (SSD) and / or combinations thereof.

[0037] Communication network 108 may include a communication medium through which electronic device 102 can communicate with display device 104 and other devices omitted from disclosure for brevity. Communication network 108 may be either a wired or wireless connection. Examples of communication network 108 may include, but are not limited to, the Internet, cloud networks, Wi-Fi networks, personal area networks (PANs), local area networks (LANs), or metropolitan area networks (MANs). Various devices in network environment 100 may be configured to connect to communication network 108 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Li-Fi, 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access points (APs), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0038] In operation, electronic device 102 can initiate an XAI workflow for the detection of one or more objects in one or more image frames and for the thermal visualization of regions including the detected objects in one or more such image frames. In an embodiment, the XAI workflow can be initiated based on user input received via display device 104.

[0039] As the XAI workflow is initiated, electronic device 102 can receive a first image frame (such as image 112). The first image frame can be associated with a static scene with fixed foreground or background objects or a dynamic scene with one or more moving objects. The first image frame may include at least one object of interest. For example, image 112 includes a football player 114 as the object of interest.

[0040] Electronic device 102 can input a first image frame into a DNN model 106 that can be trained for an object detection task. The object detection task can be associated with single-class object detection or multi-class object detection. For example, if the object of interest is a football player 114, then the object detection task can be associated with single-class object detection of football player 114 (i.e., a human object). Otherwise, if there are more than one object of interest (such as football player 114 and a football), then the object detection task can be associated with multi-class object detection of both football player 114 and the football. In at least one embodiment, the DNN model 106 can be trained to detect multiple moving or static objects of the same or different object categories in the first image frame.

[0041] DNN model 106 can receive an input first image frame through its input layer. In some embodiments, the size of the input first image frame can be adjusted to match the size of the input layer of DNN model 106. For example, if the first image frame is a high-resolution image (i.e., 1280x720 pixels) but the size of the input layer is 416x416, then the electronic device 102 can adjust the size of the first image frame to match the size of the input layer. Through the output layer of DNN model 106, DNN model 106 can output object detection results associated with objects of interest detected in the input first image frame. For example, if football player 114 in image 112 is detected by DNN model 106, then the object detection result can be a vector including the object score of football player 114, bounding box coordinates predicted to include the bounding box of football player 114, and the class score (or class probability) of football player 114. The object score can specify the confidence level (based on a probability value between 0 and 1) of DNN model 106 when detecting football player 114 in image 112. A higher object score indicates a higher confidence level for the DNN model 106. In some embodiments, a threshold object score (e.g., 0.95) can be set to filter out object detections with a confidence level below the threshold object score. Class scores can include soft or hard labels to represent the object classes that the DNN model 106 can detect. An object of interest can belong to one of these classes. For example, if the DNN model 106 supports three classes, such as human objects, car objects, and cat objects, then the class score for the human object class could be 1 (or a number close to 1), while the class scores for each of the car and cat object classes would be 0 (or close to 0).

[0042] In some embodiments, the DNN model 106 may employ a region-based or grid-based detection method, thereby dividing the input first image frame into multiple grid cells and performing a cell-by-cell search for the detection of objects of interest in one or more of these grid cells. When multiple objects of interest overlap in a shared grid cell, anchor boxes can be specified to create longer grid cells, and multiple objects can be associated with such grid cells. When multiple objects of interest exist in different grid cells, the DNN model 106 can output object detection results for each of the different grid cells associated with the multiple objects of interest. The electronic device 102 can extract the object detection results associated with the objects of interest as the output of the DNN model 106 for the first image frame.

[0043] For heatmap visualization, electronic device 102 can segment a first image frame into a first plurality of image portions based on bounding box coordinates. Subsequently, for each of the first plurality of image portions, electronic device 102 can determine a weight value indicating the probability that the corresponding image portion belongs to an object of interest. For example, in Figure 3A and 3B The details of segmentation and weight value determination for the first image frame are further described below.

[0044] The electronic device can generate a visualization framework based on weight values ​​determined for each of the first plurality of image portions. The visualization framework may include a heatmap visualization of a region included in the input first image frame and defined by bounding box coordinates, for example in... Figure 3A and 3B As described in [the document]. An example of the visualization framework 116 is shown as a heatmap including region 118, which includes football player 114.

[0045] Color schemes based on heatmap visualizations can map each weight value to a color intensity. For example, a color scheme can utilize a color palette, such as a rainbow palette or a color scale from warm to cool colors. Higher weight values ​​can be mapped to warmer colors (e.g., red, orange, yellow, or their hues and shades), which can fill the corresponding image portion of the visualization frame. Similarly, lower weight values ​​can be mapped to cooler colors (e.g., blue, green, purple, or their hues and shades), which can fill the corresponding image portion of the visualization frame.

[0046] Figure 2 This is a block diagram illustrating an exemplary electronic device for thermal image visualization of object detection according to an embodiment of the present disclosure. Figure 2 Combination Figure 1 The elements in the text will be explained. (See reference.) Figure 2 A block diagram 200 of an electronic device 102 is shown. The electronic device 102 may include a circuit system 202, a memory 204, an input / output (I / O) device 206, and a network interface 208. The circuit system 202 may be communicatively coupled to the memory 204, the I / O device 206, and the network interface 208. In some embodiments, the I / O device may include a display device (such as...) Figure 1 Display device 104).

[0047] Circuit system 202 may include suitable logic, circuitry, and / or interfaces that can be configured to execute program instructions associated with different operations to be performed by electronic device 102. Circuit system 202 may include one or more dedicated processing units, which may be implemented as integrated processors or clusters of processors that collectively perform the functions of one or more dedicated processing units. Circuit system 202 may be implemented based on a variety of processor technologies known in the art. Examples of implementations of circuit system 202 may include x86-based processors, graphics processing units (GPUs), reduced instruction set computing (RISC) processors, application-specific integrated circuit (ASIC) processors, complex instruction set computing (CISC) processors, microcontrollers, central processing units (CPUs), and / or other computing circuitry.

[0048] Memory 204 may include suitable logic, circuitry, and / or interfaces configured to store program instructions to be executed by circuitry 202. In at least one embodiment, memory 204 may be configured to store a first image frame, a generated visualization framework, and a DNN model 106. Memory 204 may be configured to store a color scheme to be used for heatmap visualization. Example implementations of memory 204 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drive (HDD), solid-state drive (SSD), CPU cache, and / or secure digital card (SD card).

[0049] I / O device 206 may include suitable logic, circuitry, interfaces, and / or code that can be configured to receive input and provide output based on the received input. I / O device 206 may include various input and output devices that can be configured to communicate with circuitry 202. For example, electronic device 102 may receive user input via I / O device 206 to select an object of interest from multiple objects in an image frame and initiate an XAI workflow (which includes an object detection workflow and a heatmap visualization workflow). Examples of I / O device 206 may include, but are not limited to, a touchscreen, keyboard, mouse, joystick, display device (e.g., display device 104), microphone, or speaker.

[0050] Network interface 208 may include appropriate logic, circuitry, interfaces, and / or code that can be configured to facilitate communication between circuitry 202 and display device 104 and / or other communication devices via communication network 108. Network interface 208 can be implemented using various known technologies to support wireless communication of electronic device 102 via communication network 108. Network interface 208 may include, for example, an antenna, radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, an encoder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, local buffer circuitry, etc.

[0051] Network interface 208 can be configured to communicate wirelessly with networks such as the Internet, intranets, wireless networks, cellular telephone networks, wireless local area networks (LANs), or metropolitan area networks (MANs). Wireless communication can be configured to use one or more of various communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Li-Fi, or Wi-MAX.

[0052] like Figure 1 The functions or operations performed by electronic device 102 as described herein can be performed by circuit system 202. For example, in Figure 3A and 3B , Figure 4 , Figure 5 , Figure 6 and Figure 7 The operation performed by the circuit system 202 is described in detail.

[0053] Figure 3A and 3B These are diagrams illustrating exemplary operations for thermal mapping visualization of object detection according to embodiments of the present disclosure. Figure 3A and 3B Combination Figure 1 and Figure 2 The elements in the text will be explained. (See reference.) Figure 3A and 3B A block diagram 300 illustrates exemplary operations from 302 to 322 as described herein. The exemplary operations shown in block diagram 300 may begin at 302 and can be performed by any computing system, apparatus, or device, such as by [unclear text - likely a typo]. Figure 1 or Figure 2 The electronic device 102 performs the operation. Although illustrated with discrete blocks, exemplary operations associated with one or more blocks of block diagram 300 may be partitioned into additional blocks, combined into fewer blocks, or eliminated, depending on the implementation of the exemplary operations.

[0054] At 302, image data can be acquired. In an embodiment, circuitry 202 can acquire image data from a data source. For example, the data source may be an onboard image sensor, persistent storage on electronic device 102, an image capture device, a cloud server, or a combination thereof. Image data may include image frames that may be captured at different times. Alternatively, image data may include a series of image frames from a video feed. Image data may represent a static scene with fixed foreground or background objects or a dynamic scene with moving objects. Each image frame of the acquired image data may include at least one object of interest. From the acquired image data, circuitry 202 can receive a first image frame 302A. As shown, for example, the first image frame 302A depicts a scene from a sports match and includes a football player as an object of interest 302B.

[0055] After acquisition, the circuit system 202 can input the first image frame 302A into the DNN model 106. In some embodiments, the circuit system 202 can adjust the size of the input first image frame 302A to match the size of the input layer of the DNN model 106, and then pass the resized first image frame 302A to the input layer of the DNN model 106.

[0056] At point 304, the detection result can be extracted from the DNN model 106. In an embodiment, the circuit system 202 can extract the object detection result associated with the object of interest 302B as the output of the DNN model 106 for the input first image frame 302A (or the resized first image frame 302A). For example, the object detection result can be a vector that may include the object score 304A of the object of interest 302B, the bounding box coordinates 304B predicted to include the bounding box of the object of interest 302B, and the class score 304C (or class probability) for the object of interest 302B.

[0057] For example, for 3-class object detection, the vector (Y) can be an 8-dimensional vector, as shown below:

[0058]

[0059] in,

[0060] O cThe object score 304A indicates whether the object of interest 302B exists in the input first image frame 302A (or the resized first image frame 302A).

[0061] bx, by, bh, bw specify the bounding box coordinates as 304B, and

[0062] c1, c2, and c3 represent the class fraction 304C.

[0063] The object score 304A can specify the confidence level of the DNN model 106 in detecting the object of interest 302B in the input first image frame 302A (or the resized first image frame 302A). In some cases, the object score 304A can be a binary value (0 or 1) or a floating-point number representing the probability between 0 and 1. A higher object score can indicate a higher level of confidence for the DNN model 106. In some embodiments, a threshold object score (e.g., 0.95) can be set to filter out object detection results with a confidence level lower than the threshold object score. The class score 304C can include soft labels or hard labels to indicate the object class that can be detected by the DNN model 106. The object of interest 302B can belong to one of such classes. For example, if the DNN model 106 detects the object of interest 302B in class c1, then the class score 304C of the object of interest 302B can be 1 (or a number close to 1) for class c1 and 0 (or close to 0) for each of classes c2 and c3.

[0064] In some embodiments, the DNN model 106 may employ a region-based or grid-based detection method, whereby the DNN model 106 can divide the input first image frame 302A (or a resized first image frame 302A) into multiple grid cells and perform a cell-by-cell search to detect one or more objects of interest 302B in the multiple grid cells. When multiple objects of interest overlap in a shared grid cell, anchor boxes can be specified to create longer grid cells, and multiple objects can be associated with the shared grid cell. When multiple objects of interest exist in different grid cells, the DNN model 106 can output object detection results for each of the different grid cells associated with the multiple objects of interest. After extracting the object detection results, operations from 306 to 322 can be performed for heatmap visualization of the object of interest 302B, as described herein.

[0065] At point 306, image segmentation can be performed. In one embodiment, circuitry 202 can segment the first image frame 302A into a first plurality of image portions 306A...306N based on bounding box coordinates 304B. In another embodiment, the size of the first image frame 302A can be adjusted to match the size of the input layer of the DNN model 106 before segmentation is performed. For example, if the first image frame 302A is a high-resolution image (1280x720 pixels) but the size of the input layer is 416x416, then the size of the first image frame 302A can be adjusted to 416x416 pixels.

[0066] The first image frame 302A (or a resized first image frame 302A) can be passed to a dynamic window segmentation function, which segments the first image frame 302A (or the resized first image frame 302A) into a first plurality of image segments 306A...306N based on the window size. In an embodiment, the circuit system 202 can select the window size for segmentation based on the input dimension (or pixel-level size) of the first image frame 302A (or the resized first image frame 302A) and the bounding box coordinates 304B. Based on the selected window size, the circuit system 202 can segment the first image frame 302A (or the resized first image frame 302A) into the first plurality of image segments 306A...306N. A unique fragment ID can be assigned to the pixels belonging to each image segment.

[0067] In this embodiment, the dynamic window segmentation function can divide the first image frame 302A (or the resized first image frame 302A) into a square or rectangular grid, wherein each grid cell can include one of a first plurality of image portions 306A...306N. Depending on the size of the object or the image scale, the object of interest 302B can exist in a single grid cell or can span multiple grid cells of a square or rectangular grid, for example in... Figure 7 As described in [the text].

[0068] At position 308, a perturbed image (i.e., a unique neighborhood image) can be generated. In an embodiment, circuit system 202 can generate multiple perturbed images 308A…308N. To generate each perturbed image, a random sample selection operation and a masking operation can be performed, as described herein. These two operations can be performed iteratively to obtain multiple perturbed images 308A…308N.

[0069] Initially, circuit system 202 can perform random sample selection from a first set of image portions 306A...306N. For example, the input first image frame 302A is shown as being divided into a 5x5 grid of 25 image portions using a window segmentation function. Circuit system 202 can randomly select 8 grid cells to select 8 image portions from the 5x5 grid of 25 image portions. The selection of grid cells (e.g., 8) can be based on the random selection of fragment IDs, which can be assigned to each image portion at 306. It should be noted that the grid size (5x5) mentioned above is provided only as an example. This disclosure can be applied to any suitable grid size without departing from the scope of this disclosure.

[0070] After performing random sample selection, the circuit system 202 can select a portion of a first image frame 302A (or a resized first image frame 302A) comprising a first set of image portions (obtained through random sample selection). The circuit system 202 can then apply a mask to the selected portion to generate a first perturbed image 308A among a plurality of perturbed images 308A…308N. For example, if the first image frame 302A (or the resized first image frame 302A) has an 8-bit color depth, the mask can be applied by setting all pixels of the selected portion to “126” or other suitable values ​​between 0 and 255.

[0071] At 310, label data can be extracted. The label data may include multiple object detection scores 310A corresponding to the generated multiple perturbed images 308A…308N. For this extraction, the cell tracking operation at 312, as described herein, can be performed while the DNN model 106 is reapplied to the multiple perturbed images 308A…308N to extract label data from the DNN model 106.

[0072] Initially, the circuit system 202 can determine the first active cell from among multiple active cells of the DNN model 106 that can be responsible for the object detection result (obtained from the DNN model 106 at 304). Each active cell and anchor in the DNN model 106 can be assigned a unique ID. One of the cell anchors (i.e., the first active cell responsible for detection) can be selected and used as a reference point. Subsequently, the circuit system 202 can input the first perturbed image 308A from multiple perturbed images 308A…308N into the DNN model 106, and can extract a first object detection score for the first perturbed image 308A from the first output of the selected first active cell of the DNN model 106. The first object detection score can be one of the following: multiple object detection scores 310A corresponding to the multiple perturbed images 308A…308N.

[0073] As part of the iterative feeding process, each of the multiple perturbed images 308A…308N can be passed as input to the DNN model 106, and a reference point (i.e., the first active cell or the selected cell-anchor) can be tracked throughout the iterative feeding process. The output value from the reference point (i.e., the first active cell) (such as the first object detection score) can be regarded as the first sample of the labeled data.

[0074] If multiple objects of interest (non-overlapping) exist in the first image frame 302A, the DNN model 106 can identify and select multiple activation cells, each of which can be associated with one of the multiple objects of interest. In this case, the circuit system 202 can track such multiple activation cells to detect the object detection score of each of the multiple objects of interest throughout the iterative feed.

[0075] At position 314, multiple feature vectors can be extracted. These feature vectors can also be referred to as unique segmentation features. In an embodiment, circuit system 202 can generate a first feature vector from a plurality of feature vectors for a first perturbed image 308A among a plurality of perturbed images 308A…308N. The generated first feature vector may include multiple vector elements. Each vector element may correspond to a corresponding image portion (or a fragment ID of the corresponding image portion) among the first plurality of image portions 306A…306N. For example, if a first image frame 302A (or a resized first image frame 302A) is segmented into a 10x10 grid comprising 100 image portions, each image portion having a unique fragment ID, then the first feature vector may be a 1x100 vector with 100 vector elements. Each of the multiple vector elements may store a value that indicates whether the corresponding image portion is masked in the first perturbed image 308A. For example, for a 5x5 window segmentation, if the first perturbed image 308A comprises 10 and 15 image parts respectively as masked and unmasked, then the first feature vector can store the value (0) in 10 vector elements (corresponding to the 10 image parts) and the value (1) in 15 vector elements (corresponding to the 15 image parts). Similarly, feature vectors for all perturbed images can be generated.

[0076] At point 316, a regression model can be trained. In an embodiment, circuit system 202 can train the regression model on multiple feature vectors (extracted at 314) and multiple object detection scores 310A (extracted as label data at 310). The regression model can be described by a response variable, explanatory variables, and unknown model parameters (called weights) that quantify the strength of the relationship between the explanatory and response variables. Here, the multiple feature vectors can be referred to as explanatory variables (i.e., independent variables) and the multiple object detection scores 310A can be referred to as response variables. During training, the values ​​of the unknown model parameters, i.e., the weights, can be estimated by fitting the regression model on the multiple feature vectors and the multiple object detection scores 310A. Cost functions, such as least squares error (LS), can be used to minimize the fitting error associated with the regression model.

[0077] At point 318, weight values ​​can be determined. In an embodiment, circuit system 202 can determine the weight value of each of the first plurality of image portions 306A...306N based on a trained regression model. The weight value can indicate the probability that a corresponding image portion among the first plurality of image portions 306A...306N belongs to the object of interest 302B.

[0078] By using a unique fragment ID, each weight value can be mapped to a corresponding image portion of the first image frame 302A (or a resized first image frame 302A). All image portions belonging to the object of interest 302B (obtained after segmentation at 306) can be associated with high weight values ​​(e.g., values ​​close to 1), while other image portions can be associated with low weight values ​​(e.g., values ​​close to 0).

[0079] At position 320, an interpretation generation operation can be performed. Within this operation, heatmap generation can be performed at position 322, followed by metric calculation at position 324.

[0080] At 322, a heatmap can be generated. In an embodiment, circuit system 202 can generate a visualization frame 322A based on weight values ​​determined for each of the first plurality of image portions 306A...306N. The visualization frame 322A may include a heatmap visualization 322B of the region included in the first image frame 302A and defined by bounding box coordinates 304B.

[0081] In an embodiment, circuit system 202 may select color values ​​representing the determined weight values ​​from a heatmap color palette. The color values ​​may be selected based on a heatmap color scheme for the range of weight values. For example, the heatmap color palette may be a rainbow palette, which may represent low weight values ​​with cool colors (e.g., blue, green, purple, shadow, or hues thereof) and high weight values ​​with warm colors (e.g., red, yellow, orange, shadow, or hues thereof). Other examples of heatmap color palettes may include, but are not limited to, sequential palettes, divergent palettes, or qualitative palettes. Subsequently, based on the color values ​​selected for each of the first plurality of image portions 306A...306N, a visualization frame 322A may be generated. After generating the visualization frame 322A, circuit system 202 may display the visualization frame 322A on a display device (such as display device 104) to provide a user-friendly and reliable heatmap visualization of the object of interest 302B (as detected in the first image frame 302A).

[0082] At position 324, metric calculations for heatmap visualization can be performed. In an embodiment, circuitry 202 can select a bounding box (i.e., predicted by DNN model 106) surrounding the object of interest 302B in the input first image frame 302A (or a resized first image frame 302A). The selected bounding box can be determined by the bounding box coordinates in the object detection results of DNN model 106.

[0083] Circuit system 202 can determine a first count (N) of weight values ​​that can be greater than zero and are located within the selected bounding box. inside+ Circuit system 202 can also determine a second count (N) of weight values ​​that can be equal to zero within the selected bounding box. inside0 ). In the first count of the weight values ​​(N) inside+ The second count of the weighted values ​​(N) can indicate the number of image portions in the first image frame 302A that can be part of the object of interest 302B and can be located within the selected bounding box. inside0 This can indicate the remaining number of image portions that may not be part of the object of interest 302B but may still be located within the selected bounding box. Both a first count and a second count of the weight values ​​can be determined based on weight values ​​determined for each of the first plurality of image portions 306A...306N. Subsequently, the circuit system 202 can transmit the first visual metric (F... weight The first visual measure (F) is calculated as the ratio of a determined first count to the sum of a determined first count and a determined second count. weight The formula for ) is provided by equation (1), as follows:

[0084]

[0085] The first visual metric can represent the accuracy of determining each weight value through a regression model. Higher accuracy can require a lower second count of weight values ​​compared to the first count. This can mean that the heatmap visualization can be required to have maximum coverage of the object of interest within the focal block of the visualization framework, at a rate of 302B.

[0086] In an embodiment, circuit system 202 can determine a first set (W) of weight values ​​that can be greater than zero within a selected bounding box. x x∈{0,M}and W x The first sum of weights greater than 0. Circuit system 202 can also determine a second set (W) of weight values ​​that can be greater than zero outside the selected bounding box. y ,y∈{0,N}, And W y The second sum of >0). Both the first and second sets of weight values ​​can be determined based on the weight values ​​determined for the first plurality of image portions 306A...306N. Circuit system 202 can transmit the second visual metric (F) spread The second visual measure (F) is calculated as the ratio of a determined first sum to the sum of the determined first sum and the determined second sum. spread The formula for ) is provided by equation (2), as follows:

[0087]

[0088] in,

[0089] x can represent the index of a weight value in the first set of weight values, and

[0090] y can represent the index of the weight value in the second set of weight values.

[0091] Such metrics (first-visual metrics or second-visual metrics) can help users or program developers evaluate or retrain DNN models106 to obtain better and more accurate object detection results, which can lead to accurate heatmap visualization.

[0092] Figure 3C This is a diagram illustrating an exemplary operation for calculating heatmap metrics for video according to an embodiment of the present disclosure. Figure 3C Combination Figure 1 , Figure 2 , Figure 3A and Figure 3B The elements in the text will be explained. (See reference.) Figure 3C This illustrates a series of exemplary operations from 324A to 324C, which are Figure 3B This is part of the measurement calculation at position 324. The operation can begin at 324A and can be performed by any computing system, device, or equipment, such as... Figure 1 or Figure 2 The electronic device 102 performs the operation.

[0093] At 324A, a series of visualization frames 326A, 326B, 326C, 326D...326N can be extracted. In an embodiment, circuit system 202 can extract a series of visualization frames 326A, 326B, 326C, 326D...326N from memory 204. The series of visualization frames 326A, 326B, 326C, 326D...326N can be generated frame-by-frame based on the heatmap visualization operation at 322. Each visualization frame in the series of visualization frames 326A, 326B, 326C, 326D...326N can include a heatmap visualization of a region that can be included in a corresponding input image frame of the video and can be defined by bounding box coordinates. For example, the first visualization frame 326A can include a heatmap visualization of a region that can be included within a bounding box 328A. This region can correspond to an object of interest (such as the body of a cyclist).

[0094] At 324B, per-frame metric calculation can be performed. In an embodiment, circuitry 202 can calculate a first visual metric (F) for each of a series of visualization frames 326A, 326B, 326C, 326D...326N. weight ) and second visual metric (F spread ).

[0095] Each visualization frame can be divided into multiple segments, the colors of which can depend on the weight values ​​associated with the corresponding segments and color palettes. For example, if the weight value is close to 1, the colors of the associated segments of the visualization frame can be close to warm colors, such as red. However, if the weight value is close to 0, the colors of the associated segments of the visualization frame can be close to cool colors, such as blue. The number of segments determines the resolution of the heatmap visualization. More segments result in a higher resolution heatmap visualization, while fewer segments result in a lower resolution.

[0096] For example, as shown in the figure, the first visualization frame 326A includes 10x10 segments or 100 segments, and the region within the delimiting box 328 includes 12 segments. For the first visualization frame 326A, the count of positive weight values ​​within the delimiting box 328 is 7, and the count of zero-value weight values ​​within the delimiting box 328 is 5. First visual metric (F) weight The following can be determined using equation (1):

[0097]

[0098] For the first visualization framework 326A, the second visual metric (F) can be determined using equation (2). spread ),as follows:

[0099]

[0100]

[0101] From the aforementioned example, the first visualization frame 326A and F = 0.583 weight And 0.477 F spread Related. F spread A higher F value indicates more high-weight segments within the bounding box 328, thus indicating more weighted blocks on the object of interest. weight The value can indicate how many high-weight segments are located within the bounding box 328, thus indicating how many focus blocks are on the object of interest.

[0102] At point 324C, a total metric calculation can be performed for a series of visualization frames 326A, 326B, 326C, 326D...326N. Afterwards, F can be used for each of the visualization frames 326A, 326B, 326C, 326D...326N. spread and F weight The values ​​are recorded in the database, as shown in Table 1 below:

[0103] Visualization framework# <![CDATA[F spread ]]> <![CDATA[F weight ]]> 326A 0.477 0.583 326B 0.412 0.625 326C 0.503 0.590 ... ... ... 326N 0.725 0.610

[0104] Table 1: F spread and F weight value

[0105] Based on the first visual metric and the second visual metric (i.e., F) weight and F spread By analyzing the values ​​(values), the circuit system 202 can determine whether a heatmap visualization has been correctly generated. If such values ​​are above a threshold, then it can be determined that the object of interest is identified in all corresponding frames of the video. In some embodiments, the circuit system 202 can generate a graph representing such values ​​for each of a series of visualization frames 326A, 326B, 326C, 326D...326N. Based on the peaks in the graph, the circuit system 202 can determine the next action, such as inferences related to the object of interest. Even if the object is small, the graph can help in interpretation and conclusions.

[0106] Figure 4 The illustration shows the use of an embodiment according to this disclosure. Figure 2 A block diagram illustrating an exemplary operation of an end-to-end artificial intelligence (XAI) solution for electronic devices. Figure 4 Combination Figure 1 , 2 Explain the elements in 3A and 3B. (See reference.) Figure 4 A block diagram 400 illustrates exemplary operations from 402 to 420 as described herein. The exemplary operations shown in block diagram 400 can begin at 402 and can be performed by any computing system, apparatus, or device, such as those described herein. Figure 1 or Figure 2 The electronic device 102 performs the operation.

[0107] Although illustrated using discrete blocks, exemplary operations associated with one or more blocks of block diagram 400 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the implementation of the exemplary operations.

[0108] At position 402, a perturbed image can be generated. In an embodiment, circuitry 202 can generate multiple perturbed images from the first image frame 412A. To generate each perturbed image, a random sample selection operation and a masking operation can be performed. Details of such operations are described in... Figure 3A Provided in, for example, from Figure 3A 306 to 308. For example, in Figure 6 The text provides a detailed description of random sample selection and masking operations.

[0109] At position 404, a cell plugin can be used to determine the cell. In an embodiment, circuitry 202 can determine the first active cell from a plurality of active cells in the DNN model 106 that can be responsible for object detection results (such as the object detection result at position 304). Each active cell and anchor in the DNN model 106 can be assigned a unique ID. One of the cell anchors (i.e., the first active cell responsible for detection) can be selected and used as a reference point.

[0110] At 406, interpretation generation can be performed. In an embodiment, circuitry 202 can generate an interpretation as input, associated with the object detection results of the DNN model 106 for the first image frame 412A. The operation at 406 may include a first operation of heatmap generation at 408 and a second operation of metric calculation at 410.

[0111] At point 408, a heatmap can be generated. In an embodiment, circuit system 202 can be based on... Figure 3B The operation at point 322 is performed to generate the visualization frame 412C. The visualization frame 412C may include elements included in the first image frame 412A and defined by bounding box coordinates (by...). Figure 4 The heatmap visualization of the region 412B is represented by the bounding box in the image.

[0112] At 410, a metric calculation can be performed. In an embodiment, circuit system 202 can determine a first visual metric (F) of visualization frame 412C based on the weight values ​​of the segments used for visualization frame 412C. weight ) and second visual metric (F spread In the presence of a range of visualization frameworks (e.g., such as...), Figure 3C As described in [the document], the circuit system 202 can determine a first visual metric (F) for each of a series of visualization frames. weight ) and second visual metric (F spread For example, in Figure 3B and Figure 3C The details of the metric calculation are provided in the document.

[0113] At 412, a selection can be made. After generating the visualization frame 412C, the user can be provided with the option to view a heatmap visualization of the region 412B including the object of interest on the display device 104. In the presence of multiple visualization frames (such as a series of visualization frames 326A, 326B, 326C, 326D...326N) corresponding to multiple image frames, the circuit system 202 can display such visualization frames on the display device 104. In some embodiments, the circuit system 202 can also display a graph that may include a first visual metric and a second visual metric for each such visualization frame. The user can decide to select any particular visualization frame among the displayed visualization frames. For example, the user can select the visualization frame corresponding to the peak of the first or second visual metric in the graph.

[0114] In the case where visualization frame 412C includes heatmap visualizations of multiple objects, a user can select and view a heatmap visualization of one of the multiple objects. In an embodiment, circuitry system 202 can select an object of interest from visualization frame 412C based on user input. A selection can be performed to display a heatmap of the selected object of interest. Circuitry system 202 can update the visualization frame to include a heatmap visualization of region 412B, which includes the selected object of interest. In some embodiments, if a series of visualization frames (such as a heatmap video) exists, the heatmap visualization of the selected object of interest can be updated across the series of visualization frames, while heatmap visualizations of regions including other objects (not selected by the user) can be removed from the series of visualization frames.

[0115] At point 414, when the visualization frame 412C is displayed on the display device 104, a user query can be received. In this embodiment, the circuit system 202 can receive the user query via the display device 104. The user query enables the end user to make a choice regarding selecting an object of interest for explanation.

[0116] At 416, the inference result can be determined. In an embodiment, circuit system 202 can determine the inference result associated with the selected object of interest based on a user query (received at 412). In an embodiment, circuit system 202 can extract a set of observations associated with the selected object of interest. The set of observations can be extracted based on the generated visualization framework 412C. In an embodiment, the set of observations can also be extracted based on a first visual metric and a second visual metric used for visualization framework 412C. For example, in Figure 3B and Figure 3C The details associated with the first and second visual metrics are provided. Circuit system 202 can determine inferences associated with a selected object of interest based on the extracted set of observations. Examples of the determined inferences may include, but are not limited to, numerical values ​​of the state or visual attributes of the selected object of interest. Examples of state or visual attributes may include, but are not limited to, changes in object size, object movement or rotation, object deformation (e.g., changes in human posture), or object visibility (such as blur values), object motion (such as joint movements of a human object), or object interaction with other objects (such as a human riding a bicycle).

[0117] For example, when the extracted observations include an increase in the visibility of the object of interest in successive image frames, the determined inference result may include a higher (above a threshold) value. Conversely, when the observations include a decrease in the visibility of the object of interest in successive image frames, the determined inference result may include a lower (below a threshold) value. In an embodiment, the circuit system 202 may generate a graphical representation of the determined inference result.

[0118] At point 418, an explanation can be generated. In an embodiment, circuit system 202 can generate an explanation based on a determined inference result. The generated explanation may include a natural language description of the determined inference result. In an embodiment, the circuit system may apply natural language processing (NLP) techniques to generate the natural language description. In an embodiment, circuit system 202 may control a display device (such as display device 104) to display the generated explanation. For example, based on the inference result for a series of visualization frames (including heatmap visualization), it can be determined that the visibility of an object of interest (selected by the user) increases in successive frames, and the explanation may include the natural language description "object visibility is increasing".

[0119] In an embodiment, circuit system 202 can determine multiple recommendations associated with the generated interpretation based on the determined inference result. The multiple recommendations may include, for example, feedback to a data source or feedback to an external system. Circuit system 202 can control display device 104 to display the generated multiple recommendations associated with the determined interpretation. Circuit system 202 can also receive input including selecting a first recommendation from the multiple recommendations. Circuit system 202 can select the first recommendation from the multiple recommendations based on the received input, and can adjust certain image parameters associated with the input first image frame 410A based on the selected first recommendation. For example, in Figure 5 The details described are related to the recommendations.

[0120] At 422, input adjustment can be performed. In an embodiment, circuitry 202 can receive user input, which includes the selection of a visualization frame with a lower value for a first visual metric or a second visual metric. In this case, circuitry 202 can generate instructions to adjust at least one imaging parameter of the input source (such as a camera that initially acquires multiple images). For example, if a visualization frame with a lower value for a first visual metric or a second visual metric is selected from an underexposed image frame, then circuitry 202 can generate instructions to adjust the sensor gain of the input source.

[0121] Figure 5 This is a diagram illustrating an exemplary user interface of an electronic device for thermal image visualization of object detection according to an embodiment of the present disclosure. Figure 5 Combination Figure 1 , 2 Explain the elements in 3A, 3B, and 4. (See reference.) Figure 5 This shows the user interface (UI) 500, which can display... Figure 4 The definitive inference results, and Figure 4 The circuit system 202 controls the display device 104 to display the UI 500. For example, as shown in the figure, the UI 500 can display an interface for user queries 502, inference results 504, and explanations 506.

[0122] In one embodiment, the input first image frame 508 may depict a person riding a bicycle. The circuit system 202 may receive input including the selection of an object of interest 508A. The circuit system 202 may select the object of interest 508A based on the received input. For example, the selected object of interest 508A may be the pedals of a bicycle.

[0123] In an embodiment, circuit system 202 can determine an inference result 504 associated with a selected object of interest 508A. For example, the inference result 504 associated with the selected object of interest 508A is shown as a graphical representation 510. In the graphical representation 510, the inference result 504 includes a numerical value representing the state of the selected object of interest 508A. For example, the numerical value may include a revolutions per minute (RPM) measurement associated with the movement state of the pedal. Such a measurement may be based on... Figure 3B or Figure 3C The metric calculation operation at position 324 is used for calculation.

[0124] In an embodiment, circuitry 202 may generate an explanation 506 based on a determined inference result 504. Explanation 506 may include a natural language description of the determined inference result 504. As shown, explanation 506 may include a description of "the input being a portion of a 5-minute video feed of a cyclist riding a bicycle at an average speed of 10 revolutions per minute." It should be noted that the information shown in UI 500 is presented as an example only and should not be construed as limiting the scope of this disclosure. In some embodiments, circuitry 202 may transmit feedback to an image source, such as an image capture device, to adjust imaging parameters associated with the source (such as aperture value, sensor gain, or shutter speed).

[0125] Figure 6 This is a diagram illustrating an exemplary scene for generating a first perturbed image according to an embodiment of the present disclosure. Figure 6 Combination Figure 1 , 2 Explain the elements in 3A, 3B, 4, and 5. (See reference.) Figure 6 The image shows a first image frame 602 and a first disturbed image 604 that can be obtained from the first image frame 602.

[0126] In an embodiment, the first image frame 602 may have a size of 300x50 pixels and the window size of the input layer of the DNN model 106 may be 10x10. The first image frame 602 may be passed to a dynamic window segmentation function (as described at 306) that can segment the first image frame 602 into a first plurality of image portions based on the window size. For example, the first image frame 602 is shown as being segmented into a 5x30 grid of 150 image portions using the window segmentation function. The DNN model 106 may assign a unique ID to each segment, such as 0, 1, 2, 3, ..., 149, as shown in the figure.

[0127] In an embodiment, circuit system 202 can perform random sample selection from a first set of image portions (labeled 0, 1, 2, 3, ..., 149) (as shown in FIG3). After performing random sample selection, circuit system 202 can select a portion of a first image frame 602, which includes the first set of image portions (obtained through random sample selection). Thereafter, circuit system 202 can apply a mask on the selected portion to generate a first perturbed image 604. For example, if the first image frame 602 has an 8-bit color depth, then the mask can be applied by setting all pixels of the selected portion to "126" or other suitable values ​​between 0 and 255. For example, for a 5x30 window segmentation, if the first perturbed image 604 includes 22 image portions and 128 image portions as masked and unmasked, respectively, then the first feature vector can store the value (0) in 22 vector elements (corresponding to the 22 image portions) and the value (1) in 128 vector elements (corresponding to the 128 image portions). exist Figure 6 The diagram illustrates a feature vector that can store a value (1) for a segment with a unique ID "0" (to indicate unmasked). The feature vector can also store a value (0) for another segment with a unique ID "7" (to indicate masked). Similarly, feature vectors can be generated for all perturbed images. For example, the feature vector could have a dimension of 1 x num_segment, where num_segment is the number of unique segments. It is a binary vector with elements that are either 1 or 0. If the corresponding segment is masked, the value at the index in the feature vector is 0. If the corresponding segment is not masked, the value at the index in the feature vector is 1.

[0128] It should be noted that the first image frame 602 and the first disturbed image 604 are presented as examples only and should not be construed as limiting the scope of this disclosure.

[0129] Figure 7 This is a diagram illustrating an exemplary relationship between a dynamic window segmentation function according to an embodiment of the present disclosure and heatmaps of objects with different window sizes. Figure 7 Combination Figure 1 , 2 Explain the elements in 3A, 3B, 4, 5, and 6. (See reference.) Figure 7The diagram illustrates a first visualization frame 702, a second visualization frame 704, and a third visualization frame 706. The first visualization frame 702 may include a heatmap visualization 708A of an object of interest 708, and a delimiting box 710A surrounding the region including the heatmap visualization 708A of the object of interest 708. The second visualization frame 704 may include a heatmap visualization 708B of the object of interest 708, and a delimiting box 710B surrounding the region including the heatmap visualization 708B of the object of interest 708. The third visualization frame 706 may include a heatmap visualization 708C of the object of interest 708, and a delimiting box 710C surrounding the region including the heatmap visualization 708C of the object of interest 708. Enlarged views of heatmap visualizations 708A, 708B, and 708C are also shown.

[0130] Dynamic window segmentation allows control over the coarseness of heatmap visualizations. Smaller window sizes result in finer heatmap visualizations, and vice versa. To achieve the optimal window size, a method can be implemented where the bounding box dimensions and input image dimensions contribute to the dynamic selection of the window size. This helps provide more refined heatmaps.

[0131] In the first visualization frame 702, the window size is larger and the heatmap visualization 708A is coarser. This results in more noise in the focus area. In the second visualization frame 704, the window size is smaller than that of the first visualization frame 702. Therefore, there is less noise in the focus area than in the first visualization frame 702, and the heatmap visualization 708B is more accurate than the heatmap visualization 708A. In the third visualization frame 706, the window size is smaller than that of the second visualization frame 704. Therefore, there is less noise in the focus area than in the second visualization frame 704, and the heatmap visualization 708C is more accurate than the heatmap visualization 708B. For bounding boxes of the same size, when the window is larger, metric calculations (such as...) Figure 3B The calculation at position 322 is not very accurate. However, the measurement is more accurate for smaller window sizes.

[0132] Object size can be one of the factors determining window size. Therefore, in at least one embodiment, the circuit system 202 changes the window size based on the object size. In multi-object scenes, such as when two objects exist in the input image, heatmap visualizations can be generated for both objects. However, because the window sizes used for the two objects are different, the heatmap visualization of one object may be coarser than that of the other. These different window sizes can depend on the object sizes of the two objects.

[0133] Figure 8This is a diagram illustrating exemplary operations for selecting a single object from multiple object detections and for heatmap visualization based on such selection, according to embodiments of the present disclosure. Figure 8 Combination Figure 1 , 2 Explain the elements in 3A, 3B, 4, 5, 6, and 7. (See reference.) Figure 8 Block diagram 800 is shown. In block diagram 800, a sequence of operations from 802 to 806 is shown for a first image frame 808 that may include multiple objects, such as a first object 810, a second object 812, and a third object 814.

[0134] At 802, the circuit system 202 can input the first image frame 808 into a DNN model (such as DNN model 106) and can extract multiple object detection results associated with multiple objects as the output of DNN model 106 for the input first image frame 808.

[0135] At 804, circuitry 202 can control a display device (such as display device 104) to display multiple bounding boxes (e.g., bounding box 804A, bounding box 804B, and bounding box 804C) surrounding multiple objects (such as first object 810, second object 812, and third object 814) in the input first image frame 808. The multiple bounding boxes can be displayed based on the extracted object detection results, for example, as in... Figure 3A As described herein, circuit system 202 can receive input that instructs selection of a first bounding box (such as bounding box 804B) from a plurality of displayed bounding boxes. Based on the received input, circuit system 202 can select an object of interest (such as a second object 812) from a plurality of objects.

[0136] At 806, circuit system 202 can be based on Figure 3A and Figure 3B The execution of the set of operations from 306 to 322 generates a visualization framework 806A for a selected object of interest (such as the second object 812). The visualization framework 806A may include a heatmap visualization of a region 806B (which may include the selected object of interest (such as the second object 812)), for example in... Figure 3B As described in [the text].

[0137] Figure 9 This is a diagram illustrating an exemplary scenario for thermal visualization of two partially overlapping objects in an image frame according to an embodiment of the present disclosure. Figure 9 Combination Figure 1 , 2 Explain the elements in 3A, 3B, 4, 5, 6, 7, and 8. (See reference) Figure 9The image shows a first image frame 902 that may include multiple objects, such as a first object 904 and a second object 906. As shown, the first object 904 (i.e., the tram) is at least partially overlapped or occluded by the second object 906 (i.e., the car). When the first image frame 902 is passed as input to a DNN model 106, the DNN model 106 can output object detection results for each of the first object 904 and the second object 906.

[0138] If there is no input from a user (such as user 110), the circuit system 202 may select both the first object 904 and the second object 906 as two separate objects of interest. Alternatively, the circuit system 202 may treat the first object 904 and the second object 906 as a lumped object and may generate a visualization framework with a heatmap visualization of the region including the lumped object.

[0139] In at least one embodiment, circuit system 202 can control a display device (such as display device 104) to display an image 908, which shows a bounding box 910A surrounding a first object 904 and a bounding box 910B surrounding a second object 906. For the first object 904 and the second object 906, the corresponding bounding boxes (i.e., bounding boxes 910A and 910B) can be shown based on the bounding box coordinates in the corresponding object detection results, for example, as in... Figure 3A As described herein, circuit system 202 can receive input that may indicate the selection of a bounding box surrounding either a first object 904 or a second object 906. Based on the received input, circuit system 202 can select an object of interest as an object surrounded by the selected bounding box. For the selected object of interest, operations from 306 to 322 can be performed to generate a visualization framework that includes a heatmap visualization of a focus block or region containing the selected object of interest. For example, if the first object 904 is selected, then visualization framework 912 may include a heatmap visualization of a focus block 912A or region containing the first object 904.

[0140] It should be noted that the above description can also be applied to another scenario in which two or more objects at least partially overlap or occlude each other.

[0141] Figure 10 This is a diagram illustrating an exemplary scenario for performing heatmap visualization of an object located within a region of another object in an image frame, according to an embodiment of the present disclosure. Figure 10 Combination Figure 1 , 2 Explain the elements in 3A, 3B, 4, 5, 6, 7, 8, and 9. (See reference.) Figure 10The image shows a first image frame 1002 including a first object 1004 and a second object 1006, wherein the second object 1006 is included within the boundary 1008 of the first object 1004. When the first image frame 1002 is passed as input to the DNN model 106, the DNN model 106 can output object detection results for each of the first object 1004 and the second object 1006.

[0142] If there is no input from the user, the circuit system 202 can select both the first object 1004 and the second object 1006 as two separate objects of interest. Alternatively, the first object 1004 and the second object 1006 can be selected as a lumped object. The circuit system 202 can perform operations from 306 to 322 to generate a visualization framework that includes a heatmap visualization of the region containing both the first object 1004 and the second object 1006.

[0143] In at least one embodiment, circuitry 202 can control a display device (such as display device 104) to display an image 1010 showing a bounding box 1012A surrounding a first object 1004 and a bounding box 1012B surrounding a second object 1006. For each of the first object 1004 and the second object 1006, the corresponding bounding box (i.e., bounding box 1012A and bounding box 1012B) can be shown based on the bounding box coordinates in the corresponding object detection results, for example, in addition to Figure 3A As described in [the text]. Circuit system 202 can receive input that may indicate the selection of a bounding box surrounding either a first object 1004 or a second object 1006. Based on the received input, circuit system 202 can select the object of interest as the object enclosed by the selected bounding box. For the selected object of interest, operations from 306 to 322 can be performed to generate a visualization framework that may include a heatmap visualization of a focus block or region containing the selected object of interest. For example, if the second object 1006 (i.e., the smaller object) is selected, then visualization framework 1014 may include a heatmap visualization of a focus block or region containing the second object 1006.

[0144] Figure 11 This is a diagram illustrating exemplary operations for thermal visualization of one or more objects detected in a series of frames of a video feed, according to embodiments of the present disclosure. Figure 11 Combination Figure 1 , 2 Explain the elements in 3A, 3B, 4, 5, 6, 7, 8, 9, and 10. (See reference.) Figure 11A block diagram 1100 illustrates exemplary operations from 1102 to 1106 as described herein. The exemplary operations shown in block diagram 1100 may begin at 1102 and can be performed by any computing system, apparatus, or device, such as by... Figure 1 or Figure 2 The electronic device 102 performs the operation. Although illustrated with discrete blocks, exemplary operations associated with one or more blocks of block diagram 1100 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the implementation of the exemplary operations.

[0145] At 1102, a video feed can be acquired. In an embodiment, the circuit system 202 can acquire the video feed and can extract a series of image frames (such as frame 1102A, frame 1102B, and frame 1102C) from the acquired video feed. If the first image frame 302A is part of the video feed, then the series of image frames can be before or after the first image frame 302A in the video feed. Each of the series of image frames can include an object of interest 1102D. In some embodiments, the series of image frames can capture a static scene with one or more static objects of interest or a dynamic scene with one or more moving objects of interest.

[0146] At 1104, a series of image frames can be fed sequentially (i.e., frame-by-frame) as input to the DNN model 106. In an embodiment, the circuit system 202 can input each of the series of image frames into the DNN model 106. The DNN model 106 can track the object of interest 1102D in the series of image frames and can output a series of object detection results associated with the tracked object of interest 1102D based on the input series of image frames.

[0147] When a series of image frames includes object of interest 1102D as the only object of interest, the circuit system 202 can extract a series of object detection results associated with object of interest 1102D as the output of the DNN model 106 for the input series of image frames. When the input series of image frames includes multiple objects, the circuit system 202 can extract multiple object detection results associated with the multiple objects as the output of the DNN model 106 for each image frame in the input series of image frames. Since each of the image frames in the series is sequentially fed into the DNN model 106, the operation for extracting object detection results (or multiple object detection results) can be the same as the operation for the first image frame 302A, as in... Figure 3A The descriptions at points 302 and 304.

[0148] When object detection is performed frame by frame, the DNN model 106 can track and uniquely identify the object of interest 1102D across the entire series of image frames. In an embodiment, for each of the input series of image frames, the circuit system 202 can extract information such as bounding box coordinates, activated cells (cell-anchors), and feature vectors (also as... Figure 3A and 3B (As described in the text). This type of information can be stored in a database. The circuit system 202 can determine the region of interest (ROI) in each of the input series of image frames based on this information. Subsequently, the determined ROI of the first image frame can be compared with the ROI of the second image frame following the first image frame in the input series of image frames. The above comparison can be iterated for all subsequent image frame pairs, and a tracking ID can be assigned to each ROI based on the similarity between the compared ROIs. The tracking ID can be used to track the object of interest 1102D throughout the entire input series of image frames.

[0149] For multi-object tracking, each object of interest can be identified as a corresponding tracking ID determined based on its bounding box coordinates and unique cell anchor values. The tracking ID can be associated with the object's position and cell anchor values ​​for differentiation. In some embodiments, tracking IDs can be used alone or in combination to implement other object tracking methods. Examples of other object tracking methods may include, but are not limited to, Kalman filter-based tracking, particle filter-based tracking, kernel-based tracking, contour tracking, support vector machines, and shape matching.

[0150] For heatmap visualization, circuit system 202 can segment each of a series of image frames into a second or more image parts based on the corresponding object detection results in a series of object detection results. For example, in Figure 3A The details of the segmentation are provided. Subsequently, operations 308 to 316 can be performed for each of the series of image frames. The circuit system 202 can then determine multiple weight values ​​corresponding to the second plurality of image portions. Multiple weight values ​​(i.e., weight vectors) can be determined for each image frame in the series of image frames. Each weight value can indicate the probability that the corresponding image portion in the second plurality of image portions belongs to the object of interest 1102D.

[0151] Circuit system 202 can generate a series of visualization frames (such as frame 1108A, frame 1108B, and frame 1108C) based on multiple weight values ​​determined for each of a series of image frames. Each visualization frame in the series can include a heatmap visualization of a region (or ROI) containing an object of interest. For example, Figure 3BThe details of generating the visualization frames are provided. A series of visualization frames can follow the visualization frames of the first image frame 302A, for example, in... Figure 3B As described in [the text].

[0152] At 1106, a series of visualization frames can be displayed. In an embodiment, the circuit system 202 can control the display device 104 to display the generated series of visualization frames (such as frames 1108A, 1108B, and 1108C). Each of frames 1108A, 1108B, and 1108C may include a heatmap visualization of a region that includes the object of interest 1102D and is defined by corresponding bounding box coordinates.

[0153] If the DNN model 106 fails to detect the object of interest 1102D in an image frame, a visualization frame for such an image frame can be generated based on the interpolation of neighboring visualization frames in a series of visualization frames. Based on the output series of object detection results, the circuit system 202 can determine a first object detection result indicating the absence of the object of interest 1102D in an intermediate image frame of the input series of image frames. For example, the object of interest may be completely overlapped or occluded by at least one of several objects. In this case, the circuit system 202 can select neighboring visualization frames from the generated series of visualization frames based on the determined first object detection result, and can interpolate a first visualization frame for the intermediate image frame based on the selected neighboring visualization frames. The interpolation can be based on suitable interpolation techniques, such as, but not limited to, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, B-spline interpolation, Lanczos interpolation, and discrete wavelet transform (DWT) interpolation. Subsequently, the circuit system 202 can insert the interpolated first visualization frame into the generated series of visualization frames.

[0154] Figure 12 This is a diagram illustrating an exemplary scenario of controlling an unmanned aerial vehicle to autonomously monitor a farm according to an embodiment of the present disclosure. Figure 12 Combination Figure 1 , 2 Explain the elements in 3A, 3B, 4, 5, 6, 7, 8, 9, 10, and 11. (See reference.) Figure 12 An exemplary scenario 1200 is illustrated. Scenario 1200 may include a farm 1202, a pest control setup 1204, and an unmanned aerial vehicle (UAV) 1206. The UAV 1206 may include an imaging device 1208, such as a camera mounted on or integrated with the UAV 1206. In this document, the UAV 1206 may be... Figure 1 An exemplary embodiment of the electronic device 102.

[0155] When the unmanned aerial vehicle (UAV) 1206 is in flight, it can control the imaging device 1208 to capture an input image frame 1210 of the farm 1202. The UAV 1206 can receive the input image frame 1210, which may include an object of interest, such as crops 1212 of the farm 1202. For the received input image frame 1210, the UAV 1206 can also receive a visualization framework, which includes a heatmap visualization of a region including the crops in the input image frame 1210 of the farm 1202. In an embodiment, using a machine learning model, the UAV 1206 can extract a set of observations associated with the object of interest (i.e., crop 1212) and determine inferences associated with the object of interest (e.g., such as...). Figure 4 (As described in the text). For example, if the set of observations includes numerical values ​​indicating crop health, then the inference can identify blocks of farmland that include damaged crop 1214 (as shown in the figure). In another embodiment, the unmanned aerial vehicle 1206 can use... Figure 3B or Figure 3C The metric calculation operation at position 324 determines the metrics associated with the visualization framework (such as F). spread or F weight Subsequently, the unmanned aerial vehicle 1206 can determine the inference results based on the established metrics.

[0156] The unmanned aerial vehicle 1206 can receive input corresponding to a user's query for the inference result to obtain the reason for the inference result regarding the damaged crop 1214 in the determined block. Based on the received input, the unmanned aerial vehicle 1206 can control the pest nozzle 1204A of the pest control setting 1204 to adjust the release of pest spray on the damaged crop 1214.

[0157] Figure 13A and 13B These are diagrams illustrating exemplary operations for thermal mapping visualization of object detection according to embodiments of the present disclosure. Figure 13A and 13B Combining from Figure 1 , 2 The components in 3A, 3B, 4, 5, 6, 7, 8, 9, 10, 11, and 12 are explained. (Refer to...) Figure 13A and 13B A flowchart 1300 illustrates exemplary operations from 1302 to 1326 as described herein. The exemplary operations shown in flowchart 1300 may begin at 1302 and can be performed by any computing system, device, or apparatus, such as by... Figure 1 or Figure 2The electronic device 102 performs the operation. Although illustrated with discrete blocks, exemplary operations associated with one or more blocks of flowchart 1300 may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the implementation of the exemplary operations.

[0158] At position 1304, the first input image frame 302A can be acquired. In an embodiment, the circuit system 202 can be based on... Figure 3A The operation at 302 involves acquiring a first image frame 302A as input from a data source (such as a camera). For example, the first image frame 302A could include a soccer player as the object of interest.

[0159] At position 1306, the detection result can be extracted from the DNN model 106. In an embodiment, the circuit system 202 can be based on... Figure 3A The operation at 304 is performed to extract object detection results associated with the object of interest, as the output of the DNN model 106 for the input first image frame 302A. Along with the object detection results, the circuit system 202 can receive information such as bounding box coordinates, confidence scores, resized images (i.e., resized versions of the input first image frame 302A), and reference cell anchors.

[0160] At 1308, it can be determined whether the number of bounding boxes is less than or equal to a threshold (k). In an embodiment, circuitry 202 can be configured to determine whether the number of bounding boxes is less than or equal to the threshold (k) based on object detection results and received information. The threshold can be predefined or set via user input (via I / O device 206). If the number of bounding boxes is less than or equal to the threshold, control can proceed to the end. Otherwise, control can proceed to 1310.

[0161] At 1310, image segmentation can be performed. In an embodiment, circuit system 202 can segment the first image frame into a first plurality of image portions based on bounding box coordinates, for example in Figure 3A The 306th description.

[0162] At position 1312, a perturbed image can be generated. In an embodiment, circuit system 202 can generate multiple perturbed images based on a first plurality of image portions. For example, in Figure 3A The operation of generating the perturbed image is described at position 308.

[0163] At position 1314, label data can be extracted. The label data may include multiple object detection scores 1314A corresponding to the generated multiple perturbed images. For this extraction, the cell tracking operation at position 1316 can be performed simultaneously with reapplying the DNN model 106 to the multiple perturbed images to extract label data from the DNN model 106. Details of this operation are provided, for example, in... Figure 3B 310 locations provided.

[0164] At position 1318, multiple feature vectors can be extracted. In an embodiment, circuit system 202 can generate a first feature vector from multiple feature vectors for a first perturbed image among multiple perturbed images. For example, in Figure 3B Details of the extraction are provided at point 314.

[0165] At position 1320, a regression model can be trained. In an embodiment, circuit system 202 can train the regression model on multiple feature vectors (extracted at position 1318) and multiple object detection scores (extracted as label data at position 1314). For example, in Figure 3B The training details are provided at point 316.

[0166] At position 1322, the weight values ​​can be determined. In an embodiment, circuit system 202 can determine the weight value of each of the first plurality of image portions based on a trained regression model. For example, in Figure 3B Details on weight determination are provided at point 318.

[0167] At 1324, interpretation generation can be performed. In an embodiment, circuitry 202 can generate an interpretation associated with the input first image frame 302A. Interpretation generation may include a first operation of heatmap generation at 1324A and a second operation of metric calculation at 1324B.

[0168] At 1324A, a visualization framework can be generated. In an embodiment, circuitry 202 can generate the visualization framework based on weight values ​​determined for each of the first plurality of image portions. The visualization framework can include a heatmap visualization of a region included in the input first image frame 302A and defined by bounding box coordinates. For example, in Figure 3B Details for generating the visualization framework are provided at point 322.

[0169] At 1324B, metric calculations can be performed. In an embodiment, circuit system 202 can determine (or calculate) visualization metrics, such as a first visual metric and a second visual metric, based on a visualization framework (generated at 1320). For example, in Figure 3B or Figure 3CDetails of the metric calculation operation are provided at point 324. In an embodiment, circuitry 202 may receive user input, including the selection of a visualization frame with a lower value for a first visual metric or a second visual metric. In this case, circuitry 202 may generate instructions to adjust at least one imaging parameter of an input source (such as a camera that initially acquires multiple images). For example, if a visualization frame with a lower value for a first visual metric or a second visual metric is selected from an underexposed image frame, then circuitry 202 may generate instructions to adjust the sensor gain of the input source. In this case, the input source may acquire a new image frame after adjusting at least one imaging parameter. Operations from 1304 to 1306 may be performed on the new image frame, and control may be passed to 1308.

[0170] At 1326, a selection can be performed. In an embodiment, circuitry 202 can control display device 104 to display a visualization frame and calculated metrics. Based on user input, circuitry 202 can select an object (such as a football player in the input first image frame 302A) and can display a visualization frame on display device 104. The visualization frame may include a heatmap visualization of the selected object. For example, in Figure 4 Further details of the selection are provided at point 412. Control can be passed to the end.

[0171] Figure 14 This is a flowchart illustrating an exemplary method for thermal image visualization of object detection according to an embodiment of the present disclosure. Figure 14 Combination Figure 1 , 2 The elements in 3A, 3B, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13A, and 13B are explained. (See reference.) Figure 14 The flowchart 1400 is shown. The method shown in flowchart 1400 can be executed by any computing system, such as by electronic device 102 or circuit system 202. The method can start at 1402 and proceed to 1404.

[0172] At 1404, a first image frame (such as first image frame 302A) can be input into DNN model 106. In one or more embodiments, circuitry 202 can be configured to input the first image frame 302A (which includes an object of interest 302B) into DNN model 106. DNN model 106 can be trained for an object detection task.

[0173] At 1406, object detection results can be extracted from DNN model 106. In one or more embodiments, circuitry 202 can be configured to extract object detection results associated with object of interest 302B as output of DNN model 106 for input first image frame 302A. The extracted object detection results may include bounding box coordinates (such as bounding box coordinates 304B) of object of interest 302B.

[0174] At 1408, the input first image frame 302A can be segmented. In one or more embodiments, the circuit system 202 can be configured to segment the input first image frame 302A into a first plurality of image portions 306A...306N based on the bounding box coordinates 304B.

[0175] At 1410, a weight value can be determined for each of the first plurality of image portions. The weight value can indicate the probability that the corresponding image portion belongs to the object of interest 302B. In one or more embodiments, the circuit system 202 can be configured to determine the weight value for each of the first plurality of image portions 306A...306N.

[0176] At 1412, a visualization framework 322A can be generated. The visualization framework 322A may include a heatmap visualization of a region included in the input first image frame 302A and defined by bounding box coordinates 304B. In one or more embodiments, the circuit system 202 may be configured to generate the visualization framework 322A based on weight values ​​determined for each of the first plurality of image portions 306A. Control may be passed to the end.

[0177] Although flowchart 1400 is shown as discrete operations, such as 1402, 1404, 1406, 1408, 1410, and 1412, this disclosure is not limited thereto. Thus, in some embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated, depending on the particular implementation without departing from the nature of the disclosed embodiments.

[0178] Various embodiments of this disclosure may provide a non-transitory computer-readable medium and / or storage medium storing computer-executable instructions executable by a machine and / or computer to operate an electronic device. The computer-executable instructions may cause a machine and / or computer to perform operations including inputting a first image frame, which may include an object of interest, into a deep neural network (DNN) model trained for an object detection task. The operations may further include extracting object detection results associated with the object of interest as output of the DNN model for the input first image frame. The extracted object detection results may include bounding box coordinates of the object of interest. The operations may further include segmenting the input first image frame into a first plurality of image portions based on the bounding box coordinates, and determining a weight value for each of the first plurality of image portions indicating the probability that the corresponding image portion belongs to the object of interest. The operations may further include generating a visualization framework, based on the weights determined for each of the first plurality of image portions, that may include a heatmap visualization of a region included in the input first image frame and defined by the bounding box coordinates.

[0179] Exemplary aspects of this disclosure may provide electronic devices (such as circuit systems) including circuit systems (e.g., circuit system 202). Figure 1 The electronic device 102. The circuitry can be configured to input a first image frame (such as first image frame 302A) that may include an object of interest (such as object of interest 302B) into a deep neural network (DNN) model (such as DNN model 106), which is trained for an object detection task. The circuitry can be configured to extract object detection results associated with the object of interest from the output of the DNN model for the input first image frame. According to an embodiment, the extracted object detection results may include bounding box coordinates of the object of interest (such as bounding box coordinates 304B). The circuitry can be configured to segment the input first image frame into a first plurality of image portions (such as the first plurality of image portions 306A...306N) based on the bounding box coordinates. For each of the first plurality of image portions, the circuitry can be configured to determine a weight value indicating the probability that the corresponding image portion belongs to the object of interest. Based on the weight values ​​determined for each of the first plurality of image portions, the circuitry can also be configured to generate a visualization framework (such as visualization framework 322A) that includes a heatmap visualization of the region included in the input first image frame and defined by the bounding box coordinates.

[0180] According to an embodiment, the input first image frame may include multiple objects (such as a first object 410, a second object 412, and a third object 414). The circuitry can be configured to extract multiple object detection results associated with the multiple objects from the output of a DNN model for the input first image frame. In some embodiments, the object of interest may be at least partially overlapped or occluded by the first object among the multiple objects. Based on the extracted multiple object detection results, the circuitry can be configured to control a display device (such as display device 104) to display multiple bounding boxes around the multiple objects in the input first image frame. The circuitry can be configured to receive input that may include selecting a first bounding box from the displayed multiple bounding boxes. The circuitry can also be configured to select the object of interest from the multiple objects based on the received input and further generate a visualization framework based on that selection.

[0181] According to an embodiment, the circuit system may also be configured to perform random sample selection from a first set of image portions among a first plurality of image portions. The circuit system may also be configured to apply a mask to a portion of an input first image frame, which may include the first set of image portions, and to generate a first perturbed image (such as the first perturbed image 308A) among a plurality of perturbed images (such as a plurality of perturbed images 308A...308N) based on the application of the mask.

[0182] According to an embodiment, the circuit system can also be configured to determine a first active cell from a plurality of active cells of the DNN model that can be responsible for object detection results in the output of the DNN model. The circuit system can be configured to input a first perturbed image from a plurality of perturbed images into the DNN model. From the first output of the determined first active cell of the DNN model for the input first perturbed image, the circuit system can be configured to extract a first object detection score from a plurality of object detection scores that can correspond to the plurality of perturbed images.

[0183] According to an embodiment, the circuit system can also be configured to generate a first feature vector from a plurality of feature vectors for a first perturbed image among a plurality of perturbed images. The generated first feature vector may include a plurality of vector elements corresponding to a first plurality of image portions of the input first image frame, and each of the plurality of vector elements may store a value that may indicate whether the corresponding image portion among the first plurality of image portions is masked in the first perturbed image.

[0184] According to an embodiment, the circuit system can also be configured to train a regression model on multiple feature vectors and multiple object detection scores (such as multiple object detection scores 310A). Weight values ​​for each of the first plurality of image portions can be determined based on the trained regression model.

[0185] According to an embodiment, the circuit system can also be configured to select a color value from a heatmap color palette that can represent the determined weight value. The color value can be selected based on a heatmap color scheme for the range of weight values. Based on the color value selected for each of the first plurality of image portions, the circuit system can be configured to generate a visualization framework that includes a heatmap visualization of the region defined by bounding box coordinates.

[0186] According to an embodiment, the circuitry can also be configured to select bounding boxes surrounding an object of interest in the input first image frame. The selected bounding boxes can be determined by bounding box coordinates in the object detection results of the DNN model. The circuitry can be configured to determine a first count of weight values ​​that can be greater than zero within the selected bounding box based on weight values ​​determined for each of the first plurality of image portions, and a second count of weight values ​​that can be equal to zero within the bounding box based on weight values ​​determined for each of the first plurality of image portions. The circuitry can be configured to calculate a first visual metric as the ratio of the determined first count to the sum of the determined first count and the determined second count.

[0187] According to an embodiment, the circuit system can also be configured to select a bounding box surrounding the object of interest in the input first image frame. The circuit system can be configured to determine a first sum of a first set of weight values ​​that can be greater than zero within the selected bounding box. The circuit system can also be configured to determine a second sum of a second set of weight values ​​that can be greater than zero outside the selected bounding box. Each of the first and second sums can be determined based on weight values ​​determined for the first plurality of image portions. Thereafter, the circuit system can be configured to determine a second visual metric as the ratio of the determined first sum to the sum of the determined first sum and the determined second sum.

[0188] According to an embodiment, the circuit system can be configured to receive input including selecting an object of interest from a visualization frame. Based on the received input, the circuit can be configured to select the object of interest from the visualization frame. The circuit system can be configured to extract a set of observations associated with the selected object of interest. The circuit system can also be configured to determine an inference result (such as inference result 504) associated with the selected object of interest based on the extracted set of observations. The determined inference result includes numerical values ​​of visual attributes or states of the selected object of interest.

[0189] According to an embodiment, the circuit system can be configured to generate an explanation (such as explanation 506) based on the determined inference result. The generated explanation includes a natural language description of the determined inference result. The circuit system can also be configured to control a display device to display the generated explanation.

[0190] According to an embodiment, the circuit system can be configured to: determine multiple recommendations associated with the generated interpretation based on the determined inference result. The circuit system can be configured to control a display device to display the generated multiple recommendations associated with the determined interpretation. The circuit system can be configured to receive input including selecting a first recommendation from the multiple recommendations. Based on the received input, the circuit system can be configured to select the first recommendation from the multiple recommendations. The circuit system can also be configured to adjust image parameters associated with the input first image frame based on the selected first recommendation.

[0191] According to an embodiment, the circuitry is further configured to extract from the video feed a series of image frames (such as frames 702A, 702B, and 702C) that may follow a first image frame in the video feed and include an object of interest. The circuitry can also be configured to input the series of image frames into a DNN model. The DNN model can be configured to track the object of interest in the series of image frames and output a series of object detection results associated with the tracked object of interest based on the input series of image frames.

[0192] According to an embodiment, the circuit system can also be configured to segment each of a series of image frames into a second plurality of image portions based on a corresponding object detection result in a series of object detection results. The circuit system can be configured to determine a plurality of weight values ​​corresponding to the second plurality of image portions of each of the image frames in the series of image frames. According to an embodiment, each of the plurality of weight values ​​can indicate the probability that the corresponding image portion in the second plurality of image portions belongs to an object of interest. The circuit system can be configured to generate a series of visualization frames (e.g., frames 1108A, 1108B, and 1108C) that can be used after a visualization frame for a first image frame used as input, based on the plurality of weight values ​​determined for each of the image frames in the series of image frames.

[0193] According to an embodiment, the circuit system can also be configured to determine a first object detection result from a series of output object detection results, the first object detection result indicating the absence of an object of interest in an intermediate image frame of an input series of image frames. The circuit system can be configured to select adjacent visualization frames from a generated series of visualization frames based on this determination, and to interpolate a first visualization frame for the intermediate image frame based on the selected adjacent visualization frames.

[0194] This disclosure can be implemented in hardware or a combination of hardware and software. It can be implemented in a centralized manner, on at least one computer system, or in a distributed manner, wherein different components can be distributed across multiple interconnected computer systems. A computer system or other apparatus suitable for performing the methods described herein may be appropriate. The combination of hardware and software can be a general-purpose computer system having a computer program that, when loaded and executed, can control the computer system to perform the methods described herein. This disclosure can be implemented in hardware that includes a portion of an integrated circuit that also performs other functions.

[0195] This disclosure can also be embedded in a computer program product that includes all features enabling the implementation of the methods described herein and, when loaded into a computer system, is capable of executing those methods. In this document, a computer program means any expression of a set of instructions represented in any language, code, or notation, which is intended to cause a system with information processing capabilities to directly perform a particular function, or to perform a particular function after one or both of the following: a) being translated into another language, code, or notation; or b) being copied in a different material form.

[0196] While this disclosure has been described with reference to certain embodiments, those skilled in the art will understand that various changes and substitutions can be made without departing from the scope of this disclosure. Furthermore, many modifications can be made to suit particular situations or materials to the teachings of this disclosure without departing from the scope of this disclosure. Therefore, it is intended that this disclosure be limited to the specific embodiments disclosed, but rather that this disclosure will include all embodiments falling within the scope of the appended claims.

Claims

1. An electronic device comprising: The circuit system is configured as follows: The first image frame, including the object of interest, is input into a deep neural network (DNN) model trained for the object detection task. Extract object detection results associated with the object of interest from the output of the DNN model for the first input image frame. The extracted object detection results include the coordinates of the bounding box for the object of interest; The input first image frame is segmented into a first plurality of image parts based on the bounding box coordinates; For each of the first plurality of image parts, determine a weight value indicating the probability that the corresponding image part belongs to the object of interest; as well as Based on the weight values ​​determined for each of the first plurality of image portions, a visualization framework for heatmap visualization is generated, comprising a region included in the input first image frame and defined by bounding box coordinates. The circuit system is further configured as follows: In the first input image frame, a bounding box is selected around the object of interest. The selected bounding box is determined by the bounding box coordinates in the object detection results of the DNN model. Based on the weight values ​​determined for each of the first plurality of image portions, a first count of weight values ​​greater than zero within the selected bounding box is determined; Based on the weight values ​​determined for each of the first plurality of image parts, a second count is determined of the weight values ​​that are equal to zero within the bounding box; as well as The first visual metric is calculated as the ratio of a determined first count to the sum of the determined first count and the determined second count.

2. The electronic device of claim 1, wherein the input first image frame comprises a plurality of objects.

3. The electronic device according to claim 2, wherein the circuit system is further configured as follows: Extract multiple object detection results associated with the multiple objects from the output of the DNN model for the input first image frame; Based on the extracted object detection results, the display device is controlled to display multiple bounding boxes surrounding the multiple objects in the input first image frame; Receive input including selecting a first delimiter from the displayed plurality of delimiter boxes; Select the object of interest from the plurality of objects based on the received input; as well as A visualization framework is then generated based on the selection.

4. The electronic device of claim 3, wherein the object of interest is at least partially overlapped or obscured by a first object of the plurality of objects.

5. The electronic device of claim 3, wherein the object of interest at least partially overlaps with or obscures the first object of the plurality of objects.

6. The electronic device of claim 1, wherein the circuit system is further configured as follows: Perform random sample selection of a first set of image portions from the first plurality of image portions; Apply a mask to a portion of a first set including image portions of the input first image frame; and The mask-based application generates the first perturbed image among multiple perturbed images.

7. The electronic device of claim 6, wherein the circuitry is further configured to select a window size based on the input dimension and bounding box coordinates of the first input image frame, and The first input image frame is divided into a first plurality of image parts based on the selected window size.

8. The electronic device of claim 6, wherein the circuit system is further configured as follows: From multiple active cells of the DNN model, determine the first active cell responsible for the object detection result in the output of the DNN model; The first perturbed image among the plurality of perturbed images is input into the DNN model; as well as Extract the first object detection score from the first output of the first perturbed image corresponding to the first object detection score of the first activated cell of the DNN model for the input first perturbed image.

9. The electronic device of claim 8, wherein the circuit system is further configured to generate a first feature vector of a plurality of feature vectors for a first perturbed image among the plurality of perturbed images, wherein The generated first feature vector includes multiple vector elements corresponding to the first plurality of image portions of the input first image frame, and Each of the plurality of vector elements stores a value indicating whether a corresponding image portion in the first plurality of image portions is masked in the first perturbed image.

10. The electronic device of claim 9, wherein the circuit system is further configured as follows: Train a regression model on the plurality of feature vectors and the plurality of object detection scores; and The weight values ​​for each image part in the first plurality of image parts are determined based on the trained regression model.

11. The electronic device of claim 1, wherein the circuit system is further configured as follows: Select a color value from the heatmap color palette that represents the determined weight value. The color values ​​are selected based on a heatmap color scheme chosen for the range of weight values; and Based on the color values ​​selected for each of the first plurality of image parts, a visualization framework is generated that includes a heatmap visualization of the region defined by bounding box coordinates.

12. The electronic device of claim 1, wherein the circuit system is further configured as follows: Determine the first sum of the first set of weight values ​​greater than zero within the selected bounding box; A second sum is determined from a second set of weight values ​​greater than zero outside the selected bounding box, wherein each of the first and second sums is determined based on weight values ​​determined for the first plurality of image portions; and The second visual metric is calculated as the ratio of the determined first sum to the sum of the determined first sum and the determined second sum.

13. The electronic device of claim 1, wherein the circuit system is further configured as follows: It receives input including selecting objects of interest from a visualization framework; Select objects of interest from the visualization framework based on the received input; Extract the set of observations associated with the selected object of interest; as well as Based on the extracted set of observations, inferences associated with the selected object of interest are determined, wherein the determined inferences include numerical values ​​for the visual attributes or states of the selected object of interest.

14. The electronic device of claim 13, wherein the circuit system is further configured as follows: An explanation is generated based on the determined inference results, wherein the generated explanation includes a natural language description of the determined inference results; and Control the display device to display the generated explanation.

15. The electronic device of claim 14, wherein the circuit system is further configured as follows: Based on the determined inference results, multiple recommendations are identified that are associated with the generated explanations; The control display device displays the determined plurality of recommendations associated with the generated interpretation; Receive input including selecting a first recommendation from the plurality of recommendations; The first recommendation is selected based on the received input; as well as Adjust the image parameters associated with the first input image frame based on the selected first recommendation.

16. The electronic device of claim 1, wherein the circuit system is further configured as follows: Extract a series of image frames from the video feed that follow the first image frame and include the object of interest; and The series of image frames is input into the DNN model, wherein the DNN model is configured as follows: Tracking the object of interest in the series of image frames; and Based on the input series of image frames, a series of object detection results associated with the tracked object of interest are output.

17. The electronic device of claim 16, wherein the circuit system is further configured as follows: Based on the corresponding object detection result in the series of object detection results, each of the series of image frames is divided into a second plurality of image parts; Determine multiple weight values ​​corresponding to a second plurality of image portions of each of the series of image frames. Each of the plurality of weight values ​​indicates the probability that the corresponding image portion in the second plurality of image portions belongs to the object of interest; as well as Based on the plurality of weight values ​​determined for each of the series of image frames, a series of visualization frames are generated following the visualization frame of the first image frame used as input.

18. The electronic device of claim 17, wherein the circuit system is further configured as follows: A first object detection result is determined from the output series of object detection results, the first object detection result indicating that an object of interest is missing in an intermediate image frame of the input series of image frames; as well as Based on this determination, adjacent visualization frames are selected from the generated series of visualization frames; as well as Based on the selected adjacent visualization frames, interpolation is applied to the first visualization frame of the intermediate image frame.

19. A method for heatmap visualization of object detection, comprising: The first image frame, including the object of interest, is input into a deep neural network (DNN) model trained for the object detection task. Extract object detection results associated with the object of interest from the output of the DNN model for the first input image frame. The extracted object detection results include the coordinates of the bounding box for the object of interest; The input first image frame is segmented into a first plurality of image parts based on the bounding box coordinates; For each of the first plurality of image parts, determine a weight value indicating the probability that the corresponding image part belongs to the object of interest; as well as Based on the weight values ​​determined for each of the first plurality of image portions, a visualization framework for heatmap visualization is generated, comprising a region included in the input first image frame and defined by bounding box coordinates. The method further includes: In the first input image frame, a bounding box is selected around the object of interest. The selected bounding box is determined by the bounding box coordinates in the object detection results of the DNN model. Based on the weight values ​​determined for each of the first plurality of image portions, a first count of weight values ​​greater than zero within the selected bounding box is determined; Based on the weight values ​​determined for each of the first plurality of image portions, a second count is determined of weight values ​​that are equal to zero within the bounding box; and The first visual metric is calculated as the ratio of a determined first count to the sum of the determined first count and the determined second count.

20. A non-transitory computer-readable medium having computer-executable instructions stored thereon, the computer-executable instructions causing the electronic device to perform operations when executed by the electronic device, the operations including: The first image frame, including the object of interest, is input into a deep neural network (DNN) model trained for the object detection task. Extract object detection results associated with the object of interest from the output of the DNN model for the first input image frame. The extracted object detection results include the coordinates of the bounding box for the object of interest; The input first image frame is segmented into a first plurality of image parts based on the bounding box coordinates; For each of the first plurality of image parts, determine a weight value indicating the probability that the corresponding image part belongs to the object of interest; as well as Based on the weight values ​​determined for each of the first plurality of image portions, a visualization framework for heatmap visualization is generated, comprising a region included in the input first image frame and defined by bounding box coordinates. The operation also includes: In the first input image frame, a bounding box is selected around the object of interest. The selected bounding box is determined by the bounding box coordinates in the object detection results of the DNN model. Based on the weight values ​​determined for each of the first plurality of image portions, a first count of weight values ​​greater than zero within the selected bounding box is determined; Based on the weight values ​​determined for each of the first plurality of image portions, a second count is determined of weight values ​​that are equal to zero within the bounding box; and The first visual metric is calculated as the ratio of a determined first count to the sum of the determined first count and the determined second count.

Images