Electronic device and method for smoke level estimation

Abstract

An electronic device for smoke estimation is provided. The electronic device receives a first image of a plurality of images of a physical space. The electronic device detects smoke in the physical space based on application of a trained neural network model to the received first image. The electronic device generates a heat map of the physical space based on the detected smoke in the physical space and an output of the trained neural network model corresponding to the detection of the smoke. The electronic device estimates a level of smoke in the physical space based on a normalization of the generated heat map.

Description

[Technical Field]

[0001] Cross-reference to related applications / incorporation by reference none.

[0002] Various embodiments of the present disclosure relate to smoke level estimation, and more particularly to electronic devices and methods for smoke level estimation. [Background technology]

[0003] Advances in medical technology have led to various developments in health and medical services, including surgery. Traditionally, surgeons have utilized electrosurgical devices (e.g., electrocautery devices) during surgical procedures. Surgical smoke is a common by-product of electrocautery device use. Surgical smoke can be generated during surgery due to tissue destruction caused by heat from the electrocautery device probe. Surgical smoke typically resides in high concentrations at the surgical site where the procedure is being performed, potentially impairing the visibility of the surgical site during surgery. Furthermore, frequent exposure to surgical smoke can adversely affect the health of personnel, including surgeons and ancillary staff, who inhale the smoke. In some cases, high concentrations of surgical smoke can cause irritation to the eyes and upper respiratory tract of surgeons and ancillary staff.

[0004] Conventional smoke detection methods in operating rooms or surgical sites may require additional hardware, such as specialized sensors, which can increase installation and maintenance costs for such hardware. Such methods also result in inaccurate real-time smoke detection due to the strong light in the operating room. Conventional methods may require the occasional manual activation of a smoke evacuation system to remove smoke from the operating room. Manual activation of a smoke evacuation system during surgery can be considered inconvenient for surgeons and supporting staff. Furthermore, such manual smoke evacuation methods may not be accurate or effective in removing smoke from the surgical site or operating room. Summary of the Invention [Problem to be solved by the invention]

[0005] The limitations and disadvantages of conventional approaches will become apparent to those skilled in the art by comparing the described system with certain aspects of the present disclosure illustrated in the remainder of this application and with reference to the drawings. [Means for solving the problem]

[0006] An electronic smoke level estimation apparatus and method is provided substantially as hereinbefore illustrated and / or described in connection with at least one of the drawings and more fully set forth in the claims.

[0007] These and other features and advantages of the present disclosure will become apparent from a consideration of the following detailed description of the disclosure when taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary network environment for smoke level estimation, according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating an exemplary electronic device for smoke level estimation, according to an embodiment of the present disclosure. [Figure 3A] 3B together with an exemplary operation for smoke level estimation according to an embodiment of the present disclosure. [Figure 3B] 3B illustrates, in conjunction with FIG. 3A, an exemplary operation for smoke level estimation according to an embodiment of the present disclosure. [Figure 4A] FIG. 1 illustrates an exemplary method for visibility enhancement of a physical space based on smoke level estimation, according to an embodiment of the present disclosure. [Figure 4B] FIG. 1 illustrates an exemplary method for visibility enhancement of a physical space based on smoke level estimation, according to an embodiment of the present disclosure. [Figure 5] 1 is a flowchart illustrating an exemplary method for smoke level estimation, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] The disclosed electronic device and method for estimating a smoke level in a physical space may find implementation as described below. An exemplary aspect of the present disclosure provides an electronic device that can be configured to receive a first image of a plurality of images of a physical space (e.g., a surgical site or operating room). The electronic device can be configured to apply a trained neural network model to the received first image. The electronic device can be configured to detect smoke in the physical space based on application of the trained neural network model. The electronic device can be configured to generate a heat map of the physical space based on the detected smoke in the physical space and an output of the trained neural network model corresponding to the detection of the smoke. The electronic device can be configured to estimate the level of smoke in the physical space based on normalizing the generated heat map.

[0010] According to an embodiment, the electronic device can be configured to classify smoke corresponding to each pixel subset of the received first image into a specific category of smoke intensity. The classification can be based on a confidence score of a trained neural network model corresponding to each pixel subset. The electronic device can be configured to assign weights to the heat map based on the classification of the smoke. The electronic device can be further configured to determine a plurality of regions, each corresponding to a specific category of the plurality of categories, based on the assigned weights. Each region of the plurality of regions can include a pixel subset classified into a specific category (e.g., light smoke, medium smoke, or heavy smoke).

[0011] According to some embodiments, the electronic device can be further configured to determine one or more overlapping regions between the determined regions. The electronic device can be further configured to eliminate one or more overlapping regions from the heat map. The electronic device can be further configured to estimate a level of smoke in the physical space based on the elimination of the one or more overlapping regions. According to some embodiments, the electronic device can be configured to control a smoke evacuation device to exhaust smoke from the physical space based on the estimated level of smoke.

[0012] The electronic device can estimate smoke levels by utilizing a trained neural network model, thereby eliminating the need for additional hardware (such as a dedicated sensor) for smoke detection. Using the trained neural network model can improve the accuracy of real-time smoke estimation during a surgical procedure. The electronic device can automatically control a smoke evacuation device (e.g., automatically switch on or off the smoke evacuation device) to evacuate smoke from a physical space based on the estimated smoke level, thereby improving visibility of the surgical site during the surgical procedure and providing a healthier working environment for the surgeon and supporting staff. Automatic control of the smoke evacuation device also enables smoke to be evacuated from the operating room without manual effort by the surgical staff, thereby improving user convenience and the accuracy and precision of smoke removal from the physical space. Thus, the electronic device can provide a robust method for smoke level estimation and smoke evacuation from a physical space (such as a surgical site or operating room).

[0013] FIG. 1 is a block diagram illustrating an exemplary network environment for smoke level estimation, according to an embodiment of the present disclosure. FIG. 1 illustrates a network environment 100. The network environment 100 may include an electronic device 102, a server 104, an image capture device 114, and a smoke evacuation device 116. The electronic device 102 may include a neural network model 108 configured to generate a heat map 110. The image capture device 114 may capture multiple images 106, such as a first image 106A, a second image 106B, and an Nth image 106N. The network environment 100 may further include a communications network 112. The electronic device 102, the server 104, the image capture device 114, and the smoke evacuation device 116 may communicate with each other via the communications network 112.

[0014] The electronic device 102 may include suitable logic, circuitry, and / or interfaces that may be configured to estimate a level of smoke in a physical space (e.g., a surgical site or operating room) by applying the trained neural network model 108. The electronic device 102 may be further configured to control a smoke evacuation device 116 to evacuate smoke from the physical space based on the estimated level of smoke. In some embodiments, the electronic device 102 may be configured to receive the trained neural network model 108 from the server 104. In other embodiments, the electronic device 102 may be further configured to generate the neural network model 108 and train the neural network model based on training data. In some cases, the electronic device 102 may receive training data (e.g., one or more characteristics related to smoke) from the server 104. Additionally, the electronic device 102 may control the operation of the image capture device 114 and one or more imaging parameters of the image capture device 114, and receive a plurality of images 106 from the image capture device 114 based on the control. Examples of electronic devices 102 may include, but are not limited to, computing devices, smartphones, electronic healthcare devices, mobile phones, gaming devices, mainframe machines, servers, computer workstations, and / or consumer electronics (CE) devices.

[0015] The server 104 may include suitable logic, circuitry, interfaces, and / or code that can be configured to store multiple neural network models for various applications. For example, the multiple neural network models may include a neural network model 108 that can be trained using one or more features for smoke detection. The server 104 may transmit the trained neural network model 108 to the electronic device 102. The server 104 may communicate with multiple electronic devices (such as the electronic device 102) to collect information to periodically generate new training data. The server 104 may periodically transmit the generated training data to the electronic device 102. The server 104 may be implemented as a cloud server and may perform operations through web applications, cloud applications, Hypertext Transfer Protocol (HTTP) requests, repository operations, file transfers, and the like. Other implementations of the server 104 may include, but are not limited to, a database server, a file server, a web server, a media server, an application server, a mainframe server, or a cloud computing server.

[0016] In at least one embodiment, server 104 may be implemented as a number of distributed cloud-based resources using techniques known to those skilled in the art. Those skilled in the art will appreciate that the scope of the present disclosure is not limited to implementing server 104 and electronic device 102 as two separate entities. In some embodiments, the functionality of server 104 may be incorporated, in whole or at least in part, into electronic device 102 without departing from the scope of the present disclosure.

[0017] The neural network model 108 (e.g., a deep neural network) can be defined by hyperparameters such as activation function(s), number of weights, cost function, regularization function, input size, and number of layers. The neural network model 108 can be a computational network or system in which nodes are artificial neurons arranged in multiple layers. The multiple layers of the neural network model 108 can include an input layer, one or more hidden layers, and an output layer. Each of the multiple layers can include one or more nodes (or artificial neurons, represented by circles, for example). The output of every node in the input layer can be connected to at least one node in the hidden layer(s). Similarly, the input of each hidden layer can be connected to the output of at least one node in another layer of the neural network model 108. The output of each hidden layer can be connected to the input of at least one node in another layer of the neural network model 108. The node(s) in the final layer can receive input from at least one hidden layer and output a result. The number of layers and the number of nodes in each layer can be determined from the hyperparameters of the neural network model 108. Such parameters may be set before, during, or after training of the neural network model 108. According to one embodiment, the electronic device 102 may be configured to train the neural network model 108 for smoke detection based on multiple images 106, smoke classification (such as light smoke, moderate smoke, or heavy smoke) based on a heat map, and smoke level estimation. In another embodiment, the server 104 may be configured to train the neural network model 108 and periodically update the neural network model 108 hosted on the electronic device 102.

[0018] Each node of the neural network model 108 may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be adjusted during training of the neural network model 108. The set of parameters may include, for example, weight parameters and regularization parameters. Each node may calculate an output using a mathematical function based on one or more inputs from nodes in other layer(s) (e.g., previous layer(s)) of the neural network model 108. All or some of the nodes of the neural network model 108 may correspond to the same or different mathematical functions.

[0019] According to one embodiment, the electronic device 102 can train the neural network model 108 based on one or more features associated with one or more pixel subsets in the received first image 106A to obtain a trained neural network model 108. The neural network model 108 can be trained to detect smoke and generate a heat map 110. The one or more features can include at least one of the intensity of smoke in the one or more pixel subsets, the shape of smoke in the one or more pixel subsets, or the color of smoke in the one or more pixel subsets. For example, the electronic device 102 can input multiple images to the neural network model 108 to train the neural network model 108. For example, the images can include images of smoke of different shapes, different colors, and different types of smoke (e.g., heavy smoke, medium smoke, light smoke, etc.). The neural network model 108 can be trained to detect smoke, classify smoke, and estimate smoke levels based on input images. In another embodiment, the neural network model 108 can be pre-trained to detect smoke based on features of a training dataset, such as smoke intensity, smoke shape, or smoke color.

[0020] Training the neural network model 108 may involve updating one or more parameters of each node of the neural network model 108 based on whether the output of the final layer (e.g., a smoke level estimate) for a given input (e.g., the first image 106A of smoke) matches the correct result (e.g., a ground truth image corresponding to the smoke level estimate) based on a loss function of the neural network model 108. The above process may be repeated for the same or different inputs until a minimum of the loss function is achieved, thereby minimizing the training error. Several training methods are known in the art, including, but not limited to, gradient descent, stochastic gradient descent, batch gradient descent, gradient boosting, and metaheuristic methods.

[0021] In some embodiments, the neural network model 108 may comprise electronic data that may be implemented, for example, as a software component of an application executable on the electronic device 102. The neural network model 108 may rely on libraries, external scripts, or other logic / instructions for execution by a processing device, such as the electronic device 102. The neural network model 108 may enable a computing device, such as the electronic device 102, to perform one or more operations associated with detecting smoke and generating the heat map 110. Additionally or alternatively, the neural network model 108 may be implemented using hardware, including a processor, a coprocessor (such as an artificial intelligence (AI) accelerator chip), a microprocessor, a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some embodiments, the neural network model 108 may be implemented using a combination of both hardware and software.

[0022] Examples of the neural network model 108 may include, but are not limited to, a deep neural network (DNN), a convolutional neural network (CNN), a CNN-recurrent neural network (CNN-RNN), R-CNN, Fast R-CNN, Faster R-CNN, a You Only Look Once (YOLO) network (such as YOLOv3), a CNN+ANN, a generative adversarial network (GAN), and / or a combination of such networks. In some embodiments, the neural network model 108 may be based on a hybrid architecture of multiple deep neural networks (DNNs).

[0023] 1 illustrates neural network model 108 integrated with electronic device 102, the present disclosure is not so limited. Thus, in some embodiments, neural network model 108 may be separate from electronic device 102 without departing from the scope of the present disclosure.

[0024] The heat map 110 may be a graphical representation of the magnitude or intensity of detected smoke in the physical space based on the first image 106A of the multiple images 106 using a color-coding scheme. The electronic device 102 may be configured to generate the heat map 110 based on the detected smoke in the physical space and a classification of the detected smoke (e.g., heavy smoke, moderate smoke, or light smoke) by the trained neural network model 108. The heat map 110 may utilize a color-coding scheme to represent different values ​​of pixels in the first image 106A. For example, the heat map 110 may include a different color for each pixel subset of the multiple pixels of the first image 106A received as an output of the trained neural network model 108. In some embodiments, the heat map 110 may be weighted based on a confidence score of the smoke detection by the trained neural network model 108. For example, a weight may be assigned to the heat map 110 based on the classification of the smoke into a particular category among multiple categories of smoke intensity. The weighted heat map 110 can be used to quantify smoke in a physical space for smoke level estimation. Details of the heat map 110 and the assignment of weights are further described, for example, in Figures 3A and 3B.

[0025] The communication network 112 may include a communication medium that enables the electronic device 102, the server 104, the image capture device 114, and the smoke evacuation device 116 to communicate with one another. For example, the electronic device 102 may receive multiple images 106 (e.g., a first image 106A) from the image capture device 114 via the communication network 112. In another example, the electronic device 102 may receive training data for the neural network model 108 from the server 104 via the communication network 112. The communication network 114 may be either a wired or wireless connection. Examples of the communication network 112 may include, but are not limited to, the Internet, a cloud network, a Wireless Fidelity (Wi-Fi) network, a mobile wireless network (e.g., Long Term Evolution or Fifth Generation New Radio (5G NR)), a personal area network (PAN), a local area network (LAN), or a metropolitan area network (MAN). The various devices in the network environment 100 may be configured to connect to the communication network 112 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols include, but are not limited to, at least one of Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE802.11, Light Fidelity (Li-Fi), 802.16, IEEE802.11s, IEEE802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocols.

[0026] The image capture device 114 may include suitable logic, circuitry and interfaces, and / or code that may be configured to capture multiple images 106 based on control signals from the electronic device 102. The image capture device 114 may include one or more parameters such as resolution, field of view, frame rate (e.g., frames per second), and magnification. The parameters of the image capture device 114 may be controlled by the electronic device 102. In one or more embodiments, the image capture device 114 may record multiple frames (e.g., multiple images 106) of a video of a surgical procedure and transmit them to the electronic device 102 or server 104 for storage. Examples of image capture device 114 may include, but are not limited to, an image sensor, a wide-angle camera, an action camera, a closed-circuit television (CCTV) camera, a camcorder, a digital camera, a camera phone, a time-of-flight camera (ToF camera), a night vision camera, a spherical camera, a panoramic camera, an action camera, a wide-angle camera, a closed-circuit television (CCTV) camera, other image capture devices with 360-degree image capture capabilities, or other image capture devices.

[0027] The smoke evacuation device 116 may include suitable logic, circuits, and interfaces, hardware components, and / or code for evacuating smoke from a physical space (such as a surgical site or operating room) based on an estimated smoke detection level. The smoke evacuation device 116 may include a housing, one or more suction fans, a variable speed motor coupled to the suction fans, one or more suction tubes of different diameters removably attached to the housing, one or more air filters downstream of the suction tubes, and other components depending on the application (e.g., laparoscopic surgery applications). Those skilled in the art will appreciate that the smoke evacuation device 116 may include other suitable components or systems in addition to those described herein. A detailed description of the other components or systems is omitted from this disclosure for the sake of brevity. In one embodiment, the electronic device 102 may control the activation and deactivation of the smoke evacuation device 116 based on an estimated smoke detection level in the physical space. In another embodiment, the electronic device 102 may control the speed of the motor and suction fan of the smoke evacuation device 116 based on the estimated smoke detection level in the physical space.

[0028] During operation, the image capture device 114 can be configured to capture multiple images 106 of a physical space (e.g., a surgical site or an operating room) during a surgical procedure. For example, the surgical procedure can be performed by one or more surgeons and / or ancillary staff. In another example, the surgical procedure can be robotically assisted, in which a robotic system can assist the surgeon in performing the surgical procedure remotely or in the operating room. During the surgical procedure, an electrosurgical device (e.g., an electrocautery device) can be used to cauterize tissue in the patient. In such cases, the use of the electrocautery device can generate smoke (e.g., surgical smoke) at the surgical site. In some embodiments, the image capture device 114 can capture video of the surgical procedure in real time to assist the surgical procedure.

[0029] The electronic device 102 can be configured to receive a first image 106A of a plurality of images 106 of a physical space (e.g., a surgical site or operating room) from an image capture device 114. For example, the plurality of images 106 can be different frames of a video of a surgical procedure captured by the image capture device 114. According to an embodiment, the electronic device 102 can be configured to receive the first image 106A of the plurality of images 106 from the image capture device 114 in real time, from a memory of the electronic device 102, or from the server 104 over the communication network 112. Details of receiving the first image 106A are further described, for example, in FIG. 3A .

[0030] The electronic device 102 may be further configured to apply a trained neural network model 108 to the received first image 106A. The electronic device 102 may be configured to detect smoke in the physical space based on application of the trained neural network model 108. Details of smoke detection are further shown, for example, in FIG. 3A.

[0031] The electronic device 102 can be configured to generate a heat map 110 of a physical space (such as a surgical site or operating room) based on the detected smoke in the physical space and the output of the trained neural network model 108 corresponding to the detection of the smoke. According to an embodiment, the electronic device 102 can be configured to obtain, as an output of the trained neural network model 108, a confidence score corresponding to each pixel subset of a plurality of pixels in the received first image 106A. The confidence score can be related to the detection of smoke corresponding to each pixel subset in the received first image 106A. The electronic device 102 can generate the heat map 110 based on the confidence score for each pixel subset. Details of the generation of the heat map 110 are further shown, for example, in FIG. 3A .

[0032] According to one embodiment, the electronic device 102 can be configured to classify the smoke corresponding to each pixel subset of the received first image 106A into a particular category among multiple smoke intensity categories based on the trained neural network model 108. The classification can be based on a confidence score corresponding to each pixel subset. For example, the multiple smoke intensity categories can include light smoke, medium smoke, and heavy smoke. Furthermore, the electronic device 102 can assign weights to the heat map 110 based on the smoke classification. Details of the smoke classification and weight assignment are further shown, for example, in FIG. 3A.

[0033] According to an embodiment, the electronic device 102 may further determine a plurality of regions, each corresponding to a particular category among the plurality of categories, based on the assigned weights. Each region of the plurality of regions may include a pixel subset classified into a particular category. The electronic device 102 may further determine one or more overlapping regions among the determined plurality of regions and eliminate the overlapping regions from the heat map 110. Details of the determination of the plurality of regions and the elimination of the overlapping regions are further shown, for example, in Figures 3A and 3B.

[0034] The electronic device 102 may further normalize the heat map 110 after eliminating the overlapping regions. For example, normalizing the heat map may include adjusting the values ​​of the heat map to a range of 0 to 1. The electronic device 102 may estimate a level of smoke in the physical space based on the normalization. In some embodiments, the estimated level of smoke in the first image 106A of the physical space may range from 0 to 1. Details of estimating the level of smoke are further shown, for example, in FIG. 3B.

[0035] According to some embodiments, the electronic device 102 may be further configured to control the smoke evacuation device 116 to evacuate the smoke from the physical space based on the estimated smoke level. Details of the control of the smoke evacuation device 116 are further shown, for example, in FIG. 3B.

[0036] According to one embodiment, electronic device 102 can be configured to control the display screen to display first image 106A indicating a first visibility level of the physical space based on the estimated level of smoke. Electronic device 102 can further control smoke evacuation device 116 to evacuate smoke from the physical space if the first visibility level is below a visibility threshold. Electronic device 102 can control the display screen to display a second image (e.g., second image 106B) of the plurality of images 106 of the physical space based on the evacuation of smoke. Second image 106B can indicate a second visibility level that is equal to or greater than the visibility threshold. Details of controlling smoke evacuation device 116 to evacuate smoke are further shown, for example, in FIG. 4A .

[0037] According to an embodiment, the electronic device 102 can be configured to control the display screen to display a first image 106A indicating a first visibility level of the physical space based on the estimated level of smoke. The electronic device 102 can further apply a dehazing process to the first image 106A if the first visibility level is below a visibility threshold. The electronic device 102 can further control the display screen to display a second image 106B of the plurality of images 106 of the physical space based on the application of the dehazing process. The second image 106B can indicate a second visibility level that is equal to or greater than the visibility threshold. For example, the second image 106B can be the first image 106A with the haze removed. Details of the application of the dehazing process are further illustrated, for example, in FIG. 4B .

[0038] FIG. 2 is a block diagram illustrating an exemplary electronic device for smoke level estimation, according to an embodiment of the present disclosure. The description of FIG. 2 is provided with reference to elements in FIG. 1. FIG. 2 illustrates a block diagram 200 of an electronic device 102. The electronic device 102 may include a circuit 202, a memory 204, an input / output (I / O) device 206, and a network interface 208. The circuit 202 may be communicatively coupled to the memory 204, the I / O device 206, and the network interface 208. In some embodiments, the memory 204 may include the neural network model 108. The I / O device 206 may further include a display screen 206A.

[0039] Circuit 202 may include suitable logic, circuits, and interfaces that may be configured to execute program instructions associated with different operations performed by electronic device 102. For example, circuit 202 may be configured to apply trained neural network model 108 to an image of the physical space (e.g., first image 106A) to detect smoke in the physical space, generate a weighted heat map (e.g., heat map 110) based on the detected smoke, and estimate the level of smoke based on normalizing the weighted heat map. Circuit 202 may include one or more specialized processing units that may be implemented as independent processors. In some embodiments, one or more specialized processing units may be implemented as an integrated processor or group of processors that collectively perform the functions of the one or more specialized processing units. Circuit 202 may be implemented based on multiple processor technologies known in the art. An example implementation of circuitry 202 may be an X86-based processor, a graphics processing unit (GPU), a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, a microcontroller, a central processing unit (CPU), and / or other control circuitry.

[0040] The memory 204 may include suitable logic, circuitry, and interfaces that may be configured to store program instructions executed by the circuitry 202. The memory 204 may be configured to store the neural network model 108. The memory 204 may be configured to store the images 106 received from the image capture device 114 and the generated heat map 110. The memory 204 may further be configured to store training data received from the server 104 for training the neural network model 108. Example implementations of the memory 204 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drive (HDD), solid-state drive (SSD), CPU cache, and / or a secure digital (SD) card.

[0041] The I / O device 206 may include suitable logic, circuitry, and interfaces that can be configured to receive input from a user (e.g., a surgeon or surgical staff) and provide output (e.g., displaying the first image 106A) based on the received input. The I / O device 206, which may include a variety of input and output devices, may be configured to communicate with the circuit 202. For example, the I / O device 206 may receive user input to detect smoke of a user-defined category among multiple categories of smoke intensity (e.g., heavy smoke, moderate smoke, or light smoke). The I / O device 206 may further include a display screen 206A that can display the first image 106A indicating a first visibility level of the physical space. The display screen 206A may further display the second image 106B that can indicate a second visibility level of the physical space that is different from the first visibility level. Examples of the I / O device 206 may include, but are not limited to, a touch screen, a display device, a keyboard, a mouse, a joystick, a microphone, and a speaker.

[0042] The network interface 208 may include suitable logic, circuits, and interfaces that may be configured to facilitate communication between the circuit 202, the server 104, the image capture device 114, and the smoke evacuation device 116 over the communications network 112. The network interface 208 may be further configured to connect the circuit 202 to the image capture device 114 and the smoke evacuation device 116 via a wireless connection (such as Bluetooth™) or a wired connection (such as a local area network (LAN) cable). The network interface 208 may be implemented to support wired or wireless communication of the electronic device 102 to and from the communications network 112 using various known technologies. The network interface 208 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder-decoder (CODEC) chipset, a subscriber identity module (SIM) card, or a local buffer circuit. The network interface 208 may be configured to communicate via wireless communication with networks such as the Internet, an intranet, or wireless networks such as cellular telephone networks, wireless local area networks (LANs), and metropolitan area networks (MANs).The wireless communications may be configured to use one or more of a number of communications standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), 5G NR, Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth™, Wireless Fidelity (WiFi) (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, or IEEE 802.11n), Voice over Internet Protocol (VoIP), Light Fidelity (Li-Fi), Worldwide Interoperability for Microwave Access (Wi-MAX), protocols for email, instant messaging, and short message service (SMS).

[0043] The functions or operations performed by electronic device 102 as described in Figure 1 may be performed by circuitry 202. The operations performed by circuitry 202 are described in detail in, for example, Figures 3A, 3B, 4A, and 4B.

[0044] Figure 3A illustrates exemplary operations for smoke level estimation, according to an embodiment of the present disclosure. The description of Figure 3A is provided with reference to elements of Figures 1 and 2. Figure 3A illustrates a block diagram 300A illustrating exemplary operations 302-316 described herein. The exemplary operations illustrated in block diagram 300A may begin at 302 and may be performed by any computer system, apparatus, or device, such as electronic device 102 of Figure 1 or circuit 202 of Figure 2.

[0045] At 302, a first image 302A of a plurality of images 106 of a physical space may be received. According to an embodiment, the circuit 202 may be configured to receive the first image 302A of the plurality of images 106 from the image capture device 114. For example, the first image 302A may be an image of a surgical field (e.g., a surgical site on a patient) for a surgical procedure. In another example, the first image 302A may be an image of an operating room in which a surgical procedure may be performed. The first image 106A may correspond to a first frame of a video captured by the image capture device 114.

[0046] According to one embodiment, circuitry 202 can be configured to receive first image 302A in real time from image capture device 114 during an online mode of operation of electronic device 102. Circuitry 202 can receive each frame of a plurality of frames (e.g., plurality of images 106) of video in real time from image capture device 114. In another embodiment, circuitry 202 can be configured to receive first image 302A from memory 204 of electronic device 102 or server 104 during an offline mode of operation of electronic device 102.

[0047] At 304, the trained neural network model 108 may be applied to the received first image 302A. According to an embodiment, the circuit 202 may be configured to apply the trained neural network model 108 to the received first image 302A for smoke detection within a physical space (such as a surgical site or operating room). The circuit 202 may input the first image 302A to the trained neural network model 108. In some embodiments, the circuit 202 may apply the trained neural network model 108 to different regions (e.g., different pixel subsets) within the received first image 302A for smoke detection.

[0048] At 306, smoke can be detected in the physical space based on application of the trained neural network model 108 to the first image 302A. According to an embodiment, the circuit 202 can be configured to detect smoke in the physical space based on application of the trained neural network model 108 to the first image 302A.

[0049] The trained neural network model 108 can detect smoke in the physical space by detecting characteristics such as smoke intensity, shape, and color in one or more pixel subsets of the received first image 302A. In an exemplary scenario, the first image 302A can show smoke emitted from an electrocautery device. The neural network model 108 can detect smoke in the physical space based on detecting the smoke in the first image 302A.

[0050] At 308, a confidence score corresponding to each pixel subset of the received first image 302A can be obtained. According to one embodiment, the circuit 202 can be configured to obtain a confidence score corresponding to each pixel subset of the received first image 302A as an output of the trained neural network model 108. The confidence score can be related to the detection of smoke corresponding to each pixel subset of the received first image 302A. In one example, the confidence score can specify the confidence that the neural network model 108 detected smoke in the input first image 302A. In some instances, the confidence score can be a binary value (0 or 1) or a floating-point number representing a likelihood between 0 and 1. A higher confidence score can indicate a higher reliability of the neural network model 108.

[0051] For example, a surgical procedure may be performed in a patient's abdominal region in an operating room. The first image 302A may include a scene in which the surgical procedure is being performed in the patient's abdominal region and smoke from an electrocautery device. The first image 302A may further include light reflections from one or more light sources incident on the patient's abdominal region. In some embodiments, one or more regions (e.g., one or more pixel subsets) of the first image 302A may include smoke, and other regions of the first image 302A may include light reflections. The circuit 202 may obtain a confidence score of the detection of smoke for each pixel subset of the received first image 302A as an output of the trained neural network model 108. In some embodiments, the confidence score may be in the range of 0 to 1.

[0052] For example, the neural network model 108 may detect smoke in a first subset of pixels of the first image 302A and output a confidence score of 0.95 corresponding to the detection of smoke in the first subset of pixels of the first image 302A. The circuit 202 may obtain a confidence score of 0.95 corresponding to the detected smoke in the first subset of pixels. Further, the neural network model 108 may detect the absence of smoke in a second subset of pixels of the first image 302A and output a confidence score of 0.90 corresponding to the detected absence of smoke in the second subset of pixels of the first image 302A. The circuit 202 may obtain a confidence score of 0.90 corresponding to the detected absence of smoke in the second subset of pixels of the first image 302A.

[0053] In some embodiments, the neural network model 108 can compare smoke features, such as smoke intensity, shape, and color, to features in the third pixel subset of the plurality of pixels. For example, the neural network model 108 may detect smoke in the third pixel subset based on smoke of matching intensity and matching shape in the third pixel subset, but the color of the smoke may not match. In such a case, the neural network model 108 may output a low confidence score (e.g., 0.86) corresponding to the detection of smoke in the third pixel subset. In one embodiment, the neural network model 108 can output a confidence score for each pixel of the plurality of pixels.

[0054] At 310, a heat map 310A of the physical space can be generated based on the detected smoke in the physical space and the output (e.g., confidence scores) of the trained neural network model 108 corresponding to the detection of the smoke. According to an embodiment, circuitry 202 can be configured to generate a heat map 310A of the physical space based on the detected smoke in the physical space and the output (e.g., confidence scores per pixel subset) of the trained neural network model 108 corresponding to the detection of the smoke.

[0055] In some embodiments, the neural network model 108 can be trained to generate the heat map 310A. In an exemplary scenario, the heat map 310A can be generated based on features associated with a subset of pixels in the received first image 302A. For example, the heat map 310A can be a visual representation of smoke in a physical space. If the smoke is localized to a specific spot, such as a surgical site, a specific color scheme in the heat map 310A can be used to distinguish the concentration of smoke at the surgical site from other areas of the physical space. The circuit 202 can assign a color corresponding to each subset of pixels in the first image 302A based on the detected concentration of smoke in different areas of the first image 302A. For example, the color scheme of the heat map 310A can include a dark red color for a high concentration of smoke, a light orange color for a medium concentration of smoke, and a yellow color for a low concentration of smoke. The color scheme of the heat map 310A can further include a blue color corresponding to pixels where smoke is not believed to be present. For example, circuitry 202 may assign a dark blue color to pixels that are believed to have no trace of detected smoke. In another example, circuitry 202 may assign a light blue color to pixels that are believed to have only a slight trace of smoke. Other color schemes may be used in the heat map to represent the concentration or intensity of smoke without departing from the scope of this disclosure.

[0056] At 312, the smoke corresponding to each pixel subset of the received first image 302A can be classified into a particular one of a plurality of smoke intensity categories based on the trained neural network model 108. According to an embodiment, the circuit 202 can be configured to classify the smoke corresponding to each pixel subset of the received first image 302A into a particular one of a plurality of smoke intensity categories based on the trained neural network model 108. The classification can be based on a confidence score corresponding to each pixel subset.

[0057] The multiple categories may include, for example, light smoke, medium smoke, and heavy smoke. In an exemplary scenario, smoke (e.g., thick smoke) may be concentrated near an electrocautery device during a surgical procedure. The neural network model 108 may output a confidence score of 0.92 for pixels corresponding to smoke detected near the electrocautery device. Based on the smoke detection and the confidence score, the neural network model 108 may classify the smoke near the electrocautery device in the first image 302A as heavy smoke.

[0058] Furthermore, the smoke around the electrocautery device may be less dense than the smoke in the vicinity of the electrocautery device. The neural network model 108 may output a confidence score of 0.90 for pixels corresponding to the smoke detected around the electrocautery device in the first image 302A. Based on the smoke detection and the confidence score, the neural network model 108 may classify the smoke around the electrocautery device in the first image 302A as medium smoke.

[0059] Additionally, the smoke away from the electrocautery device may be less dense than the smoke surrounding the electrocautery device. The neural network model 108 may output a confidence score of 0.86 for pixels corresponding to the smoke detected away from the electrocautery device in the first image 302A. Based on the smoke detection and the confidence score, the neural network model 108 may classify the smoke away from the electrocautery device in the first image 302A as light smoke.

[0060] According to one embodiment, the circuit 202 can receive user input corresponding to the detection of smoke in a user-defined category of a plurality of categories. In an exemplary scenario, the circuit 202 can receive user input from a surgeon or ancillary staff performing a surgical procedure. For example, the surgeon can require the evacuation of smoke from a physical space if the smoke is heavy smoke. The circuit 202 can receive user input corresponding to the detection of "heavy smoke" from the surgeon. Based on the user input, the circuit 202 can detect heavy smoke using classification of the smoke by the neural network model 108 for subsequent evacuation.

[0061] At 314, weights can be assigned to the heat map 310A based on the classification of the smoke. According to one embodiment, the circuit 202 can be configured to assign weights to the heat map 310A based on the classification of the smoke. In an exemplary scenario, the assigned weights can range from 0 to 1. For example, one or more first regions of the heat map 310A corresponding to pixels containing smoke classified as heavy smoke can be assigned a weight of "1." One or more second regions of the heat map 310A corresponding to pixels containing smoke classified as moderate smoke can be assigned a weight of "0.5." One or more third regions of the heat map 310A corresponding to pixels containing detected smoke classified as light smoke can be assigned a weight of "0.1." One or more fourth regions of the heat map 310A corresponding to pixels not containing detected smoke can be assigned a weight of "0." It should be noted that the weights on a scale of 0 to 1 are provided as an example only, and other scales can be used to assign weights to heatmap 310A without departing from the scope of this disclosure.

[0062] At 316, a plurality of regions may be determined based on the assigned weights. Each region of the plurality of regions may correspond to a particular category of the plurality of categories. According to one embodiment, the circuit 202 may be configured to determine a plurality of regions, each region corresponding to a particular category of the plurality of categories, based on the assigned weights. Each region of the plurality of regions may include a pixel subset classified into a particular category. In some embodiments, the circuit 202 may determine the plurality of regions based on bounding boxes predicted by the trained neural network model 108 to contain an object of interest (e.g., smoke). The trained neural network model 108 may predict bounding boxes to detect objects (e.g., smoke) in the first image 302A. For example, the bounding boxes may define boundaries (rectangular or circular boundaries) of a plurality of regions that may contain detected smoke of different categories. According to one embodiment, the output of the trained neural network model 108 may include bounding box coordinates of the object of interest (e.g., smoke).

[0063] For example, the circuit 202 may determine multiple regions, such as a first region 316A, a second region 316B, a third region 316C, a fourth region 316D, a fifth region 316E, and a sixth region 316F, within the generated heat map 310A. In an exemplary scenario, the determined first region 316A may correspond to a heavy smoke category based on a weight of 1 assigned to the pixel corresponding to the first region 316A. The determined second region 316B may correspond to a moderate smoke category based on a weight of 0.5 assigned to the pixel corresponding to the second region 316B. The third region 316C may correspond to a light smoke category based on a weight of 0.1 assigned to the pixel corresponding to the third region 316C. The fourth region 316D may correspond to a moderate smoke category based on a weight of 0.5 assigned to the pixel corresponding to the fourth region 316D. The fifth region 316E may correspond to a light smoke category based on a weight of 0.1 assigned to the pixel corresponding to the fifth region 316E. The sixth region 316F may correspond to a medium smoke category based on a weight of 0.5 assigned to the pixel corresponding to the sixth region 316F.

[0064] Although example operations associated with one or more blocks in block diagram 300A are shown as discrete blocks, they may be divided into further blocks, combined into fewer blocks, or eliminated depending on the implementation of the example operations.

[0065] Figure 3B illustrates exemplary operations for smoke level estimation, according to an embodiment of the present disclosure. The description of Figure 3B is provided with reference to elements of Figures 1 and 2. Figure 3B includes a block diagram 300B illustrating exemplary operations 318-328 described herein. Control may proceed to block 318 from block 316 of Figure 3A.

[0066] At 318, one or more overlap regions between the determined regions (e.g., first region 316A, second region 316B, third region 316C, fourth region 316D, fifth region 316E, and sixth region 316F) may be determined. According to an embodiment, circuit 202 may be configured to determine overlap regions such as first overlap region 318A, second overlap region 318B, and third overlap region 318C.

[0067] For example, the circuit 202 may determine a first overlap region 318A between the first region 316A and the second region 316B. The circuit 202 may determine a second overlap region 318B between the third region 316C and the fourth region 316D. The circuit 202 may determine a third overlap region 318C between the first region 316A, the fifth region 316E, and the sixth region 316F. For example, the first region 316A and the second region 316B may be considered to overlap by the proximity of one or more pixel subsets of one region (e.g., the first region 316A) to one or more pixel subsets of the other region (e.g., the second region 316B). In some cases, the first region 316A and the second region 316B of the heat map 310A may overlap, such that smoke in the same region of physical space is classified by the neural network model 108 as both heavy smoke and light smoke.

[0068] At 320, one or more overlapping regions can be removed from heatmap 310A. According to an embodiment, circuit 202 can be configured to remove overlapping regions from heatmap 310A. For example, circuit 202 can remove first overlapping region 318A, second overlapping region 318B, and third overlapping region 318C from heatmap 310A to obtain heatmap 320A.

[0069] For example, the circuit 202 may determine a first overlap region 318A (such as an overlap region between a first region 316A and a second region 316B) as containing both categories of smoke (such as heavy smoke and moderately intense smoke). The circuit 202 may reject the first overlap region 318A to remove areas of the heat map 310A that may indicate smoke classifications with less than 100% accuracy. Similarly, the circuit 202 may reject the second overlap region 318B and the third overlap region 318C to obtain the heat map 320A.

[0070] At 322, the heatmap 320A can be normalized. According to one embodiment, the circuit 202 can be configured to normalize the heatmap 320A. Examples of normalization methods utilized by the circuit 202 include, but are not limited to, min-max feature scaling, coefficient of variation, standardized moment, sigmoid function, studentized residual, and standard score methods. The values ​​in the heatmap 320A can be normalized to obtain a single value for smoke estimation within a range of 0 to 1. In one embodiment, the normalization of the heatmap 320A can use the assigned weights of each of the multiple regions. In another embodiment, the normalization of the heatmap 320A can further include an averaging process that obtains an average of the weights assigned to multiple regions (e.g., regions 316A-316F) in the heatmap 320A.

[0071] At 324, the level of smoke can be estimated. According to an embodiment, the circuit 202 can be configured to estimate the level of smoke based on eliminating overlapping regions and normalizing the heat map 320A. In some embodiments, the estimated level of smoke in the first image 302A of the physical space can be in the range of 0 to 1, with a value of 0 indicating the lowest level of smoke and a value of 1 indicating the highest level of smoke.

[0072] For example, the heat map 320A may include six regions, such as a first region 316A, a second region 316B, a third region 316C, a fourth region 316D, a fifth region 316E, and a sixth region 316F. The first region 316A may be associated with a weight of 1, the second region 316B may be associated with a weight of 0.5, and the third region 316C may be associated with a weight of 0.1. The fourth region 316D may be associated with a weight of 0.5, the fifth region 316E may be associated with a weight of 0.1, and the sixth region 316F may be associated with a weight of 0.5. Based on the assigned weights of the six regions, the circuit 202 may estimate the level of smoke throughout the physical space, ranging from 0 to 1. For example, the circuit 202 may estimate the level of smoke to be 0.55 based on a normalization process and an averaging process.

[0073] The estimated level of smoke may be determined to be greater than or equal to a threshold level (e.g., 0.5) at 326. According to an embodiment, the circuit 202 may be configured to determine the estimated level of smoke to be greater than or equal to a threshold level.

[0074] At 328, the smoke evacuation device 116 can be controlled based on the determination that the level of smoke is at or above a threshold level. According to one embodiment, the circuitry 202 can be configured to control the smoke evacuation device 116 based on the determination that the estimated level of smoke is at or above a threshold level. For example, the estimated level of smoke can be 0.55, which is higher than the threshold level of 0.5. The circuitry 202 can control the smoke evacuation device 116 to evacuate the smoke from the physical space to reduce the level of smoke below the threshold level.

[0075] According to one embodiment, the circuit 202 can be configured to turn on the smoke evacuator 116 to evacuate the smoke when the estimated level of smoke in the physical space is equal to or greater than a threshold level. The circuit 202 can turn on the smoke evacuator 116 when the smoke level is 0.55. Additionally, the circuit 202 can turn off the smoke evacuator 116 when the estimated level of smoke in the physical space is less than the threshold level. After evacuating the smoke, the circuit 202 can process subsequent frames (e.g., the second image 106B through the Nth image 106N) from the video stream of the surgical procedure to determine the smoke level in real time. For example, the circuit 202 can turn off the smoke evacuator 116 when the level of smoke in the physical space reaches 0.45, which is below a threshold level (e.g., 0.5). The circuit 202 can repeat the process of turning on the smoke evacuator 116 when the estimated level of smoke in the physical space (e.g., the surgical site or operating room) exceeds a threshold level during a surgical procedure.

[0076] Although example operations associated with one or more blocks of block diagram 300B are shown as discrete blocks, they may be divided into further blocks, combined into fewer blocks, or eliminated depending on the implementation of the example operations.

[0077] FIG. 4A illustrates an exemplary method for enhancing visibility of a physical space based on smoke level estimation, according to an embodiment of the present disclosure. The description of FIG. 4A is provided with reference to elements in FIGS. 1, 2, 3A, and 3B. FIG. 4A illustrates a block diagram 400A illustrating exemplary operations 402-408 described herein. The exemplary operations illustrated in block diagram 400A may begin at 402 and may be performed by any computer system, apparatus, or device, such as electronic device 102 of FIG. 1 or circuit 202 of FIG. 2.

[0078] At 402, display screen 206A can be controlled to display a first image (e.g., first image 106A or first image 302A) based on an estimated level of smoke (as described at 324 in FIG. 3B ). According to an embodiment, circuit 202 can be configured to control display screen 206A to display first image 302A. In an exemplary scenario, circuit 202 can receive first image 302A in real time from image capture device 114 (e.g., in online mode) during a surgical procedure and display first image 302A in real time for assistance to personnel, such as a surgeon, performing the surgical procedure. First image 302A can indicate a first visibility level of a physical space (e.g., a surgical site or operating room) based on an estimated level of smoke (as described at 324 in FIG. 3B ). For example, the estimated level of smoke can be 0.7. In some embodiments, the first visibility level can range from 0 to 1, with "0" being the lowest visibility level and "1" being the highest visibility level. Note that the higher the estimated level of smoke, the lower the visibility level of the physical space, and vice versa. In an exemplary scenario, the first visibility level of the physical space can be lowered (e.g., 0.43) based on the estimated level of smoke.

[0079] At 404, it may be determined that the first visibility level is lower than a visibility threshold. According to one embodiment, the circuit 202 may determine that the first visibility level is lower than a visibility threshold. For example, the visibility threshold may be 0.8. In one embodiment, the visibility threshold may be set by user input based on the preferences of the surgeon and / or supporting staff. In another embodiment, the visibility threshold is set by the neural network model 108 and may vary for different applications (e.g., type of surgery) or environmental conditions (e.g., lighting conditions) of the physical space. In another embodiment, the visibility threshold may be set based on experimentation with simulating different levels of smoke and selecting a visibility threshold at which visibility of the surgical field is not affected.

[0080] At 406, based on a determination that the first visibility level (e.g., 0.43) may be less than the visibility threshold (e.g., 0.8), the smoke evacuation device 116 may be controlled to evacuate the smoke from the physical space. The circuitry 202 may control the smoke evacuation device 116 in real time during a surgical procedure (e.g., in an online mode). Details of the control of the smoke evacuation device 116 are shown, for example, in FIG. 3B.

[0081] At 408, the display screen 206A can be controlled to display a second image (e.g., second image 106B) of the plurality of images 106 of the physical space based on the evacuation of smoke. According to an embodiment, the circuitry 202 can be configured to control the display screen 206A to display the second image 106B of the plurality of images 106 of the physical space based on the evacuation of smoke. The second image 106B can exhibit a second visibility level that is equal to or greater than a visibility threshold (e.g., 0.8). For example, the second visibility level of the second image 106B can be 0.9 based on the evacuation of smoke from the physical space. In this manner, the electronic device 102 can improve visibility of the surgical site in real time during a surgical procedure, providing a healthier working environment for the surgeon and supporting staff.

[0082] Although example operations associated with one or more blocks of block diagram 400A are shown as discrete blocks, they may be divided into further blocks, combined into fewer blocks, or eliminated depending on the implementation of the example operations.

[0083] FIG. 4B illustrates an exemplary method for enhancing visibility of a physical space based on smoke level estimation, according to an embodiment of the present disclosure. The description of FIG. 4B is provided with reference to elements of FIGS. 1, 2, 3A, 3B, and 4A. FIG. 4B illustrates a block diagram 400B illustrating exemplary operations 410-416 described herein. The exemplary operations illustrated in block diagram 400B may begin at 410 and may be performed by any computer system, apparatus, or device, such as electronic device 102 of FIG. 1 or circuit 202 of FIG. 2.

[0084] At 410, the display screen 206A can be controlled to display a first image 302A based on the estimated level of smoke (as described at 324 in FIG. 3B ). According to an embodiment, the circuit 202 can be configured to control the display screen 206A to display the first image 302A. In an exemplary scenario, the circuit 202 can receive the first image 302A from the image capture device 114 in offline mode and display the first image 302A after a surgical procedure. The first image 302A can indicate a first visibility level of the physical space based on the estimated level of smoke. The display of the first image 302A at 410 can be the same as the display of the first image 302A at 402. Details of the display of the first image 302A are shown, for example, at 402 in FIG. 4A .

[0085] At 412, the first visibility level may be determined to be less than a visibility threshold. According to an embodiment, the circuit 202 may determine that the first visibility level is less than a visibility threshold (e.g., 0.8).

[0086] At 414, a haze removal process (e.g., image haze removal) can be applied to the first image 302A based on determining that the first visibility level is likely to be lower than the visibility threshold. According to an embodiment, the circuit 202 can be configured to apply the haze removal process to the first image 302A based on determining that the first visibility level is likely to be lower than the visibility threshold. The haze removal process can be applied to the first image 302A to remove smoke effects from the first image 302A and provide a clear image with improved contrast and color revelation. For example, the haze removal process can be based on deep learning-based techniques. The haze removal process can be based on computer vision techniques, deep multi-model fusion network methods, densely connected pyramid dehazing network (DCPDN) methods, feature fusion attention network (FFA-Net) architecture methods, etc. The circuit 202 can apply the haze removal process in an offline mode, for example, after a surgical procedure. The haze removal process can be utilized for image enhancement applications, and in some embodiments, the circuit 202 can also apply the haze removal process in an online mode for image processing of the first image 302A during a surgical procedure.

[0087] At 416, the display screen 206A can be controlled to display a second image 106B of the plurality of images 106 of the physical space based on the application of the haze removal process. According to an embodiment, the circuitry 202 can be configured to control the display screen 206A to display a second image 106B of the plurality of images 106 of the physical space based on the application of the haze removal process. The second image 106B can exhibit a second visibility level that is equal to or greater than the visibility threshold. For example, after the application of the haze removal process, the second visibility level of the second image 106B can be 0.95.

[0088] Although example operations associated with one or more blocks of block diagram 400B are shown as discrete blocks, they may be divided into further blocks, combined into fewer blocks, or eliminated depending on the implementation of the example operations.

[0089] FIG. 5 is a flowchart illustrating an exemplary method for smoke level estimation, according to an embodiment of the present disclosure. Figure 5 will be described with reference to elements in Figures 1, 2, 3A, 3B, 4A, and 4B. Figure 5 shows a flowchart 500. The method shown in flowchart 500 can be performed by any computer system, such as electronic device 102 or circuit 202. The method can start at 502 and proceed to 504.

[0090] At 504, a first image (e.g., first image 106A) of a plurality of images (e.g., plurality of images 106) of a physical space (e.g., a surgical site or operating room) may be received. According to an embodiment, circuitry 202 may be configured to receive first image 106A of the plurality of images 106 of the physical space. Details of receiving first image 106A are further shown, for example, in FIG. 3A.

[0091] At 506, smoke may be detected in the physical space based on application of the trained neural network model (e.g., neural network model 108) to the received first image 106A. According to an embodiment, circuit 202 may be configured to detect smoke in the physical space based on application of the trained neural network model 108 to the received first image 106A. Details of smoke detection are further shown, for example, in FIG. 3A.

[0092] At 508, a heat map (such as heat map 110) of the physical space may be generated based on the detected smoke in the physical space and the output of trained neural network model 108 corresponding to the detection of smoke. According to an embodiment, circuitry 202 may be configured to generate heat map 110 of the physical space based on the detected smoke in the physical space and the output of trained neural network model 108 corresponding to the detection of smoke. Details of the generation of heat map 110 are further shown, for example, in FIG. 3A.

[0093] At 510, a level of smoke within the physical space can be estimated based on normalization of the generated heat map 110. According to an embodiment, the circuit 202 can be configured to estimate a level of smoke within the physical space based on normalization of the generated heat map 110. Details of estimating the level of smoke are further shown, for example, in FIG. 3B. Control can proceed to an end.

[0094] Although flowchart 500 depicts discrete operations such as 502, 504, 506, 508, and 510, the disclosure is not so limited. Accordingly, in some embodiments, such discrete operations may be further divided into additional operations, combined into fewer operations, or eliminated depending on the particular embodiment without departing from the essence of the disclosed embodiments.

[0095] Various embodiments of the present disclosure may provide a non-transitory computer-readable medium and / or storage medium having stored thereon instructions executable by a machine and / or computer (e.g., electronic device 402). The instructions may cause the machine and / or computer to perform operations that may include receiving a first image (e.g., first image 106A) of a plurality of images (e.g., plurality of images 106) of a physical space. The operations may further include detecting smoke in the physical space based on application of a trained neural network model (e.g., neural network model 108) to the received first image 106A. The operations may further include generating a heat map (e.g., heat map 110) of the physical space based on the detected smoke in the physical space and an output of trained neural network model 108 corresponding to the detection of the smoke. The operations may further include estimating a level of smoke in the physical space based on normalizing the generated heat map 110.

[0096] An exemplary aspect of the present disclosure may provide an electronic device (such as electronic device 102 of FIG. 1 ) including a circuit (such as circuit 202). Circuit 202 may be configured to receive a first image (such as first image 106A) of a plurality of images (such as plurality of images 106) of a physical space. Circuit 202 may be further configured to detect smoke in the physical space based on application of a trained neural network model (such as neural network model 108) to the received first image 106A. Circuit 202 may be further configured to generate a heat map (such as heat map 110) of the physical space based on the detected smoke in the physical space and an output of trained neural network model 108 corresponding to the detection of the smoke. Circuit 202 may be further configured to estimate a level of smoke in the physical space based on normalizing generated heat map 110.

[0097] According to an embodiment, the circuit 202 may be further configured to obtain a confidence score corresponding to each pixel subset of the plurality of pixels of the received first image 106A as an output of the trained neural network model 108. The confidence score may be related to the detection of smoke corresponding to each pixel subset of the received first image 106A. The circuit 202 may further generate a heat map 110 based on the confidence score of each pixel subset.

[0098] According to one embodiment, the circuitry 202 can be further configured to classify the smoke corresponding to each pixel subset of the received first image 106A into a particular category among a plurality of smoke intensity categories based on the trained neural network model 108. The classification can be based on a confidence score corresponding to each pixel subset. The circuitry 202 can further assign weights to the heat map 110 based on the classification of the smoke.

[0099] According to one embodiment, the circuit 202 may be further configured to determine a plurality of regions (e.g., first region 316A, second region 316B, ..., sixth region 316F), each corresponding to a particular category among the plurality of categories, based on the assigned weights. Each region of the plurality of regions may include a pixel subset classified into a particular category. The circuit 202 may further determine one or more overlapping regions (e.g., first overlapping region 318A, second overlapping region 318B, third overlapping region 318C) between the determined plurality of regions (e.g., first region 316A, second region 316B, ..., sixth region 316F). Furthermore, the circuit 202 may remove one or more overlapping regions from the heat map 110. The circuit 202 may further estimate the level of smoke in the physical space based on the removal of one or more overlapping regions (e.g., first overlapping region 318A, second overlapping region 318B, third overlapping region 318C).

[0100] According to an embodiment, the circuitry 202 may be further configured to receive a user input, which may correspond to the detection of smoke in a user-defined category of a plurality of categories.

[0101] According to one embodiment, the estimated level of smoke in the first image of the physical space can be in the range of 0 to 1.

[0102] According to one embodiment, the circuit 202 can be configured to receive a first image 106A of the plurality of images 106 in real time from an image sensor, from a memory of the electronic device 102, and / or from a server (such as the server 104).

[0103] According to an embodiment, the circuitry 202 may be further configured to control a smoke exhaust device (such as the smoke exhaust device 116) to exhaust the smoke from the physical space based on the estimated smoke level.

[0104] According to one embodiment, the circuitry 202 can be further configured to turn on the smoke evacuation device 116 to evacuate the smoke when the estimated level of smoke in the physical space is equal to or greater than a threshold level. The circuitry 202 can be further configured to turn off the smoke evacuation device 116 when the estimated level of smoke in the physical space is less than a threshold level. According to one embodiment, the threshold level can be 0.5.

[0105] According to an embodiment, circuitry 202 may be further configured to control a display screen (e.g., display screen 206A) to display a first image 106A indicating a first visibility level of the physical space based on the estimated level of smoke. Circuitry 202 may further control smoke evacuation device 116 to evacuate smoke from the physical space if the first visibility level is below a visibility threshold. Circuitry 202 may further control display screen 206A to display a second image (e.g., second image 106B) of the plurality of images 106 of the physical space based on the evacuation of smoke. Second image 106B may indicate a second visibility level that is equal to or greater than the visibility threshold.

[0106] According to an embodiment, the circuitry 202 can be further configured to control the display screen 206A to display a first image 106A indicative of a first visibility level of the physical space based on the estimated level of smoke. The circuitry 202 can further apply a haze removal process to the first image 106A if the first visibility level is below a visibility threshold. The circuitry 202 can further control the display screen 206A to display a second image 106B of the plurality of images 106 of the physical space based on the application of the haze removal process. The second image 106B can indicate a second visibility level that is equal to or greater than the visibility threshold.

[0107] According to an embodiment, the circuit 202 may be further configured to train a neural network model based on one or more features associated with one or more pixel subsets in the received first image 106A to obtain a trained neural network model 108. The neural network model 108 may be trained to detect smoke and generate a heat map 110. The one or more features include at least one of an intensity of the smoke in the one or more pixel subsets, a shape of the smoke in the one or more pixel subsets, or a color of the smoke in the one or more pixel subsets.

[0108] The present disclosure can be implemented in hardware or a combination of hardware and software. The present disclosure can be implemented in a centralized manner in at least one computer system, or in a distributed manner where different elements can be distributed across several interconnected computer systems. Any computer system or other device adapted to perform the methods described herein can be suitable. The combination of hardware and software can be a general-purpose computer system that includes a computer program that, when loaded and executed, can control the computer system to perform the methods described herein. The present disclosure can be implemented in hardware, including portions of integrated circuits that also perform other functions.

[0109] The present disclosure may also be embodied in a computer program product, which includes all features that enable the implementation of the methods described herein and which is capable of executing these methods when loaded into a computer system. A computer program in this context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having information processing capabilities to perform a particular function, either directly, or after a) conversion into another language, code or notation, or b) reproduction in a different content form, or both.

[0110] While the present disclosure has been described with reference to several embodiments, those skilled in the art will recognize that various modifications may be made and equivalents may be substituted without departing from the scope of the disclosure. Additionally, many modifications may be made to adapt a particular situation or material to the teachings of the disclosure without departing from the scope of the disclosure. Therefore, it is not intended that the disclosure be limited to the particular embodiments disclosed, but rather, it is intended to include all embodiments falling within the scope of the appended claims. [Explanation of symbols]

[0111] 100 Network Environment 102 Electronic equipment 104 Server 106 Multiple Images 106A First Image 106B Second Image 106N Image N 108 Neural Network Model 110 Heatmap 112 Communication Network 114 Image capture device 116 Smoke exhaust system

Claims

1. 1. An electronic device, comprising: receiving a first image of a plurality of images of a physical space; detecting smoke in the physical space based on application of a trained neural network model to the received first image; generating a heat map of the physical space based on the detected smoke in the physical space and an output of the trained neural network model corresponding to the detection of the smoke; obtaining, as an output of the trained neural network model, a confidence score corresponding to each pixel subset of a plurality of pixels of the received first image, the confidence score associated with the detection of the smoke corresponding to each pixel subset of the received first image; generating the heat map based on the confidence scores for each pixel subset; classifying the smoke corresponding to each pixel subset of the received first image into a particular category among a plurality of smoke intensity categories based on the confidence score corresponding to each pixel subset based on the trained neural network model; assigning weights to the heatmap based on the classification of the smoke; determining a plurality of regions each corresponding to a particular one of the categories based on the assigned weights, the plurality of regions each including the pixel subset classified into the particular category; determining one or more overlapping regions between the determined regions; estimating the level of smoke in the physical space based on normalizing the heat map from which the one or more overlapping regions have been removed; a circuit configured to: An electronic device characterized by:

2. the circuitry is further configured to receive a user input; the user input corresponds to the detection of the smoke in a user-defined category of the plurality of categories. The electronic device of claim 1 .

3. the estimated level of smoke in the first image of the physical space is in the range of 0 to 1; The electronic device of claim 1 .

4. the circuitry is configured to receive the first image of the plurality of images in real time from an image sensor, from a memory of the electronic device, or from a server. The electronic device of claim 1 .

5. the circuitry is further configured to control a smoke evacuation device to exhaust the smoke from the physical space based on the estimated level of the smoke. The electronic device of claim 1 .

6. The circuit comprises: turning on the smoke evacuation device to evacuate the smoke if the estimated level of the smoke in the physical space is equal to or greater than a threshold; turning off the smoke evacuation device when the estimated level of smoke in the physical space falls below a threshold level; The electronic device of claim 5 further configured to:

7. the threshold level is 0.5; 7. The electronic device of claim 6.

8. The circuit comprises: controlling a display screen to display the first image indicative of a first visibility level of the physical space based on the estimated level of smoke; controlling a smoke evacuation device to exhaust the smoke from the physical space when the first visibility level is below a visibility threshold; controlling the display screen to display a second image of the plurality of images of the physical space that indicates a second visibility level that is equal to or greater than the visibility threshold based on the exhaust of the smoke. The electronic device of claim 1 further configured to:

9. The circuit comprises: controlling a display screen to display the first image indicative of a first visibility level of the physical space based on the estimated level of smoke; applying a haze removal process to the first image if the first visibility level is below a visibility threshold; controlling the display screen to display a second image of the plurality of images of the physical space that exhibits a second visibility level that is equal to or greater than the visibility threshold based on application of the haze removal process. The electronic device of claim 1 further configured to:

10. the circuitry is further configured to train a neural network model based on one or more features associated with one or more pixel subsets in the received first image to obtain the trained neural network model; the neural network model is trained to detect the smoke and generate the heat map; the one or more features include at least one of: an intensity of the smoke in the one or more pixel subsets; a shape of the smoke in the one or more pixel subsets; or a color of the smoke in the one or more pixel subsets. The electronic device of claim 1 .

11. receiving a first image of a plurality of images of a physical space; detecting smoke within the physical space based on application of a trained neural network model to the received first image; and generating a heat map of the physical space based on the detected smoke within the physical space and an output of the trained neural network model corresponding to the detection of the smoke. classifying the smoke corresponding to each pixel subset of the received first image into a particular category among a plurality of smoke intensity categories based on a confidence score corresponding to each pixel subset based on the trained neural network model; assigning weights to the heatmap based on the classification of the smoke; determining a plurality of regions each corresponding to a particular one of the categories based on the assigned weights, the plurality of regions each including the pixel subset classified into the particular category; determining one or more overlapping regions between the determined regions; removing the one or more overlapping regions from the heat map; and estimating the level of smoke in the physical space based on a normalization of the heat map with the one or more overlapping regions removed; A method comprising:

12. and controlling a smoke evacuation device to exhaust the smoke from the physical space based on the estimated level of the smoke. The method of claim 11.

13. controlling a display screen to display the first image indicative of a first visibility level of the physical space based on the estimated level of smoke; controlling a smoke evacuation device to exhaust the smoke from the physical space when the first visibility level is below a visibility threshold; controlling the display screen to display a second image of the plurality of images of the physical space that indicates a second visibility level that is equal to or greater than the visibility threshold based on the exhaust of the smoke; The method of claim 11 further comprising:

14. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by a processor, cause the processor to: receiving a first image of a plurality of images of a physical space; detecting smoke in the physical space based on application of a trained neural network model to the received first image; generating a heat map of the physical space based on the detected smoke in the physical space and an output of the trained neural network model corresponding to the detection of the smoke; classifying the smoke corresponding to each pixel subset of the received first image into a particular category among a plurality of smoke intensity categories based on a confidence score corresponding to each pixel subset based on the trained neural network model; assigning weights to the heatmap based on the classification of the smoke; determining a plurality of regions each corresponding to a particular one of the categories based on the assigned weights, the plurality of regions each including the pixel subset classified into the particular category; determining one or more overlapping regions between the determined regions; removing the one or more overlapping regions from the heat map; and estimating the level of smoke in the physical space based on a normalization of the heat map with the one or more overlapping regions removed; 10. A non-transitory computer-readable medium for causing a computer to perform operations including:

Images