Electronic device, and method for capturing image in electronic device

Abstract

An electronic device according to various embodiments may be configured to include a camera module, a display, a plurality of detection modules, and a processor, wherein the processor detects at least one first region of interest in a first image input through the camera module using a first detection module for detecting a first region of interest among the plurality of detection modules, and if the processor fails to detect at least one first region of interest in a second image input through the camera using the first detection module that matches the at least one first region of interest detected in the first image, it detects at least one second region of interest in the second image using a second detection module for detecting a second region of interest among the plurality of detection modules, estimates at least one first region of interest based on the at least one second region of interest, and if the at least one first region of interest detected in the first image matches the at least one first region of interest estimated, it updates the at least one first region of interest estimated to at least one first region of interest, and changes the location of a preview area including the updated at least one first region of interest based on the location of the at least one first region of interest estimated that matches the at least one first region of interest detected in the first image. There are various other embodiments.

Description

Technology Field

[0001] Various embodiments relate to an electronic device and a method for capturing an image in an electronic device. Background Technology

[0002] Various services and additional functions provided through electronic devices, such as portable electronic devices like smartphones, are gradually increasing. To enhance the utility value of these devices and satisfy the needs of diverse users, telecommunications service providers and electronic device manufacturers are competitively developing devices to offer various functions and differentiate themselves from competitors. Consequently, the various functions provided through electronic devices are also becoming increasingly sophisticated.

[0003] Among the various functions provided by an electronic device, in the image capture function (e.g., video call, photo shooting, and / or video shooting), when a region of interest (e.g., face area) is set while capturing an image, functions such as zooming / panning are applied so that the set region of interest is always positioned at the center of the display screen. The problem to be solved

[0004] If a subject is not detected while capturing an image on an electronic device, for example, if the face of a subject set as the region of interest is partially obscured by a mask or the like, functions such as zooming or panning to position the region of interest at the center of the display screen may be disabled. means of solving the problem

[0005] An electronic device according to various embodiments comprises a camera module, a display, a plurality of detection modules, and a processor, wherein the processor detects at least one first region of interest in a first image input through the camera module using a first detection module for detecting a first region of interest among the plurality of detection modules, and if at least one first region of interest matching the at least one first region of interest detected in the first image is not detected in a second image input through the camera using the first detection module, the processor detects at least one second region of interest in the second image using a second detection module for detecting a second region of interest among the plurality of detection modules, estimates at least one first region of interest based on the at least one second region of interest, and if the at least one first region of interest detected in the first image matches the at least one first region of interest estimated, the estimated at least one first region of interest is updated to at least one first region of interest, and the at least one first region of interest matching the at least one first region of interest detected in the first image Based on the location of at least one first region of interest estimated above, the location of the preview region including the updated at least one first region of interest may be set to be changed.

[0006] According to various embodiments, a method for capturing an image in an electronic device comprises: detecting at least one first region of interest in a first image input through a camera module using a first detection module for detecting a first region of interest among a plurality of detection modules; if at least one first region of interest matching the at least one first region of interest detected in the first image is not detected in the second image input through the camera using the first detection module, detecting at least one second region of interest in the second image using a second detection module for detecting a second region of interest among the plurality of detection modules; estimating at least one first region of interest based on the at least one second region of interest; and if the at least one first region of interest detected in the first image matches the at least one first region of interest estimated, updating the at least one first region of interest estimated to at least one first region of interest; and matching the at least one first region of interest detected in the first image The method may include changing the location of a preview area including the updated at least one first region of interest based on the location of the estimated at least one first region of interest. Effects of the invention

[0007] According to various embodiments, there is an effect of continuously detecting a region of interest while capturing an image in an electronic device and providing the detected region of interest at the center of a display screen. Brief explanation of the drawing

[0008] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 2 is a block diagram illustrating an electronic device according to various embodiments. Figure 3 is a block diagram illustrating an image capturing module of an electronic device. FIGS. 4a and 4b are drawings illustrating an estimation function for estimating a first region of interest based on a second region of interest in an electronic device. FIG. 5 is a diagram illustrating an estimation function for estimating a second region of interest based on a first region of interest in an electronic device. FIGS. 6a to 6d are drawings illustrating an estimation function for estimating a first region of interest based on a third region of interest in an electronic device. Figure 7 is a diagram illustrating the movement of a preview image according to the movement of a subject in an electronic device. FIG. 8 is a flowchart illustrating the operation of detecting a region of interest during image capture of an electronic device according to various embodiments. FIGS. 9a and 9b are flowcharts illustrating the operation of detecting a region of interest during image capture of an electronic device according to various embodiments. FIG. 10 is a flowchart illustrating a region of interest detection operation during image capture of an electronic device according to various embodiments. . Specific details for implementing the invention

[0009] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0010] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0011] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0012] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0013] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0014] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0015] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0016] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0017] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0018] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0019] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0020] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0021] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0022] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0023] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0024] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0025] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0026] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0027] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0028] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0029] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0030] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0032] FIG. 2 is a block diagram (200) illustrating an electronic device according to various embodiments.

[0033] Referring to FIG. 2 above, an electronic device (201) (e.g., the electronic device (101) of FIG. 1) may include a processor (220), memory (230), a display (260), and a camera module (280).

[0034] According to various embodiments, the processor (220) can control the overall operation of the electronic device (201) and may be identical to the processor (120) of FIG. 1, or may perform at least one function or operation performed by the processor (120).

[0035] According to various embodiments, the processor (220) may include an image capturing module (310) capable of continuously detecting a region of interest using a plurality of detection modules while capturing an image and changing a preview image displayed on a display (260) according to a change in the region of interest.

[0036] According to one embodiment, the processor (220) can control the image capturing module (310) or perform the same function as the image capturing module (310).

[0037] According to one embodiment, the image capturing module (310) may be included in the processor (220) or included in the electronic device (201) as a separate module. The image capturing module (310) can be described in detail in FIG. 3 below.

[0038] According to various embodiments, the memory (230) may be implemented substantially identically or similarly to the memory (130) of FIG. 1.

[0039] According to various embodiments, the display (260) may be implemented substantially identically or similarly to the display module (160) of FIG. 1. According to one embodiment, the display (260) may display a preview image including a region of interest while capturing an image.

[0040] According to various embodiments, the camera module (280) may be implemented substantially identically or similarly to the camera module (180) of FIG. 1.

[0042] FIG. 3 is a block diagram (300) illustrating an image capturing module of an electronic device, FIG. 4a to FIG. 4b are drawings (400a to FIG. 400b) illustrating an estimation function for estimating a first region of interest based on a second region of interest in an electronic device, FIG. 5 is a drawing (500) illustrating an estimation function for estimating a second region of interest based on a first region of interest in an electronic device, FIG. 6a to FIG. 6d are drawings (600a to FIG. 600d) illustrating an estimation function for estimating a first region of interest based on a third region of interest in an electronic device, and FIG. 7 is a drawing (700) illustrating the movement of a preview image according to the movement of a subject in an electronic device.

[0043] Referring to FIG. 3 above, the image capturing module (310) may include an interest region determining module (330), a target region determining module (350), and a preview region determining module (370).

[0044] According to various embodiments, the region of interest determination module (330) (e.g., the processor (220) of FIG. 2) can detect a region of interest in an image input through the camera module (280).

[0045] According to one embodiment, the region of interest determination module (330) can detect a region of interest in an image input from the camera module (280) using a plurality of detection modules while capturing an image, crop the region of interest in the image, and perform a zooming function or / and a panning function so that the cropped region of interest is positioned at the center of a display (e.g., the display (260) of FIG. 2) at a certain ratio.

[0046] According to one embodiment, the region of interest determination module (330) may include a plurality of detection modules, and the plurality of detection modules may include Region of Interest (ROI) detection modules for detecting different regions of interest. For example, the plurality of detection modules may include a face detector for detecting a face, a body detector for detecting a body, and a face attribute detector for detecting face attributes, and may include other feature detection modules. Each of the plurality of detection modules may detect coordinate information (e.g., left coordinate, right coordinate, upper coordinate, and lower coordinate) of each of the plurality of regions of interest, and the plurality of detection modules may operate in a set order or simultaneously.

[0047] According to one embodiment, when a first region of interest is set among a plurality of regions of interest, the region of interest determination module (330) may determine a first detection module for detecting the first region of interest among the plurality of detection modules as the main detection module. The region of interest determination module (330) may determine at least one detection module excluding the first detection module determined as the main detection module among the plurality of detection modules as an auxiliary detection module, and may verify the application order of the at least one detection module determined as the auxiliary detection module for detecting the first region of interest based on the order set when the first detection module is determined as the main detection module. The region of interest determination module (330) may pre-set and store the application order of at least one auxiliary detection module for detecting the region of interest according to the type of the determined main detection module.

[0049] According to various embodiments, the region of interest determination module (330) (e.g., the processor (220) of FIG. 2) can sequentially use a plurality of detection modules to detect and update at least one first region of interest in each of at least one image through the camera module (280).

[0050] According to various embodiments, the region of interest determination module (330) can update the detected at least one first region of interest (e.g., N face regions) in a first image input through the camera module by using a first detection module (e.g., face detector) for detecting a first region of interest (e.g., face region) set as the main detection module among a plurality of detection modules. The first region of interest (e.g., face region) may represent a set region of interest among a plurality of regions of interest.

[0051] According to one embodiment, the interest region determination module (330) detects at least one first interest region (e.g., N face regions) in the first image, and then uses the first detection module (e.g., face detector) to detect at least one first interest region (e.g., M face regions) in the second image input through the camera module (280) after the first image.

[0052] Using a region matching function, at least one first region of interest (e.g., N face regions) detected in the first image and at least one first region of interest (e.g., M face regions) detected in the second image can be compared. The region matching function can be explained in detail by the following <Equation 1> to <Equation 2>.

[0053] The above interest region determination module (330) can update at least one first interest region to at least one first interest region (e.g., N face regions) detected in the first image and at least one first interest region (e.g., M face regions) detected in the second image when they all match (e.g., all faces of the same subject).

[0054] The above interest region determination module (330) can update at least one first interest region to a part of the at least one first interest region (e.g., K face regions) among N (K + J) face regions detected in the first image and a part of the at least one first interest region (e.g., K face regions) among M (K + L) face regions detected in the second image when the two parts of the first interest region (e.g., K face regions) match (e.g., some faces of the same subject).

[0055] The above interest region determination module (330) can determine some first interest regions (e.g., L face regions) that do not match at least one first interest region (e.g., N face regions (K + J face regions) detected in the first image among at least one first interest region (e.g., M face regions (K + L face regions)) detected in the second image as a new first interest region (e.g., face regions of a new subject), and update at least one first interest region by adding the new first interest region (e.g., L face regions).

[0056] According to one embodiment, the region of interest determination module (330) can detect at least one second region of interest (e.g., P body regions) in the second image by using a second detection module (e.g., body detector) for detecting a second region of interest (e.g., P body regions) which is set as an auxiliary detection module among a plurality of detection modules, in order to detect some first regions of interest (e.g., J face regions) among at least one first region of interest (e.g., N (K + J) face regions) detected in the first image that were not detected by the first detection module, and can estimate at least one first region of interest (e.g., P face regions) based on the at least one second region of interest (e.g., P body regions).

[0057] The above-described region of interest determination module (330) can estimate at least one first region of interest (e.g., P face regions) from at least one second region of interest (e.g., P body regions) using an estimation function. The estimation function can be described in detail in FIGS. 4a to 4b, FIGS. 5, and FIGS. 6a to 6d below.

[0058] The above interest region determination module (330) can compare some first interest regions (e.g., J face regions) detected in the first image with at least one estimated first interest region (e.g., P face regions) using a region matching function.

[0059] The above interest region determination module (330) can update at least one first interest region to the estimated at least one first interest region (e.g., P face regions) if at least some first interest regions (e.g., J face regions) detected in the first image and at least one first interest region (e.g., P face regions) all match.

[0060] The above interest region determination module (330) can update at least one first interest region to the at least one first interest region (e.g., Q face regions) among some first interest regions (e.g., J (Q + S) face regions) detected in the first image if the at least one first interest region (Q face regions) among some first interest regions (e.g., P (Q + S) face regions) estimated matches the at least one first interest region (e.g., Q face regions) among some first interest regions (e.g., P (Q + S) face regions).

[0061] The above interest region determination module (330) can determine some first interest regions (e.g., R face regions) that do not match some first interest regions (e.g., J face regions (Q+S face regions) detected in the first image among at least one estimated first interest region (e.g., P (Q+R) face regions) as new first interest regions (e.g., face regions of a new subject), and update at least one first interest region by adding the new first interest region (e.g., R face regions).

[0063] According to one embodiment, the interest region determination module (330) can detect at least one third interest region (e.g., T face feature regions) in the second image by using a second detection module (e.g., face attribute detector) for detecting a third interest region (e.g., T face feature regions) set as an auxiliary detection module among a plurality of detection modules, in order to detect some first interest regions (e.g., J face regions) among at least one first interest region (e.g., N face regions (K + J face regions) detected in the first image that were not detected by the first detection module, and can estimate at least one first interest region (e.g., T face regions) based on the at least one third interest region (e.g., T face feature regions).

[0064] The above-described region of interest determination module (330) can estimate at least one first region of interest (e.g., T face regions) from at least one third region of interest (e.g., T face feature regions) using an estimation function. The estimation function can be described in detail in FIGS. 4a to 4b, FIGS. 5, and FIGS. 6a to 6d below.

[0065] The above interest region determination module (330) can compare some first interest regions (e.g., J face regions) detected in the first image with at least one estimated first interest region (e.g., T face regions) using a region matching function.

[0066] The above interest region determination module (330) can update at least one first interest region to the estimated at least one first interest region (e.g., T face regions) if at least some first interest regions (e.g., J face regions) detected in the first image and at least one first interest region (e.g., T face regions) all match.

[0067] The above interest region determination module (330) can update at least one first interest region to the at least one first interest region (e.g., Q face regions) among some first interest regions (e.g., J (Q + S) face regions) detected in the first image if the at least one first interest region (Q face regions) among some first interest regions (e.g., T (Q + V) face regions) estimated matches the at least one first interest region (e.g., Q face regions) among some first interest regions (e.g., T (Q + V) face regions).

[0068] The above interest region determination module (330) can determine some first interest regions (e.g., V face regions) that do not match some first interest regions (e.g., J face regions (Q+S face regions) detected in the first image among at least one estimated first interest region (e.g., T face regions (Q+V face regions)) as new first interest regions (e.g., face regions of a new subject), and update at least one first interest region by adding the new first interest region (e.g., V face regions).

[0070] According to various embodiments, the region of interest determination module (330) can update the detected at least one second region of interest (e.g., N body regions) in a first image input through the camera module by using a second detection module (e.g., body detector) for detecting a second region of interest (e.g., body region) set as the main detection module among a plurality of detection modules. The second region of interest (e.g., body region) may represent a set region of interest among a plurality of regions of interest.

[0071] According to one embodiment, the region of interest determination module (330) can detect at least one second region of interest (e.g., N body regions) in the first image, and then, using the second detection module (e.g., body detector), detect at least one second region of interest (e.g., M body regions) in the second image input through the camera module (280) after the first image, compare the at least one second region of interest (e.g., N body regions) detected in the first image with the at least one second region of interest (e.g., M body regions) detected in the second image using a region matching function. The region matching function can be explained in detail in the following <Equation 1> to <Equation 2>.

[0072] The above interest region determination module (330) can update at least one second interest region to at least one second interest region (e.g., M body regions) detected in the second image when at least one second interest region (e.g., N body regions) detected in the first image and at least one second interest region (e.g., M body regions) detected in the second image match (e.g., all body regions of the same subject).

[0073] The above interest region determination module (330) can update at least one second interest region to a part of the at least one second interest region (e.g., K body regions) among the N (K + J) body regions detected in the first image and a part of the second interest region (e.g., K body regions) among the M (K + L) body regions detected in the second image when the two parts of the second interest region (e.g., K body regions) match (e.g., some body of the same subject).

[0074] The above interest region determination module (330) can determine some second interest regions (e.g., L body regions) that do not match at least one second interest region (e.g., M body regions (K + L) detected in the first image) among at least one second interest region (e.g., N body regions (K + J) detected in the second image) as new second interest regions (e.g., body regions of a new subject), and update at least one second interest region by adding the new second interest region (e.g., L body regions).

[0075] According to one embodiment, the interest region determination module (330) can detect at least one first interest region (e.g., P face regions) in the second image by using a first detection module (e.g., face detector) for detecting a first interest region (e.g., P face regions) set as an auxiliary detection module among a plurality of detection modules, in order to detect some second interest regions (e.g., J body regions) among at least one second interest region (e.g., N body regions (K + J)) detected in the first image that were not detected by the second detection module, and can estimate at least one second interest region (e.g., P body regions) based on the at least one first interest region (e.g., P face regions).

[0076] The above-mentioned region of interest determination module (330) can estimate at least one second region of interest (e.g., P body regions) from at least one first region of interest (e.g., P face regions) using an estimation function. The estimation function can be described in detail in FIGS. 4a to 4b, FIGS. 5, and FIGS. 6a to 6d below.

[0077] The above interest region determination module (330) can compare some second interest regions (e.g., J body regions) detected in the first image with at least one estimated second interest region (e.g., P body regions) using a region matching function.

[0078] The above interest region determination module (330) can update at least one second interest region to the estimated at least one second interest region (e.g., P body regions) if at least some second interest regions (e.g., J body regions) detected in the first image and at least one second interest region (e.g., P body regions) all match.

[0079] The above interest region determination module (330) can update at least one second interest region to the at least one second interest region (e.g., Q body regions) among the at least one second interest region (e.g., P body regions (Q body regions)) among the at least one second interest region (e.g., Q body regions) among the at least one second interest region (e.g., Q body regions) among the at least one second interest region (e.g., P body regions (Q body regions)) if the at least one second interest region (Q body regions) among the at least one second interest region (e.g., Q body regions) detected in the first image matches.

[0080] The above interest region determination module (330) can determine some second interest regions (e.g., R body regions) that do not match some second interest regions (e.g., J body regions (Q+S)) detected in the first image among at least one estimated second interest region (e.g., P (Q+R) body regions) as new second interest regions (e.g., body regions of a new subject), and update at least one second interest region by adding the new second interest region (e.g., R body regions).

[0081] According to one embodiment, the interest region determination module (330) can detect at least one third interest region (e.g., P face feature regions) in the second image by using a third detection module (e.g., face attribute detector) for detecting a third interest region (e.g., P face feature regions) set as an auxiliary detection module among a plurality of detection modules to detect some first interest regions (e.g., J body regions) that are not detected by the second detection module among at least one second interest region (e.g., N (K + J) body regions) detected in the first image, and can estimate at least one first interest region (e.g., P face regions) based on the at least one third interest region (e.g., P face feature regions) using an estimation function.

[0082] The above interest region determination module (330) can estimate at least one second interest region (e.g., P body regions) based on at least one first interest region (e.g., P face regions) using an estimation function.

[0083] The above estimation function can be explained in detail in FIGS. 4a to 4b, FIG. 5, and FIGS. 6a to 6d below.

[0084] The above interest region determination module (330) can compare some second interest regions (e.g., J body regions) detected in the first image with at least one estimated second interest region (e.g., P body regions) using a region matching function.

[0085] The above interest region determination module (330) can update at least one second interest region to the estimated at least one second interest region (e.g., P body regions) if at least some second interest regions (e.g., J body regions) detected in the first image and at least one second interest region (e.g., P body regions) all match.

[0086] The above interest region determination module (330) can update at least one second interest region to the at least one second interest region (e.g., Q body regions) among the at least one second interest region (e.g., P body regions (Q body regions)) among the at least one second interest region (e.g., Q body regions) among the at least one second interest region (e.g., Q body regions) among the at least one second interest region (e.g., P body regions (Q body regions)) if the at least one second interest region (Q body regions) among the at least one second interest region (e.g., Q body regions) detected in the first image matches.

[0087] The above interest region determination module (330) can determine some second interest regions (e.g., R body regions) that do not match some second interest regions (e.g., J body regions (Q+S)) detected in the first image among at least one estimated second interest region (e.g., P (Q+R) body regions) as new second interest regions (e.g., body regions of a new subject), and update at least one second interest region by adding the new second interest region (e.g., R body regions).

[0089] According to various embodiments, the region of interest determination module (330) (e.g., the processor (220) of FIG. 2) can detect and update at least one first region of interest in each of at least one image through the camera module (280) by simultaneously using a plurality of detection modules.

[0090] According to one embodiment, when the region of interest determination module (330) determines all of the plurality of detection modules as main detection modules, it can detect at least one first region of interest in an image input through the camera module by simultaneously using the plurality of detection modules.

[0091] According to one embodiment, the region of interest determination module (330) can simultaneously use a plurality of detection modules (e.g., a first detection module (e.g., a face detector), a second detection module (e.g., a body factor), and a third detection module (e.g., a face attribute detector)) to detect at least one first region of interest (e.g., N face regions) in a first image input through the camera module, and then update the detected at least one first region of interest (e.g., N face regions) to at least one first region of interest. The first region of interest (e.g., a face region) may represent a set region of interest among the plurality of regions of interest.

[0092] According to one embodiment, the region of interest determination module (330) can detect at least one first region of interest (e.g., N face regions) in the first image, and then use a first detection module (e.g., face detector) among the plurality of detection modules to detect at least one first region of interest (e.g., M face regions) in the second image input through the camera module (280) following the first image. The region of interest determination module (330) can detect at least one second region of interest (e.g., P body regions) in the second image using a second detection module (e.g., body detector) among the plurality of detection modules, and use an estimation function to detect at least one first region of interest (e.g., P face regions) based on the at least one second region of interest. The above interest region determination module (330) can detect at least one third interest region (e.g., T face feature regions) in the second image using a third detection module (e.g., face attribute detector) among the plurality of detection modules, and can detect at least one first interest region (e.g., T face regions) based on the at least one third interest region using an estimation function.

[0093] According to one embodiment, the region of interest determination module (330) can compare at least one first region of interest (e.g., N face regions) detected in the first image using a region matching function with at least one first region of interest (e.g., M face regions, P face regions, and T face regions) detected using the plurality of detection modules.

[0094] According to one embodiment, the interest region determination module (330) can update at least one first interest region (e.g., M face regions, P face regions, and T face regions) that matches at least one first interest region (e.g., N face regions) detected in the first image among at least one first interest region (e.g., M face regions, P face regions, and T face regions) detected using the plurality of detection modules as at least one first interest region.

[0095] According to one embodiment, the region of interest determination module (330) can update at least one first region of interest by adding some or all of the at least one first region of interest (e.g., M face regions, P face regions, and T face regions) that do not match the at least one first region of interest detected in the first image (e.g., N face regions) among the at least one first region of interest detected using the plurality of detection modules (e.g., M face regions, P face regions, and T face regions) as a new first region of interest.

[0096] According to various embodiments, the region of interest determination module (330) (e.g., the processor (220) of FIG. 2) can transmit at least one updated first region of interest to the target determination region (350).

[0097] According to various embodiments, the region of interest determination module (330) (e.g., the processor (220) of FIG. 2) can determine whether the first region of interest detected in the first image matches the first region of interest detected in the second image by using a region matching function.

[0098] According to one embodiment, the region of interest determination module (330) can perform a region matching function using at least one of a plurality of equations using an Interaction of Union (IOU) included in the following <Equation 1>. The region of interest determination module (330) can determine that the first region of interest (A region of interest) detected in the first image and the first region of interest (B region of interest) detected in the second image are the same first region of interest if the result of matching using at least one of the plurality of equations using an Interaction of Union (IOU) included in the following <Equation 1> is greater than or equal to a threshold value (e.g., 75%).

[0099] <Equation 1>

[0100]

[0102] According to one embodiment, the region of interest determination module (330) can calculate the size of the first region of interest by expanding it to a certain size or larger, taking into account that the subject detecting the region of interest moves.

[0103] According to one embodiment, when the interest region determination module (330) detects a plurality of first interest regions that match one first interest region, it may determine the first interest region having the largest IOU value among the plurality of first interest regions, or the first interest region that is closest in center distance to one of the plurality of first interest regions, as the first interest region that matches one first interest region.

[0104] According to one embodiment, the region of interest determination module (330) can perform a region matching function using a Euclidean distance that can calculate similarity as in <Equation 2>. The region of interest determination module (330) can, through the following <Equation 2>, if the Euclidean distance between the center (A RECT center) of the first region of interest (A region of interest) detected in the first image and the center (B RECT center) of the first region of interest (B region of interest) detected in the second image is smaller than the value obtained by multiplying the width (BodyWidth) of the second region of interest detected by the second interest module (e.g., body detector) by a certain ratio, or if it is smaller than the value obtained by multiplying the width (BodyWidth) of the second region of interest detected by the second interest module (e.g., body detector) by a certain ratio

[0105] The first region of interest (A region of interest) detected in the first image and the first region of interest (B region of interest) detected in the second image can be determined to be the same first region of interest.

[0106] <Equation 2>

[0107]

[0109] According to various embodiments, the region of interest determination module (330) (e.g., processor (220) of FIG. 2) may, by using an estimation function, estimate a first region of interest (e.g., face region) from a second region of interest (e.g., body region), estimate a second region of interest (e.g., body region) from a first region of interest (e.g., face region), or estimate a first region of interest (e.g., face region) from a third region of interest (e.g., face feature region).

[0110] According to one embodiment, the region of interest determination module (330) can estimate a first region of interest based on a second region of interest detected using a second detection module. The region of interest determination module (330) can estimate the first region of interest (413) from the second region of interest (411) by using an estimation function that calculates the size (W*R1, H*R2) of the first region of interest (e.g., face region) at a certain ratio (R1, R2) to the size (W(width), H(height)) of the second region of interest (411) (e.g., body region) as shown in FIG. 4a, and calculates the position (height*R4 at left, width*R3 at bottom) of the first region of interest (413) (e.g., face region) at a certain ratio (R3, R4) to the size (W(width), H(height)) of the second region of interest (411) (e.g., body region).

[0111] The above-mentioned region of interest determination module (330) can calculate the size and location of a first region of interest (e.g., face region) using location information between skeletons when human body skeleton information is stored in the memory of the electronic device (e.g., memory (230) of FIG. 2). The above-mentioned region of interest determination module (330) can estimate the size and location of the face region using the coordinates of the left and right shoulder skeletons among the human body skeleton information. The above-mentioned region of interest determination module (330) can determine the location of the estimated first region of interest (e.g., face region) (413) by multiplying the Euclidean distance between the left and right shoulder coordinates by specific ratios (e.g., Ratio5 and Ratio6), respectively, as in FIG. 4b, and estimating the size (e.g., a1*R6 (width) and a1*R5 (height)) of the first region of interest (e.g., face region) so that the center coordinates (a1 / 2) of the left and right shoulders match the center coordinates of the estimated first region of interest (e.g., face region) (413), and so that the first region of interest (e.g., face region) is located on the shoulders.

[0113] According to one embodiment, the region of interest determination module (330) can estimate a second region of interest (e.g., body region) based on a first region of interest (413) (e.g., face region) detected using a first detection module. The above interest region determination module (330) can estimate the second interest region (e.g., body region) by using an estimation function in which the upper side (TOP) of the second interest region (e.g., body region) coincides with the upper side (TOP) of the first interest region (e.g., face region) and the center (center) of the second interest region (e.g., body region) coincides with the center (center) of the first interest region (e.g., face region), as shown in FIG. 5.

[0114] According to one embodiment, the region of interest determination module (330) can estimate a first region of interest (e.g., face region) based on a third region of interest (e.g., face feature information) detected using a third detection module. The region of interest determination module (330) can estimate the first region of interest (e.g., face region) based on a second region of interest (e.g., body region) or estimate the second region of interest (e.g., body region) based on the first region of interest (e.g., face region) by using relative positions and sizes based on information such as eyes, nose, and mouth detected in the third region of interest (e.g., face feature information), using a similar method of estimating the first region of interest (e.g., face region) using relative positions and sizes based on the third region of interest (e.g., face feature information). When the region of interest determination module (330) detects both eyes (631, 633) as a third region of interest as in FIG. 6a, the size and position of the first region of interest (e.g., face region) are R' and R' at the distance (W) between both eyes (631, 633) and the height (H) of the eyes (631, 633). ,, , R ,,,A first region of interest (e.g., face region) (613) can be estimated using a ratio-multiplying estimation function.

[0115] When the region of interest determination module (330) detects both eyes (631, 633) and the nose (635) as the third region of interest as in FIG. 6b, the size and position of the first region of interest (e.g., face region) are R' and R' at the distance (W) between both eyes (631, 633) and the height (H) between the eyes (631, 633) and the nose (631). ,, , R ,,, A first region of interest (e.g., face region) (613) can be estimated using a ratio-multiplying estimation function.

[0116] When the interest region determination module (330) detects the nose (635) and mouth (637) as the third interest region as shown in FIG. 6c, the size and position of the first interest region (e.g., face region) are R' and R' in relation to the left-right distance (W) of the mouth (637) and the height (H) of the nose (635) and mouth (637). ,, , R ,,, A first region of interest (e.g., face region) (613) can be estimated using a ratio-multiplying estimation function.

[0117] When the above-described region of interest determination module (330) detects the mouth (637) as the third region of interest as in FIG. 6d, the size and position of the first region of interest (e.g., face region) are R' and R' at the distance (W) and height (H) of the mouth (637). ,, , R ,,, A first region of interest (e.g., face region) (613) can be estimated using a ratio-multiplying estimation function.

[0118] The above interest region determination module (330) can estimate a first interest region based on a third interest region as shown in FIGS. 6a to 6d, and estimate a second interest region in the same way as in FIG. 5a based on the estimated first interest region.

[0119] According to one embodiment, the region of interest determination module (330) can update the size of the estimated region of interest differently according to a certain scale used in the estimation function.

[0120] For example, the above-mentioned region of interest determination module (330) can detect a second region of interest (e.g., body area) using a second detection module (e.g., body detector) which is an auxiliary detector, when the first detection module for detecting a first region of interest (e.g., face area) is the main detection module, and the first region of interest (e.g., face area) is partially obscured by a mask or the like, resulting in a failure to detect the first region of interest (e.g., face area). The above-mentioned region of interest determination module (330) can estimate the region of interest as only the face area, up to the upper body area, or the entire body area according to a certain ratio (e.g., R1, R2, R3, R4) with respect to the detected second region of interest (e.g., body area). Accordingly, when multiple subjects (e.g., people) are displayed on a display (e.g., the display (260) of FIG. 2), all of the multiple subjects may be displayed cropped with the face area as the center, or the face area of ​​some subjects and the upper body area of ​​some subjects may be cropped and displayed together, or the face area of ​​some subjects, the upper body area of ​​some subjects, and the body area of ​​some subjects may be cropped and displayed together. That is, according to a certain ratio used in the estimation function, multiple subjects (e.g., people) with different crop ranges applied, such as the face area, the upper body area, and the body area, may be displayed on the display.

[0121] For example, the above-mentioned interest region determination module (330) may change the main interest region to an estimated upper body region or a second interest region (e.g., body region) by operating a second detection module (e.g., body detector) when the first interest region (e.g., face region) is set as the main interest region and the first interest region (e.g., face region) is partially obscured by a mask or the like. When the above-mentioned interest region determination module (330) changes the main interest region to an upper body region or a second interest region (e.g., body region), the zooming centered on the face region may be changed to zooming centered on the upper body region or body region.

[0122] According to one embodiment, the region of interest determining module (330) can apply different regions of interest to each of all subjects included in the image input through the camera module (e.g., the camera module (280) of FIG. 2).

[0124] According to various embodiments, the region of interest determination module (330) (e.g., the processor (220) of FIG. 2) may determine that at least one first region of interest (e.g., N face regions) detected in a first image input through a camera module (e.g., the camera module (280) of FIG. 2) and at least one first region of interest that continuously matches for more than a certain number of frames input through the camera module (e.g., the camera module (280) of FIG. 2) after the first image matches the at least one first region of interest (e.g., N face regions) detected in the first image. Accordingly, it is possible to prevent a subject passing in front of the camera module (e.g., a person) or a subject included in an image input through the camera module for only a short period of time (e.g., a person) from being set as a region of interest.

[0125] According to one embodiment, the region of interest determination module (330) may use all regions of interest or use only a certain number of regions of interest among all regions of interest by applying different regions of interest to each of the plurality of subjects included in the image input through the camera module.

[0126] For example, the interest region determination module (330) may select to use only a certain number of interest regions with large sizes among all interest regions, and the method of selecting a certain number of interest regions with large sizes may be to select them by classifying them in order of size or to select a certain number close to the center of the image. The interest region determination module (330) may also select a certain number by applying weights to the distance from the center of the image and the size of the interest region. The interest region determination module (330) may determine a main subject among a plurality of subjects included in the image based on the selected certain number of interest regions, and perform zooming and / or panning functions by setting only the main subject as an interest region.

[0128] According to various embodiments, when the main region of interest determination module (330) (e.g., processor (220) of FIG. 2) fails to detect the main region of interest (e.g., first region of interest) in the main detection module (e.g., first detection module) and detects the main region of interest (e.g., first region of interest) using a plurality of auxiliary detection modules, the order between the plurality of auxiliary detection modules can be changed and a new auxiliary detection module can be added.

[0129] According to various embodiments, the region of interest determination module (330) (e.g., the processor (220) of FIG. 2) may perform a region matching function after performing an estimation function when detecting a primary region of interest (e.g., a first region of interest) using a plurality of auxiliary detection modules, or may perform an estimation function after performing a region matching function.

[0130] According to one embodiment, the region of interest determination module (330) can update previous regions of interest with different regions of interest detected using a plurality of auxiliary detection modules when it does not perform an estimation function.

[0131] According to various embodiments, the region of interest determination module (330) (e.g., processor (220) of FIG. 2) may detect a region of interest for all subjects included in an image input through a camera module (e.g., camera module (280) of FIG. 2) using a plurality of detection modules, or may detect a region of interest limited to a surrounding area of ​​a set region of interest.

[0133] According to various embodiments, the target region determination module (350) (e.g., the processor (220) of FIG. 2) can change the target region displaying at least one first region of interest based on a change in the first region of interest.

[0134] According to one embodiment, the target region determination module (350) can calculate a new second target region based on a change in at least one first region of interest detected in a second image input after the first image while displaying a first preview image as a first target region for displaying at least one first region of interest detected in the first image.

[0135] According to one embodiment, the target area determination module (350) can update the first target area for displaying at least one first area of ​​interest to a new second target area when at least one of the following conditions is satisfied: a first condition in which the number of at least one first area of ​​interest changes as a new subject is added or an existing subject disappears from the image; a second condition in which the position of some or all of the at least one first area of ​​interest changes as a result of the movement of some or all of the subjects (e.g., when some or all of the at least one first area of ​​interest moves out of the current target area (e.g., first target area)); or a third condition in which the size of the union area (union RECT) containing at least one first area of ​​interest changes as a result in the size of the union area becoming smaller than a certain ratio (Th1) relative to the current target area (e.g., first target area) (Target RECT) and satisfying <Equation 3> below.

[0136] <Equation 3>

[0137]

[0138] Union_RECT : At least one region of interest( N Minimum area including (ROI RECT)

[0140] For example, the target area determination module (350) may update the first target area to the second target area if it additionally includes a first area of ​​interest for a new subject trackable in the image (e.g., the second image) according to the first condition. If the target area determination module (350) fails to detect a first area of ​​interest for a subject previously tracked in the image (e.g., the second image) according to the first condition, it may update the first target area to the second target area so as to display only the first areas of interest of the remaining subjects.

[0141] For example, the target area determination module (350) can provide the effect of moving along with the subject by updating the first target area to the second target area when the subject, to which zooming and / or panning is applied according to the setting of the first area of ​​interest according to the second condition, moves up / down / left / right on the screen and approaches the boundary of the first target area. The target area determination module (350) can update the first target area to the second target area if it determines that at least one subject will move out of the first target area by making a judgment for each of the areas of interest set within the image (e.g., the second image).

[0142] For example, the target area determination module (350) may update the first target area to a second target area when, according to the third condition, the position of at least one first area of ​​interest included in the first target area changes above a threshold value. For example, the target area determination module (350) may update the first target area to a second target area so as to reduce the margin within the display and zoom in on the first area of ​​interest more significantly when subjects standing in the display (e.g., the display (260) of FIG. 2) sit down, or when at least one first area of ​​interest that was widely distributed in the first image gathers to one side in the second image.

[0143] According to one embodiment, the target area determination module (350) can update the first target area (current target area) to the second target area (new target RECT) through the following <Equation 4> when at least one of the first to third conditions is satisfied. The target area determination module (350) can calculate the new target area by adding a margin of a certain ratio to the top, bottom, left, and right of the union area (union RECT) that includes at least one first area of ​​interest.

[0144] <Equation 4>

[0145]

[0147] According to one embodiment, the target area determination module (350) can determine a new preview area by adjusting the size of the new target area in the same ratio as the image input through the camera module (e.g., the camera module (280) of FIG. 2), and transmit the determined new preview area to the preview area determination module. The target area determination module (350) can compare the determined new preview area with the input image, and if the size of the new preview area is greater than or equal to the size of the input image or extends beyond the boundaries of the input image, the size of the new preview area can be adjusted to match the size of the input image.

[0149] According to various embodiments, the preview area determination module (370) can provide a new preview image by gradually moving from the previous preview image.

[0150] According to one embodiment, the preview area determination module (370) can generate an intermediate preview image (713) containing an intermediate preview area to be displayed between the current preview image (711) and the new preview image (715) when changing and displaying the current preview image (711) containing the current preview area received from the target area determination module (e.g., 350), as shown in FIG. 7. The preview area determination module (370) can naturally change and display the current preview image to the new preview image (715) by sequentially displaying the current preview image (711), the intermediate preview image (713), and the new preview image (715).

[0151] According to one embodiment, the preview area determination module (370) can generate an intermediate preview image (713) including an intermediate preview area by calculating center coordinate information (NextCenter) as location information of the intermediate preview area through the following <Equation 5> and calculating size information (NextRectSize) of the intermediate preview area through the following <Equation 6>, when the maximum distance that can be moved at once is d1 and the ratio that can be enlarged or reduced at once is r1.

[0152] <Equation 5>

[0153]

[0154] ( : Center coordinate information of the current preview area, : Center coordinate information of the new preview area)

[0155]

[0157] <Equation 6>

[0158]

[0159]

[0160] (CurrentRectSize: Size of the current preview area, PreviewRectSize: Size of the new preview area)

[0162] According to one embodiment, the preview area determination module (370) can adaptively apply d1, which is the maximum distance that can be moved at once, and r1, which is the ratio that can be enlarged or reduced at once, so that when moving from a current preview image (711) containing a current preview area to a new preview image containing a new preview area, the new preview image can be displayed while simultaneously terminating the independently operating movement function and the zoom or reduction function. The preview area determination module (370) can adjust the number of zooming (enlargement or reduction) and movement of the preview area while comparing the number of zooming (enlargement or reduction) and movement required at the moment.

[0164] According to one embodiment, the preview area determination module (370) can adjust the amount of movement of the preview area (e.g., the current preview area of ​​the current preview image or the new preview area of ​​the new preview image). The preview area determination module (370) can gradually increase d1, which is the maximum distance that can be moved at once, when the preview area moves above a threshold value and the amount of change is large, and gradually decrease d1 when the preview area moves below a threshold value and the amount of change is small.

[0165] The above preview area determination module (370) can adaptively apply r1, which is a ratio that can be enlarged or reduced at once according to the amount of change of zooming (enlargement or reduction) in the same way as the amount of movement of the preview area.

[0166] According to one embodiment, the preview area determination module (370) can adjust the new preview area to a resolution set in the electronic device and display it on a display (e.g., the display (260) of FIG. 2).

[0168] According to various embodiments, an electronic device (e.g., electronic device (201)) comprises a camera module (e.g., camera module (280) of FIG. 2), a display (e.g., display (260) of FIG. 2), a plurality of detection modules, and a processor (e.g., processor (220) of FIG. 2). The processor detects at least one first region of interest in a first image input through the camera module using a first detection module for detecting a first region of interest among the plurality of detection modules, and if the processor fails to detect at least one first region of interest in a second image input through the camera using the first detection module that matches the at least one first region of interest detected in the first image, it detects at least one second region of interest in the second image using a second detection module for detecting a second region of interest among the plurality of detection modules, estimates at least one first region of interest based on the at least one second region of interest, and if the at least one first region of interest detected in the first image matches the estimated at least one first region of interest, the estimated at least one first region of interest is at least one Updated to a first region of interest, and matched with at least one first region of interest detected in the first image Based on the location of at least one first region of interest estimated above, the location of the preview region including the updated at least one first region of interest may be set to be changed.

[0169] According to various embodiments, the processor may be configured to detect at least one first region of interest in the second image input through the camera module using the first detection module, determine whether the at least one first region of interest detected in the first image matches the at least one first region of interest detected in the second image, and if the at least one first region of interest detected in the first image matches the at least one first region of interest detected in the second image, update the at least one first region of interest detected in the second image to at least one first region of interest.

[0170] According to various embodiments, the processor may be configured to update at least one first region of interest by adding the at least one first region of interest detected in the second image as a new first region of interest if the at least one first region of interest detected in the first image does not match the at least one first region of interest detected in the second image.

[0171] According to various embodiments, the processor may be configured to update at least one first region of interest by adding the estimated at least one first region of interest as a new first region of interest if at least one first region of interest detected in the first image and at least one first region of interest estimated based on the second region of interest do not match.

[0172] According to various embodiments, the processor may be configured to detect at least one third region of interest in the second image using a third detection module for detecting a third region of interest among the plurality of detection modules when the at least one first region of interest detected in the first image does not match the at least one first region of interest estimated based on the at least one second region of interest, estimate at least one first region of interest based on the estimated at least one second region of interest, and when the at least one first region of interest detected in the first image matches the estimated at least one first region of interest, update the estimated at least one first region of interest to at least one first region of interest.

[0173] According to various embodiments, the processor may be configured to calculate the size of the first region of interest based on the size of the second region of interest and a certain ratio, calculate the location of the first region of interest based on the size of the second region of interest and a certain ratio, and estimate the first region of interest in the second region of interest based on the calculated size and location of the first region of interest.

[0174] According to various embodiments, the processor may be configured to determine the first detection module for detecting the first region of interest among the plurality of regions of interest as the main detection module when confirming the setting of the first region of interest among the plurality of detection modules.

[0175] According to various embodiments, the processor may be configured to determine at least one detection module, excluding the first detection module determined as the main detection module among the plurality of detection modules, as an auxiliary detection module, and to check the application order of the at least one detection module determined as the auxiliary detection module for detecting the first region of interest based on the order set when the first detection module is determined as the main detection module.

[0176] According to various embodiments, when the processor determines all of the plurality of detection modules as main detection modules, it may be configured to use the plurality of detection modules simultaneously to detect at least one first region of interest that matches at least one first region of interest detected in the first image in a second image input through the camera.

[0177] According to various embodiments, the processor may be configured to update the at least one first region of interest detected using the plurality of detection modules to the at least one first region of interest if the at least one first region of interest detected using the plurality of detection modules matches the at least one first region of interest detected using the plurality of detection modules.

[0179] FIG. 8 is a flowchart (800) for explaining a region of interest detection operation during image capture of an electronic device according to various embodiments. The region of interest detection operation may include operations 801 through 815 and may be understood to be performed by a processor (e.g., processor (120) of FIG. 1 or processor (220) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2). According to one embodiment, at least one of operations 801 through 815 may be omitted, the order of some operations may be changed, or other operations may be added.

[0180] In operation 801, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can detect at least one first region of interest in a first image input through a camera module (e.g., camera module (280)) using a first detection module.

[0181] According to one embodiment, the electronic device may use a first detection module (e.g., face detector) for detecting a first region of interest (e.g., face region) among a plurality of detection modules as a primary detection module to detect at least one first region of interest in a first image input through the camera module, and may update the detected at least one first region of interest to at least one first region of interest. The first region of interest (e.g., face region) may represent a set region of interest among a plurality of regions of interest.

[0182] According to one embodiment, the plurality of detection modules may include Region of Interest (ROI) detection modules for detecting different regions of interest. For example, the plurality of detection modules may include a face detector for detecting a face, a body detector for detecting a body, and a face attribute detector for detecting face features, and may include other feature detection modules. Each of the plurality of detection modules can detect coordinate information (e.g., left coordinate, right coordinate, upper coordinate, and lower coordinate) of each of the plurality of regions of interest.

[0183] In operation 803, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can check whether at least one first region of interest is detected in a second image input through a camera that matches at least one first region of interest detected in the first image using a first detection module.

[0184] In the above 803 operation, if the electronic device detects at least one first region of interest that matches at least one first region of interest detected in the first image in the second image input through the camera using the first detection module, in the 805 operation, the electronic device can update at least one first region of interest detected in the second image input through the camera using the first detection module to at least one first region of interest.

[0185] According to one embodiment, the electronic device can change the location of a preview region including the updated at least one first region of interest based on the location of the estimated at least one first region of interest.

[0186] According to one embodiment, the electronic device may update a target area for displaying at least one first region of interest when satisfying at least one of the following conditions: a first condition in which the number of at least one first region of interest changes, a first condition in which the location of some or all of at least one first region of interest changes, or a condition in which the size of the integrated area of ​​at least one first region of interest changes. The electronic device may update the target area by adding a margin of a certain ratio to the top, bottom, left, and right of the integrated area containing the at least one first region of interest. The electronic device may calculate a new preview area based on the updated target area. The electronic device may determine the location and size of an intermediate preview area based on the coordinate information of the current preview area, the coordinate information of the new preview area, a set maximum distance that can be moved at once, and a set ratio for zooming in or out at once. The electronic device may sequentially display a current preview image including the current preview area, an intermediate preview image including the intermediate preview area, and a new preview image including the new preview area.

[0187] In the above 803 operation, if the electronic device fails to detect at least one first region of interest that matches at least one first region of interest detected in the first image using the first detection module in the second image input through the camera, in the 807 operation, the electronic device may detect at least one second region of interest in the second image using the second detection module.

[0188] According to one embodiment, if the electronic device fails to detect a first region of interest (e.g., a face region) in a second image received through the camera module after the first image using the first detection module (e.g., a face detector), it may determine that at least one first region of interest matching at least one first region of interest detected in the first image was not detected in the second image.

[0189] According to one embodiment, the electronic device detects a first region of interest (e.g., a face region) in the second image received through the camera module after the first image using the first detection module (e.g., a face detector), and if it is determined that at least one first region of interest detected in the first image and at least one first region of interest detected in the second image do not match using a region matching function based on <Equation 1> and <Equation 2>, it can be determined that at least one first region of interest matching the at least one first region of interest detected in the first image was not detected in the second image.

[0190] According to one embodiment, the electronic device can update at least one first region of interest by using the region matching function to add at least one first region of interest detected in the second image, which is determined not to match at least one first region of interest detected in the first image, as a new at least one first region of interest.

[0191] According to one embodiment, the electronic device can detect at least one second region of interest in the second image by using a second detection module (e.g., body detector) for detecting a second region of interest (e.g., body region) among a plurality of detection modules as an auxiliary detection module.

[0192] In operation 809, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can estimate at least one first region of interest (e.g., face region) based on at least one second region of interest (e.g., body region).

[0193] According to one embodiment, the electronic device can estimate at least one first region of interest (e.g., face region) from at least one second region of interest (e.g., body region) using an estimation function based on FIGS. 4a and 4b.

[0194] In operation 811, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can determine whether at least one first region of interest detected in the first image matches at least one first region of interest estimated above.

[0195] In the above 811 operation, if the electronic device confirms that at least one first region of interest detected in the first image matches the estimated at least one first region of interest, in the 813 operation, the electronic device may update the estimated at least one first region of interest to at least one first region of interest.

[0196] According to one embodiment, the electronic device can update the estimated at least one first region of interest to at least one first region of interest by using a region matching function based on <Equation 1> and <Equation 2> when the at least one first region of interest detected in the first image matches the estimated at least one first region of interest.

[0197] According to one embodiment, the electronic device can update at least one first region of interest by using the region matching function, if at least one first region of interest detected in the first image does not match the estimated at least one first region of interest, by adding the estimated at least one first region of interest as a new at least one first region of interest.

[0198] According to one embodiment, the electronic device may update a target area for displaying at least one first region of interest when satisfying at least one of the following conditions: a first condition in which the number of at least one first region of interest changes, a first condition in which the location of some or all of at least one first region of interest changes, or a condition in which the size of the integrated area of ​​at least one first region of interest changes. The electronic device may update the target area by adding a margin of a certain ratio to the top, bottom, left, and right of the integrated area containing the at least one first region of interest. The electronic device may calculate a new preview area based on the updated target area. The electronic device may determine the location and size of an intermediate preview area based on the coordinate information of the current preview area, the coordinate information of the new preview area, a set maximum distance that can be moved at once, and a set ratio for zooming in or out at once. The electronic device may sequentially display a current preview image including the current preview area, an intermediate preview image including the intermediate preview area, and a new preview image including the new preview area.

[0199] In the above 811 operation, if the electronic device confirms that at least one first region of interest detected in the first image does not match the estimated at least one first region of interest, in the 815 operation, the electronic device may execute an operation to detect at least one first region of interest that matches the at least one first region of interest detected in the first image in the second image using the next detection module among the plurality of detection modules.

[0201] FIGS. 9a and 9b are flowcharts (900a to 900b) for explaining a region of interest detection operation during image capture of an electronic device according to various embodiments. The region of interest detection operation may include operations 901 to 913 of FIG. 9a and operations 921 to 933 of FIG. 9b and may be understood to be performed by a processor (e.g., processor (120) of FIG. 1 or processor (220) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2). According to one embodiment, at least one of operations 901 to 913 of FIG. 9a and operations 921 to 933 of FIG. 9b may be omitted, the order of some operations may be changed, or other operations may be added.

[0202] Referring to FIGS. 9a and FIGS. 9b above, when the main detection module is set as a first detection module (e.g., face detector), the operation of detecting a first region of interest (e.g., face region) is as follows.

[0203] Referring to FIG. 9a above, in operation 901, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can input an image (e.g., a second image).

[0204] According to one embodiment, the electronic device can input an image (e.g., a second image) through a camera module (e.g., a camera module (280) of FIG. 2).

[0205] In operation 903, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can detect at least one first region of interest (e.g., M face regions) in an image (e.g., a second image) through a first detection module (e.g., a face detector).

[0206] According to one embodiment, at least one first region of interest (e.g., M face regions) can be detected in the image (e.g., a second image) using a first detection module (e.g., a face detector) for detecting a first region of interest among a plurality of detection modules.

[0207] In operation 905, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can determine whether there is a partial or complete match between at least one first region of interest detected in an image (e.g., the second image) and at least one first region of interest detected in a previous image (e.g., the first image).

[0208] According to one embodiment, the electronic device can determine whether there is a partial or complete match between at least one first region of interest (e.g., M face regions) detected in an image (e.g., a second image) and at least one first region of interest (e.g., N face regions) detected in a previous image (e.g., a first image) by using a region matching function based on <Equation 1> to <Equation 2>.

[0209] In the above 905 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can update at least one first region of interest (e.g., N face regions) detected in the previous image (e.g., the first image) and at least one first region of interest (e.g., M face regions) detected in the image (e.g., the second image) in the 907 operation to at least one first region of interest (e.g., M (K+L) face regions) detected in the image (e.g., the second image) when both match (e.g., all are faces of the same subject).

[0210] In the above 905 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) may, in the above 907 operation, update at least one first region of interest (e.g., K face regions) among at least one first region of interest (e.g., N (K + J) face regions) detected in the previous image (e.g., the first image) and some first region of interest (e.g., K face regions) among at least one first region of interest (e.g., M (K + L) face regions) detected in the image (e.g., the second image) to some first region of interest (e.g., K face regions) among at least one first region of interest (e.g., some faces of the same subject) in the above 907 operation.

[0211] In the above 905 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) determines as a new first region of interest (e.g., L face regions) that does not match the at least one first region of interest (e.g., N (K + J) face regions) detected in the previous image (e.g., the first image) among at least one first region of interest (e.g., M (K + L) face regions) detected in the image (e.g., the second image), and in the above 907 operation, the at least one first region of interest may be updated by adding the new first region of interest (e.g., L face regions).

[0212] In the above 905 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can perform the 909 operation if it confirms the existence of some first regions of interest (e.g., J face regions) that were not detected using the first detection module among at least one first region of interest (e.g., N (K + J) face regions) detected in the previous image (e.g., the first image).

[0213] In the above 909 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can detect at least one second region of interest (e.g., P body regions) in an image (e.g., the second image) using a second detection module (e.g., a body detector).

[0214] According to one embodiment, the electronic device can detect at least one second region of interest (e.g., P body regions) in an image (e.g., a second image) using a second detection module (e.g., a body detector) for detecting a second region of interest among a plurality of detection modules.

[0215] In operation 911, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can estimate at least one first region of interest (e.g., P face regions) based on at least one second region of interest (e.g., P body regions).

[0216] According to one embodiment, the electronic device can estimate the at least one first region of interest (e.g., P face regions) in the at least one second region of interest (e.g., P body regions) using an estimation function based on FIGS. 4a and 4b.

[0217] In operation 913, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can determine whether there is a partial or complete match between at least one estimated first region of interest (e.g., P face regions) and some first regions of interest (e.g., J (Q+S) face regions) detected in a previous image (e.g., the first image).

[0218] According to one embodiment, the electronic device can determine whether there is a partial or complete match between the estimated at least one first region of interest (e.g., P face regions) and some first regions of interest (e.g., J (Q + S) face regions) detected in a previous image (e.g., the first image) by using a region matching function based on <Equation 1> or <Equation 2>.

[0219] In the above 913 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) may, in the above 907 operation, update at least one first region of interest to the estimated at least one first region of interest (e.g., P face regions) if at least some first regions of interest (e.g., J face regions) detected in the previous image (e.g., the first image) and at least one estimated first region of interest (e.g., P face regions) all match.

[0220] In the above 913 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) may, in the above 907 operation, update at least one first region of interest to the above first region of interest (e.g., Q face regions) among some first regions of interest (e.g., J (Q + S) face regions) detected in the previous image (e.g., the first image) and some first regions of interest (Q face regions) among some first regions of interest (e.g., P (Q + S) face regions) estimated, if the above first regions of interest (Q face regions) match the above estimated at least one first region of interest (e.g., P (Q + S) face regions).

[0221] In the above 913 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) determines as a new first region of interest (e.g., R face regions) that does not match some first regions of interest (e.g., J face regions (Q+S face regions) detected in the first image among at least one estimated first region of interest (e.g., P (Q+R) face regions), and in the above 907 operation, the at least one first region of interest may be updated by adding the new first region of interest (e.g., R face regions).

[0222] Referring to FIG. 9b above, in operation 921, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can input an image (e.g., a second image).

[0223] According to one embodiment, the electronic device can input an image (e.g., a second image) through a camera module (e.g., a camera module (280) of FIG. 2).

[0224] In operation 923, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can detect at least one first region of interest (e.g., M face regions) in an image (e.g., a second image) through a first detection module (e.g., a face detector).

[0225] According to one embodiment, at least one first region of interest (e.g., M face regions) can be detected in the image (e.g., a second image) using a first detection module (e.g., a face detector) for detecting a first region of interest among a plurality of detection modules.

[0226] In operation 925, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can determine whether there is a partial or complete match between at least one first region of interest detected in an image (e.g., a second image) and at least one first region of interest detected in a previous image (e.g., a first image).

[0227] According to one embodiment, the electronic device can determine whether there is a partial or complete match between at least one first region of interest (e.g., M face regions) detected in an image (e.g., a second image) and at least one first region of interest (e.g., N face regions) detected in a previous image (e.g., a first image) by using a region matching function based on <Equation 1> to <Equation 2>.

[0228] In the above 925 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can update at least one first region of interest (e.g., N face regions) detected in the previous image (e.g., the first image) and at least one first region of interest (e.g., M face regions) detected in the image (e.g., the second image) in the 927 operation to at least one first region of interest (e.g., M (K+L) face regions) detected in the image (e.g., the second image) when both match (e.g., all are faces of the same subject).

[0229] In the above 925 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) may update at least one first region of interest (e.g., K face regions) among at least one first region of interest (e.g., N (K + J) face regions) detected in the previous image (e.g., the first image) and some first region of interest (e.g., K face regions) among at least one first region of interest (e.g., M (K + L) face regions) detected in the image (e.g., the second image) to some first region of interest (e.g., K face regions) among at least one first region of interest (e.g., some faces of the same subject) in the above 927 operation.

[0230] In the above 925 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) determines as a new first region of interest (e.g., L face regions) that does not match the at least one first region of interest (e.g., N (K + J) face regions) detected in the previous image (e.g., the first image) among at least one first region of interest (e.g., M (K + L) face regions) detected in the image (e.g., the second image), and in the above 927 operation, the at least one first region of interest may be updated by adding the new first region of interest (e.g., L face regions).

[0231] In the above 925 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can perform the 929 operation if it confirms the existence of some first regions of interest (e.g., J face regions) that were not detected using the first detection module among at least one first region of interest (e.g., N (K + J) face regions) detected in the previous image (e.g., the first image).

[0232] In the above 929 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can detect at least one third region of interest (e.g., T face feature regions) in an image (e.g., a second image) using a third detection module (e.g., a face attribute detector).

[0233] According to one embodiment, the electronic device can detect at least one third region of interest (e.g., T face feature regions) in an image (e.g., a second image) using a third detection module (e.g., a face attribute detector) for detecting a third region of interest among a plurality of detection modules.

[0234] In operation 931, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can estimate at least one first region of interest (e.g., T face regions) based on at least one third region of interest (e.g., T face feature regions).

[0235] According to one embodiment, the electronic device can estimate the at least one first region of interest (e.g., T face regions) in the at least one third region of interest (e.g., T face feature regions) using an estimation function based on FIGS. 4a and 4b.

[0236] In operation 933, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can determine whether there is a partial or complete match between at least one estimated first region of interest (e.g., T face regions) and some first regions of interest (e.g., N (K + J) face regions) detected in a previous image (e.g., the first image).

[0237] According to one embodiment, the electronic device can determine whether there is a partial or complete match between the estimated at least one first region of interest (e.g., T face regions) and some first regions of interest (e.g., J face regions) detected in a previous image (e.g., the first image) by using a region matching function based on <Equation 1> or <Equation 2>.

[0238] In the above 933 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) may update at least one first region of interest to the estimated at least one first region of interest (e.g., T face regions) in the above 927 operation if at least some first regions of interest (e.g., J face regions) detected in the previous image (e.g., the first image) and at least one estimated first region of interest (e.g., T face regions) all match.

[0239] In the above 933 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) may, in the above 927 operation, update at least one first region of interest to the part of the estimated at least one first region of interest (e.g., U face regions) if the part of the first region of interest (e.g., J face regions) detected in the previous image (e.g., the first image) matches the part of the first region of interest (e.g., U face regions) among the estimated at least one first region of interest (e.g., T face regions (U face regions)).

[0240] In the above 933 operation, the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) determines as a new first region of interest (e.g., V face regions) that does not match some first regions of interest (e.g., J face regions) detected in the first image among at least one estimated first region of interest (e.g., T (U + V) face regions), and in the above 927 operation, the at least one first region of interest may be updated by adding the new first region of interest (e.g., V face regions).

[0242] FIG. 10 is a flowchart (1000) for explaining a region of interest detection operation during image capture of an electronic device according to various embodiments. The region of interest detection operation may include operations 1001 through 1009 and may be understood to be performed by a processor (e.g., processor (120) of FIG. 1 or processor (220) of FIG. 2) of an electronic device (e.g., electronic device (101) of FIG. 1 or electronic device (201) of FIG. 2). According to one embodiment, at least one of operations 1001 through 1009 may be omitted, the order of some operations may be changed, or other operations may be added.

[0243] In operation 1001, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can input an image (e.g., a second image).

[0244] According to one embodiment, the electronic device can input an image (e.g., a second image) through a camera module (e.g., a camera module (280) of FIG. 2).

[0245] In 1003a, 1003b, and 1003c, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can simultaneously detect a first region of interest in an image (e.g., a second image) using a plurality of detection modules.

[0246] In the above 1003a, the electronic device can detect at least one first region of interest (e.g., M face regions) in an image (e.g., a second image) through a first detection module (e.g., a face detector) for detecting a first region of interest.

[0247] In the above 1003b, the electronic device detects at least one second region of interest (e.g., P body regions) in an image (e.g., a second image) through a second detection module (e.g., a body detector) for detecting a second region of interest, and in 1005b, can detect at least one first region of interest (e.g., P body regions) in the at least one second region of interest (e.g., P body regions) using an estimation function.

[0248] In the above 1003c, the electronic device detects at least one third region of interest (e.g., T face feature regions) in an image (e.g., a second image) through a third detection module (e.g., a face attribute detector) for detecting a third region of interest, and in the above 1005c, can detect at least one first region of interest (e.g., P body regions) in at least one third region of interest (e.g., T face feature regions) using an estimation function.

[0249] In operation 1007, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) can determine whether there is a partial or complete match between at least one first region of interest (e.g., N face regions) detected in a previous image (e.g., the first image) and at least one first region of interest (e.g., M face regions, P face regions and T face regions) detected using a plurality of detection modules.

[0250] According to one embodiment, the electronic device can determine whether there is a partial or complete match between at least one first region of interest (e.g., N face regions) detected in a previous image (e.g., a first image) and at least one first region of interest (e.g., M face regions) detected using a first detection module by using a region matching function.

[0251] According to one embodiment, the electronic device can determine whether there is a partial or complete match between at least one first region of interest (e.g., N face regions) detected in a previous image (e.g., a first image) and at least one first region of interest (e.g., P face regions) estimated using a second detection module by using a region matching function.

[0252] According to one embodiment, the electronic device can determine whether there is a partial or complete match between at least one first region of interest (e.g., N face regions) detected in a previous image (e.g., a first image) and at least one first region of interest (e.g., T face regions) estimated using a third detection module by using a region matching function.

[0253] In the above operation 1007, if the electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2) determines at least one first region of interest (e.g., M face regions, P face regions and T face regions) that matches at least one first region of interest (e.g., N face regions) detected in a previous image (e.g., the first image) among at least one first region of interest (e.g., M face regions, P face regions and T face regions) detected using a plurality of detection modules, in the above operation 1009, the electronic device can update at least one first region of interest (e.g., M face regions, P face regions and T face regions) determined by the above-mentioned at least one first region of interest (e.g., M face regions, P face regions and T face regions) as at least one first region of interest.

[0255] According to various embodiments, a method for capturing an image in an electronic device comprises: detecting at least one first region of interest in a first image input through a camera module using a first detection module for detecting a first region of interest among a plurality of detection modules; if at least one first region of interest matching the at least one first region of interest detected in the first image is not detected in the second image input through the camera using the first detection module, detecting at least one second region of interest in the second image using a second detection module for detecting a second region of interest among the plurality of detection modules; estimating at least one first region of interest based on the at least one second region of interest; if the at least one first region of interest detected in the first image matches the at least one first region of interest estimated, updating the at least one first region of interest estimated to at least one first region of interest; and matching the at least one first region of interest detected in the first image. It includes an operation to change the location of a preview area including the updated at least one first region of interest based on the location of the estimated at least one first region of interest.

[0256] According to various embodiments, when the at least one first region of interest is detected in the second image input through the camera module using the first detection module, the method may further include an operation of determining whether the at least one first region of interest detected in the first image matches the at least one first region of interest detected in the second image, and an operation of updating the at least one first region of interest detected in the second image to at least one first region of interest if the at least one first region of interest detected in the first image matches the at least one first region of interest detected in the second image.

[0257] According to various embodiments, if the at least one first region of interest detected in the first image and the at least one first region of interest detected in the second image do not match, the operation of updating the at least one first region of interest by adding the at least one first region of interest detected in the second image as a new first region of interest may be further included.

[0258] According to various embodiments, if at least one first region of interest detected in the first image and at least one first region of interest estimated based on the second region of interest do not match, the operation of updating at least one first region of interest by adding the estimated at least one first region of interest as a new first region of interest may be further included.

[0259] According to various embodiments, if at least one first region of interest detected in the first image and at least one first region of interest estimated based on the second region of interest do not match, the method may further include: detecting at least one third region of interest in the second image using a third detection module for detecting a third region of interest among the plurality of detection modules, and estimating at least one second region of interest based on the at least one third region of interest; estimating at least one first region of interest based on the estimated at least one second region of interest; and if at least one first region of interest detected in the first image and at least one first region of interest estimated match, updating the estimated at least one first region of interest to at least one first region of interest.

[0260] According to various embodiments, the operation of estimating the first region of interest may include the operation of calculating the size of the first region of interest based on the size of the second region of interest and a certain ratio, the operation of calculating the location of the first region of interest based on the size of the second region of interest and a certain ratio, and the operation of estimating the first region of interest from the second region of interest based on the calculated size and location of the first region of interest.

[0261] According to various embodiments, when the setting of the first region of interest among a plurality of regions of interest is confirmed, the operation of determining the first detection module for detecting the first region of interest among the plurality of detection modules as the main detection module may be further included.

[0262] According to various embodiments, the method may further include an operation of determining at least one detection module, excluding the first detection module determined as the main detection module among the plurality of detection modules, as an auxiliary detection module, and an operation of confirming the application order of the at least one detection module determined as the auxiliary detection module for detecting the first region of interest based on the order set when determining the first detection module as the main detection module.

[0263] According to various embodiments, when all of the plurality of detection modules are determined as main detection modules, the operation of detecting at least one first region of interest that matches at least one first region of interest detected in the first image in the second image input through the camera by simultaneously using the plurality of detection modules may be further included.

[0264] According to various embodiments, if the at least one first region of interest detected in the first image matches the at least one first region of interest detected using the plurality of detection modules, the operation of updating the at least one first region of interest detected using the plurality of detection modules to the at least one first region of interest may be further included.

[0266] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0267] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., by wire), wirelessly, or through a third component.

[0268] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0269] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0270] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0271] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

Claim 1 In an electronic device, the camera module: display; multiple detection modules; at least one processor; The electronic device includes a memory for storing instructions, wherein when the instructions are executed individually or collectively by the at least one processor, the electronic device determines a first detection module among the plurality of detection modules as the main detection module, determines at least one detection module among the plurality of detection modules excluding the first detection module as the auxiliary detection module, checks the order set in the plurality of detection modules including the first detection module and the at least one detection module, detects at least one first region of interest in a first image input through the camera module using a first detection module for detecting a first region of interest among the plurality of detection modules, and if at least one first region of interest matching the at least one first region of interest detected in the first image is not detected in a second image input through the camera module using the first detection module, detects at least one second region of interest in the second image using a second detection module for detecting a second region of interest among the at least one detection module determined as the auxiliary detection module, estimates at least one first region of interest based on the at least one second region of interest, and at least An electronic device configured to update the estimated at least one first region of interest to at least one first region of interest when one first region of interest matches the estimated at least one first region of interest, and to change the location of a preview region including the updated at least one first region of interest based on the location of the estimated at least one first region of interest that matches the at least one first region of interest detected in the first image. Claim 2 An electronic device configured such that, when the instructions are executed individually or collectively by the at least one processor, the electronic device detects the at least one first region of interest in the second image input through the camera module using the first detection module, determines whether the at least one first region of interest detected in the first image matches the at least one first region of interest detected in the second image, and if the at least one first region of interest detected in the first image matches the at least one first region of interest detected in the second image, updates the at least one first region of interest detected in the second image to the at least one first region of interest. Claim 3 An electronic device configured such that, when the instructions are executed individually or collectively by the at least one processor, the at least one first region of interest detected in the first image does not match the at least one first region of interest detected in the second image, and the at least one first region of interest detected in the second image is added as a new first region of interest to update the at least one first region of interest. Claim 4 An electronic device configured such that, when the instructions are executed individually or collectively by the at least one processor, the electronic device updates the at least one first region of interest by adding the estimated at least one first region of interest as a new first region of interest if the at least one first region of interest detected in the first image and the at least one first region of interest estimated based on the second region of interest do not match. Claim 5 An electronic device configured such that, when the instructions are executed individually or collectively by the at least one processor, the electronic device detects at least one first region of interest detected in the first image and at least one first region of interest estimated based on the second region of interest, using a third detection module for detecting a third region of interest among the at least one detection modules determined by the auxiliary detection module, detects at least one third region of interest in the second image, estimates at least one second region of interest based on the at least one third region of interest, estimates at least one first region of interest based on the estimated at least one second region of interest, and, if at least one first region of interest detected in the first image and at least one estimated first region of interest match, updates the estimated at least one first region of interest to at least one first region of interest. Claim 6 An electronic device configured such that, when the instructions are executed individually or collectively by the at least one processor, the electronic device calculates the size of the first region of interest based on the size of the second region of interest and a certain ratio, calculates the location of the first region of interest based on the size of the second region of interest and a certain ratio, and estimates the first region of interest in the second region of interest based on the calculated size and location of the first region of interest. Claim 7 delete Claim 8 delete Claim 9 An electronic device configured such that, when the instructions are executed individually or collectively by the at least one processor, the electronic device detects at least one first region of interest corresponding to at least one first region of interest detected in the first image by simultaneously using the plurality of detection modules when the plurality of detection modules are determined as the main detection modules. Claim 10 An electronic device configured such that, when the instructions are executed individually or collectively by the at least one processor, the at least one first region of interest detected by the at least one detection module matches the at least one first region of interest detected by the at least one detection module, thereby updating the at least one first region of interest detected by the at least one detection module to the at least one first region of interest. Claim 11 A method for capturing images of an electronic device comprises: an operation of determining a first detection module among a plurality of detection modules as a main detection module; an operation of determining at least one detection module among the plurality of detection modules, excluding the first detection module, as an auxiliary detection module; an operation of confirming the order set in the plurality of detection modules including the first detection module and the at least one detection module; an operation of detecting at least one first region of interest in a first image input through a camera module using a first detection module among the plurality of detection modules for detecting a first region of interest; if at least one first region of interest matching the at least one first region of interest detected in the first image is not detected in the second image input through the camera module using the first detection module, an operation of detecting at least one second region of interest in the second image using a second detection module among the at least one detection module determined as the auxiliary detection module for detecting a second region of interest; and an operation of estimating at least one first region of interest based on the at least one second region of interest. If at least one first region of interest detected in the first image matches the estimated at least one first region of interest, the operation of updating the estimated at least one first region of interest to at least one first region of interest; and the at least one first region of interest that matches the first region of interest detected in the first image A method comprising changing the location of a preview region including the updated at least one first region of interest based on the location of the estimated at least one first region of interest. Claim 12 A method according to claim 11, further comprising: an operation of determining whether the at least one first region of interest detected in the first image and the at least one first region of interest detected in the second image match when the at least one first region of interest detected in the first image is detected in the second image when the at least one first region of interest detected in the first image and the at least one first region of interest detected in the second image match; and an operation of updating the at least one first region of interest detected in the second image to at least one first region of interest when the at least one first region of interest detected in the first image and the at least one first region of interest detected in the second image match. Claim 13 A method according to claim 12, further comprising the operation of updating at least one first region of interest by adding the at least one first region of interest detected in the second image as a new first region of interest if the at least one first region of interest detected in the first image does not match the at least one first region of interest detected in the second image. Claim 14 A method according to claim 11, further comprising the operation of updating at least one first region of interest by adding the estimated at least one first region of interest as a new first region of interest if at least one first region of interest detected in the first image and at least one first region of interest estimated based on the second region of interest do not match. Claim 15 A method according to claim 11, further comprising: an operation in which, if at least one first region of interest detected in the first image and at least one first region of interest estimated based on the second region of interest do not match, at least one third region of interest is detected in the second image using a third detection module for detecting a third region of interest among the at least one detection module determined by the auxiliary detection module, and at least one second region of interest is estimated based on the at least one third region of interest; an operation in which at least one first region of interest is estimated based on the estimated at least one second region of interest; and, if at least one first region of interest detected in the first image and at least one first region of interest estimated match, at least one first region of interest is updated to at least one first region of interest. Claim 16 A method according to claim 11, wherein the operation of estimating the first region of interest comprises: an operation of calculating the size of the first region of interest based on the size of the second region of interest and a certain ratio; an operation of calculating the location of the first region of interest based on the size of the second region of interest and a certain ratio; and an operation of estimating the first region of interest from the second region of interest based on the calculated size and location of the first region of interest. Claim 17 delete Claim 18 delete Claim 19 A method according to claim 11, wherein when all of the plurality of detection modules are determined as main detection modules, the method further includes the operation of detecting at least one first region of interest that matches at least one first region of interest detected in the first image in a second image input through the camera module by simultaneously using the plurality of detection modules. Claim 20 A method according to claim 19, further comprising the operation of updating the at least one first region of interest detected using the plurality of detection modules to the at least one first region of interest if the at least one first region of interest detected using the plurality of detection modules matches the at least one first region of interest detected using the plurality of detection modules.

Images