Method for controlling angle of view in dis mode operation

Abstract

An apparatus and a method for image processing are disclosed. The present specification combines an optical zoom operation in a digital image stabilization (DIS) processing process and a digital zoom-in operation in a DIS mode to solve a problem of deterioration of image resolution and inconsistency of angles of view. The present specification comprises an image sensor for outputting an image captured through a lens, and when a DIS mode is initiated when an optical zoom-in magnification is equal to or greater than a predetermined magnification, performs control to match angles of view before and after the DIS mode operation by performing optical zoom-out and performing digital zoom-in according to the DIS mode. Accordingly, a heterogeneous phenomenon before and after the DIS mode operation can be improved. In the present specification, one or more among a surveillance camera, an autonomous vehicle, a user terminal, and a server may be linked to an artificial intelligence module, a robot, an augmented reality (AR) device, a virtual reality (VR) device, a device related to 5G services, and the like.

Description

How to control the angle of view when DIS mode is in operation

[0001] This specification relates to digital image stabilization, and more particularly to a method for improving field of view mismatch caused by digital zoom in the DIS mode of a camera.

[0002] When shooting camera video, shake reduction can be broadly divided into optical (OIS: Optical Image Stabilization) and electronic (EIS: Electronic Image Stabilization). OIS is a technology that reduces vibration or shaking using hardware, and compensates for shaking by shifting camera components such as the lens or sensor. EIS crops a portion of an image or video to obtain an image smaller than the original, and uses the cropped edges as a margin for correction. Even if there is shaking in the video frame, the cropped screen provided to the user is not affected, and the image is processed through software.

[0003] While EIS and DIS share the same image processing method, they differ in how they detect shaking. EIS utilizes a gyro sensor to detect shaking, while DIS can detect shaking or shaking based on changes in the position of the subject within the image.

[0004] DIS is an image stabilization function. It performs digital zoom beyond a certain magnification to process the edges of the image, and compensates for the shaking by shifting the area of ​​the original image that is not visible due to cropping. During this process, there is a problem that the image angle before and after the camera operates in DIS mode is different, and because of this, when the images before and after DIS mode are synthesized, the image remains heterogeneous.

[0005] In order to solve the above-mentioned problem, the present specification aims to provide an image processing method and device that can improve the heterogeneous phenomenon before and after DIS mode operation by supplementing the optical zoom operation with the phenomenon of angle mismatch due to digital zoom operation according to DIS mode operation.

[0006] In addition, according to one embodiment of the present specification, it is an object to provide an image processing method and device that can more efficiently implement a DIS function without deteriorating image quality (resolution) due to a digital zoom operation according to a DIS mode operation.

[0007] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the detailed description of the invention below.

[0008] An image processing device according to one embodiment of the present specification includes an image sensor that outputs an image captured through a lens; and a processor that performs an optical zoom out when an operation signal of a DIS (Digital Image Stabilization) mode is detected when an optical zoom in magnification of the lens is greater than a predetermined magnification, and performs a digital zoom in according to the DIS mode so that the field of view of the image before and after the operation of the DIS mode is matched.

[0009] The above processor can control the digital zoom-in operation to be performed simultaneously with the optical zoom-out operation.

[0010] Accordingly, the optical zoom and digital zoom operate simultaneously, making the optical zoom operation invisible to the user.

[0011] The above processor can perform shake correction on an image acquired in the optical zoom out state, and perform digital zoom in on an image on which shake correction has been performed.

[0012] The digital zoom-in ratio related to the above digital zoom-in can be dynamically determined based on sensor data of the gyro sensor.

[0013] The optical zoom in magnification has a value greater than the digital zoom in magnification.

[0014] The processor can control the digitally zoomed-in image in the DIS mode to have a size according to the predetermined optical zoom-in magnification so as to match the angle of view of the image.

[0015] The above DIS mode can be set to On or Off through user input.

[0016] The image processing device includes a surveillance camera, and when a plurality of presets are set so that different optical zoom-in magnifications are applied to a plurality of sites photographed through the surveillance camera, the processor, when performing an operation according to the DIS mode on an image acquired from at least one of the plurality of sites, can perform the digital zoom-in so that the size is according to the optical zoom-in magnification defined for each site.

[0017] An image processing method according to one embodiment of the present specification may include a step of detecting an operation signal of a DIS (Digital Image Stabilization) mode during image capture; a step of performing an optical zoom out when an optical zoom in magnification of a lens is greater than a predetermined magnification; and a step of matching a field of view of the image before and after the operation of the DIS mode as a digital zoom in according to the DIS mode is performed.

[0018] The digital zoom-in operation can be performed simultaneously with the optical zoom-out operation.

[0019] The step of matching the field of view of the image may include a step of performing shake correction on an image acquired in the optical zoom-out state; and a step of performing the digital zoom-in on an image on which shake correction has been performed.

[0020] According to another embodiment of the present specification, an image processing device includes an image sensor that outputs an image captured through a lens; and a processor that, when detecting an operation signal of a DIS (Digital Image Stabilization) mode, performs an optical zoom out when the position of a zoom lens is in an optical zoom in state greater than a predetermined optical zoom magnification, and processes the image so that the field of view of the image is maintained the same as the field of view before the DIS operation through a digital zoom in in the optical zoom out state.

[0021] The processor is controlled to perform the digital zoom-in in the DIS mode based on a digital zoom ratio determined based on gyro sensor data, wherein the digital zoom ratio can be dynamically varied according to the shaking of the image processing device.

[0022] The above processor can be controlled to perform the digital zoom in simultaneously with performing the optical zoom out.

[0023] According to one embodiment of the present specification, the phenomenon of field of view mismatch due to digital zoom operation according to DIS mode operation can be improved by supplementing the optical zoom operation, thereby improving the heterogeneous phenomenon before and after DIS mode operation.

[0024] Meanwhile, according to one embodiment of the present specification, the DIS function can be implemented more efficiently without degradation of image quality (resolution) due to digital zoom operation according to DIS mode operation.

[0025] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the description below.

[0026] The accompanying drawings, which are incorporated in and constitute a part of the detailed description to aid in understanding the present specification, provide embodiments of the present specification and, together with the detailed description, also describe the technical features of the present specification.

[0027] FIG. 1 is a drawing for explaining a surveillance camera system for implementing an image processing method of a surveillance camera according to one embodiment of the present specification.

[0028] FIG. 2 is a schematic block diagram of a surveillance camera according to one embodiment of the present specification.

[0029] Figure 3 is a flowchart of an image processing method according to one embodiment of the present specification.

[0030] FIG. 4 is a drawing for explaining an example of an image processing method according to one embodiment of the present specification.

[0031] FIGS. 5 and 6 are drawings for explaining an example of an image processing method when there are multiple sites for which presets are set according to one embodiment of the present specification.

[0032] The accompanying drawings, which are incorporated in and constitute a part of the detailed description to aid in the understanding of the present invention, provide embodiments of the present invention and, together with the detailed description, explain the technical features of the present invention.

[0033] Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0034] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0035] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0036] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0037] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0038] FIG. 1 is a drawing for explaining a surveillance camera system for implementing an image processing method of a surveillance camera according to one embodiment of the present specification.

[0039] Referring to FIG. 1, a surveillance camera system (10) according to one embodiment of the present specification may include a photographing device (100) and an image management server (200). The photographing device (100) may be an electronic photographing device placed at a fixed location in a specific location, an electronic photographing device that can be automatically or manually moved along a predetermined path, or an electronic photographing device that can be moved by a person or a robot. The photographing device (100) may be an IP camera that is connected to a wired or wireless Internet. The photographing device (100) may be a PTZ camera having pan, tilt, and zoom functions. The photographing device (100) may have a function of recording or taking pictures of a monitored area. The photographing device (100) may have a function of recording sounds generated in the monitored area. The photographing device (100) may have a function of generating a notification or performing recording or taking pictures when a change, such as movement or sound, occurs in the monitored area.

[0040] The image management server (200) may be a device that receives and stores the image captured by the photographing device (100) and / or the image obtained by editing the image. The image management server (200) may analyze the received image to correspond to the intended use. For example, the image management server (200) may detect an object using an object detection algorithm to detect an object in the image. The object detection algorithm may be an AI-based algorithm, and may detect an object by applying a pre-trained artificial neural network model.

[0041] Meanwhile, the image management server (200) may store various learning models suitable for the purpose of image analysis. In addition to the learning model for object detection described above, it may also store a model capable of obtaining the movement speed of the detected object. Here, the learned models may include a learning model that outputs a shutter speed corresponding to the movement speed of the object and a maximum sensor gain value when entering a low-speed shutter. In addition, the learned models may include a learning model that outputs a noise removal strength adjustment value corresponding to the movement speed of the object.

[0042] Additionally, the video management server (200) can analyze the received video to generate metadata and index information for the metadata. The video management server (200) can analyze video information and / or audio information included in the received video together or separately to generate metadata and index information for the metadata.

[0043] The image management system (10) may further include an external device (300) capable of performing wired or wireless communication with the photographing device (100) and / or the image management server (200).

[0044] The external device (300) can transmit an information provision request signal requesting provision of all or part of an image to the image management server (200). The external device (300) can transmit an information provision request signal requesting the image management server (200) for information on the presence or absence of an object based on image analysis results, the object's movement speed, a shutter speed adjustment value based on the object's movement speed, a noise removal value based on the object's movement speed, etc. In addition, the external device (300) can transmit an information provision request signal requesting metadata obtained by analyzing an image and / or index information for the metadata to the image management server (200).

[0045] The image management system (10) may further include a communication network (400) which is a wired or wireless communication path between the photographing device (100), the image management server (200), and / or the external device (300). The communication network (400) may include wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), ISDNs (Integrated Service Digital Networks), or wireless networks such as wireless LANs, CDMA, Bluetooth, and satellite communication, but the scope of the present specification is not limited thereto.

[0046] FIG. 2 is a schematic block diagram of a surveillance camera according to one embodiment of the present specification.

[0047] FIG. 2 is a block diagram showing the configuration of the camera illustrated in FIG. 1. Referring to FIG. 2, the camera (100) is described as a network camera that performs an intelligent image analysis function to generate the image analysis signal, but the operation of the network surveillance camera system according to an embodiment of the present invention is not necessarily limited thereto.

[0048] The camera (100) includes an image sensor (110), an encoder (120), a memory (130), a communication unit (140), an AI processor (150), and a processor (160).

[0049] The image sensor (110) performs the function of capturing an image by photographing a surveillance area, and can be implemented as, for example, a CCD (Charge-Coupled Device) sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, etc.

[0050] The encoder (120) performs an operation of encoding an image acquired through an image sensor (110) into a digital signal, which may follow, for example, H.264, H.265, MPEG (Moving Picture Experts Group), M-JPEG (Motion Joint Photographic Experts Group) standards, etc.

[0051] The memory (130) can store video data, audio data, still images, metadata, etc. As mentioned above, the metadata may be data including object detection information (movement, sound, intrusion into a designated area, etc.) captured in the surveillance area, object identification information (person, car, face, hat, clothing, etc.), and detected location information (coordinates, size, etc.).

[0052] In addition, the still image is generated together with the metadata and stored in the memory (130), and can be generated by capturing image information for a specific analysis area among the image analysis information. For example, the still image can be implemented as a JPEG image file.

[0053] For example, the still image may be generated by cropping a specific area of ​​image data determined to be an identifiable object among image data of the surveillance area detected in a specific area and for a specific period of time, and this may be transmitted in real time together with the metadata.

[0054] The communication unit (140) transmits the video data, audio data, still images, and / or metadata to the video receiving / searching device (300). According to one embodiment, the communication unit (140) can transmit the video data, audio data, still images, and / or metadata to the video receiving device (300) in real time. The communication unit (140) can perform at least one communication function among wired / wireless Local Area Network (LAN), Wi-Fi, ZigBee, Bluetooth, and Near Field Communication.

[0055] The AI ​​processor (150) is for artificial intelligence image processing, and can apply a deep learning-based object detection algorithm learned from images acquired through a surveillance camera system. The AI ​​processor (150) may be implemented as one module with the processor (260) that controls the entire system, or as an independent module. As described above, the photographing device (100) disclosed in one embodiment of the present specification can learn various learning models necessary for image processing, such as object detection, shutter speed, sensor gain value control, and noise removal emphasis adjustment, and apply them to image processing, and can implement artificial intelligence image processing through the operation of the AI ​​processor (150) and / or the processor (160).

[0056] The AI ​​processor (150) may include a learning data acquisition unit (151) and a model learning unit (152).

[0057] The AI ​​processor (150) may be mounted on an electronic device including an AI module capable of performing AI processing, a server including an AI module, or the like, or may be implemented as an independent device. AI processing may include all operations related to the control unit of a surveillance camera or video management server. For example, a surveillance camera or video management server may perform AI processing of an acquired video signal to perform processing / judgment and control signal generation operations.

[0058] The AI ​​processor (150) can be installed and operated in a client device that directly utilizes the AI ​​processing results, or in a device in a cloud environment that provides the AI ​​processing results to other devices. The AI ​​processor (150) can be included in a computing device capable of learning a neural network, and can be implemented in various electronic devices such as a server, desktop PC, laptop PC, tablet PC, etc.

[0059] The AI ​​processor (150) can learn a neural network for recognizing data related to surveillance cameras. Here, the neural network for recognizing data related to surveillance cameras can be designed to simulate the structure of the human brain on a computer and can include a plurality of network nodes having weights that simulate neurons of a human neural network. The plurality of network modes can exchange data according to their respective connection relationships so as to simulate the synaptic activity of neurons that exchange signals through synapses. Here, the neural network can include a deep learning model developed from a neural network model. In the deep learning model, the plurality of network nodes can be located in different layers and exchange data according to convolutional connection relationships. Examples of neural network models include various deep learning techniques such as deep neural networks (DNNs), convolutional deep neural networks (CNNs), recurrent Boltzmann machines (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), and deep Q-networks, which can be applied to fields such as computer vision, speech recognition, natural language processing, and speech / signal processing.

[0060] Meanwhile, the processor performing the functions described above may be a general-purpose processor (e.g., CPU), but may also be an AI-specific processor for artificial intelligence learning (e.g., GPU).

[0061] The memory (130) can store various programs and data required for the operation of the AI ​​processor (150). The memory (130) can be implemented as a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SDD), etc. The memory (130) is accessed by the AI ​​processor (150), and data can be read / written / modified / deleted / updated, etc. by the AI ​​processor (150). In addition, the memory (130) can store a neural network model (e.g., a deep learning model) generated through a learning algorithm for image data classification / recognition.

[0062] Meanwhile, the AI ​​processor (150) may include a data learning unit that learns a neural network for data classification / recognition. The data learning unit may learn criteria regarding which learning data to use to determine data classification / recognition and how to classify and recognize data using the learning data. The data learning unit may acquire learning data to be used for learning, and learn a deep learning model by applying the acquired learning data to the deep learning model. The data learning unit may be manufactured in the form of at least one hardware chip and mounted on the image processing device (100). For example, the data learning unit may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of a general-purpose processor (CPU) or a graphics processor (GPU) and mounted on the image processing device (100). In addition, the data learning unit may be implemented as a software module. When implemented as a software module (or a program module including instructions), the software module may be stored on a non-transitory computer-readable recording medium that can be read by a computer. In this case, at least one software module may be provided by an operating system (OS) or an application.

[0063] The data learning unit may include a learning data acquisition unit (151) and a model learning unit (152).

[0064] The learning data acquisition unit (151) can acquire learning data required for a neural network model for classifying and recognizing data. The model learning unit (152) can use the acquired learning data to train the neural network model to have judgment criteria regarding how to classify predetermined data. At this time, the model learning unit (152) can train the neural network model through supervised learning that uses at least some of the learning data as judgment criteria. Alternatively, the model learning unit (152) can train the neural network model through unsupervised learning that discovers judgment criteria by learning on its own using the learning data without guidance. In addition, the model learning unit (152) can train the neural network model through reinforcement learning using feedback on whether the results of situational judgment according to learning are correct. In addition, the model learning unit (152) can train the neural network model using a learning algorithm including error back-propagation or gradient descent.

[0065] When the neural network model is learned, the model learning unit (152) can store the learned neural network model in the memory (130). The model learning unit (152) can also store the learned neural network model in the memory of a server connected to the image processing device (100) via a wired or wireless network.

[0066] The data learning unit may further include a learning data preprocessing unit (not shown) and a learning data selection unit (not shown) to improve the analysis results of the recognition model or to save resources or time required for generating the recognition model.

[0067] The learning data preprocessing unit can preprocess the acquired data so that it can be used for learning to determine situations. For example, the learning data preprocessing unit can process the acquired data into a preset format so that the model learning unit (152) can utilize the acquired learning data for learning image recognition.

[0068] In addition, the learning data selection unit can select data required for learning from among the learning data acquired from the learning data acquisition unit (151) or the learning data preprocessed from the preprocessing unit. The selected learning data can be provided to the model learning unit (152).

[0069] Additionally, the data learning unit may further include a model evaluation unit (not shown) to improve the analysis results of the neural network model.

[0070] The model evaluation unit inputs evaluation data into the neural network model, and if the analysis results output from the evaluation data do not satisfy a predetermined standard, it can cause the model learning unit (152) to relearn. In this case, the evaluation data may be predefined data for evaluating the recognition model. For example, the model evaluation unit can evaluate that the predefined standard is not satisfied if the number or ratio of evaluation data with inaccurate analysis results among the analysis results of the learned recognition model for the evaluation data exceeds a preset threshold.

[0071] This specification may be linked to one or more of a surveillance camera, an autonomous vehicle, a user terminal, and a server, including an artificial intelligence module, a robot, an augmented reality (AR) device, a virtual reality (VT) device, and a device related to a 5G service.

[0072] FIG. 3 is a flowchart of an image processing method according to one embodiment of the present specification. The image processing method illustrated in FIG. 3 may be implemented through a surveillance camera system, a surveillance camera device, a processor or a control unit included in the surveillance camera device described with reference to FIGS. 1 and 2. For convenience of explanation, the image processing method is described on the premise that various functions can be controlled through the processor (160) of the surveillance camera (100) illustrated in FIG. 2; however, it should be noted that the present specification is not limited thereto. Meanwhile, an image processing device (100) according to one embodiment of the present specification may take a surveillance camera as an example, and the image processing device (100) may of course include an external device (server) in addition to the surveillance camera.

[0073] Referring to FIG. 3, the processor (160) detects a DIS mode operation signal of the image processing device (100) (S300).

[0074] DIS mode performs digital image stabilization. Image stabilization technology corely tracks and compensates for pixel movement within the captured image frame. Pixel movement represents image shake, and tracking and compensating for this allows for more stable images. During this process, software tracks individual pixels in the image and moves them to the correct positions. Meanwhile, digital image stabilization (DIS) inherently involves digital zooming in to secure the edge area of ​​the image. In other words, DIS performs shake correction based on the second angle image zoomed in from the first angle of the original image. Accordingly, if the camera suddenly detects shake, DIS mode is activated at the time of detection, which may result in differences between the images before (first angle) and after (second angle) DIS mode. Furthermore, if it is necessary to synthesize images before and after DIS mode operation, or even outside of the image synthesis process, the discrepancy in angles may cause a sense of incongruity when providing images before and after DIS mode to the user.

[0075] Therefore, in order to resolve the above-mentioned sense of incongruity, the present specification controls the angle of view itself when starting the digital zoom in the DIS mode to generate an image having a larger angle of view than before the DIS mode operation (the first angle of view), and to acquire an image having the first angle of view due to the digital zoom performed in the DIS mode, thereby minimizing the problem of angle of view mismatch due to the DIS mode operation described above. As a method of acquiring an image having a larger angle of view than before the DIS mode operation (the first angle of view), an optical zoom out is used in one embodiment of the present specification. That is, when the DIS mode is started, it is necessary to acquire a zoomed-out image having a larger angle of view than the angle of view of the original image (the first angle of view) through the optical zoom out function. Here, in the case of the digital zoom out as a method of acquiring a zoomed-out image, the problem of image quality (resolution) deterioration occurs, so the present specification implements the optical zoom out through direct movement of the lens through the optical zoom out.

[0076] Therefore, when the DIS mode operation is initiated, in order to perform optical zoom out, zoom out cannot be performed if the currently acquired image is a full size image. Accordingly, the processor (160) first checks whether the zoom lens magnification of the image processing device (100) is set to a predetermined magnification or higher (S310).

[0077] That is, when the zoom lens is in the DIS mode and the image is optically zoomed in at a predetermined magnification, for example, a zoom magnification of x1.2 (S310:Y), the processor (160) performs optical zoom out (S320).

[0078] Accordingly, the image processing method according to one embodiment of the present specification can be applied to all images acquired in a state where the optical zoom magnification of the image processing device (100) is set to a size greater than the digital zoom magnification in the DIS mode applied to the present specification.

[0079] The processor (160) performs a digital zoom-in operation according to the DIS mode operation (S330). Even if the digital zoom-in in the DIS mode is performed while the angle of view is expanded due to the optical zoom-out described above and the angle of view is reduced again, the angle of view can be restored to the angle of view before the DIS mode operation. The processor (160) of the present specification controls the image angles before and after the DIS operation to be the same by adjusting the digital zoom magnification (S340).

[0080] Meanwhile, in the present specification, when the DIS mode starts, an optical zoom-out may be performed by changing the position of the lens, and then a digital zoom-in operation according to the DIS mode may be performed, but the optical zoom-out and digital zoom-in operations may be controlled to be performed simultaneously. More specifically, the processor (110) controls the digital zoom-in operation to start at the time the optical zoom-out starts, and to be completed at the time the optical zoom-out is completed, thereby preventing the user from substantially detecting a change in the image size according to the optical zoom-out operation.

[0081] Meanwhile, the processor (160) can perform shake correction in DIS mode on an image frame acquired while performing optical zoom out in S320. The processor (160) can perform digital zoom in on an image frame for which shake correction has been performed, thereby controlling the angle of view before and after DIS mode operation to be identical.

[0082] Additionally, the digital zoom ratio applied to digital zoom in in DIS mode can be calculated by referring to gyro sensor data.

[0083] Meanwhile, in the process of matching the image angle before and after the DIS mode operation, a method may be applied in which the processor (160) matches the image angle so that the DIS mode digitally zoomed-in image has a size according to a predetermined optical zoom-in magnification.

[0084] FIG. 4 is a drawing for explaining an example of an image processing method according to one embodiment of the present specification.

[0085] The processor (160) of the image processing device (100) can output an original image (400) through an image sensor from an image captured through a lens. The original image has an original field of view (FO). When the processor (160) detects a DIS mode operation signal, it acquires a first image (410) in a state where the magnification of the lens is optically zoomed out by a predetermined magnification or more instead of the original image (400) (Fig. 4 (a)). The first field of view (F1) of the first image (410) may have a value greater than the original field of view (FO) due to the optical zoom out for the original image. That is, when the DIS mode operation signal is detected, since the image acquired by the camera must be optically zoomed out, the original image must be an image acquired in a predetermined optical zoom-in state.

[0086] The processor (160) acquires a second image (420) by digitally zooming in on the first image (410) according to the DIS mode operation (FIG. 4(b)). The processor (160) can control the digital zoom-in so that the angle of view (F2) of the second image (420) matches the original angle of view (FO) of the original image before the DIS mode operation. For example, if the optical zoom-in magnification before the DIS mode operation is x2 and the digital zoom magnification according to the DIS mode operation is x1.2, the processor (160) performs the optical zoom-out of the image within a range greater than x1.2 and less than x2 when the DIS mode is started. In addition, the processor (160) can correct the image so that the angle of view matches the angle of view before the DIS mode operation by digitally zooming in on the optically zoomed-out image. As described above, the processor (160) can perform the digital zoom-in process simultaneously with the optical zoom-out when the DIS mode is started.

[0087] FIGS. 5 and 6 are drawings for explaining an example of an image processing method when there are multiple sites for which presets are set according to one embodiment of the present specification.

[0088] Referring to FIG. 5, a surveillance camera (100) according to one embodiment of the present specification can be operated with multiple preset settings. Here, the preset may mean that the camera's video capturing characteristics are set to meet certain conditions when a user wants to acquire video for a specific site. For example, the first site may be an area 10 m away from the surveillance camera (100), the second site may be an area 40 m away, and the third site may be an area 20 m away. The surveillance camera (100) may be a PTZ camera capable of monitoring a wide area through pan / tilt / zoom operations for the first to third sites. The greater the distance from the surveillance camera (100), the greater the optical zoom magnification of the camera before DIS mode operation can be set. Accordingly, the surveillance camera (100) can shoot at an optical zoom-in magnification of x20 when shooting the first site, change the optical zoom-in magnification to x40 when shooting the second site, and change the optical zoom-in magnification to x30 when shooting the third site.

[0089] Accordingly, according to one embodiment of the present specification, the optical zoom-in magnification before DIS mode operation may be different for each preset set according to the shooting conditions of a specific site, and this means that the digital zoom-in magnification to match the image angle before DIS operation in DIS mode also varies for each site.

[0090] Referring to FIG. 6, the processor (160) may receive preset settings for each of the first to Nth sites, reflecting the user's area of ​​interest, environment of interest, etc., for sites that can be captured by the surveillance camera (100). The preset settings may include the optical zoom magnification of the camera, but the present specification is not limited thereto.

[0091] When the processor (160) detects a DIS mode operation signal (S610), it can determine whether preset setting information exists. Preset setting information may not exist separately, in which case a global setting value may be applied. The global setting value may mean, for example, an optical zoom-in magnification of x2. When the optical zoom-in magnification according to the global setting value is x2, since it is greater than the digital zoom-in magnification x1.2 that operates in a normal DIS mode, an optical zoom-out operation according to an embodiment of the present specification is performed when entering the DIS mode, so that image processing is possible.

[0092] When preset setting information exists, the processor (160) determines that the currently acquired image is an image acquired from the first site and that a DIS operation signal for the first image corresponding to the first site has been detected (S621). When the DIS mode operation is started, the processor (160) performs an optical zoom out on the first image according to the first optical zoom magnification (S623). Then, the processor (160) performs a digital zoom in operation on the first image according to the first digital zoom magnification (S625). As described above, the optical zoom out and digital zoom in operations can be performed simultaneously. The processor (160) can control the angle of view before and after the DIS mode operation to be the same while adjusting the size of the first image to a size according to the optical zoom magnification set at the first site through the digital zoom in operation according to the DIS operation processing (S627).

[0093] When preset setting information exists, the processor (160) determines that the currently acquired image is an image acquired from the Nth site and that a DIS operation signal for the Nth image corresponding to the Nth site has been detected (S631). When the DIS mode operation is started, the processor (160) performs an optical zoom out operation according to the Nth optical zoom magnification for the Nth image (S633). Then, the processor (160) performs a digital zoom in operation according to the Nth digital zoom magnification for the Nth image (S635). The processor (160) can control the angle of view before and after the DIS mode operation to be the same while adjusting the size of the Nth image to a size according to the optical zoom magnification set at the Nth site through the digital zoom in operation according to the DIS operation processing (S637).

[0094] The present invention described above can be implemented as computer-readable code on a medium having a program recorded thereon. Computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also include media implemented in the form of carrier waves (e.g., transmission via the Internet). Therefore, the above detailed description should not be construed as limiting in all respects, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are intended to be included in the scope of the present invention.

Claims

1. An image sensor that outputs an image captured through a lens; and A processor that performs optical zoom out when the optical zoom in magnification of the lens detects an operation signal of the DIS (Digital Image Stabilization) mode at a predetermined magnification or higher, and performs digital zoom in according to the DIS mode, thereby matching the field of view of the image before and after the operation of the DIS mode; An image processing device including:

2. In paragraph 1, The above processor, An image processing device characterized in that it controls the digital zoom-in operation to be performed simultaneously with the optical zoom-out operation.

3. In paragraph 1, The above processor, An image processing device characterized in that shake correction is performed on an image acquired in the optical zoom out state, and digital zoom in is performed on an image on which shake correction has been performed.

4. In paragraph 1, An image processing device characterized in that the digital zoom-in magnification related to the above digital zoom-in is dynamically determined based on sensor data of a gyro sensor.

5. In paragraph 4, An image processing device characterized in that the optical zoom-in magnification has a value greater than the digital zoom-in magnification.

6. In paragraph 1, The above processor, An image processing device characterized in that the digitally zoomed-in image in the DIS mode is sized to a size according to the predetermined optical zoom-in ratio so as to match the angle of view of the image.

7. In paragraph 1, An image processing device characterized in that the above DIS mode is set to On or Off through user input.

8. In paragraph 1, The above image processing device includes a surveillance camera, When multiple presets are set to apply different optical zoom magnifications to multiple sites filmed by the above surveillance camera, The above processor, An image processing device characterized in that, when performing an operation according to the DIS mode on an image acquired from at least one of the plurality of sites, the digital zoom-in is performed so that the size is according to the optical zoom-in magnification defined for each site.

9. In paragraph 1, The above processor, A surveillance camera image processing device characterized in that, when movement of the object is detected or the moving speed of the object increases, the noise control intensity is immediately reduced according to a predetermined reduction ratio from the current noise control intensity.

10. In paragraph 1, The above processor, A surveillance camera image processing device characterized in that, when the movement of the detected object disappears or the movement speed of the object decreases, after a predetermined pause time has elapsed, the noise removal intensity is linearly increased to reflect the movement speed of the object.

11. In paragraph 1, Including the Department of Communications; The above processor, A surveillance camera video processing device characterized in that the video data acquired through the video capturing unit is transmitted to an external server through the communication unit, and the artificial intelligence-based object recognition results are received from the external server through the communication unit.

12. Video recording department; and A processor that linearly controls the noise removal intensity in an image acquired through the image capturing unit in a low-light environment based on the sensor gain amplification amount, and dynamically controls the noise removal intensity according to the movement speed of an object recognized in the image; The above processor, A surveillance camera image processing device characterized in that the image acquired from the above image capturing unit is set as input data, object recognition is set as output data, and a pre-learned neural network model is applied to recognize the object.

Images