Image analysis method and camera for analyzing image

Abstract

The present invention provides an image analysis method comprising the steps of: acquiring a first image corresponding to a live video recorded using a camera; analyzing the first image by using a pre-trained AI engine provided in the camera; measuring a load value of the camera on the basis of the usage of resources of the camera; and configuring an operation failure time of the camera on the basis of the load value so as to store a second image captured during the operation failure time, and analyzing the second image by using the AI engine after the operation failure time.

Description

Video analysis methods and cameras that analyze video

[0001] Embodiments of the present invention relate to an image analysis method and a camera for analyzing images.

[0002] When a large volume of video data is temporarily input into a camera, the camera must analyze all of the incoming data within a short period. However, due to limitations in the camera's hardware resources (e.g., CPU, memory, power consumption, etc.), not all objects in the video data can be analyzed, potentially leading to the loss of critical security data that requires analysis and causing significant damage to the security system.

[0003] For example, problems can easily arise when a camera monitoring a highway uses License Plate Recognition (LPR) image analysis to recognize the license plates of passing vehicles and stores the data for management. If dozens or hundreds of vehicles are rapidly passing within the camera's field of view, it is impossible to perform simultaneous LPR analysis for all of them. Furthermore, if various other image analysis functions, such as object detection and heat mapping, are performed concurrently alongside LPR analysis, hardware resource issues can easily occur, potentially leading to the loss of image analysis data.

[0004] The present invention aims to solve various problems, including those mentioned above, by providing an image analysis method and a camera for analyzing images. However, these problems are exemplary and do not limit the scope of the present invention.

[0005] According to one aspect of the present invention, a video analysis method is provided, comprising the steps of: acquiring a first video, which is a live video captured using a camera; analyzing the first video using a pre-trained AI engine equipped in the camera; measuring a load value of the camera based on the usage of resources of the camera; and setting a failure time of the camera based on the load value, storing a second video captured during the failure time, and analyzing the second video using the AI ​​engine after the failure time.

[0006] The step of measuring the load value may include measuring whether the load value exceeds a preset threshold based on the use of resources of the camera, including hardware usage, network traffic, or power consumption of the camera.

[0007] The step of analyzing the second image may include setting the time during which the load value exceeds a preset threshold or the time during which the AI ​​engine does not operate as the operation failure time, and, when the load value recovers to below the preset threshold and the AI ​​engine operates normally, analyzing the second image stored during the operation failure time using the AI ​​engine.

[0008] The step of analyzing the second image may include a step of proceeding with the analysis of the second image at a preset speed if, during the analysis of the second image, the load value exceeds a preset threshold or the AI ​​engine does not operate.

[0009] The step of analyzing the second image may include reducing the ratio of the analysis resources of the AI ​​engine for the second image by a preset ratio when the first image is input during the analysis of the second image, and stopping the analysis of the second image when the analysis resources of the AI ​​engine allocated to the first image are less than a preset threshold.

[0010] According to one aspect of the present invention, a camera for analyzing images is provided, comprising a processor, wherein the processor acquires a first image, which is a live image captured using the camera, analyzes the first image using a pre-trained AI engine provided in the camera, measures a load value of the camera based on the usage of resources of the camera, sets a failure time of operation of the camera based on the load value, stores a second image captured during the failure time, and analyzes the second image using the AI ​​engine after the failure time.

[0011] Other aspects, features, and advantages other than those described above will become clear from the following specific details, claims, and drawings for implementing the invention.

[0012] According to one embodiment of the present invention as described above, an image analysis method and a camera for analyzing images can be implemented, which can effectively analyze images by utilizing a failure response scenario for analyzing objects for which image analysis is missing. Of course, the scope of the present invention is not limited by such effects.

[0013] FIG. 1 is a diagram illustrating the operation of a camera that analyzes images according to an embodiment of the present invention.

[0014] FIG. 2 is a drawing for explaining the configuration and operation of a camera according to an embodiment of the present invention.

[0015] FIG. 3 is a diagram illustrating the operation of a camera that analyzes images according to another embodiment of the present invention.

[0016] FIG. 4 is a diagram illustrating a method for checking the resource usage of a camera according to an embodiment of the present invention.

[0017] FIG. 5 is a flowchart illustrating an image analysis method according to an embodiment of the present invention.

[0018] FIGS. 6 and FIGS. 7 are drawings for explaining an image analysis method according to an embodiment of the present invention.

[0019] The present invention is capable of various modifications and may have various embodiments; specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described below in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various forms.

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0021] In the following embodiments, terms such as "first," "second," etc. are used not in a limiting sense, but for the purpose of distinguishing one component from another. Also, singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, terms such as "include" or "have" mean that the feature or component described in the specification exists, and do not exclude the possibility that one or more other features or components may be added.

[0022] In the drawings, the size of components may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are depicted arbitrarily for convenience of explanation, so the present invention is not necessarily limited to what is illustrated.

[0023] In the following embodiments, when a part such as a region, component, section, block, or module is described as being on or above another part, it includes not only cases where it is directly on top of the other part, but also cases where another region, component, section, block, or module is interposed therein. Furthermore, when a region, component, section, block, or module is described as being connected, it includes not only cases where the region, component, section, block, or module is directly connected, but also cases where other regions, components, sections, blocks, or modules are interposed therein to indirectly connect them.

[0024] Hereinafter, in order to enable a person skilled in the art to easily practice the present invention, various embodiments of the present invention will be described in detail with reference to the attached drawings.

[0025] FIG. 1 is a diagram illustrating the operation of a camera for analyzing images according to one embodiment of the present invention. FIG. 2 is a diagram illustrating the configuration and operation of a camera according to one embodiment of the present invention. FIG. 3 is a diagram illustrating the operation of a camera for analyzing images according to another embodiment of the present invention.

[0026] Referring to FIGS. 1 to 3 together, an image analysis system according to one embodiment of the present invention may include a camera (100). However, the present invention is not limited thereto, and an image analysis system according to one embodiment of the present invention may include additional components, or some components may be omitted. Some components of an image analysis system according to one embodiment of the present invention may be separated into a plurality of devices, or a plurality of components may be merged into a single device. For example, as shown in FIG. 3, an image analysis system according to one embodiment of the present invention may include a camera (100) and a server (300). Also, although one camera (100) is shown in FIG. 3, an image analysis system according to one embodiment of the present invention may include one camera or two or more cameras.

[0027] The camera (100) may represent a camera device that captures images. Additionally, the camera (100) may analyze captured images. For example, the camera (100) may be an image analysis security camera. For example, as illustrated in FIG. 1, the camera (100) may recognize an object (10) from the captured images and analyze the object (10). For example, the camera (100) may analyze a vehicle's license plate. Additionally, the camera (100) may analyze a person's movement or face.

[0028] The server (300) may be a server device provided in the image analysis system of the present invention. For example, the server (300) may be connected to the camera (100) via a network to exchange data with each other.

[0029] Referring to FIG. 2, a camera (100) according to one embodiment of the present invention may include a memory (130), a processor (140), a communication unit (110), and a user interface unit (120). However, the present invention is not limited thereto, and an image analysis system according to one embodiment of the present invention may include other components, and some components may be omitted.

[0030] The communication unit (110) may provide a function for communicating with an external device via a network. For example, a request generated by the processor (140) of the camera (100) according to program code stored in a recording device such as memory (130) may be transmitted to an external device via a network under the control of the communication unit (110). Conversely, control signals, commands, content, files, etc. provided by an external device may be received by the camera (100) via the communication unit (110) through the network. For example, control signals or commands, etc. from an external device received through the communication unit (110) may be transmitted to the processor (140) or memory (130).

[0031] The communication method is not limited and may include not only communication methods utilizing communication networks that the network may include (e.g., mobile communication networks, wired internet, wireless internet, broadcasting networks), but also short-range wireless communication between devices. For example, the network may include any one or more networks such as a PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Additionally, the network may include any one or more network topologies such as a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, but is not limited thereto.

[0032] Additionally, the communication unit (110) can communicate with an external server via a network. The communication method is not limited, but the network may be a short-range wireless communication network. For example, the network may be a Bluetooth, BLE (Bluetooth Low Energy), or Wi-Fi network.

[0033] Additionally, the camera (100) according to the present invention may include a user interface unit (120). The user interface unit (120) may be a means for interfacing with an input / output device. For example, the input device may include a device such as a keyboard or a mouse, and the output device may include a device such as a display for displaying a communication session of an application. As another example, the user interface unit (120) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. As a more specific example, the processor (140) of the camera (100) may process instructions of a computer program loaded in memory (130), and a service screen or content configured using data provided by an external device may be displayed on a display through the user interface unit (120).

[0034] The memory (130) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Additionally, program code for controlling a camera may be stored temporarily or permanently in the memory (130).

[0035] The processor (140) can control the overall operation of the camera (100). For example, the processor (140) may be implemented in a form that optionally includes a processor, an ASIC (Application-Specific Integrated Circuit), other chipsets, logic circuits, registers, communication modems and / or data processing devices known in the art to perform the above-described operation. For example, the processor (140) may perform basic arithmetic, logic, and input / output operations and, for example, execute program code stored in memory (130). The processor (140) may store data in memory (130) or load data stored in memory (130).

[0036] A processor (140) according to one embodiment of the present invention can acquire a first video, which is a live video captured using a camera. Additionally, the processor (140) can analyze the first video using a pre-trained AI engine equipped in the camera. Additionally, the processor (140) can measure the load value of the camera based on the usage of the camera's resources. Additionally, the processor (140) can set a camera failure time based on the camera's load value and store a second video captured during the failure time. Additionally, the processor (140) can analyze the second video using the AI ​​engine after the failure time.

[0037] These processors (140) and components of the processor (140) may be implemented to execute instructions according to the code of an operating system contained in memory (130) and the code of at least one program. Here, the components of the processor (140) may be representations of different functions of the processor (140) that are performed by the processor (140) according to instructions provided by the program code stored in memory (130).

[0038] The processor (140) can train an AI engine using a program stored in memory (130). In particular, the processor (140) can train an AI engine for analyzing image-related data. Here, the AI ​​engine for analyzing image-related data can be designed to simulate the structure of the human brain on a computer and may include multiple network nodes having weights that simulate neurons of a human neural network. The multiple network modes can exchange data according to their respective connection relationships to simulate the synaptic activity of neurons that exchange signals through synapses. Here, the AI ​​engine may include a deep learning model. In the deep learning model, multiple network nodes can exchange data according to convolutional connection relationships while located in different layers.

[0039] For example, examples of AI engines may include deep neural networks (DNN), convolutional deep neural networks (CNN), recurrent Boltzmann machines (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), deep Q-networks, etc.

[0040] A server (300) according to one embodiment of the present invention may include a cloud server. In addition, a cloud server according to one embodiment of the present invention may be equipped with an AI engine and may analyze images using the AI ​​engine.

[0041] A camera (100) according to one embodiment of the present invention can store image data on a cloud server. For example, the camera (100) can store image data and metadata, which is the result of analyzing the image data, on a cloud server.

[0042] A camera (100) according to one embodiment of the present invention can detect a lack of resource based on the usage of the camera's resources. For example, when the camera (100) analyzes video data, if there are many processing objects, a lack of resources may occur, causing an analysis failure. A camera (100) according to one embodiment of the present invention can transmit a lack of resource event to a cloud server when a camera operation failure occurs due to camera overload.

[0043] A camera (100) according to one embodiment of the present invention can transmit the camera's operational failure time (e.g., RTP timestamp) as an event to a cloud server when the camera's resources are recovered and it is restored to a normal state. In this case, the cloud server can analyze the video during the camera's operational failure time using its own AI engine. For example, the cloud server according to one embodiment of the present invention can re-analyze objects in the video using the AI ​​engine according to the operational failure time event and store the analysis results in metadata. For example, the cloud server can exclude object data previously analyzed by the camera during the operational failure time, additionally analyze objects omitted by the camera, and additionally store the additional analysis results in metadata. For example, a re-analysis data flag may be set for objects additionally analyzed by the cloud server. For example, after the camera (100)'s resources are normalized and the failure is restored, the re-analysis data flag may be distinguished and displayed in the failure recovery report or the object analysis results of the camera (100).

[0044] A camera (100) according to one embodiment of the present invention can analyze images during the operation failure time when the camera's resources are normalized. In this case, a cloud server can transmit additional analysis results during the operation failure time to the camera (100).

[0045] FIG. 4 is a diagram illustrating a method for checking the resource usage of a camera according to an embodiment of the present invention.

[0046] Referring to FIG. 4, an embodiment is illustrated in which resource shortage detection criteria are set for each item of the camera's CPU, GPU, and Memory. For example, while the CPU, GPU, and Memory are illustrated in FIG. 4, the present invention is not limited thereto.

[0047] For example, a camera (100) according to one embodiment of the present invention may generate a lack of resource event when a resource threshold is reached for each item of the camera's CPU, GPU, and Memory. FIG. 5 is a flowchart for explaining an image analysis method according to one embodiment of the present invention. For example, an image analysis method according to one embodiment of the present invention may be performed by the processor (140) of the camera shown in FIG. 2.

[0048] Referring to FIG. 5, in an image analysis method according to one embodiment of the present invention, in step S110, a step of acquiring a first image, which is a live image captured using a camera, may be performed.

[0049] In step S120, the processor (140) can analyze the first image using a pre-trained AI engine equipped in the camera.

[0050] In step S130, the processor (140) can measure the load value of the camera based on the usage of the camera's resources. For example, the processor (140) can measure whether the load value of the camera exceeds a preset threshold based on the usage of the camera's resources, including the camera's hardware usage, network traffic, or power consumption.

[0051] In step S140, the processor (140) sets a camera failure time based on the camera's load value, stores a second video captured during the failure time, and analyzes the second video using an AI engine after the failure time. For example, the processor (140) may set the failure time as the time when the camera's load value exceeds a preset threshold or the time when the AI ​​engine does not operate. Additionally, the processor (140) may analyze the second video stored during the failure time using an AI engine when the camera's load value recovers to below the preset threshold and the AI ​​engine operates normally.

[0052] A processor (140) according to one embodiment of the present invention may proceed with the analysis speed of the second image at a preset speed if, during the analysis of the second image stored during the operation failure time, the load value of the camera exceeds a preset threshold or the AI ​​engine does not operate. In this case, the preset speed may be set to a speed slower than the existing analysis speed.

[0053] A processor (140) according to one embodiment of the present invention may reduce the ratio of the AI ​​engine's analysis resources for the second image by a preset ratio when the first image is input during the analysis of the second image. Additionally, the processor (140) may stop the analysis of the second image if the AI ​​engine's analysis resources allocated to the first image are below a preset threshold.

[0054] FIGS. 6 and FIGS. 7 are drawings for explaining an image analysis method according to an embodiment of the present invention.

[0055] First, referring to FIG. 6, in step S200, the camera (100) can capture a first video, which is a live video. For example, the camera (100) can be installed in a location where video analysis is required to capture a live video.

[0056] In step S210, the camera (100) can analyze the first video. For example, the camera (100) can analyze an object within the live video. For example, the camera (100) can analyze the object using a pre-trained AI engine.

[0057] In step S220, the camera (100) measures the load value of the camera based on the usage of the camera's resources and can set the camera's failure time based on the camera's load value. For example, the camera (100) can set the time when the camera's load value exceeds a preset threshold or the time when the AI ​​engine does not operate as the failure time and measure the failure time.

[0058] In step S230, the camera (100) can capture a second image during the motion failure time.

[0059] In step S240, the camera (100) may store a second image captured during the operation failure time. For example, in one embodiment of the present invention, the camera (100) may store the second image in a memory (130) provided in the camera (100). Or in another embodiment of the present invention, the camera (100) may store the second image in a server (300).

[0060] In step S250, the camera (100) can terminate the failure time when the camera's load value recovers to below a preset threshold or when the AI ​​engine operates normally.

[0061] In steps S260 and S270, the camera (100) can analyze the second image using an AI engine after the motion failure time.

[0062] Referring to FIG. 7, in steps S310 and S320, the camera (100) can analyze the second image using an AI engine after the operation failure time.

[0063] In step S330, the camera (100) can capture the first video, which is a live video, while analyzing the second video using an AI engine after the motion failure time.

[0064] In step S340, the camera (100) can set the ratio of the AI ​​engine's analysis resources when analysis of the first image is required while analyzing the second image using the AI ​​engine after the operation failure time. For example, when analysis of the first image is required while analyzing the second image using the AI ​​engine after the operation failure time, the camera (100) can reduce the ratio of AI analysis resources for the second image to prioritize analysis of the first image. Alternatively, if the camera (100) lacks AI analysis resources for the first image, it can stop analysis of the second image, perform analysis of the first image, and resume analysis of the remaining second image later.

[0065] Additionally, if the camera (100) starts another motion failure time while analyzing the second video using an AI engine after the motion failure time, the speed of AI analysis for the second video can be slower than the existing speed.

[0066] The device and / or system described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. The device and component described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on the operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0067] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0068] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0069] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0070] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A step of acquiring a first video, which is a live video captured using a camera; A step of analyzing the first image using a pre-trained AI engine equipped in the camera; A step of measuring the load value of the camera based on the resource usage of the camera; and A video analysis method comprising the step of setting a camera operation failure time based on the above load value, storing a second video captured during the operation failure time, and analyzing the second video using the AI ​​engine after the operation failure time.

2. In Paragraph 1, A video analysis method comprising the step of measuring the load value, which includes measuring whether the load value exceeds a preset threshold based on the use of resources of the camera, including hardware usage, network traffic, or power consumption of the camera.

3. In Paragraph 2, The step of analyzing the second image above is, A step of setting the time during which the load value exceeds a preset threshold or the time during which the AI ​​engine does not operate as the operation failure time; and A video analysis method comprising the step of analyzing the second video stored during the operation failure time using the AI ​​engine when the load value recovers to below a preset threshold and the AI ​​engine operates normally.

4. In Paragraph 2, A method for analyzing images, wherein the step of analyzing the second image includes the step of proceeding with the analysis speed of the second image at a preset speed when the load value exceeds a preset threshold or the AI ​​engine does not operate during the analysis of the second image.

5. In Paragraph 2, The step of analyzing the second image above is, A step of reducing the ratio of the analysis resources of the AI ​​engine for the second image by a preset ratio when the first image is input during the analysis of the second image; and An image analysis method comprising the step of stopping the analysis of the second image when the analysis resources of the AI ​​engine allocated to the first image are less than a preset threshold.

6. In a camera that analyzes video, Includes a processor, The above processor acquires a first video, which is a live video captured using a camera, analyzes the first video using a pre-trained AI engine equipped in the camera, measures a load value of the camera based on the usage of resources of the camera, sets a failure time of operation of the camera based on the load value, stores a second video captured during the failure time, and analyzes the second video using the AI ​​engine after the failure time.

Images