Adaptive update of neural network model for object re-identification
Patent Information
- Application Number
- PCT/KR2024/004601
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-03
- Filing Date
- 2024-04-08
- Publication Date
- 2024-11-07
AI Technical Summary
Existing object recognition neural networks for surveillance systems face challenges in adapting to varying image characteristics across different camera installations, requiring costly data collection and lengthy training processes, especially when new cameras are installed with different viewing conditions.
A method for efficiently updating a pre-trained neural network model by acquiring image characteristic parameters from new camera locations without labeling learning data or calculating a loss function, using batch normalization layers to normalize and update image features such as edge changes, skewness, noise, and illumination.
Enables the neural network to adapt to diverse image characteristics from multiple surveillance cameras, reducing data collection costs and learning time, and allowing for efficient updates even in edge device environments, improving object recognition reliability without the need for extensive retraining.
Smart Images

Figure KR2024004601_07112024_PF_FP_ABST
Abstract
Description
Adaptive updating of neural network models for object recognition
[0001] This specification relates to a method for updating an object recognition neural network model.
[0002] As detection algorithms for people and vehicles using AI technology have become more sophisticated, research is also underway on neural network algorithms for person / vehicle re-identification (Re-ID), which utilize detection results. Person / vehicle re-identification algorithms distinguish between people or vehicles detected by a detection system and identify them as the same person. They are primarily used in surveillance systems such as CCTV, particularly in locations like supermarkets, hospitals, and hotels, where multiple cameras are installed to search for individuals.
[0003] Training a recognition neural network requires first constructing a training data set of identical people or vehicles captured by multiple cameras. This incurs costs for camera installation, data extraction, and data purification. Furthermore, once the training data set is complete, training and validating the neural network using it can take several days, or even months. Furthermore, embedding the trained person or vehicle recognition neural network to enable it to operate on edge devices is also necessary.
[0004] The characteristics of the captured images may vary depending on the CCTV camera's installation location or position. In other words, the characteristics recognized in the image (such as brightness, background, and blur) may vary depending on the installation location or position of the surveillance camera. To address these differences, data collected from the relevant site and camera must be refined to train a recognition model. Specifically, creating trainable data can be time-consuming and expensive, requiring the construction of a set of images of the same person. Furthermore, the costs of installing multiple cameras, the time and labor required to process the collected data, and other factors are incurred. If images acquired from surveillance cameras installed in a new location exhibit characteristics different from those previously observed, the costs of collecting and refining data from that location, as well as the training time, are significant.
[0005] To address the aforementioned issues, the present specification aims to provide a method for updating a recognition neural network so as to adapt to image characteristics presented by multiple surveillance camera devices.
[0006] In addition, the present specification aims to provide a method for efficiently updating a pre-trained re-recognition neural network model without labeling training data or calculating a loss function.
[0007] In addition, the present specification aims to provide a method for efficiently updating a re-recognition neural network model even in an edge device environment (surveillance camera device).
[0008] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the detailed description of the invention below.
[0009] According to one embodiment of the present specification, a surveillance camera device includes: an image acquisition unit for acquiring an image; a memory for storing a neural network model for object re-recognition learned in advance; a processor for detecting an object from the image acquired through the image acquisition unit and acquiring learning data for updating the neural network model among the detected objects according to a predetermined standard; wherein the processor can obtain image characteristic parameters corresponding to the learning data by feedforward inputting the acquired learning data into the neural network model, and update the neural network model by reflecting the image characteristic parameters into the neural network model.
[0010] The above processor can configure a batch normalization layer for normalizing the image characteristic parameters within the neural network model, and update the image characteristic parameters by feeding forward the learning data for a plurality of layers included in the neural network model.
[0011] The processor can update the image characteristic parameters by updating the average and variance of the learning data and the average and variance between the plurality of layers.
[0012] The above-determined criteria may include at least one of the size of the detection box of the object, the shape of the object, and the movement path of the object. The image characteristic parameter may include at least one of the edge variation, skewness, noise, illumination component, and reflectance component.
[0013] The first image used for training the pre-trained neural network model and the second image used for updating the neural network model may be images acquired through surveillance cameras installed in different locations.
[0014] At least one element of the image characteristic parameters of the first image and the second image may have different characteristics.
[0015] A method for updating a neural network model for object re-recognition according to another embodiment of the present specification includes the steps of: storing a neural network model for object re-recognition that has been learned in advance; obtaining an image from the surveillance camera device; detecting an object from the image and obtaining learning data for updating the neural network model among the detected objects according to a predetermined criterion; and obtaining image characteristic parameters corresponding to the learning data by feedforward inputting the obtained learning data into the neural network model, and reflecting the image characteristic parameters into the neural network model to update the neural network model.
[0016] The step of updating the neural network model may include a step of configuring a batch normalization layer for normalizing the image characteristic parameters within the neural network model; and a step of updating the image characteristic parameters by feeding forward the learning data for a plurality of layers included in the neural network model.
[0017] The step of updating the image specific parameters may update the image characteristic parameters by updating the average and variance of the learning data and the average and variance between the plurality of layers.
[0018] The above neural network model update method may further include a step of transmitting the updated neural network model to the surveillance camera via a wireless communication unit.
[0019] The first image used for training the pre-trained neural network model and the second image used for updating the neural network model may be images acquired through surveillance cameras installed in different locations.
[0020] Another method for updating a neural network model for object re-recognition according to one embodiment of the present specification includes: training a neural network model for object re-recognition based on a first image acquired through a first surveillance camera installed at a first location; obtaining a second image by applying the trained neural network model to a second surveillance camera to be installed at a second location; obtaining training data for updating the neural network model based on an object detected from the second image; and updating the neural network model based on image characteristic parameters of the second image acquired by feedforward inputting the acquired training data to the neural network model.
[0021] The image characteristic parameter of the first image and the image characteristic parameter of the second image differ in at least one element, and the image characteristic parameter may include at least one of edge variation, skewness, noise, illumination component, and reflectance component.
[0022] The first surveillance camera and the second surveillance camera may be cameras installed at different locations with different view points with respect to the same object.
[0023] The method for updating the neural network model further includes a step of performing object re-recognition on the second image for the first object recognized in the first image; and, as a result of the object re-recognition, if the same object is recognized as a different object or if different objects are recognized as the same object, a step of acquiring learning data for updating the neural network model can be performed.
[0024] The above neural network model update step may include: a step of performing object re-recognition using a neural network model for object re-recognition learned based on the first image after acquiring the second image; and a step of updating the neural network model when a predetermined performance is not obtained as a result of the object re-recognition operation.
[0025] According to one embodiment of the present disclosure, a recognition neural network can be updated to adapt to image characteristics presented by multiple surveillance camera devices.
[0026] Additionally, according to one embodiment of the present specification, a pre-trained re-recognition neural network model can be efficiently updated without labeling data or calculating a loss function.
[0027] Additionally, according to one embodiment of the present specification, a re-recognition neural network model can be updated efficiently even in an edge device environment (a surveillance camera device).
[0028] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.
[0029]
[0030] The accompanying drawings, which are incorporated in and constitute a part of the detailed description to aid in understanding the present specification, provide embodiments of the present specification and, together with the detailed description, also describe the technical features of the present specification.
[0031] FIG. 1 is a drawing for explaining a surveillance camera system for implementing an image processing method of a surveillance camera according to one embodiment of the present specification.
[0032] FIG. 2 is a schematic block diagram of a surveillance camera according to one embodiment of the present specification.
[0033] FIG. 3 is a diagram illustrating an AI device (module) applied to analysis of surveillance camera images according to one embodiment of the present specification.
[0034] FIG. 4 is a flowchart of a method for updating a neural network model for object re-recognition according to one embodiment of the present specification.
[0035] FIG. 5 is a diagram illustrating a method for updating a neural network model according to one embodiment of the present specification.
[0036] FIG. 6 is a flowchart illustrating another example of a method for updating a neural network model for object re-recognition according to one embodiment of the present specification.
[0037] The accompanying drawings, which are incorporated in and constitute a part of the detailed description to aid in the understanding of the present invention, provide embodiments of the present invention and, together with the detailed description, explain the technical features of the present invention.
[0038]
[0039] Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.
[0040] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.
[0041] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.
[0042] Singular expressions include plural expressions unless the context clearly indicates otherwise.
[0043] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.
[0044] FIG. 1 is a drawing for explaining a surveillance camera system for implementing an image processing method of a surveillance camera according to one embodiment of the present specification.
[0045] Referring to FIG. 1, an image management system (10) according to one embodiment of the present specification may include a photographing device (100a, 100b, 100c, hereinafter referred to as 100 for convenience of explanation) and an image management server (200). The photographing device (100) may be an electronic photographing device placed at a fixed position in a specific location, an electronic photographing device that can be moved automatically or manually along a predetermined path, or an electronic photographing device that can be moved by a person or a robot.
[0046] The recording device (100) may be an IP camera connected to a wired or wireless Internet. The recording device (100) may be a PTZ camera having pan, tilt, and zoom functions. The recording device (100) may have a function to record or take pictures of the monitored area. The recording device (100) may have a function to record sounds generated in the monitored area. The recording device (100) may have a function to generate a notification or perform recording or taking pictures when a change, such as movement or sound, occurs in the monitored area.
[0047] The above-described photographing device (100) may include a plurality of photographing devices (100a, 100b, 100c) installed in different spaces. For example, the first photographing device (100a) and the second photographing device (100b) may be spaced apart by a first interval, and the second photographing device (100b) and the third photographing device (100c) may be spaced apart by a second interval. That is, each of the photographing devices (100a, 100b, 100c) may be a system implemented in the form of a CCTV, each positioned at a position capable of photographing the same person at a predetermined time interval.
[0048] The above-described plurality of photographing devices (100a, 100b, 100c) may be devices that each collect image data for image management in a single image management server (200). Accordingly, even if the same object is included in each image acquired from the plurality of photographing devices (100a, 100b, 100c), they may be recognized as different objects depending on the lighting, background, viewpoint of each photographing device, etc. of the location where the plurality of photographing devices (100a, 100b, 100c) are installed. That is, after a first object is detected by the first photographing device (100a), it may be sequentially detected by the second photographing device (100b) and the third photographing device (100c) through object movement. The image management server (200) may perform an object re-recognition operation to check whether the second photographing device (100b) and the third photographing device (100c) also recognize the first object as the same first object. As a result of performing the object re-recognition operation, if the first photographing device (100a), the second photographing device (100b), and the third photographing device (100c) all recognize the first object as different objects, it is necessary to update the neural network model for object re-recognition included in the image management server (200) and / or each photographing device (100a, 100b, 100c). The image management server (200) may be a device having a storage and / or search function for receiving the image captured by the photographing device (100) and / or the image obtained by editing the image. The image management server (200) may analyze the received image so as to correspond to the intended use. For example, the image management server (200) may detect an object using an object detection algorithm to detect the object in the image. The object detection algorithm may be an AI-based algorithm, and may detect the object by applying a pre-learned artificial neural network model.
[0049] According to one embodiment of the present specification, the image management server (200) can function as an image search device. The image search device can quickly and easily search for images acquired from multiple surveillance camera channels by inputting a specific image, an object included in a specific image, or a specific channel as a search condition. The image search device must first build a database so that a user can easily search for images, and one embodiment of the present specification proposes a method of limiting the amount of computation by limiting the size of a search target when performing an image search according to a specific search condition.
[0050] Meanwhile, the video management server (200) may be an NVR (Network Video Recorder) or DVR (Digital Video Recorder) that performs the function of storing videos acquired through a network. Alternatively, it may be a CMS (Central Management System) that comprehensively manages and controls videos and enables remote video monitoring. Meanwhile, the video management server (200) is not limited thereto, and may be a personal computer or a portable terminal. However, this is merely an example, and the technical idea of the present specification is not limited thereto, and any device capable of receiving, displaying, and / or storing multimedia objects from one or more surveillance cameras through a network may be used without limitation.
[0051] Meanwhile, the image management server (200) may store various learning models suitable for the purpose of image analysis. In addition to the learning model for object detection described above, it may also store a model capable of obtaining the movement speed of a detected object. Here, the learned models may include a learning model that takes images captured by the plurality of photographing devices (100a, 100b, 100c), that is, images with different shooting times and shooting locations, as input data, and outputs the gender of a person included in the captured images and the feature vector values of the images.
[0052] Additionally, the video management server (200) can analyze the received video to generate metadata and index information for the metadata. The video management server (200) can analyze the video information and / or audio information contained in the received video, either together or separately, to generate metadata and index information for the metadata. The metadata may further include information on the time the video was shot, the shooting location, etc.
[0053] The image management system (10) may further include an external device (300) capable of performing wired or wireless communication with the photographing device (100) and / or the image management server (200).
[0054] The external device (300) can transmit an information provision request signal requesting provision of all or part of an image to the image management server (200). The external device (300) can transmit an information provision request signal requesting the image management server (200) for information on the presence or absence of an object based on an image analysis result, the object's moving speed, a shutter speed adjustment value based on the object's moving speed, a noise removal value based on the object's moving speed, a sensor gain value, etc. In addition, the external device (300) can transmit an information provision request signal requesting metadata obtained by analyzing an image and / or index information for the metadata to the image management server (200).
[0055] The image management system (10) may further include a communication network (400) which is a wired / wireless communication path between the photographing device (100), the image management server (200), and / or the external device (300). The communication network (400) may include wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), and ISDNs (Integrated Service Digital Networks), or wireless networks such as wireless LANs, CDMA, Bluetooth, and satellite communication, but the scope of the present specification is not limited thereto.
[0056] FIG. 2 is a schematic block diagram of a surveillance camera according to one embodiment of the present specification.
[0057] FIG. 2 is a block diagram showing the configuration of the camera illustrated in FIG. 1. Referring to FIG. 2, the camera (100) is described as a network camera that performs an intelligent image analysis function to generate the image analysis signal, but the operation of the network surveillance camera system according to an embodiment of the present invention is not necessarily limited thereto.
[0058] The camera (100) includes an image sensor (110), an encoder (120), a memory (130), an event sensor (140), a processor (140), and a communication unit (150).
[0059] The image sensor (110) performs the function of capturing an image by photographing a surveillance area, and can be implemented as, for example, a CCD (Charge-Coupled Device) sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, etc.
[0060] The encoder (120) performs an operation of encoding an image acquired through an image sensor (110) into a digital signal, which may follow, for example, H.264, H.265, MPEG (Moving Picture Experts Group), M-JPEG (Motion Joint Photographic Experts Group) standards, etc.
[0061] The memory (130) can store video data, audio data, still images, metadata, etc. As mentioned above, the metadata may be data including object detection information (movement, sound, intrusion into a designated area, etc.) captured in the surveillance area, object identification information (person, car, face, hat, clothing, etc.), and detected location information (coordinates, size, etc.).
[0062] In addition, the still image is generated together with the metadata and stored in the memory (130), and can be generated by capturing image information for a specific analysis area among the image analysis information. For example, the still image can be implemented as a JPEG image file.
[0063] For example, the still image may be generated by cropping a specific area of image data determined to be an identifiable object among image data of the surveillance area detected in a specific area and for a specific period of time, and this may be transmitted in real time together with the metadata.
[0064] The memory (130) can store a neural network model trained for object recognition. The neural network model can be configured and trained by considering brightness and variance, which are image characteristic parameters. When features are extracted from each neural network layer, the feature values can be trained by considering brightness and variance. Once neural network training is completed, the image characteristic parameters determined according to the training data are fixed and may not be changed. The neural network model can be received from the image management server (200) of FIG. 1 and stored in the memory (130) by the processor (160). Additionally, the neural network model can be trained independently in the image capturing device (100) and stored in the memory (130).
[0065] One embodiment of the present specification can apply the YOLO (You Only Lock Once) algorithm to object detection. YOLO is an AI algorithm suitable for surveillance cameras that process real-time video because of its fast object detection speed. Unlike other object-based algorithms (e.g., Faster R-CNN, R_FCN, FPN-FRCN), the YOLO algorithm resizes a single input image and passes it through a single neural network once, outputting a bounding box indicating the location of each object and a classification probability of what the object is. Finally, non-max suppression is performed to detect one object once.
[0066] Meanwhile, it is to be noted that the object recognition algorithm disclosed in this specification is not limited to the aforementioned YOLO and can be implemented using various deep learning algorithms.
[0067] The communication unit (140) transmits the video data, audio data, still images, and / or metadata to the video receiving / searching device (300 of FIG. 1). According to one embodiment, the communication unit (140) can transmit the video data, audio data, still images, and / or metadata to the video receiving device (300 of FIG. 1) in real time. The communication unit (140) can perform at least one communication function among wired / wireless Local Area Network (LAN), Wi-Fi, ZigBee, Bluetooth, and Near Field Communication.
[0068] Meanwhile, object recognition for images acquired through a surveillance camera according to one embodiment of the present specification and learning of a neural network model for object recognition may be performed under the control of the processor (160) illustrated in FIG. 2, but may also be performed by an AI device (module) independently provided for artificial intelligence image analysis. For convenience of explanation, the AI device (module) is described in FIG. 3, but it goes without saying that the function performed by the module of FIG. 3 may also be performed by the processor (160) of FIG. 2.
[0069] FIG. 3 is a drawing for explaining an AI device (module) applied to an image search device according to one embodiment of the present specification.
[0070] Referring to FIG. 3, the AI device (20) may include an electronic device including an AI module capable of performing AI processing, a server including an AI module, or the like. In addition, the AI device (20) may be included as part of at least a portion of a surveillance camera or an image management server, and may be configured to perform at least a portion of the AI processing.
[0071] AI processing can encompass all operations related to the control unit (processor) of a surveillance camera or video management server. For example, a surveillance camera or video management server can perform AI processing of acquired video signals to perform processing / judgment and control signal generation operations.
[0072] The AI device (20) may be a client device that directly utilizes the AI processing results, or may be a device in a cloud environment that provides the AI processing results to other devices. The AI device (20) is a computing device capable of learning a neural network and may be implemented as various electronic devices such as a server, desktop PC, laptop PC, tablet PC, etc.
[0073] The AI device (20) may include an AI processor (21), a memory (25) and / or a communication unit (27).
[0074] The AI processor (21) can learn a neural network using a program stored in the memory (25). In particular, the AI processor (21) can learn a neural network for recognizing data related to surveillance cameras. Here, the neural network for recognizing data related to surveillance cameras can be designed to simulate the structure of the human brain on a computer, and can include a plurality of network nodes having weights that simulate neurons of a human neural network. The plurality of network modes can exchange data according to their respective connection relationships so as to simulate the synaptic activity of neurons that exchange signals through synapses. Here, the neural network can include a deep learning model developed from a neural network model. In the deep learning model, a plurality of network nodes are located in different layers and can exchange data according to convolutional connection relationships. Examples of neural network models include various deep learning techniques such as deep neural networks (DNNs), convolutional deep neural networks (CNNs), recurrent Boltzmann machines (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), and deep Q-networks, which can be applied to fields such as computer vision, speech recognition, natural language processing, and speech / signal processing.
[0075] Meanwhile, the processor performing the functions described above may be a general-purpose processor (e.g., CPU), but may also be an AI-specific processor for artificial intelligence learning (e.g., GPU).
[0076] The memory (25) can store various programs and data required for the operation of the AI device (20). The memory (25) can be implemented as a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SDD), etc. The memory (25) is accessed by the AI processor (21), and data reading / writing / modifying / deleting / updating, etc. can be performed by the AI processor (21). In addition, the memory (25) can store a neural network model (e.g., a deep learning model (26)) generated through a learning algorithm for data classification / recognition according to one embodiment of the present invention.
[0077] Meanwhile, the AI processor (21) may include a data learning unit (22) that learns a neural network for data classification / recognition. The data learning unit (22) may learn criteria regarding which learning data to use to determine data classification / recognition and how to classify and recognize data using the learning data. The data learning unit (22) may acquire learning data to be used for learning and apply the acquired learning data to the deep learning model, thereby learning the deep learning model.
[0078] The data learning unit (22) may be manufactured in the form of at least one hardware chip and mounted on the AI device (20). For example, the data learning unit (22) may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of a general-purpose processor (CPU) or a graphics processor (GPU) and mounted on the AI device (20). In addition, the data learning unit (22) may be implemented as a software module. When implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer readable medium that can be read by a computer. In this case, at least one software module may be provided by an operating system (OS) or an application.
[0079] The data learning unit (22) may include a learning data acquisition unit (23) and a model learning unit (24).
[0080] The learning data acquisition unit (23) can acquire learning data required for a neural network model to classify and recognize data.
[0081] The model learning unit (24) can use the acquired learning data to learn the neural network model to have a judgment criterion on how to classify a given data. At this time, the model learning unit (24) can train the neural network model through supervised learning that uses at least some of the learning data as a judgment criterion. Alternatively, the model learning unit (24) can train the neural network model through unsupervised learning that discovers a judgment criterion by learning on its own using the learning data without guidance. In addition, the model learning unit (24) can train the neural network model through reinforcement learning using feedback on whether the result of the situation judgment according to the learning is correct. In addition, the model learning unit (24) can train the neural network model using a learning algorithm including error back-propagation or gradient descent.
[0082] Once the neural network model is trained, the model training unit (24) can store the trained neural network model in memory. The model training unit (24) can also store the trained neural network model in the memory of a server connected to the AI device (20) via a wired or wireless network.
[0083] The data learning unit (22) may further include a learning data preprocessing unit (not shown) and a learning data selection unit (not shown) to improve the analysis results of the recognition model or to save resources or time required for creating the recognition model.
[0084] The learning data preprocessing unit can preprocess the acquired data so that it can be used for learning to determine situations. For example, the learning data preprocessing unit can process the acquired data into a preset format so that the model learning unit (24) can utilize the acquired learning data for learning image recognition.
[0085] In addition, the learning data selection unit can select data required for learning from among the learning data acquired from the learning data acquisition unit (23) or the learning data preprocessed from the preprocessing unit. The selected learning data can be provided to the model learning unit (24).
[0086] Additionally, the data learning unit (22) may further include a model evaluation unit (not shown) to improve the analysis results of the neural network model.
[0087] The model evaluation unit inputs evaluation data into the neural network model, and if the analysis results output from the evaluation data do not satisfy a predetermined standard, it can cause the model learning unit (22) to relearn. In this case, the evaluation data may be predefined data for evaluating the recognition model. For example, the model evaluation unit can evaluate that the predetermined standard is not satisfied if the number or ratio of evaluation data with inaccurate analysis results among the analysis results of the learned recognition model for the evaluation data exceeds a preset threshold.
[0088] The communication unit (27) can transmit the AI processing results by the AI processor (21) to an external electronic device. For example, the external electronic device may include a surveillance camera, a Bluetooth device, an autonomous vehicle, a robot, a drone, an AR device, a mobile device, a home appliance, etc.
[0089] Meanwhile, the AI device (20) illustrated in FIG. 3 is functionally divided into an AI processor (21), a memory (25), a communication unit (27), etc., but it should be noted that the aforementioned components may be integrated into one module and referred to as an AI module.
[0090] This specification may be linked to one or more of a surveillance camera, an autonomous vehicle, a user terminal, and a server, including an artificial intelligence module, a robot, an augmented reality (AR) device, a virtual reality (VT) device, and devices related to 5G and 6G services.
[0091] FIG. 4 is a flowchart of a method for updating a neural network model for object re-recognition according to one embodiment of the present specification. The method for updating a neural network model in FIG. 4 may be implemented through a surveillance camera system, a surveillance camera device, a processor or a control unit included in the surveillance camera device described with reference to FIGS. 1 to 3. The operations for updating a neural network model for object re-recognition in the present specification may be performed solely by the surveillance camera device or may be implemented by a combination of the surveillance camera and the video management server. For convenience of explanation, FIG. 4 is implemented in a surveillance camera device (100a, 100b, 100c of FIG. 1, 100 of FIG. 2) and is described assuming that it is implemented by the processor (160) of the surveillance camera device.
[0092] Referring to FIG. 4, the surveillance camera (100) can store a pre-learned object recognition neural network model in memory (S400).
[0093] Here, the surveillance camera (100) may be identical to the object recognition model and / or object re-recognition model stored in surveillance cameras installed at different locations with different viewpoints. If the viewpoints are different, this may mean that the objects recognized in the images acquired from each camera for the same object are not recognized as the same object. For example, this may mean a camera that recognizes the front of a person and a camera that recognizes the back of a person. In addition, since the image characteristic parameters of the acquired images are different, the two cameras are more likely to recognize the same object as different objects.
[0094] Meanwhile, according to one embodiment of the present specification, a surveillance camera requiring an update of an object recognition neural network model may be a surveillance camera newly installed at a specific location or site. Accordingly, if a previously trained recognition neural network is applied as is to a camera installed at a newly installed location, there is a problem in that it may not reflect the characteristics of the image acquired at the new location. In order to overcome this problem and ensure the reliability of the object recognition results, it is necessary to update the previously trained and installed object recognition neural network model to suit the characteristics of the location where the camera will be installed. To this end, it is necessary to configure the image acquired at the new site as training data.
[0095] Accordingly, the processor (160) can acquire an image through the image acquisition unit (S410) and detect an object (S420). According to one embodiment, the processor (160) can detect objects such as people and vehicles in the image. The processor (160) can detect objects for multiple frames and analyze the movement path of the object, the shape of the object, etc.
[0096] The processor (160) may select training data for updating a neural network model according to predetermined criteria (S430). According to one embodiment, the processor (160) may select objects with a large detection box size and a clear shape as training data to be used for updating the neural network model. In order to update the re-recognition neural network model according to one embodiment of the present disclosure, the neural network model can be updated through adaptive learning only with data collected from a site where a surveillance camera is newly installed, without the need for training data used for training a previously trained neural network model.
[0097] The processor (160) can obtain image characteristic parameters of images collected from the new site by inputting the selected learning data into a pre-trained neural network model through feedforward (S440). Since the present specification enables model modification through feedforward input only for a pre-configured neural network model without the need to calculate a loss function, it can be efficiently applied even in edge device environments where computational performance may be relatively low.
[0098] The processor (160) can update the neural network model based on the acquired image characteristic parameters.
[0099] Hereinafter, with reference to FIG. 5, the process of efficiently updating a pre-trained neural network model using only a portion of newly acquired image data from a new site is described in more detail.
[0100] FIG. 5 is a diagram illustrating a method for updating a neural network model according to one embodiment of the present specification.
[0101] Referring to FIG. 5, the processor (160) configures a batch normalization layer (53) for normalizing the image characteristic parameters within the pre-learned neural network model, and feeds forward the learning data for a plurality of layers (51, 52, 54) included in the neural network model, thereby updating the image characteristic parameters.
[0102] Here, the image characteristic parameters may include at least one of edge variation, skewness, noise, illumination component, and reflectance component.
[0103] The method of reflecting the image characteristic parameters of the images acquired from the surveillance cameras installed in the new site is similar to the general batch normalization process.
[0104] Batch normalization refers to a technique in which one or more batch normalization layers are added to a neural network to normalize inputs to the layer based on the statistical characteristics of the inputs based on the training data. Batch normalization can also refer to the process of normalizing input data on a batch-by-batch basis, ensuring that even if the input data has diverse distributions, it has a uniform distribution.
[0105] A portion of a neural network (50) may include an input layer (51), a hidden layer (52), a batch normalization layer (53), and an output layer (54). During training, training data is typically divided into multiple "batches" to improve efficiency (e.g., because the entire training data set may not fit in the computer's memory, and performance can be improved by avoiding reading from a disk or other mass storage device). Samples from each batch are provided to the input layer of the neural network, and the activation values calculated by each layer for a given input are fed to the next layer. For example, the activation values calculated by the input layer (51) are fed as input to the hidden layer (52), which feeds the activation values to the batch normalization layer (53). The batch normalization layer (53) normalizes the inputs received from the previous layer (e.g., the layer (52) as illustrated in FIG. 5) for the current batch of training data. For example, the processor (160) calculates the mean and variance of the input for a batch of training data and normalizes the input so that it has the same mean and variance, thereby feeding the normalized version of the input to the next layer of the network (e.g., layer (54) as illustrated in FIG. 5). Once training is complete, the calculated mean and variance for the entire training data set are stored in the batch normalization layer (53), so that during inference, the input to the batch normalization layer (53) is adjusted based on the mean and variance of the training data set.
[0106] Therefore, each batch normalization layer of a pre-trained neural network model stores statistical feature information that reflects the statistical distribution of the previous layer's output in response to the training data (as processed by the previous layer of the neural network).
[0107] Meanwhile, the object recognition neural network model method according to one embodiment of the present specification does not actually calculate the average and variance of all training data in advance, but can update the average and variance parameters using methods such as moving averages and exponential averages as training progresses. The average and variance are not only considered for the input data itself, but also calculated between layers of the neural network, and the corresponding values are updated during training. Once training is complete and in the inference step, the moving average and moving variance parameters are no longer updated and are used as fixed parameters.
[0108] Accordingly, the present specification can additionally configure a layer that can reflect values representing the normalization of the characteristics of the image, including the mean and variance, similar to the method of updating the mean and variance parameters of the batch normalization layer. In the above description, the recognition neural network model was described as having a batch normalization layer added, but the present specification is not limited thereto, and it is obvious that the added layer can include any layer that can perform the batch normalization function.
[0109] The image characteristics mentioned here may include edge variation, skewness, noise, illumination, and reflectance. Layers are configured to normalize these characteristics, and during training, image feature parameters for each layer are learned using methods such as moving averages or exponential averaging. Once training is complete, all parameters are frozen and used during inference.
[0110] According to one embodiment, the present specification can update a neural network by a method of updating parameters for camera adaptation after installation on a new site camera, as shown in the flowchart of FIG. 4. According to one embodiment, first, the processor (160) can change the image characteristic parameters so that they can be learned. However, parameters other than the image characteristic parameters are not changed. Thereafter, the processor (160) inputs the learning data into the recognition neural network. The image characteristic parameters are acquired and updated by feeding the data forward within the neural network.
[0111] Accordingly, the present specification enables updating of a neural network model using only data collected from a new site without the need for training data previously used in the neural network model training process.
[0112] In addition, this specification can reduce the resources (labor costs and work time) required to update a neural network model because there is no need to individually label training data.
[0113] This allows learning with only a few hundred pieces of data, and learning can be done even in edge device environments (such as surveillance camera devices) without calculating a loss function or requiring repetitive updates.
[0114] Meanwhile, the re-recognition neural network model update process disclosed in this specification can also be efficiently performed through a combination of an edge device environment and a cloud environment.
[0115] FIG. 6 is a flowchart illustrating another example of a method for updating a neural network model for object re-recognition according to one embodiment of the present specification.
[0116] Referring to FIG. 6, the initial re-recognition neural network model can be trained by the server (200) based on the first image acquired from the first surveillance camera (100a). The first surveillance camera (100a) may be a surveillance camera installed at a specific location for a certain period of time or longer. Meanwhile, a second surveillance camera may be newly installed at the specific location at a point with a different viewpoint from that of the first surveillance camera.
[0117] The re-recognition neural network model learned in the above server (200) is provided to both the first surveillance camera (100a) and the second surveillance camera (100b). Accordingly, the newly installed second surveillance camera (100b) can initially perform object re-recognition using the previously learned re-recognition neural network model. However, since the image characteristic parameters of the images acquired by the first surveillance camera (100a) and the second surveillance camera (100b) may differ, the second surveillance camera (100b) needs to additionally update the previously learned neural network model based on the second image.
[0118] The second surveillance camera (100b) acquires a second image (S610) and selects specific data from objects detected in the second image as learning data (S620). The selected learning data may be selected according to predetermined criteria, as described above.
[0119] The second surveillance camera (100b) can obtain image characteristic parameters of the second image and efficiently update the neural network model by feeding forward learning data to the pre-trained neural network model.
[0120] Meanwhile, according to one embodiment, although the second surveillance camera (100b) is newly installed, if there is no problem with the reliability of object re-recognition even if the pre-trained re-recognition neural network model is applied as is, the neural network model can be used as is without updating. Accordingly, after the second surveillance camera (100b) is installed in a new site, after checking the object re-recognition result, only if it is determined that the same object recognized by the first surveillance camera (100a) was not correctly recognized, the neural network model can be updated based on the second image. That is, if a predetermined result is derived as a result of performing the object re-recognition operation on the second image acquired through the second surveillance camera (100b) installed in the new site, that is, if the performance of the re-recognition neural network model installed in the first surveillance camera is above a certain level, the pre-trained neural network model may not be updated through the image data acquired through the second image.
[0121] Meanwhile, although not shown in FIG. 6, the server (200) may update the neural network model for each surveillance camera and then perform optimization to suit the edge device environment and provide it to each surveillance camera.
[0122] As described above, this specification discloses a method for more easily updating a previously learned and stored recognition model through images acquired in an edge device environment (an environment of multiple surveillance camera systems installed in the field), but this specification is not limited thereto. For example, even when the previously learned and stored model is an object recognition model rather than an object re-recognition model, the concepts derived from this specification may be applied. According to one embodiment, it may be possible to update a model by applying a previously learned object recognition model, face detection model, etc. through image data acquired in an edge environment in a state where an object recognition model has been previously learned and stored.
[0123] The present invention described above can be implemented as computer-readable code on a medium having a program recorded thereon. Computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also include media implemented in the form of carrier waves (e.g., transmission via the Internet). Therefore, the above detailed description should not be construed as limiting in all respects, but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are intended to be included in the scope of the present invention.
Claims
1. Image acquisition unit for acquiring images; Memory for storing pre-trained neural network models for object recognition; A processor that detects an object from the image acquired through the image acquisition unit and acquires learning data for updating the neural network model among the detected objects according to predetermined criteria; The above processor, A surveillance camera device characterized in that the acquired learning data is fed forward to the neural network model to obtain image characteristic parameters corresponding to the learning data, and the image characteristic parameters are reflected in the neural network model to update the neural network model.
2. In paragraph 1, The above processor, A batch normalization layer is configured within the neural network model to normalize the image characteristic parameters, A surveillance camera device characterized in that the image characteristic parameters are updated by feeding forward the learning data for a plurality of layers included in the neural network model.
3. In paragraph 2, The above processor, A surveillance camera device characterized in that the image characteristic parameters are updated by updating the average and variance of the learning data and the average and variance between the plurality of layers.
4. In paragraph 1, The above predetermined criteria are, A surveillance camera device characterized in that it includes at least one of the size of the detection box of the object, the shape of the object, and the movement path of the object.
5. In paragraph 1, A surveillance camera device, characterized in that the above image characteristic parameters include at least one of edge variation, skewness, noise, illumination component, and reflectance component.
6. In paragraph 1, A surveillance camera device characterized in that the first image used for training the pre-trained neural network model and the second image used for updating the neural network model are images acquired through respective surveillance cameras installed in different locations.
7. In paragraph 6, A surveillance camera device, characterized in that at least one element of the image characteristic parameters of the first image and the second image have different characteristics.
8. Step of saving a neural network model for object re-recognition that has been learned in advance; A step of acquiring images from the above surveillance camera device; A step of detecting an object from the image and obtaining learning data for updating the neural network model among the detected objects according to predetermined criteria; and A step of obtaining image characteristic parameters corresponding to the learning data by feedforward inputting the acquired learning data into the neural network model, and updating the neural network model by reflecting the image characteristic parameters into the neural network model; A method for updating a neural network model for object re-recognition including .
9. In paragraph 8, The steps for updating the above neural network model are: A step of configuring a batch normalization layer for normalizing the image characteristic parameters within the neural network model; and A step of updating the image characteristic parameters by feeding forward the learning data for a plurality of layers included in the neural network model; A method for updating a neural network model for object re-recognition, characterized in that it includes:
10. In paragraph 9, The step of updating the above image specific parameters is: A method for updating a neural network model for object re-recognition, characterized in that the image characteristic parameters are updated by updating the average and variance of the learning data and the average and variance between the plurality of layers.
11. In paragraph 8, A step of transmitting the updated neural network model to the surveillance camera via a wireless communication unit; A method for updating a neural network model for object re-recognition, characterized in that it further includes.
12. In paragraph 8, A method for updating a neural network model for object re-recognition, characterized in that the first image used for training the pre-trained neural network model and the second image used for updating the neural network model are images acquired through respective surveillance cameras installed in different locations.
13. A step of training a neural network model for object re-recognition based on a first image acquired through a first surveillance camera installed at a first location; A step of applying the learned neural network model to a second surveillance camera to be installed at a second location to obtain a second image; A step of obtaining learning data for updating the neural network model based on an object detected from the second image; and A step of updating the neural network model based on the image characteristic parameters of the second image obtained by feedforward inputting the acquired learning data into the neural network model; A method for updating a neural network model for object re-recognition including .
14. In paragraph 12, The image characteristic parameters of the first image and the image characteristic parameters of the second image differ in at least one element, The above image characteristic parameters are, A method for updating a neural network model for object re-recognition, characterized in that it includes at least one of edge variation, skewness, noise, illumination component, and reflectance component.
15. In paragraph 12, A method for updating a neural network model for object re-recognition, characterized in that the first surveillance camera and the second surveillance camera are each installed at locations having different view points with respect to the same object.
16. In paragraph 12, Further comprising a step of performing object re-recognition in the second image for the first object recognized in the first image; A method for updating a neural network model for object re-recognition, characterized in that, as a result of the object re-recognition, if the same object is recognized as a different object or if different objects are recognized as the same object, a step of acquiring learning data for updating the neural network model is performed.
17. In paragraph 12, The step of updating the above neural network model is: After acquiring the second image, a step of performing object re-recognition using a neural network model for object re-recognition learned based on the first image; and A step of updating the neural network model when a predetermined performance is not obtained as a result of the object re-recognition operation; A method for updating a neural network model, characterized in that it includes:
Citation Information
Patent Citations
Aircraft nose gear steering actuator capable of absorbing shock from a worm shaft
KR1020230139029A
Adhesive for optical film, adhesive layer, optical member and display apparatus
KR102403279B1
KR20210020723A
KR20220138682A
KR20230010854A
