Training of Neural Networks for Object Recognition

Through transfer learning and intelligent image filtering technology, the problem of poor adaptability of neural networks in specific scenarios is solved, efficient and automatic neural network training is achieved, and the accuracy and adaptability of object recognition is improved. It is suitable for scenarios in monitoring applications. Specific object recognition.

CN113935395BActive Publication Date: 2024-12-06AXIS
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202110678459.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-06-25
Filing Date
2021-06-18
Publication Date
2024-12-06
Estimated Expiration
2041-06-18

AI Technical Summary

Technical Problem

Existing object recognition neural networks are poorly adaptable in specific scenarios and require extensive pre-training for different environments and object appearances, which makes the training process complex and time-consuming, especially in monitoring applications, making it difficult to achieve scenario-specific optimization.

Method used

Transfer learning method is adopted to intelligently generate annotated image training data, filter images using a multi-part object recognition algorithm, and transfer learning of neural networks is carried out based on proven object recognition to adapt to object recognition in specific scenarios.

Benefits of technology

It realizes efficient and automatic pre-training of neural networks in specific scenarios, reduces processing requirements, improves the accuracy and adaptability of object recognition, and reduces the resource load on image capture devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113935395B_ABST
    Figure CN113935395B_ABST
Patent Text Reader

Abstract

The present invention provides training of an object recognition neural network. A method for transfer learning of an object recognition neural network is disclosed. The method includes: acquiring an image frame set; determining multiple object recognitions in the image frame set by a first object recognition algorithm implementing the object recognition neural network; determining a verified object recognition by evaluating the multiple object recognitions by a second object recognition algorithm different from the first object recognition algorithm, wherein the object recognition with a positive result in the evaluation forms the verified object recognition; forming a training set of annotated images including image frames associated with the verified object recognition; performing transfer learning of the object recognition neural network based on the training set of annotated images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of training of neural networks for object recognition. In particular, the present invention relates to transfer learning of object recognition neural networks used in image capture devices. Background Art

[0002] Neural networks are now commonly used for a variety of graphic analysis tasks. One task is object recognition, which includes object detection, object localization, object pose estimation, and object segmentation. Neural networks used for object recognition typically determine the probability of recognizing a certain object. The object recognition candidates are then filtered to extract object recognitions with high probability, as these are most likely to be correctly recognized. The value used for filtering is often referred to as a threshold. A lower threshold will result in more detections, including more false positives, and a higher threshold will result in fewer detections, with the possible negative effect of not detecting actual objects. The neural network outputs object recognitions based on the threshold filtering.

[0003] Training of neural networks for object recognition involves providing the neural network with annotated images depicting objects and scenes of varying appearance. In order to be well suited to different object recognition applications, existing neural networks are typically pre-trained using images selected to cover a wide spectrum of environments and objects. A disadvantage of such training is that when the neural network is set to operate on images of a specific scene, the neural network is not adapted to that scene. In particular for surveillance applications where object recognition plays an important role, scene-specific training of the neural network is desirable.

[0004] Training a neural network from scratch for a specific scene would be a cumbersome solution. Furthermore, it would require a lot of work to collect images from scenes that cover all variations of the scene due to weather or seasons, such as light changes, vegetation changes (in outdoor scenes), or other appearance changes. Therefore, it is desirable to find a less complex solution for implementing scene-specific training of object recognition neural networks. Summary of the invention

[0005] The object of the present invention is to achieve a method for providing scene-specific training for an object recognition neural network. The method should preferably be time-saving, require low processing power, and be easy to implement.

[0006] The inventors have realized that the adaptation of a pre-trained neural network for object recognition to a specific installation can be achieved in an efficient and automatic manner by using transfer learning with intelligently generated annotated images as training data. The intelligent generation of annotated images includes multi-part object recognition using different recognition algorithms for filtering images for specific scenes. These images are then annotated.

[0007] According to a first aspect, the present invention is a method for transfer learning of an object recognition neural network. The method comprises the following actions:

[0008] Get image frame set,

[0009] determining, by a first object recognition algorithm implementing an object recognition neural network, a plurality of object recognitions in the set of image frames, wherein each object recognition includes position data of the recognized object in an associated image frame,

[0010] determining a verified object recognition by evaluating a plurality of object recognitions by a second object recognition algorithm different from the first object recognition algorithm, wherein object recognitions having a positive result in the evaluation form the verified object recognition,

[0011] forming a training set of annotated images comprising image frames associated with verified object recognitions,

[0012] Perform transfer learning of object recognition neural networks based on a training set of annotated images.

[0013] Thus, the method comprises a first object recognition using a first object recognition algorithm performed using a neural network, followed by a second object recognition performed by a second, different object recognition algorithm. Images are sequentially filtered from a set of images by an algorithm, annotated, and used to perform transfer learning of the neural network. From the outset, the neural network typically has a general training that is able to handle a variety of environments and object appearances. By transfer learning using a training set of images filtered in the proposed method, the neural network becomes better at detecting objects in the scene it monitors. The method is intended to be performed using a set of images representing a scene that is monitored or to be monitored by an image capture device using an object recognition neural network. Preferably, the method is performed when the image capture device is installed in its intended location.

[0014] The neural network outputs an object recognition representing an object recognized in an image of the image set. The object recognition includes position data of the recognized object in an associated image frame. Thus, the object recognition includes position data of the object and an association with an image frame in which the object has been recognized. The position data may be provided as metadata to the image frame, thereby forming an association between the position data and the image frame. Position data refers to any form of data indicating the position of an object in an image frame provided by any type of neural network. The position data may be provided in the form of image coordinates.

[0015] When a training set of annotated images is formed based on the verified object recognition and the image frames associated therewith, an annotation process is applied so that the filtered images become annotated. The annotation process is generally known and can be performed in various ways. For example, the filtered images can be annotated by manual review and annotation or by annotation automation (both of which are conventional techniques in the art).

[0016] Transfer learning is therefore also a well-known process for training neural networks. Transfer learning refers to the further training of an already trained network with the purpose of changing the results, in this case adapting the object recognition mechanism in the neural network to the content of a specific scene. The architecture of the neural network remains the same. As an example, transfer learning can include changing some or all weights in the neural network. Transfer learning can alternatively be referred to as further training or incremental training.

[0017] Non-limiting examples of object recognition included in this article are object localization, object detection, object segmentation, and pose estimation. Non-limiting examples of neural network architectures that can be used are: R-CNN, YOLO (You Only Look Once), SSD (Single Shot Detector), Deeplab, U-Net, Mask R-CNN, PoseNet, OpenPose, DeepCut, and RMPE (Regional Multi-Person Pose Estimation).

[0018] In the first embodiment, the second object recognition is performed based on the movement characteristics of the recognized object. The object movement characteristics refer to the quality of the object movement, such as walking, running, waving of a human object, or driving of a vehicle object. The quality may be related to the movement mode, speed or direction of the object movement.

[0019] To this end, the step of determining the verified object recognition may include determining whether the movement of the associated object recognition between the image frames satisfies a predetermined movement condition. The movement condition may be based on the movement mode, movement direction or speed of the identified object. How to associate object recognition between image frames is also known and can be accomplished by using, for example, object embedding.

[0020] The determined movement pattern of the associated object recognition may be compared with one or more templates defining movement patterns, wherein a similarity with the one or more templates above a certain threshold corresponds to fulfillment of a predetermined movement condition.

[0021] The determined movement direction of the associated object recognition may be compared with one or a range of template directions, wherein a sufficiently small distance to one or a range of template directions corresponds to fulfillment of a predetermined movement condition.

[0022] The determined speed value of the associated object recognition can be compared with one or a range of template speed values, wherein a sufficiently small distance to one or a range of template speed values ​​corresponds to fulfillment of a predetermined movement condition.

[0023] The variations disclosed above relating to different types of movement conditions may be combined such that the predetermined movement condition is based on two or more types of movement characteristics.

[0024] In a variation of this embodiment, each object recognition includes an object class that is utilized in the second object recognition algorithm to set a predetermined movement condition. Specifically, the predetermined movement condition is selected based on the object class and may be different for different object classes. Using this method of filtering images, a high degree of false positives from the first object recognition algorithm may be filtered in an effective manner by the second object recognition algorithm.

[0025] In a second embodiment, a second object recognition is performed based on the position of the recognized object. The object position is given by the position data of the object recognition provided by the neural network. When determining the verified object recognition performed by the second algorithm, it can be determined whether the object recognition is located within a predetermined image area. There are different image areas for different object categories. Therefore, an image area can be selected as part of the object recognition based on the object category provided by the first algorithm.

[0026] In one embodiment of the method, the neural network is configured such that over-detection of objects is achieved. Over-detection refers to the neural network detecting objects that do not necessarily correspond to real objects in the scene. With this configuration, the first algorithm works as a first filter for the image set, the first filter filtering true positive identifications and a certain amount of false positive identifications. The second algorithm aims to find true positive identifications by filtering different parameters. As illustrated above, different parameters can be selected based on the information of the object identification output from the first algorithm (e.g., object location or object category).

[0027] The first object recognition algorithm can be implemented in an image capture device such as a surveillance camera. This is a beneficial approach because many images may need to be processed by the first object recognition algorithm to achieve the desired number of object recognitions for transfer learning. If the first object recognition algorithm is executed on a remote server, a large number of images will need to be transmitted to the remote server for no reason because most of them may not result in object recognition. This is especially true for scenes where the relevant objects rarely appear for the object detection model in use. Such scenes require long periods of image acquisition before the desired number of object recognitions are reached. Therefore, both network capacity and time can be saved by executing the first object recognition algorithm in image capture recognition.

[0028] In addition, the second object recognition algorithm can be implemented in the remote server. Thus, the object recognition can be determined in the image capture recognition and transmitted to the remote server for analysis by the second object recognition algorithm. This configuration balances the benefits of keeping network utilization low with the benefits of not overloading the processing capacity of the image capture device. Another advantage is that the second algorithm can be configured to include processing-demanding operations that are impossible to perform with the limited resources of the image capture device.

[0029] In order to cover the changes in lighting conditions within a day cycle, a set of image frames may be acquired in a determined time period, which may be at least 24 hours (i.e., a day cycle). Alternatively or in addition, another condition that may be applied is that image frames should be acquired until a predetermined number of object identifications are determined from the set of image frames.

[0030] According to one embodiment, the method includes the additional act of transferring the object recognition neural network after transfer learning to another image capture device monitoring the same scene or another scene of the same scene type.

[0031] According to a second aspect, the present invention is a method of training an object recognition neural network in an image capture device monitoring a scene. The method comprises:

[0032] · running comprises executing the installation phase of the method of the first aspect or its embodiments,

[0033] During the operational phase, monitoring the overall visual appearance of the scene by analyzing the acquired image frames, and

[0034] When a change in the overall visual appearance is detected, the installation phase is run again.

[0035] Therefore, an operating scheme for an image capture device is provided, which operating scheme includes two phases: an installation phase and an operation phase. As disclosed for the first aspect above, in the installation phase, a set of images is acquired to determine object recognition. In addition, according to the second aspect, a monitoring mechanism in the operation phase is provided, which monitoring mechanism is suitable for monitoring the overall visual appearance of the scene. When a change in the overall visual appearance of the scene is detected, the image capture device enters the installation phase again to perform another round of transfer learning. By triggering transfer learning when the visual appearance of the scene changes, the object detection neural network can be trained to handle different variations of the specific scene being monitored. This allows the object recognition neural network to become even more adapted to the specific scene, and it can perform even better.

[0036] According to a third aspect, the invention is a system comprising an image capture device comprising a first object identifier implementing an object recognition neural network and a remote server comprising a second object identifier. The image capture device is arranged to:

[0037] Get image frame set,

[0038] determining, by a first object identifier, a plurality of object identifications in the set of image frames, wherein each object identification comprises coordinates of an identified object in an associated image frame, and

[0039] Transmit multiple object-recognized and associated image frames to a remote server,

[0040] The remote server is arranged as:

[0041] determining a verified object identification by evaluating a plurality of object identifications in a second object identifier by a second object identification algorithm different from the first object identification algorithm, wherein object identifications with a positive result form a verified object identification, and

[0042] • Forming a training set of annotated images comprising image frames associated with verified object recognitions.

[0043] In one embodiment, the remote server is further arranged to initiate transfer learning of the object recognition neural network based on the training set of annotated images.

[0044] With regard to variations and advantages of the third aspect, reference is made to the discussion of the first aspect above. These also apply to this third aspect.

[0045] Further scope of applicability of the present invention will become apparent from the detailed description given below. However, it should be understood that the detailed description and specific embodiments indicating preferred embodiments of the present invention are given by way of illustration only, because from this detailed description, those skilled in the art will clearly understand various changes and modifications within the scope of the present invention.

[0046] Therefore, it should be understood that the present invention is not limited to the specific components of the described devices or the steps of the described methods, as such devices and methods may vary. It should also be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to be limiting. It must be noted that the articles "a", "an", "the" and "said" as used in the specification and the appended claims are intended to indicate the presence of one or more elements, unless the context clearly indicates otherwise. Thus, for example, a reference to "an object" or "the object" may include multiple objects, etc. In addition, the term "comprising" does not exclude other elements or steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will now be described in more detail by way of example and with reference to the accompanying drawings, in which:

[0048] Figure 1 This is a general overview of a method of transfer learning according to an embodiment.

[0049] Figure 2 The figure shows a system with an image capture device implementing a method according to an embodiment.

[0050] Figure 3 The diagram illustrates a method for having different operating modes for an image capture device according to an embodiment.

[0051] Figure 4 The figure illustrates a method according to an embodiment. DETAILED DESCRIPTION

[0052] An embodiment of a transfer learning process for an object recognition neural network will now be disclosed. Figure 1 Provide an overview of the process. Figure 1 The components shown in FIG. 1 are executed. The image frames are captured by the image acquisition 100. Specifically, the image frame set 101 is acquired. Then, the image frame set 101 is filtered by different object recognition algorithms in two steps. Filtering refers to selecting in the image set 101 based on filtering criteria. The object recognition algorithm applies different filtering criteria. Therefore, as shown by Figure 1 As indicated by the stepped pyramidal shape of the image frame set 101 in , the image frame set 101 is reduced through each filtering step.

[0053] The first of the two steps is performed by a first object recognition algorithm 102 that implements an object recognition neural network. Implementation refers to providing the image frame set 101 to the neural network to perform object recognition. The first object recognition algorithm 102 may include pre-processing or post-processing of object recognition by the neural network.

[0054] Non-limiting examples of the types of object recognition included in this article are object localization, object detection, object segmentation, and pose estimation. Object localization includes determining the location of an object; object detection includes determining the location of an object and the class of an object; object segmentation includes determining the area of ​​an object; and pose estimation includes determining the shape of an object, such as how to locate a person's arms and legs. How to perform any of these types of object recognition by a neural network is well known in the art. Therefore, the details of the object recognition process will not be discussed in this article.

[0055] The output in the form of object recognition 103 comes in different forms depending on the network architecture used in the first object recognition algorithm 102. The object recognition 103 may include, for example, bounding box representations, keypoint vectors, or coordinates of segments found by semantic segmentation.

[0056] The output object identifications 103 have in common that they include location data representing in some form the location of one or more identified objects of the associated image frame. The output object identifications 103 may also include the determined object class for each identified object.

[0057] The object identification 103 is input into the second of the two steps performed by the second object identification algorithm 104 along with the first subset of the image frame set 101. The object identification 103 and the associated image frames are evaluated using a second selection criterion that may be based on motion characteristics or object location. Object identifications evaluated with a positive result (i.e., satisfying the second selection criterion) become verified object identifications 105. Thus, the first subset of the image frame set 101 is again reduced to a second subset that is smaller than the first subset of the image frame set 110.

[0058] Going into more detail of the second object identification algorithm 104 , there are different embodiments for this algorithm. The differences between the embodiments are mainly in what selection criteria the object identification in the evaluation is based on. What the embodiments have in common is the type of object identification that is performed to verify the object identification 103 .

[0059] In a first embodiment, the object recognition of the second object recognition algorithm 104 is performed based on the movement characteristics of the identified object in the object recognition being evaluated. The object movement characteristics refer to the quality of the movement of an object such as walking, running, waving, or driving of a vehicle object. The quality may be related to the movement mode, speed, or direction of the object movement. For the purpose of determining the verified object recognition 105 based on the object movement characteristics, the second object recognition algorithm 104 may include determining whether the movement of the relevant object recognition between image frames meets the predetermined movement condition. Therefore, the second object recognition algorithm 104 may consider multiple image frames for determining the object movement characteristics of the object recognition. The multiple image frames may be continuous image sequences and include image frames of the object recognition analyzed by the second object recognition algorithm 104. The continuous image sequence may include image frames before or after the image frame of the object recognition, or include image frames before and after the image frame of the object recognition. In order to determine the movement characteristics, it may be necessary to associate the object recognition of different image frames. This can be done by, for example, object embedding, which is a conventional technique in the art. In other words, object embedding can be used to determine which object recognitions in different images are associated with each other.

[0060] The predetermined movement condition may be based on movement mode, movement direction or speed.

[0061] As a first example, the determined movement pattern of the associated object recognition may be compared to one or more templates defining movement patterns, wherein a similarity to the one or more templates above a certain threshold corresponds to satisfying a predetermined movement condition.

[0062] As a second example, the determined movement direction of the associated object recognition may be compared to one or a range of template directions, wherein a sufficiently small distance to one or a range of template directions corresponds to fulfilling a predetermined movement condition.

[0063] As a third example, the determined velocity value of the associated object recognition may be compared to one or a range of template velocity values, wherein a sufficiently small distance to one or a range of template velocity values ​​corresponds to fulfillment of a predetermined movement condition.

[0064] The variations disclosed above relating to different types of movement conditions may be combined such that the predetermined movement condition is based on two or more types of movement characteristics.

[0065] In a variation of this embodiment, each object recognition in object recognition 103 includes an object category (determined by an object recognition neural network) used in the second object recognition algorithm 104 to set a predetermined movement condition. Specifically, the predetermined movement condition is selected based on the object category and can be different for different object categories. For example, if the object category is "people", the predetermined movement condition can be a movement mode corresponding to walking or running movement. Alternatively, the predetermined movement condition can be a speed corresponding to the walking or running speed. Further, for the object category "vehicle", the predetermined movement condition can be a movement mode corresponding to a traveling car or a speed corresponding to a moving vehicle. Those skilled in the art can apply this concept to other object categories and movement condition types. Using this method of filtering images, a high degree of false positives from the first object recognition algorithm can be filtered in an effective manner by the second object recognition algorithm 104. This is because only objects that move according to the movement conditions expected for its object category will be considered true positives. In this embodiment, the neural network may be set with a low threshold in the first object recognition algorithm 102 to capture all true positives as well as multiple false positives, while the second object recognition algorithm 104 performs an effective second filtering based on the object class determined by the first object recognition algorithm 102 .

[0066] In a second embodiment, the second object recognition of the second object recognition algorithm 104 is performed based on the position of the recognized object. The object position is given by the position data of the object recognition provided by the object recognition neural network. The verified object recognition 105 can be determined by determining that the object recognition is located within a predetermined image area. For example, the predetermined image area can be set to an image area corresponding to the expected moving object of the scene or the area where the specific type of object of interest for training is expected to be found. The scene area can include a sidewalk, a road, a bicycle lane, an entrance / exit door, etc. The image area can be set manually in advance or set in an automated manner by, for example, detecting an area including a road, a door, etc. Alternatively, the image area can be determined by analyzing previous image frames of the same scene to determine where the object movement exists and setting the image area to those areas. The object movement of a specific object class of interest can be determined. The image area can be set based on the position of the determined object movement. There are different image areas for different object classes. Therefore, in one embodiment, each object recognition of the object recognition 103 can include the object class of the recognized object, and the predetermined image area utilized in the second object recognition algorithm 104 can be specific to the object class of the object recognition being evaluated. The shape, size and position of the image region may be determined manually or in an automated manner.

[0067] Back to Figure 1 As an overview, verified object recognitions 105 are input to a training set compiler 106 along with their associated image frames. The training set compiler 106 forms a training set of annotated images 107 from the input. Annotation can be performed manually or automatically, both of which are recognized ways of annotating images. The annotated images 107 are input to a trainer 108, the purpose of which is to transfer train a first object recognition algorithm as indicated by 109. Transfer learning is therefore a well-known technique for shaping or adjusting the training of a neural network. Therefore, the details of transfer learning will not be elaborated in this article.

[0068] By using multiple different object recognition algorithms to filter images, the process becomes robust and less sensitive to the configuration of the recognition sensitivity in the corresponding algorithms. Further, the process of transfer training also becomes more flexible because object recognition can be divided between different devices. The first object recognition algorithm 102 can be executed in one device (e.g., in an edge device (camera)), and the second object recognition algorithm 104 can be executed in a remote device (e.g., a remote server). Therefore, the process can utilize the processing resources of two devices. This is particularly beneficial for transfer learning methods, where it is important to collect images that preferably cover all types of objects that may exist in the scene for the training set. For this purpose and according to an embodiment, the object detection neural network is configured so that over-detection of objects is achieved. Over-detection means that in addition to the objects present in the scene, the neural network also detects objects that do not correspond to the real objects in the scene. Therefore, the neural network identifies non-objects in the scene as moving objects. Therefore, the object recognition output from the neural network includes both true positive recognition and false positive recognition. Over-detection of the neural network can be achieved by configuring it with a low threshold, which is a well-known configuration of different types of object detection neural networks. By means of the second object recognition algorithm 104, a verified object recognition is determined, which is a further restriction on true positive recognitions. The second object recognition algorithm 104 can be configured and fine-tuned to optimize its filtering. In the case where the second object recognition algorithm 104 is located on a remote server, it does not need to be connected to the edge device (camera) used to configure the first object recognition algorithm 102. In some implementations, establishing such a connection may be undesirable or even impossible.

[0069] Notice, Figure 1 It is not intended to illustrate any physical division between components. In other words, different components may be implemented in a single or different physical devices. Figure 2 Embodiments of implementations are disclosed.

[0070] The digital camera 20 monitors a scene 21 by means of an image sensor 22 (e.g., a conventional CMOS sensor). The scene 22 is typically an outdoor scene including an environment and objects that define its appearance. In this example, the scene 21 includes a moving person 21a (as indicated by the trajectory of the shadow) and a tree 21b. The image processor 23 of the camera 20 processes the captured image through different processing algorithms 23 (including a recognizer 23a). Other non-limiting examples include a white balancer 23b, an exposure controller 23c, and a noise filter 23d.

[0071] Identifier 23a performs the same Figure 1The first object recognition algorithm corresponding to the algorithm 102 of the camera 20 is implemented. That is, the recognizer 23a implements an object detection neural network to determine object recognition in the captured image of the scene 21. These object recognitions can be associated with their image frames by including them as metadata to the image frames. The association can be maintained through a subsequent encoding process performed by the encoder 24 of the camera 20.

[0072] The camera 20 is suitable for receiving a signal from the camera 20 (e.g., Figure 2 1 and 2 in the figure. In the first operating mode 1, the camera 20 (specifically the encoder 20) outputs encoded image frames with object identification as indicated by 25a. The output is received by a server 26 located remote from the camera 20. The server 26 includes a processor 26a arranged to determine a verified object identification by evaluating the received object identification and their associated image frames. The server 26 may include other processing components.

[0073] Thereafter, the verified object recognitions and their associated image frames form a training set of annotated images. This step may be performed within the server 26, for example by another component thereof or by a separate training device 27. The annotated images are used for transfer learning of the object recognition neural network implemented in the recognizer 23a. For this purpose, the server 26 or the trainer 27 may be adapted to be connected to the recognizer 23a of the camera 20 via a wired or wireless connection.

[0074] In the second operating mode 2, the camera 20 outputs the encoded image frame as indicated by 25b to a remote device 28, such as a recorder or streamer. The second operating mode 2 represents a mode in which the camera 20 operates as intended in use. That is, the camera 20 captures images / videos and transmits the images / videos to a receiver for recording and / or real-time viewing. The object detection neural network can be activated by determining object recognition that can be used for internal image processing within the camera 20 and / or for external analysis such as event analysis in a separate device. Therefore, object recognition can be transmitted along with the image stream of the camera 20 in the second operating mode 2. The object detection neural network is preferably configured with different thresholds in different operating modes. Specifically, when the camera 20 is set in the first operating mode 1, the object detection neural network is preferably set with a low threshold to achieve over-detection as discussed. Further, when the camera 20 is set in the second operating mode 2, the object detection neural network is preferably set with a higher threshold. In this operating mode, the object detection neural network is adjusted to a certain extent for a specific scene and should detect relevant targets with a high probability.

[0075] exist Figure 3The concept of two operating modes is further illustrated in FIG. Figure 3 The figure shows a method of switching between different operating modes. In the installation stage 301, the camera 20 is Figure 2 The camera operates in a first operating mode 1 of . That is, object recognition is determined within the camera and sent to the server 26 for evaluation, wherein a verified object recognition is determined. In order to cover changes in lighting conditions within a day cycle, a set of image frames can be acquired in a determined time period, which can be at least 24 hours (i.e. a day cycle). Another condition that can be applied is that image frames should be acquired until a predetermined number of object recognitions are determined from the set of image frames. The image frames can be continuously evaluated by the object recognition neural network, and when a predetermined number of object recognitions is reached, image acquisition can be ended and analysis of the object recognition by a second object recognition algorithm can be initiated. The predetermined number of object recognitions is preferably 1000 or more. The number of object recognitions required can depend on the type and size of the object detection neural network.

[0076] When the transfer learning of the object recognition neural network is completed in the installation phase 301 , the camera 20 enters the operation phase 302 .

[0077] In addition to the normal function of image acquisition of the camera 20 in the operation phase 302, a monitoring mechanism is provided. The purpose of the monitoring mechanism is to monitor the overall visual appearance of the scene 21. The monitoring of the overall visual appearance can be performed via image analysis or via measurement by a sensor (not shown) such as a light sensor. The monitoring mechanism can be performed by using components of the image processor 23. When a change in the overall visual appearance of the scene is detected, the camera 20 re-enters the installation phase 301 as indicated by 304 to perform another round of transfer learning. By triggering transfer learning when the visual appearance of the scene changes, the object detection neural network of the first object recognition algorithm 102 can be trained to handle different changes of the specific scene 21 being monitored.

[0078] Reasons for changes in overall visual appearance include weather and seasons. For example, the vegetation in an outdoor scene may be very different during the summer and during the winter, especially in areas with large seasonal changes. The number and type of objects in motion in the scene may also vary from season to season. For example, a dock is generally full of boats and people moving during the summer, but may be quite quiet during the rest of the year. Therefore, it is an advantage to train object recognition neural networks using images from these two periods to be able to perform well year-round. The same reasoning applies to weather changes, where the movement in the scene and its appearance may be different in different weather conditions (such as snow, rain, or sun). In indoor scenes, the reorganization of furniture may change the scene to the extent that further training is required.

[0079] As discussed, the purpose of transfer learning of an object recognition neural network is to adjust the configuration of the neural network so that it performs well on the images it processes. Other cameras with object recognition neural networks that monitor similar or even the same scenes can also benefit from transfer learning. Even though the scenes may not be exactly the same, but the scene types are the same, transfer learning can be an improvement over standard image training neural networks. For this purpose and according to one embodiment, the object recognition neural network of camera 20 can be migrated to another camera after transfer learning. Migrating an object recognition neural network refers to arranging a copy of the neural network for implementation in other image capture devices. Alternatively or in addition, the network can be migrated to a storage device (e.g., on a cloud server) to become a resource for future cameras that are set up to monitor the same scene or scene type. The same scene type in this article refers to scenes that have similarities in environment (outdoor / indoor), content (class of objects detected), or geographic location.

[0080] exist Figure 4 An overview of the method steps for transfer learning according to the embodiments discussed herein is provided in . First, a set of images is acquired S401 . Object recognition is determined S402 from the set of images, and image frames including object recognition as detected by the object recognition neural network are filtered. In the previous step S402 , a verified object recognition is determined according to the object recognition of step S402 . Therefore, a second image filtering is performed. In step S404 , image frames associated with the verified object recognition are annotated. Thereafter, a training set of annotated images is formed in step S405 . Transfer learning of the object recognition neural network is then performed using the annotated images. As discussed above, method steps S401 -S406 may be performed in the installation phase of the camera. As illustrated above, method steps S401 -S406 may be triggered to be performed again based on conditions such as a change in the overall scene appearance of the monitored scene. Therefore, an iteration step S407 is included, wherein the iteration step S407 may be triggered. Alternatively, the iteration step S407 may be configured to be performed on a regular basis (e.g., once every third month or year). As yet another alternative, the iteration step S407 may be triggered manually by an operator.Of course, it is also possible to combine triggering mechanisms in a single embodiment.

[0081] It should be understood that Figure 1 and Figure 2 The components illustrated and disclosed in the specification may be implemented as hardware, software, or a combination thereof.

[0082] In a hardware implementation of a component, the component may correspond to a circuit specifically and specially designed to provide the functionality of the part. The circuit may take the form of one or more integrated circuits such as one or more application specific integrated circuits or one or more field programmable gate arrays.

[0083] In a software implementation of the component, the circuitry may however take the form of a processor such as a microprocessor, the processor being associated with computer code instructions stored on a (non-transitory) computer readable medium such as a non-volatile memory, causing the image processor 23 to perform (parts of) any of the methods disclosed herein. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, and optical disks, etc. For example, in the software case, the identifier 23a may correspond to a portion of computer code instructions stored on a computer readable medium, which, when executed by the processor, causes the image processor 23 to perform the functionality of the component.

[0084] A person skilled in the art realizes that the present invention is by no means limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. For example, the disclosed concept of transfer training can be applied to cameras based on different types of imaging technology (e.g., thermal imaging cameras). It should also be noted that the second recognition algorithm can be performed on recorded object recognition (e.g., part of a recorded video stream) or on object recognition livestreamed from a camera. In a first variant, the recorded object recognition together with the associated image frames can be stored on a storage device such as a server. Reference Figure 2 , the storage device may be located between the camera 20 and the server 26 .

Claims

1. A computer-implemented method for providing scene-specific training to a pre-trained object recognition neural network, the method comprising: acquiring, by an image sensor of a digital camera, a set of image frames of a scene monitored by the digital camera, performing a first image filtering step on the set of image frames to filter a first subset of the set of image frames by means of a first object recognition algorithm, wherein the first subset comprises image frames associated with object recognitions determined by the first object recognition algorithm, and wherein each object recognition comprises position data and an object class of the recognized object in the associated image frame, performing a second image filtering step on the first subset of image frames to reduce the first subset of the set of image frames to a second subset of the set of image frames by means of a second object recognition algorithm, wherein the second subset of the set of image frames comprises image frames associated with verified object detections determined by the second object recognition algorithm by evaluating object recognitions in the first subset of image frames to determine whether movement of the associated object recognitions between image frames satisfies a predetermined movement condition that is specific to the object class of the object recognitions being evaluated, wherein the predetermined movement condition is selected from a plurality of predetermined movement conditions depending on the object class included in the object recognitions determined by the first object recognition algorithm, and wherein object recognitions having a positive result in the evaluation form verified object recognitions, forming a training set of annotated image frames from the verified object recognitions and their associated image frames, and Based on the training set of annotated image frames, scene specific training of the pre-trained object recognition neural network is provided.

2. The method of claim 1, wherein: The first object recognition algorithm is implemented in an image capture device.

3. The method of claim 2, wherein: The second object recognition algorithm is implemented in a remote server.

4. A computer-implemented method for providing scene-specific training to a pre-trained object recognition neural network in an image capture device monitoring a scene, the method comprising: The operation comprises performing an installation phase of the method according to claim 1, In an operational phase, the overall visual appearance of the scene is monitored by analyzing the acquired image frames, and When a change in the overall visual appearance is detected, the installation phase is run again.

5. The method of claim 4, wherein: The overall visual appearance is the lighting conditions or the visual appearance due to weather or season.

6. A system comprising a digital camera and a remote server, the digital camera comprising a first object identifier, the remote server comprising a second object identifier, wherein: The digital camera is arranged to: acquiring, by an image sensor of the digital camera, a set of image frames of a scene monitored by the digital camera, determining, by the first object identifier, a plurality of object identifications in the set of image frames by performing a first image filtering step on the set of image frames by means of a first object identification algorithm to filter a first subset of the set of image frames, wherein the first subset comprises image frames associated with object identifications determined by the first object identification algorithm, wherein each object identification comprises position data and an object category of the identified object in the associated image frame, and transmitting the plurality of object identified and associated image frames to the remote server, And wherein the remote server is arranged to: determining a verified object identification by performing a second image filtering step on the first subset of image frames by means of a second object identification algorithm of the second object identifier to reduce the first subset of the set of image frames to a second subset of the set of image frames, wherein the second subset of the set of image frames includes image frames associated with a verified object identification, the verified object identification being determined by the second object identification algorithm by evaluating object identifications in the first subset of image frames to determine whether movement of the associated object identifications between image frames satisfies a predetermined movement condition that is specific to the object class of the object identification being evaluated, wherein the predetermined movement condition is selected from a plurality of predetermined movement conditions depending on the object class included in the object identification determined by the first object identification algorithm, and wherein an object identification having a positive result in the evaluation forms a verified object identification, and A training set of annotated images is formed including image frames associated with the verified object identifications.

7. The system of claim 6, wherein: The remote server is further arranged to initiate scene specific training of a pre-trained object recognition neural network based on the training set of annotated images.

Citation Information

Patent Citations

  • Target object monitoring method, device and system

    CN109919009A

  • Monitor parameter update system, monitor parameter update method and program

    JP2019205034A

  • System and method for object recognition using neural networks

    WO2019141559A1