Updating annotation points in a digital image

Abstract

The invention relates to updating annotation points in digital images. A mechanism for updating coordinates of annotation points in digital images due to camera movement is provided. A method performed by an image processing device. The method comprises obtaining a current digital image of a scene. The current digital image has been captured by a camera after movement of the camera relative to the scene. The current digital image is associated with at least one annotation point. Each of the at least one annotation point has a respective coordinate in the current digital image. The method comprises identifying a movement amount by comparing location indication information in the current digital image with location indication information in a previous digital image of the scene. The previous digital image has been captured before the movement of the camera. The method comprises updating the coordinate of each of the at least one annotation point in accordance with the identified movement amount and a camera homography.

Description

Technical Field

[0001] The embodiments presented herein relate to methods, image processing apparatus, computer programs, and computer program products for updating the coordinates of annotation points in a digital image due to camera movement. Background Technology

[0002] Generally, surveillance cameras are video cameras used to observe an area. Surveillance cameras are typically connected to recording devices or networks (such as Internet Protocol (IP) networks). In some examples, security guards or law enforcement personnel monitor images recorded by surveillance cameras in real time. Cameras and other types of recording equipment used to be relatively expensive and required manual monitoring of camera footage; however, automated software has been developed to analyze recordings based on digital images captured by digital imaging cameras. Such automated software can be configured for digital image analysis and for organizing the captured digital video recordings into searchable databases.

[0003] In some surveillance scenarios, it may be necessary to capture scenes with a field of view (FoV) that includes one or more areas where privacy should be protected. Such areas of image content should therefore be excluded from any recording in the video surveillance system. In some surveillance scenarios where a monitoring area is defined, it may be necessary to capture scenes with a FoV wider than the monitoring area. Since only image content within the monitoring area needs to be analyzed, any image content outside the monitoring area should be excluded from any analysis in the video surveillance system (although image content outside the monitoring area does not need to be excluded from any recording in the video surveillance system). For both scenarios, a region of interest (ROI) can be defined as a portion of the image to be filtered or otherwise processed. Therefore, an ROI can be defined such that one or more areas where privacy should be protected are filtered out. An ROI can be defined by a binary mask. The binary mask is the same size as the image to be processed. In the binary mask, the pixel defining the ROI is set to 1, while all other pixels are set to 0 (or vice versa, depending on the purpose of the binary mask; to include or exclude image content within the ROI).

[0004] In other monitoring scenarios, a virtual tripwire that extends between two endpoints and defines a virtual tripwire can be added to a scene for crossline detection using a video surveillance system; whenever an object (such as a person, vehicle, or other type of object) crosses the virtual tripwire in the scene, an alarm or other type of indication will be issued.

[0005] One way to define the corner points and / or endpoints of virtual lines in an ROI is to use annotation points. Each annotation point has coordinates that are typically set manually or semi-automatically. However, once set, the coordinates of annotation points can be affected by camera position drift, FoV orientation shift, temperature changes, mechanical aging, camera drift due to vibration, etc. This can cause changes in the corner points and / or endpoints of virtual lines in the ROI, thus affecting the position of the privacy mask and / or the virtual lines, which are the parts of the monitored area that are actually under surveillance. Furthermore, when cameras are replaced, annotation points need to be manually remarked.

[0006] Therefore, when a camera undergoes camera movement (e.g., caused by any of the following: camera position drift, FoV orientation shift, temperature change, mechanical aging, camera replacement, camera drift due to vibration), improved processing of annotation points is required. Summary of the Invention

[0007] The purpose of this embodiment is to solve the above-mentioned problems by providing a method, image processing apparatus, computer program and computer program product for updating the coordinates of annotation points in a digital image due to camera movement (wherein, as described above, camera movement is caused by any of the following: camera position drift, FoV orientation shift, temperature change, mechanical aging, camera replacement, camera drift due to vibration).

[0008] According to a first aspect, a method for updating the coordinates of annotation points in a digital image due to camera movement is presented. The method is performed by an image processing apparatus. The method includes obtaining a current digital image of a scene. The current digital image was captured by a camera after the camera moved relative to the scene. The current digital image is associated with at least one annotation point. Each of the at least one annotation point has corresponding coordinates in the current digital image. The method includes identifying a movement amount by comparing position indication information in the current digital image with position indication information in a previous digital image of the scene. The previous digital image was captured before the camera moved. The method includes updating the coordinates of each of the at least one annotation point based on the identified movement amount and camera homography.

[0009] According to a second aspect, an image processing apparatus is presented for updating the coordinates of annotation points in a digital image due to camera movement. The image processing apparatus includes processing circuitry. The processing circuitry is configured to enable the image processing apparatus to obtain a current digital image of a scene. The current digital image was captured by a camera after the camera moved relative to the scene. The current digital image is associated with at least one annotation point. Each of the at least one annotation point has corresponding coordinates in the current digital image. The processing circuitry is configured to enable the image processing apparatus to identify the amount of movement by comparing position indication information in the current digital image with position indication information in a previous digital image of the scene. The previous digital image was captured before the camera moved. The processing circuitry is configured to enable the image processing apparatus to update the coordinates of each of the at least one annotation point based on the identified amount of movement and camera homography.

[0010] According to a third aspect, a video surveillance system is presented. This video surveillance system includes a camera and an image processing device according to a second aspect.

[0011] According to the fourth aspect, a computer program for updating the coordinates of annotation points in a digital image due to camera movement is presented, the computer program including computer program code that, when run on an image processing apparatus, causes the image processing apparatus to perform the method according to the first aspect.

[0012] According to a fifth aspect, a computer program product is presented, comprising a computer program according to a fourth aspect and a computer-readable storage medium storing the computer program thereon. The computer-readable storage medium may be a non-transitory computer-readable storage medium.

[0013] Advantageously, these aspects solve the aforementioned problems related to camera movement.

[0014] Advantageously, these aspects provide accurate updates to the coordinates of annotation points in digital images due to camera movement.

[0015] Advantageously, these aspects enable annotation points to be automatically repositioned or recalibrated when they shift due to camera movement.

[0016] Other objects, features, and advantages of the appended embodiments will become apparent from the following detailed disclosure, from the appended dependent claims, and from the accompanying drawings.

[0017] In general, all terms used in the claims should be interpreted according to their ordinary meaning in the art, unless otherwise expressly defined herein. All references to “a / the element, device, component, apparatus, module, step, etc.” should be interpreted as referring to at least one instance of an element, device, component, apparatus, module, step, etc., unless otherwise expressly stated. The steps of any method disclosed herein need not be performed in the exact order disclosed, unless expressly stated otherwise. Attached Figure Description

[0018] The inventive concept will now be described by way of example with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustration is a schematic diagram of a scenario according to an embodiment;

[0020] Figure 2 The illustration schematically depicts annotation points and location indication information in a digital image according to an embodiment;

[0021] Figure 3 , Figure 4 and Figure 5 This is a flowchart of the method according to an embodiment;

[0022] Figure 6 and Figure 7 The illustrations schematically depict example scenarios according to embodiments;

[0023] Figure 8 This is a schematic diagram illustrating the functional units of an image processing apparatus according to an embodiment; and

[0024] Figure 9 An example of a computer program product including a computer-readable storage medium according to an embodiment is shown. Detailed Implementation

[0025] The inventive concept will now be described more fully below with reference to the accompanying drawings, in which specific embodiments of the inventive concept are illustrated. However, the inventive concept can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Throughout the specification, the same numerals refer to the same elements. Any step or feature illustrated by dashed lines should be considered optional.

[0026] Figure 1 This is a schematic diagram illustrating a scenario in which the embodiments presented herein can be applied. Figure 1 The diagram illustrates a surveillance scene. Camera 130 is configured to capture a digital image of scene 100 within a FoV 140. Figure 1In the illustrative example, scenario 100 includes vehicle 110 and person 120. Camera 130 includes image processing device 150. Camera 130 and image processing device 150 can be part of a video surveillance system. Such a video surveillance system can be applied to any of the above monitoring scenarios.

[0027] As mentioned above, when a camera undergoes camera movement (e.g., caused by any of the following: camera position drift, FoV orientation shift, temperature change, mechanical aging, camera replacement, camera drift due to vibration), improved processing of annotation points is required.

[0028] Therefore, the embodiments disclosed herein relate to a mechanism for updating the coordinates of annotation points in a digital image due to camera movement. To obtain such a mechanism, an image processing apparatus 150, a method executed by the image processing apparatus 150, and a computer program product including code (e.g., in the form of a computer program) that, when the code is run on the image processing apparatus 150, causes the image processing apparatus 150 to execute the method.

[0029] The embodiments disclosed herein enable the automatic calibration of annotation points for feature links (in previous digital images) to (current digital images), addressing the aforementioned problems related to camera position drift, FoV orientation shift, temperature variations, mechanical aging, camera replacement, and camera movement caused by camera drift due to vibration. The embodiments disclosed herein update at least one coordinate of the annotation point based on detecting movement of camera 130 and by comparing the position indication information of the current digital image with the position indication information of a previous digital image of the same scene 100. Reference will now be made to... Figure 2 This needs to be explained. Figure 2 (a) schematically illustrates annotation points a1 and a2 in the previous digital image 200a, as well as positional indications of key points k1, k2, k3, k4, k5, and k6. Figure 2 (b) schematically illustrates the same annotation points a′1, a′2 and key points k′1, k′2, k′3, k′4, k′5, k′6 as in the previous digital image 200a, but in the current digital image 200b within the same scene as the previous digital image 200a. Figure 2 The image center and the radius r extending from the image center C are also marked at point C. Possible uses of parameters C and r will be disclosed below. Assume that camera movement occurred sometime between the capture of the previous digital image 200a and the capture of the current digital image 200b. Therefore, by... Figure 2 At (b) and Figure 2A comparison at (a) shows that the camera movement has caused the coordinates of annotation points a1 and a2 (and the coordinates of key points k1:k6) to shift.

[0030] Figure 3 This is a flowchart illustrating an embodiment of a method for updating the coordinates of annotation points a1, a2 in a digital image 200b due to camera movement. Camera movement may be due to the same camera 130 being moved, causing a change in FoV 140, or one camera 130 being replaced by another (which may also cause a change in FoV 140). These methods are performed by an image processing device 150. This method is advantageously provided as a computer program 920. In particular, the image processing device 150 is configured to perform steps S102, S104, and S106.

[0031] S102: The image processing device 150 obtains a current digital image 200b of scene 100. The current digital image 200b was captured by camera 130 after camera 130 moved relative to scene 100. The current digital image 200b is associated with at least one annotation point a′1, a′2. Each of the at least one annotation point a′1, a′2 has corresponding coordinates in the current digital image 200b.

[0032] S104: The image processing device 150 identifies the amount of movement by comparing the position indication information in the current digital image 200b with the position indication information in the previous digital image 200a of scene 100. The previous digital image 200a was captured (either by the same camera 130 or by another camera 130) before the camera 130 moved.

[0033] S106: The image processing device 150 updates the coordinates of at least one annotation point (from a1, a2 to a′1, a′2) based on the identified amount of movement and the camera homography (of the camera 130).

[0034] This method allows annotation points to be automatically repositioned or recalibrated when they shift due to camera movement.

[0035] An embodiment relating to further details of updating the coordinates of annotation points a′1, a′2 in digital image 200b due to camera movement, performed by image processing device 150, will now be disclosed.

[0036] The location indication information can be various examples. In some aspects, the location indication information in the current digital image 200b and the previous digital image 200a is defined by a corresponding set of keypoints k1:k6, k′1:k′6. Specifically, in some embodiments, the location indication information in the current digital image 200b is represented by a first set of keypoints k′1:k′6 extracted from the current digital image 200b, and the location indication information in the previous digital image 200a is represented by a second set of keypoints k1:k6 extracted from the previous digital image 200a.

[0037] In some aspects, the amount of camera movement is determined by the matching between keypoints k′1:k′6 in the current digital image 200b and keypoints k1:k6 in the previous digital image 200a. Specifically, in some embodiments, the comparison involves the positional matching of a first set of keypoints k′1:k′6 with a second set of keypoints k1:k6. The amount of movement is then identified based on the difference between the coordinates of the first set of keypoints k1:k6 in the current digital image 200b and the coordinates of the second set of keypoints k1:k6 in the previous digital image 200a.

[0038] Next, it will be disclosed how to select discriminative keypoints based on ranking their similarity. In some aspects, matching is performed by minimizing the distance between the feature vectors in the current digital image 200b and the feature vectors in the previous digital image 200a. Specifically, in some embodiments, a corresponding first feature vector is determined for each keypoint k1:k6 in the first set of keypoints k′1:k′6, and a corresponding second feature vector is determined for each keypoint k1:k6 in the second set of keypoints k1:k6. Then, the first set of keypoints k′1:k′6 is matched with the second set of keypoints k1:k6 by finding a pair of pairs of first feature vectors and second feature vectors that produce the minimum distance between the first and second feature vectors among all pairs of pairs of first feature vectors and second feature vectors.

[0039] The properties of the second set of keypoints k1:k6 will now be disclosed. In some embodiments, the second set of keypoints k1:k6 is a subset of all keypoints k1:k6 that can be extracted from the previous digital image 200a. How the selection of the most suitable keypoints can be performed will now be disclosed regarding how this subset of all keypoints k1:k6 is determined.

[0040] In some aspects, any keypoints k1:k6 with similar feature vector pairs are excluded from the second set of keypoints k1:k6. That is, in some embodiments, those keypoints k1:k6 that have a second feature vector most similar to the second feature vector of another keypoint k1:k6 extracted from the previous digital image 200a are excluded from the second set of keypoints k1:k6.

[0041] In some aspects, location information is used when determining which of all keypoints k1:k6 should be included in the second set of keypoints k1:k6. Specifically, in some embodiments, which of all keypoints k1:k6 will be included in the second set of keypoints k1:k6 depends on their position in the previous digital image 200a. In this respect, as will be disclosed below, the position may be relative to the image center and / or annotation points.

[0042] In some aspects, the keypoints among all keypoints k1:k6 that are closest to the image center are to be included as candidate points in the second set of keypoints k1:k6. Specifically, in some embodiments, the second set of keypoints k1:k6 is limited to including only those located at the center of the image at a distance of 200a from the previous digital image (in... Figure 2 The predefined radius (denoted as C) in the equation (in) Figure 2 The key points k1 and k6 within r are represented in the middle.

[0043] In some aspects, those keypoints among all keypoints k1:k6 that are closest to the location of annotation points a1, a2 are to be included as candidate points in the second set of keypoints k1:k6. Specifically, in some embodiments, the previous digital image 200a is associated with at least one annotation point a1, a2, wherein each of the at least one annotation point a1, a2 has corresponding coordinates in the previous digital image 200a, and the second set of keypoints k1:k6 is limited to include only keypoints k1:k6 located within a predefined radius of the coordinates of at least one annotation point a1, a2 in the previous digital image 200a.

[0044] Therefore, the above embodiments, aspects, and examples enable the selection of the most suitable key point based on the coordinates of at least one annotation point and the image center.

[0045] In some respects, the preceding digital image 200a is selected from a set of preceding digital images 200a in scene 100. An example of how the preceding digital image 200a is selected from this set of preceding digital images 200a in scene 100 will now be disclosed.

[0046] In some embodiments, each of the previous digital images 200a in the group of previous digital images 200a has its own timestamp, and which previous digital image 200a is selected is based on a comparison of that timestamp with the timestamp of the current digital image 200b. Therefore, a previous digital image 200a can be selected based on a timestamp comparison such that the previous digital image 200a and the current digital image 200b were captured at the same time of day, or are temporally closest to the current digital image 200b.

[0047] In a further example, the selection of which previous digital image 200a to use is based on lighting, image quality, etc. Combinations of such parameters are also envisioned. For example, to obtain the optimal image quality for optimizing keypoint matching, previous digital image 200a and current digital image 200b could be extracted, for example, within a specific time interval during daytime. Furthermore, similar timestamps relative to previous digital image 200a and current digital image 200b can produce the most similar image quality and lighting.

[0048] The aspects of how to update the coordinates of at least one annotation point a1, a2 will now be disclosed. In some embodiments, the coordinates of at least one annotation point a′1, a′2 are updated by applying a homography matrix to the coordinates of at least one annotation point a′1, a′2 in the current digital image 200b. The homography matrix depends on the amount of motion identified and the camera homography.

[0049] Further aspects of the possible actions taken by the digital image processing device 150 will now be disclosed. In some examples, an alarm is triggered if camera movement causes any annotation points a′1, a′2 to move out of the image. That is, in some embodiments, the digital image processing device 150 is configured to perform (optional) step S108.

[0050] S108: If updating the coordinates of at least one annotation point a′1, a′2 results in either of the at least one annotation point a′1, a′2 having coordinates outside the current digital image 200b, then issue a notification.

[0051] Therefore, if the area of ​​interest has been moved out of the FoV 140, an alarm can be triggered.

[0052] In some respects, the embodiments disclosed herein can be described in two phases: a feature extraction phase and a calibration phase. The feature extraction phase involves processing the previous digital image, while the calibration phase is applied after the camera has moved.

[0053] Now refer to Figure 4The flowchart is used to disclose various aspects of the feature extraction stage.

[0054] S201: Set at least one annotation point in the digital image.

[0055] Each such annotation point can represent a corner of the ROI or the endpoint of a virtual line.

[0056] S202: Search for key points in a digital image.

[0057] Keypoint detectors can be used to search for keypoints. Non-limiting examples of keypoint detectors include the Harris Corner Detector, the Scale Invariant Feature Transform (SIFT) detector, and descriptors based on Convolutional Neural Networks (CNNs). A keypoint detector identifies the coordinates of each keypoint and its associated descriptor (or feature vector). In some non-limiting examples, the descriptor is provided as a vector of 32 to 128 variables. The descriptor distinguishes each keypoint from the others and makes it identifiable from them. Without loss of generality, the total number of keypoints found is denoted as K.

[0058] S203: Select the most suitable key point (descriptor).

[0059] Step S203 can be implemented based on multiple sub-steps or subroutines.

[0060] M ≤ K discriminant keypoints are selected by ranking them based on their similarity. In one example, the similarity between two keypoint pairs is calculated based on the Euclidean distance between the descriptors (or feature vectors) of the two keypoints. If the Euclidean distance is less than a preset threshold, the pair of keypoints is defined as similar and is therefore discarded.

[0061] The M key points can be randomly divided into g subsets S. g Each such subset consists of N keypoints from the original M keypoints, where N << M. In some examples, at least 4 keypoints are selected, that is, N ≥ 4. In terms of coordinates, the N keypoints are selected as having the minimum distance from the image center and / or annotation point, as defined by the minimization criterion in (1):

[0062] min(S1, S2·, ..., Sg) (1)

[0063] Wherein, the weighted distance S k Defined as:

[0064]

[0065] In (2), in terms of coordinates, d(k) i a j) is each key point k in the digital image. i To each annotation point a i The Euclidean distance between them, and in terms of coordinates, d(k i C) represents each key point k i The Euclidean distance between the image center C and the image center C, and w k and w c These are their respective weights. Further, J represents the total number of annotation points. The weights can be used to prioritize keypoints that are closer to the annotation points or keypoints that are closer to the center of the image. The minimization criterion in (1) is used because, in terms of coordinates, the closer the distance between keypoints and annotation points in the digital image is as the camera 130 moves, the lower the probability that the keypoints will be lost from or even move out of the digital image.

[0066] Furthermore, for every two key points k in each subset i k j The distance between them can satisfy:

[0067]

[0068] The distance d(k) between every two key points i k j The sum of () is greater than the threshold D t This avoids the possibility of key points clustering together. This can be achieved through different techniques, such as the RANdom Sample Consensus (RANSAC) algorithm.

[0069] For example, such as Figure 2 As illustrated in the diagram, in terms of coordinates, compared to keypoints k5 and k6, keypoint k... 1- k4 has a smaller distance to the annotation point and the image center, and therefore, the key point k is selected. 1- K4 is discarded, while key points K5 and K6 are dropped.

[0070] In monitoring applications, the location of annotation points and the location of the image center are usually the most important locations.

[0071] S204: Stores the coordinates and descriptors (feature vectors) of the selected key points, as well as the coordinates of the annotation points.

[0072] It can also store camera settings (e.g., camera focal length) to enable camera view calibration or normalization, etc.

[0073] Now refer to Figure 5 The flowchart discloses aspects of the calibration phase (such as those applied after camera 130 has been moved or replaced).

[0074] S301: Obtain the coordinates of the annotation points and the key points (and their coordinates and descriptors) of a previous digital image of the same scene.

[0075] S302: Identify key points in the current image. How to identify key points in the current image has been described above. Figure 3 The description is public.

[0076] S303: Use the feature vectors determined from the key points k′1:k′6 in the current digital image 200b and the key points k1:k6 in the previous digital image 200a to identify matching key points.

[0077] S304: Determine the homography matrix from the matching key points.

[0078] In this regard, the homography matrix is ​​applied to account for camera movement between the previous and current digital images, in terms of camera rotation and / or translation. It is assumed that the digital images are normalized relative to each other. Normalization can be achieved using information about the camera focal length used when capturing the previous digital image and the camera focal length used when capturing the current digital image. In some non-limiting examples, the homography matrix H is calculated based on matched annotation points using single-valued decomposition (SVD). Further aspects of how the homography matrix is ​​determined will be disclosed below.

[0079] S305: The homography matrix is ​​applied to at least one annotation point in the current digital image.

[0080] Then, based on the coordinates (denoted as A) of at least one corresponding annotation point in the previous digital image, the updated coordinates (denoted as A′) of at least one annotation point in the current digital image are determined:

[0081] A′=H·A (2)

[0082] Where H represents the homography matrix.

[0083] Further aspects of how to determine the homography matrix H will now be disclosed.

[0084] Without loss of generality, assume there are four pairs of corresponding annotation points. Let a1, a2, a3, a4 represent annotation points in the previous digital image, and let a′1, a′2, a′3, a′4 represent annotation points in the current digital image. Each annotation point has a corresponding x-coordinate and a corresponding y-coordinate. Further assume that the focal length is normalized relative to both the previous and current digital images. Then, the homography matrix H can be calculated as follows.

[0085] First, determine the following 2×9 matrix:

[0086]

[0087] Here, indices i = 1, 2, 3, 4 correspond to four annotation points. Then, all these 2×9 matrices are stacked into a matrix P, which satisfies:

[0088] P·h=0 (4)

[0089] Expression (4) is therefore equivalent to:

[0090]

[0091] Further assume that matrix P has the following SVD decomposition:

[0092] P=U∑VT

[0093] Among them, column V corresponds to column P. T The eigenvectors of P are obtained and solutions for h are generated. The homography matrix H can be reshaped according to h. To improve the homography matrix H, a cost function can be applied by minimizing the estimation error. Therefore, given the homography matrix H, the projected coordinates from point a(x, y) in the previous digital image to point a′(x, y) in the current digital image are given by the following equation:

[0094] a′=H·a

[0095] In other words:

[0096]

[0097] Therefore, for i = 1, 2, 3, 4, we can insert x = x into expression (5). i y = y i 、x′x′ i and y′=y′ i To find the annotation point p i The coordinates of ′.

[0098] Figure 6 A first scenario in which embodiments disclosed herein may be applied is schematically illustrated. A previous digital image 600a of scene 610 captured by camera 130 of a video surveillance system is illustrated at (a). The scene includes building 620. For privacy reasons, windows 630a: 630f of the building are... Figure 6(a) (not shown) will be excluded from monitoring. Therefore, six different privacy masks 640a:640f are applied, each covering windows 630a:630f. The lower left corner of the privacy mask is defined by annotation points a1:a6. (b) illustrates the current digital image 600b of the scene captured by camera 130 after camera movement. As seen in (b), since annotation points a1:a6 have the same coordinates as in the previous digital image 600a, camera movement causes the privacy mask to move, thus partially exposing windows 630a:630f. (c) illustrates the application of the annotation point a′ disclosed herein for updating the current digital image due to camera movement. １ In at least some embodiments of the coordinates of a′6, the current digital image 600b' of the scene captured by camera 130 after the camera moves. It can be seen that the update causes the privacy masks to move back to their correct positions, thus properly covering windows 630a:630f again.

[0099] Figure 7 A second scenario in which embodiments disclosed herein can be applied is schematically illustrated. A previous digital image 700a of the scene captured by camera 130 of a video surveillance system is illustrated at (a). This scene includes building 710 and parking lot 720. The parking lot defines an Area of ​​Interest (ROI) 730 and is under surveillance. Therefore, the ROI is defined based on the corners of the parking lot. The corners of the ROI are defined by annotation points (represented by a single annotation point a1). A current digital image 700b of the scene captured by camera 130 after the camera moves is illustrated at (b). As seen at (b), since the annotation points (again represented by a single annotation point a1) have the same coordinates as in the previous digital image 700a, the camera movement causes the ROI 730 to move, thus partially excluding parts of the parking lot 720 from the surveillance. The illustration at (c) shows the current digital image 700b' of the scene captured by camera 130 after camera movement, when at least some of the embodiments disclosed herein for updating the coordinates of annotation points (represented by a single annotation point a′1) in the current digital image due to camera movement are applied. It can be seen that the update causes ROI 730 to move back to its correct position, thus properly covering parking lot 720 again.

[0100] Figure 8 The components of the image processing apparatus 150 according to an embodiment are schematically illustrated with respect to multiple functional units. These components are capable of executing functions stored in (e.g.) Figure 9The processing circuitry 810 may be provided by any combination of one or more of the following: a suitable central processing unit (CPU), a multiprocessor, a microcontroller, a digital signal processor (DSP), etc., in the software instructions of the computer program product 910 (e.g., in the form of storage medium 830). The processing circuitry 810 may further be provided as at least one application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0101] Specifically, the processing circuit 810 is configured to cause the image processing apparatus 150 to perform the set of operations or steps described above. For example, the storage medium 830 may store the set of operations, and the processing circuit 810 may be configured to retrieve the set of operations from the storage medium 830 to cause the image processing apparatus 150 to perform the set of operations. The set of operations may be provided as a set of executable instructions.

[0102] Therefore, the processing circuitry 810 is thus configured to perform the methods disclosed herein. The storage medium 830 may also include permanent storage, such as any one or a combination of magnetic storage, optical storage, solid-state storage, or even remotely mounted storage. The image processing apparatus 150 may further include a communication interface 820, configured at least to communicate with the camera 130, and potentially with other functions, nodes, entities, and / or devices (e.g., functions, nodes, entities, and / or devices of a video surveillance system). Thus, the communication interface 820 may include one or more transmitters and receivers comprising analog and digital components. The processing circuitry 810 controls the general operation of the image processing apparatus 150, for example, by sending data and control signals to the communication interface 820 and the storage medium 830, by receiving data and reports from the communication interface 820, and by retrieving data and instructions from the storage medium 830. Other components and related functions of the image processing apparatus 150 are omitted to avoid obscuring the concepts presented herein.

[0103] The image processing device 150 may be provided as a standalone device or as part of at least one other device. For example, the image processing device 150 and the camera 130 may be part of a video surveillance system.

[0104] A first portion of the instructions executed by the image processing device 150 may be executed in a first device, and a second portion of the instructions executed by the image processing device 150 may be executed in a second device; the embodiments disclosed herein are not limited to any particular number of devices on which the instructions executed by the image processing device 150 can be executed. Therefore, the method according to the embodiments disclosed herein is suitable for execution by the image processing device 150 residing in a cloud computing environment. Therefore, although... Figure 8The diagram illustrates a single processing circuit 810, but processing circuits 810 can be distributed across multiple devices or nodes. This also applies to… Figure 9 Computer program 920.

[0105] Figure 9 An example of a computer program product 910 including a computer-readable storage medium 930 is shown. On this computer-readable storage medium 930, a computer program 920 may be stored, which may cause processing circuitry 810 and entities and means operatively coupled thereto (such as a communication interface 820 and the storage medium 830) to perform methods according to embodiments described herein. Therefore, computer program 920 and / or computer program product 910 may provide means for performing any steps as disclosed herein.

[0106] exist Figure 9 In the example, computer program product 910 is illustrated as an optical disc such as a CD (compressed disc), DVD (Digital Multifunction Optical Disc), or Blu-ray disc. Computer program product 910 may also be embodied as a memory (such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM)), and more specifically as a non-volatile storage medium (such as a USB (Universal Serial Bus) memory or flash memory such as compact flash memory) in external memory. Therefore, although computer program 920 is schematically shown herein as a track on the depicted optical disc, computer program 920 may be stored in any manner suitable for computer program product 910.

[0107] The inventive concept has been described above with reference to several embodiments. However, as will be readily understood by those skilled in the art, other embodiments besides those disclosed above are also possible within the scope of the inventive concept as defined by the appended claims.

Claims

1. A method for updating coordinates of annotation points in a digital image due to camera movement, the method being performed by an image processing device, the method comprising: obtaining a current digital image of a scene, the current digital image having been captured by a camera after the camera has moved relative to the scene, wherein the current digital image is associated with at least two annotation points , ) defining a boundary of any one of the following: a privacy mask in the current digital image, a virtual warning line in the scene and a surveillance zone in the scene, and wherein each annotation point has a respective coordinate in the current digital image; identifying a movement amount by comparing position indication information in a current digital image with position indication information in a previous digital image of the scene, the previous digital image having been captured prior to the camera movement, wherein the position indication information in the current digital image is represented by a first set of key points extracted from the current digital image, wherein the position indication information in the previous digital image is represented by a second set of key points extracted from the previous digital image, wherein the second set of key points is a subset of all key points extractable from the previous digital image, wherein the previous digital image is associated with at least two annotation points in the previous digital image having respective coordinates, and wherein the second set of key points is limited to include only key points located within a predefined radius from the coordinates of the at least two annotation points in the previous digital image; and updating the coordinates of each annotation point according to the identified movement amount and a camera homography.

2. The method of claim 1, wherein, The comparison involves a position matching of the first set of key points with the second set of key points, and wherein the movement amount is identified from how much the coordinates of the key points of the first set of key points in the current digital image differ from the coordinates of the key points of the second set of key points in the previous digital image.

3. The method of claim 1, wherein, A respective first feature vector is determined for each key point in the first set of key points, and a respective second feature vector is determined for each key point in the second set of key points, and wherein the first set of key points is matched with the second set of key points by finding a pair of one of the first feature vectors and one of the second feature vectors that results in a smallest distance between the first feature vector and the second feature vector among all pairs of one of the first feature vectors and one of the second feature vectors.

4. The method of claim 1, wherein, Those of all key points having a second feature vector that is most similar to a second feature vector of another key point extracted from the previous digital image are excluded from the second set of key points.

5. The method of claim 1, wherein, Which of all key points is to be included in the second set of key points depends on its position in the previous digital image.

6. The method of claim 1, wherein, The second set of key points is limited to include only key points located within a predefined radius from a center of the previous digital image.

7. The method of claim 1, wherein, The previous digital image is selected from a set of previous digital images of the scene, wherein each of the previous digital images in the set of previous digital images has its own time stamp, and wherein which previous digital image is selected is based on a comparison of the time stamp with a time stamp of the current digital image.

8. The method of claim 1, wherein, The coordinates of each annotation point are updated by applying a homography matrix to the coordinates of each annotation point in the current digital image, wherein the homography matrix depends on the identified amount of movement and the camera homography.

9. The method of claim 1, wherein, The method further includes: A notification is issued if updating the coordinates of each annotation point results in any one of the at least two annotation points having coordinates outside the current digital image.

10. An image processing apparatus for updating the coordinates of annotation points in a digital image due to camera movement, the image processing apparatus comprising processing circuitry configured to cause the image processing apparatus to: obtaining a current digital image of a scene, the current digital image having been captured by a camera after movement of the camera relative to the scene, wherein, The current digital image is associated with at least two annotation points, wherein the at least two annotation points define the boundaries of any of the following: a privacy mask in the current digital image, a virtual warning line in the scene, and a monitoring area in the scene, and wherein each annotation point has corresponding coordinates in the current digital image; The amount of movement is identified by comparing position indication information in the current digital image with position indication information in a previous digital image of the scene, which was captured before the camera moved. The position indication information in the current digital image is represented by a first set of keypoints extracted from the current digital image, and the position indication information in the previous digital image is represented by a second set of keypoints extracted from the previous digital image. The second set of keypoints is a subset of all keypoints that can be extracted from the previous digital image, which is associated with at least two annotation points with corresponding coordinates in the previous digital image. The second set of keypoints is limited to include only keypoints located within a predefined radius of the coordinates of the at least two annotation points in the previous digital image. The coordinates of each annotation point are updated based on the identified movement and camera homography.

11. A non-transitory computer-readable recording medium comprising a computer program for updating the coordinates of annotation points in a digital image due to camera movement, the computer program comprising computer code that, when executed on processing circuitry of an image processing apparatus, causes the image processing apparatus to: obtaining a current digital image of a scene, the current digital image having been captured by a camera after movement of the camera relative to the scene, wherein, The current digital image is associated with at least two annotation points, wherein the at least two annotation points define the boundaries of any of the following: a privacy mask in the current digital image, a virtual warning line in the scene, and a monitoring area in the scene, and wherein each annotation point has corresponding coordinates in the current digital image; The amount of movement is identified by comparing position indication information in the current digital image with position indication information in a previous digital image of the scene, which was captured before the camera moved. The position indication information in the current digital image is represented by a first set of keypoints extracted from the current digital image, and the position indication information in the previous digital image is represented by a second set of keypoints extracted from the previous digital image. The second set of keypoints is a subset of all keypoints that can be extracted from the previous digital image, which is associated with at least two annotation points having corresponding coordinates in the previous digital image. The second set of keypoints is limited to include only keypoints located within a predefined radius of the coordinates of the at least two annotation points in the previous digital image. The coordinates of each annotation point are updated based on the identified movement and camera homography.

Images