Method and image processing apparatus for determining a probability value indicating that an object captured in a stream of image frames belongs to an object type

Abstract

To provide a method and an image processing device for determining a probability value indicating that an object captured in a stream of image frames belongs to an object type.SOLUTION: A method comprises: detecting a first object or a first part of the first object in a first area of an image frame captured by a first camera of a multi-camera system; determining a first probability value indicating a first probability that the detected first object belongs to an object type; detecting a second object or a second part of the second object in a second area captured in a second stream of image frames by a second camera; determining a second probability value indicating a second probability that the detected second object belongs to the object type; and when the second probability value is below a second threshold value and the first probability value is above a first threshold value, determining an updated second probability value by increasing the second probability value.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Embodiments of the present specification relate to a method and an image processing device for determining a probability value indicating that an object captured in a stream of image frames belongs to an object type. A corresponding computer program and a computer program carrier are also disclosed.

Background Art

[0002] Public surveillance using imaging, particularly video imaging, is common in many regions around the world. Areas that may require surveillance are, for example, banks, stores, and other areas that require security such as schools and government facilities. However, it is illegal to install cameras in many places without a license / permit. Other areas that require surveillance are processing, manufacturing, and logistics applications, and video surveillance is mainly used to monitor the processing.

[0003] However, there may be a requirement that people cannot be identified from video surveillance. The requirement that people cannot be identified may be in contrast to the requirement that it is possible to determine what is happening in the video. For example, it may be important to perform a headcount or queue monitoring on anonymized image data. In practice, there is a trade-off between these two requirements, namely, satisfying non-identifiable video and extracting large amounts of data for different purposes such as headcount.

[0004] In order to avoid identifying people while being able to recognize activities, several image processing techniques have been described. For example, different types of "color blurring" such as edge detection / representation, edge enhancement, silhouetted objects, and color change or dilation are examples of such operations. Privacy masking is another image processing technique used in video surveillance to protect an individual's privacy by hiding a part of the image using a masked area.

[0005] Image processing refers to any process applied to an image. The process can include the application of various effects, masks, filters, etc. to the image. In this way, the image can be sharpened, converted to grayscale, or altered in some way. Images are typically captured by a video camera, a still image camera, etc.

[0006] As described above, one way to avoid identifying people is by masking moving people and objects within the image in real time. Masking in live video and recorded video can be done by comparing the live camera view to a set background scene and applying dynamic masking to areas of people and objects that change and are essentially moving. Color masking, which can also be called solid color masking or monochrome masking where an object is masked by a solid mask of a particular color, provides privacy protection while allowing movement to be seen. Mosaic masking, also known as pixelation, pixelated privacy masking, or transparent pixelation masking, shows moving objects at a low resolution and allows the form to be better distinguished by seeing the color of the object.

[0007] Masking of live video and recorded video is suitable for remote video surveillance or recording in areas where surveillance is a concern for privacy rules and regulations. When video surveillance is mainly used to monitor a process, it is ideal for processing, manufacturing, and logistics applications. Other potential uses are in retail, education, and government facilities.

[0008] Before masking an object, the object may need to be detected as an object to be masked, or in other words, may need to be classified as an object to be masked. Patent Document 1 discloses a method including acquiring a video of a scene, detecting an object in the scene, determining an object detection probability value indicating the possibility that the detected object belongs to an object category to be edited, and editing the video by obfuscating the detected object belonging to the object category to be edited.

[0009] A difficult problem seen when dynamic masking is used in a monitoring system is that detecting a partially occluded person gives a much lower detection score, such as a lower object detection probability value, compared to the detection score obtained for detecting an unoccluded person. The occlusion may be due to another object in the scene. The lower detection score may result in a partially occluded person not being masked in the video stream in which they are captured. For example, if half of an object (e.g., a person) captured in a video stream is occluded, the detection score obtained from the object detector may be 67% that the object is human, while if the captured object is unoccluded, the detection score may be 95%. If the threshold detection score for masking a person in the video stream is set, for example, at 80%, the partially occluded person will not be masked, but the unoccluded object will be masked. This is a problem because a partially occluded person should also be masked to avoid identification.

Prior Art Documents

Patent Documents

[0010]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0011] Accordingly, an object of embodiments of the present specification may be to prevent some of the above problems or at least reduce their impact. Specifically, an object of embodiments of the present specification may be how to detect captured objects in a stream of images blocked by other objects and how to classify them according to known object types. For example, an object of embodiments of the present specification may be to detect a human in a stream of video images even though the human is not fully visible in the stream of images. When an object is classified as a human, the classification may lead to human masking or human counting.

[0012] Accordingly, a further object of embodiments of the present specification may be to anonymize or pseudonymize a person in a stream of image frames, for example, by masking the person, while still being able to determine what is happening in the stream of image frames.

[0013] A further object may be to improve the determination of a probability value indicating whether an object captured in a stream of image frames belongs to an object type, for example, an object type to be masked, filtered, or counted. In other words, a further object may be to improve the determination of an object detection probability value indicating the likelihood that a detected object belongs to a particular object category. Means for Solving the Problems

[0014] According to one aspect, this object is achieved by a method for determining a probability value indicating whether an object captured in a stream of image frames belongs to an object type, which is executed in a multi-camera system.

[0015] The method comprises detecting a first object or a first portion of a first object within a first area of a scene captured within a first stream of image frames captured by a first camera of a multi-camera system.

[0016] The method comprises determining the probability that the detected first object or first portion belongs to the object type based on characteristics of the first object or the first portion of the first object

[0017] The method further comprises detecting a second object or a second portion of a second object within a second area of a scene captured within a second stream of image frames by a second camera of a camera system different from the first camera, wherein the second area at least partially overlaps with the first area.

[0018] The method further comprises determining a second probability value indicative of the probability that the detected second object or second portion belongs to the object type based on characteristics of the second object or the second portion of the second object.

[0019] The method further comprises determining an updated second probability value by increasing the second probability value when the second probability value is below a second threshold and the first probability value is above a first threshold.

[0020] According to another aspect, the above object is achieved by an image processing apparatus configured to execute the above method.

[0021] According to a further aspect, this object is achieved by a computer program and a computer program carrier corresponding to the above aspect.

[0022] Embodiments of the present specification use a probability value from another camera indicating that the detected first object or first portion belongs to the object type.

[0023] Since the second probability value is increased when the second probability value is less than the second threshold value and the first probability value is greater than the first threshold value, the second probability value can be compensated when the second object is shielded. By doing so, it is possible to compensate the second probability value for a specific object having a high first probability value from the first camera. This means that, despite the second object being partially shielded within the second camera, the number of second objects that are erroneously detected as the object type to be detected remains small, i.e., the number of false detections is small, and the probability of detecting that the second object belongs to a specific object type is increased.

Advantages of the Invention

[0024] Therefore, an advantage of the embodiments of the present specification is that the probability of detecting an object of a specific object type that is partially shielded within a specific camera is increased. Therefore, the detection of objects of a specific type or category can be improved. When the probability value indicates that an object captured in a stream of image frames belongs to an object type, the detection of the object can be determined. For example, when the second probability value indicates that the second object or the second part belongs to the object type to be detected, it is determined that the second object belongs to that object type. For example, a high second probability value can indicate that the second object or the second part belongs to the object type, and a low second probability value can indicate that the second object or the second part does not belong to the object type. The high probability value and the low probability value can be identified by one or more threshold values.

[0025] A further advantage is that the masking of objects of a specific type is improved. Yet another advantage is that the counting of objects of a specific type is improved.

[0026] Hereinafter, embodiments of the present invention will be described based on the drawings. Various aspects including specific features and advantages of the embodiments disclosed herein will be readily understood from the following detailed description and the accompanying drawings.

Brief Description of the Drawings

[0027]

Figure 1

Figure 2a

Figure 2b

Figure 3

Figure 4a

Figure 4b

Figure 4c

Figure 4d

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0028] The following will be described in detail.

[0029] Embodiments of this specification can be implemented in one or more image processing devices. In some embodiments of this specification, one or more image processing apparatuses may include, or may be, one or more image capturing devices such as digital cameras. FIG. 1 shows various exemplary image capture devices 110. The image capture device 110 can be, for example, a video camera 120 such as a camcorder, network video recorder, camera, surveillance camera or monitoring camera, a digital camera, a wireless communication device 130 such as a smartphone including an image sensor, or an automobile 140 including an image sensor, or can include these.

[0030] FIG. 2a shows an exemplary video network system 250 in which embodiments of this specification can be implemented. The video network system 250 may include an image capture device such as a video camera 120 that can capture a digital image 201 such as a digital video image and perform image processing thereon. The video server 260 in FIG. 2a may, for example, obtain an image from the video camera 120 via a network or the like, which is indicated by the bidirectional arrow in FIG. 2a.

[0031] The video server 260 is a dedicated computer-based device for distributing video. Video servers are used in many applications and often have additional features and capabilities to address the needs of specific applications. For example, video servers used in security, surveillance, and inspection applications are typically designed to capture video from one or more cameras and distribute the video via a computer network connection. In video production and broadcast applications, a video server may have the ability to record and play back recorded video and distribute multiple video streams simultaneously. Today, many video server functions are built into the video camera 120.

[0032] However, in FIG. 2a, the video server 260 is connected to an image capture device, here exemplified by the video camera 120, via the video network system 250. The video server 260 may be further connected to a video storage device 270 for storage of video images and / or may be connected to a monitor 280 for display of video images. In some embodiments, the video camera 120 is directly connected to the video storage 270 and / or the monitor 280 as shown by the direct arrows between these devices in FIG. 2a. In some other embodiments, the video camera 120 is connected to the video storage 270 and / or the monitor 280 via the video server 260 as shown by the arrows between the video server 260 and the other devices.

[0033] FIG. 2b shows a user device 295 connected to the video camera 120 via the video network system 250. The user device 295 may be, for example, a computer or a mobile phone. The user device 295 may, for example, control the video camera 120 and / or display video emitted from the video camera 120. The user device 295 may further include the functions of both the monitor 280 and the video storage 270.

[0034] To better understand the embodiments herein, an imaging system is first described.

[0035] FIG. 3 is a schematic diagram of an imaging system 300 that is a digital video camera such as a video camera 120 in this case. The imaging system captures a scene on an image sensor 301. The image sensor 301 may be provided with a Bayer filter so that different pixels receive radiation in a specific wavelength region in a known pattern. Typically, each pixel of the captured image is represented by one or more values representing the intensity of the captured light within a certain wavelength band. These values are usually referred to as color components or color channels. The term "image" may refer to an image frame or video frame containing information resulting from an image sensor that captured the image.

[0036] After reading the signals of the individual sensor pixels of the image sensor 301, different image processing actions can be performed by an image signal processor 302. The image signal processor 302 can include an image processing unit 302a, sometimes called an image processing pipeline, and a video post-processing unit 302b.

[0037] Typically, for video processing, the images are included in a stream of images. FIG. 3 shows a first video stream 310 from the image sensor 301. The first video stream 310 may include a plurality of captured image frames such as a first captured image frame 311 and a second captured image frame 312.

[0038] Image processing can include demosaicing, color correction, noise filtering (to remove spatial and / or temporal noise), distortion correction (e.g., to remove the effects of barrel distortion), global and / or local tone mapping (e.g., to enable imaging of scenes with a wide range of intensities), transformation (e.g., cropping and rotation), flat field correction (e.g., to remove the effects of vignetting), application of overlays (e.g., privacy masks, captions), etc. The image signal processor 302 can also be associated with an analysis engine that performs object detection, recognition, alarms, etc.

[0039] The image processing unit 302a may, for example, perform image stabilization and apply noise filtering, distortion correction, global and / or local tone mapping, conversion, and flat field correction. The video post-processing unit 302b may, for example, crop a part of the image, apply an overlay, and may include an analysis engine.

[0040] Following the image signal processor 302, the image is transferred to the encoder 303, and the information within the image frame is encoded according to an encoding protocol such as H.264. The encoded image frame is then transferred, for example, to a receiving client, exemplified here by the monitor 280, a video server 260, a storage device 270, etc.

[0041] The video encoding process generates several values that can be encoded to form a compressed bitstream. These values can include the following. · Quantized transform coefficients · Information to enable the decoder to recreate the prediction · Information regarding the structure of the compressed data and the compression tools used during encoding · - Information regarding the complete video sequence.

[0042] These values and parameters (syntax elements) are converted to a binary code using, for example, variable length coding and / or arithmetic coding. Each of these coding methods generates an efficient and compact binary representation of the information, also called the encoded bitstream. The encoded bitstream can then be stored and / or transmitted.

[0043] In a camera system having a plurality of cameras, embodiments herein propose to improve the masking quality in a camera by using detection information from surrounding cameras. For example, the masking quality is such that an object of a specific object type is partially blocked in one of the cameras, but the number of objects misdetected as the object type to be detected is still small, i.e., even when the number of false detections is small, it can be improved by increasing the probability of detecting an object of a specific object type.

[0044] The detection information may include a detection score or a detection probability. Different cameras may have different preconditions for detecting people due to camera resolution, the installation positions of different cameras, and their processing capabilities, which affect the characteristics of the video network used with the cameras. An example of these characteristics is the size of the images input to the video network. A person who is slightly blocked within the image frame of one camera may be fully visible within the image frame of another camera. A person with a pixel density too low to be detected by one camera may be detectable by another camera with a higher pixel density of the person.

[0045] · The camera system may need to understand the spatial overlap between cameras. For example, the camera system may be presented with or determine the overlap of the areas captured by the cameras that will share the detection information. This can be done via known methods of scene segmentation using information such as geolocation, pan and tilt angles, zoom, field of view, and installation height. · An image processing device such as a camera collects detection scores from surrounding cameras for objects within the area captured by two or more cameras. This area may sometimes be referred to as a spatially overlapping area. · The image processing device appropriately adjusts the detection score based on the detection scores from at least one other camera. ·The image processing device can apply a privacy mask to the detected object based on the adjusted detection score. This will be described in more detail below, for example, in relation to action 503 in FIG. 5.

[0046] Here, referring to FIGS. 4a, 4b, 4c, 4d, and 5, and further referring to FIGS. 1, 2a, 2b, and 3, exemplary embodiments of the present specification will be described.

[0047] FIG. 4a shows a scene including an object such as a person blocked by another object such as a wall and a multi-camera system 400. The multi-camera system 400 includes at least a first camera 401 and a second camera 402. The first camera 401 and the second camera 402 may be video cameras. The multi-camera system 400 may also include a video server 460. The first camera 401 and the second camera 402 can capture the subject from different angles.

[0048] In a scenario where an embodiment can be implemented, the first camera 401 can capture most of the entire person, while the second camera 402 can capture only a smaller portion of the person, such as the person's head and arms. Each of the first camera 401 and the second camera 402 can assign a detection score, for example, a probability value indicating that the detected object belongs to an object type such as a human.

[0049] FIG. 4b shows a first stream of image frames 421 captured by a first camera 401 of a multi-camera system 400 and a second stream of image frames 422 captured by a second camera 402 of the camera system 400. The second camera 402 is different from the first camera 401. The first stream 421 of image frames includes an image frame 421_2 that captures a first object 431. The first object 431 can include a first portion 431a. The second stream 422 of image frames includes an image frame 422_2 that captures a second object 432. The second object 432 can include a second portion 432a. In the scenario of this specification, the first object 431 may correspond to the second object 432 in that they are the same real object captured. Also, the first portion 431a may correspond to the second portion 432a.

[0050] FIG. 5 shows a flowchart explaining a method for determining a probability value indicating the probability that an object captured in a stream of image frames 422 belongs to an object type such as a specific object type. For example, the object type may be a human or an animal. In other words, the method may be for determining the probability that an object belongs to an object type.

[0051] This method may be executed in the camera system 400, and more specifically, may be for masking or counting an object captured in a stream of image frames. Masking or counting of the object may be performed when a probability value indicating that an object captured within the stream of image frames 422 belongs to the object type exceeds a threshold value for masking the object or a threshold value for counting the object. Also, the threshold value for masking the object and the threshold value for counting the object may be different.

[0052] In particular, the embodiments may be performed by an image processing device of the multi-camera system 400. The image processing apparatus may be either the first camera 401 and the second camera 402 or the video server 460. The first camera 401 and the second camera 402 may each be a video camera such as a surveillance camera.

[0053] The following actions may be performed in any suitable order, for example, in an order different from the order presented below.

[0054] Action 501 The method comprises detecting a first object 431 or a first portion 431a of the first object 431 within a first area 441 of a scene captured within a first stream 421 of image frames captured by the first camera 401 of the multi-camera system 400.

[0055] Action 502 The method further includes determining a first probability value indicative of a first probability that the detected first object 431 or first portion 431a belongs to an object type such as a human, based on characteristics of the first object 431 or the first portion 431a of the first object 431. In other words, the method includes determining a first probability that the first object belongs to an object type.

[0056] For example, the determination may be performed by weighting probability values for some parts of the first object 431. Each probability value indicating the probability that a detected portion 431a of the first object 431 belongs to an object part type may be determined based on characteristics of the detected portion 431a of the first object 431.

[0057] Action 503 If the first probability value exceeds the first threshold, the detected first object 431 or first part 431a may be determined to belong to an object type such as a human. Further, if the first probability value exceeds the first threshold, some further action may be performed. For example, at least a part of the first object 431, such as the first object 431 or the first part 431a of the first object 431 within the first stream 421 of the image frame, may be masked out. For example, a privacy mask, such as a masking filter or a masking overlay, may be applied to at least a part of the first object 431, such as the first object 431 or the first part 431a of the first object 431 within the first stream 421 of the image frame, in order to hide it. In particular, the privacy mask may be obtained by masking regions within each image frame, and may include color masking, also called solid color masking or monochrome masking, mosaic masking, also called pixelation, pixelated privacy masking or transparent pixelation, Gaussian blur, Sobel filter, background model masking by applying the background as a mask (the object appears transparent), or chameleon mask (a mask that changes color according to the background). As a specific example, a color mask can be obtained by ignoring the image data and using other pixel values for the pixels to be masked. For example, the other pixel values may correspond to a specific color such as red or gray.

[0058] In another embodiment, if the first probability value exceeds the first threshold, the object may be counted as a specific predetermined object. Thus, the first threshold may be a masking threshold, a counting threshold, or some other threshold related to some other function performed on the object as a result of the first probability value exceeding the first threshold.

[0059] Action 504 The method further includes detecting a second object 432 or a second portion 432a of the second object 432 within a second area 442 of a scene captured within a second stream 422 of image frames by a second camera 402 of the camera system 400. The second camera 402 is different from the first camera 401. The second area 442 at least partially overlaps with the first area 441.

[0060] Action 505 The method further includes determining a second probability value indicative of a second probability that the detected second object 432 or second portion 432a belongs to an object type, based on characteristics of the second object 432 or the second portion 432a of the second object 432. In other words, the method includes determining a second probability that the second object belongs to an object type.

[0061] If the second probability value exceeds a second threshold, the detected second object 432 or second portion 432a may be determined to belong to an object type such as a human. In other words, the detected second object 432 or second portion 432a may be detected as a human or as something belonging to a human.

[0062] If the second probability value is below the second threshold, the second object 432 may be determined not to belong to the object type. However, in embodiments herein, the method continues to evaluate whether the second object 432 belongs to the object type even when the second probability value is below the second threshold, by considering the first probability value from the first camera 401.

[0063] Action 506 Yet another condition for continued evaluation of the detection of the second object 432 based on the first probability value may be a determination of the collocation of the first object 431 and the second object 432, to ensure that the first object 431 and the second object 432 are the same object, and thus it is appropriate to consider the first probability value when evaluating the detection of the second object 432.

[0064] Thus, the method may further comprise determining that a second object 432 or a second portion 432a of the second object 432 and a first object 431 or a first portion 431a of the first object 431 are collocated within an overlapping area of a second area 442 and a first area 441.

[0065] In other words, the method may further include determining that the first object 431 and the second object 432 are the same object.

[0066] For example, it may be determined that a first person in a first stream 421 of an image frame is located at the same location as a second person in a second stream 422 of the image frame. Based on the determination of collocation, it may be determined that the first person and the second person are the same person. For example, if it is determined that the first object 431 is a person with a 90% probability and the second object 432 is further determined to be a person with a 70% probability, and there are no other objects detected within the image frame, it may be determined that the persons are likely to be the same person.

[0067] In other words, for example, when a matching level is given based on a set of constraints, it may be determined that the motion track entries associated with the first object 431 and the second object 432 are linked (i.e., determined to belong to one real-world object), thereby determining that the first object 431 and the second object 432 are linked. The track entries may be created based on the received first and second image series, the determined first and second feature description feature sets, and the movement data.

[0068] In another example, it may be determined that the second object 432 is placed at the same location as the first portion 431a of the first object 431.

[0069] In another example, it may be determined that the second part 432a of the second object 432 is disposed at the same location as the first part of the first object 431. For example, it may be determined that the head of the first object is at the same position as the head of the second object. This method may be similarly applied to other parts such as the arms, torso, and legs. Operation 506 may be performed after or before operation 507.

[0070] Action 507 If the second probability value is lower than the second threshold and the first probability value is higher than the first threshold, the method includes determining an updated second probability value by increasing the second probability value.

[0071] The second threshold may be a masking threshold, a counting threshold, or some other threshold associated with some other function to be performed on the second object 432 or the second part 432a as a result of the second probability value exceeding the second threshold.

[0072] The second threshold and the first threshold may be different. However, in some embodiments herein, they are the same. FIG. 4c shows a simple example of how the second probability value can increase from a value below the second threshold to a value above the second threshold when the first threshold and the second threshold are the same and the first probability value exceeds the first threshold.

[0073] In some embodiments herein, the first threshold is 90% and the second threshold is 70%. The second threshold may be lower than the first threshold. This may be advantageous, for example, when the second object 432 is partially blocked by some other object such as a wall.

[0074] In one scenario, the first camera 401 determines that the probability that the first object is a human is 95%. The second camera 402 determines that the probability that the second object is a human is 68%. The second probability value may then be increased by, for example, 5 or 10%. The increase may be a fixed value as long as the first probability value exceeds the first threshold. However, in some embodiments of this specification, the increase in the second probability value is based on the first probability value. For example, the increase in the second probability value may be proportional to the first probability value. In some embodiments of this specification, when the first probability value is 60%, the increase in the second probability value is 5%, when the first probability value is 70%, the increase in the second probability value is 10%, and when the first probability value is 80%, the increase in the second probability value is 15%.

[0075] Increasing the second probability value based on the first probability value may include determining the difference between the first probability value and the first threshold and increasing the second probability value based on that difference. For example, if the difference between the first probability value and the first threshold is 5%, the second probability value may be increased by 10%. In another example, if the difference between the first probability value and the first threshold is 10%, the second probability value may be increased by 20%.

[0076] In some other embodiments, increasing the second probability value based on the first probability value includes increasing the second probability value along with the difference. For example, if the difference between the first probability value and the first threshold is 5%, the second probability value may be increased by 5%. In another example, if the first probability value is 95% and a common threshold such as a common masking threshold is 80%, the difference is 15%, and thus, for example, a second probability value of 67% may be increased by 15%, resulting in 82% which exceeds the common threshold of 80%. As a result, the second object may also be masked or counted.

[0077] In some embodiments of the present specification, the updated second probability value is determined in response to determining that the second object 432 or a part 432a of the second object 432 and the first object 431 or a part 431a of the first object 431 are located at the same location within the overlapping area according to the above action 506.

[0078] In some embodiments of the present specification, each of the first and second thresholds is specific to the type of the object or a part of the object. For example, each of the first and second thresholds may be specific to different object types or different object part types such as the head, arms, torso, legs, etc., or both. Since it is likely to be more important to mask the face than the arms, the face masking threshold may be lower than another masking threshold for the arms. For example, the masking threshold for the head may be 70%, while the masking threshold for the arms may be 90%, and the masking threshold for the torso may be 80%.

[0079] A further condition for determining the updated second probability value by increasing the second probability value may be that the second probability value exceeds a third threshold in addition to being below the second threshold. Thus, it may be an additional condition that the second probability value exceeds the lower third threshold. The reason why it is advantageous for the second probability value to be higher than the lower third threshold is that when the second probability value is lower than the third threshold, for example, less than 20%, the object detector is quite certain that the object is not an object to be masked, for example, not a human. However, when the second probability value is between 20% and 80%, the object may be a masked person, and thus, if another camera is more certain in detecting the object as an object to be masked, the probability value may be increased.

[0080] In some embodiments of the present specification, the second object part type of the second part 432a is the same object part type as the first object part type of the first part 431a. For example, the method can compare a first face with a second face, a first leg with a second leg, and so on.

[0081] It may be advantageous to increase the second probability value only when the difference between the first probability value and the first threshold exceeds a fourth threshold, for example, when the difference exceeds 10%. In this way, the number of false positives can be controlled. For example, the larger the threshold difference, the fewer the number of false positives can be.

[0082] For example, a scenario where the first threshold is 70%, the second threshold is 90%, and the fourth threshold is 75%. A first probability value of 72% may be large enough to detect the first object as a human in the first stream 421 of the image frame, but may be considered too low to increase the second probability value.

[0083] Action 508 In some embodiments of this specification, in response to the updated second probability value exceeding the second threshold and in response to determining that the second object belongs to a certain object type, further actions may be taken. For example, if the object type is an object type to be masked out, the first and second thresholds may be for masking an object or a part of an object of the object type to be masked out. Then, the method further includes applying a privacy mask to at least a part of the second object 432, such as the second part 432a in the second stream 422 of the image frame, to mask out that part when the updated second probability value exceeds the second threshold. For example, if a human face is captured by both the first camera 401 and the second camera 402 and the face score is updated for the second camera 402, this can lead to privacy masking of the face in the second image stream 422. Thus, the method can further include anonymizing an unidentified person, for example, by removing identifying features by any of the methods described above in action 503.

[0084] In some other embodiments, the object type is an object type to be counted. In this case, the first and second thresholds may be for counting an object or a part of an object of the object type to be counted. Then, the method further includes incrementing a counter value associated with the second stream of the image frame 422 when the updated second probability value exceeds the second threshold. For example, the counter value may be for an object or an object part.

[0085] FIG. 4d shows a scenario where the scene includes a first human 451 and a second human 452, and a third object 453 between the two humans 451, 452. The shape of the third object 453 resembles the shape of the body parts of the humans 451, 452. The third object may be, for example, a balloon resembling a head. In one example, the second camera 402 captures the three objects 451, 452, 453, and the method of FIG. 5 can be used for each of the three objects 451, 452, 453. Thus, each of the three objects 451, 452, 453 in FIG. 4d can be the second object 432 in the second stream 422 of the image frame. Correspondingly, each of the three objects in FIG. 4d may be the first object 431 in the first stream 421 of the image frame.

[0086] The method can be executed by the second camera, or by the video server 460, or even by the first camera 401.

[0087] This method can include determining a second probability value. The second probability values can be 69%, 70%, and 80% respectively for the first human 451 (left), the third object 453 (balloon), and the second human 452 (right). In one scenario, the second threshold for detecting and masking a human is 80%. This means that without the method of FIG. 5, only the second human 452 would be masked. If a general decrease of 10% is performed on the second threshold, the third object 453 is masked, but the first human 451 is not masked. However, for each first object 431 such as the first human 451 and the second human 452, if the first probability value derived from the first stream 421 of the image frame is higher than the first threshold and the second probability value is lower than the second threshold, the second probability value is increased. Thus, in a state where the first probability value is 90% and the detected first human 451 belongs to the human type, when the first human 451 is detected within the first stream 421 of the image frame from the first camera 401, the second probability value is increased by, for example, 15%. The updated second probability value becomes 84% which exceeds the second threshold, and the first human 451 is masked. The first probability value of the detected third object 453 indicating the probability that the detected third object 453 belongs to the human object type is low, for example 18%. This means that since the first probability value of the third object 453 is below the first threshold, the second probability value is not increased. Thus, the third object 453 is not masked in either stream of images from the first camera 401 and the second camera 402.

[0088] Referring to FIG. 6, a schematic block diagram of an embodiment of the image processing apparatus 600 is shown. As described above, the image processing apparatus 600 is configured to determine a probability value indicating that an object captured within a stream of image frames belongs to an object type. Further, the image processing apparatus 600 can be part of the multi - camera system 400.

[0089] As described above, the image processing apparatus 600 can include any one of, or be any one of, cameras such as surveillance cameras, surveillance cameras, camcorders, network video recorders, and the wireless communication device 130. In particular, the image processing apparatus 600 can be a first camera 401 or a second camera 402 such as a surveillance camera, or a video server 460 that can be part of the multi-camera system 400. The method for determining a probability value indicating that an object captured in the stream of image frames belongs to an object type can also be executed in a distributed manner within several image processing devices such as within the first camera 401 and the second camera 402. For example, actions 501 to 503 may be executed by the first camera 401, while actions 504 to 508 may be executed by the second camera 402.

[0090] The image processing apparatus 600 can further include a processing module 601 such as means for executing the method described herein. The means may be embodied in the form of one or more hardware modules and / or one or more software modules.

[0091] The image processing apparatus 600 may further have a memory 602. The memory can include, for example, instructions in the form of a computing program 603, or can accommodate or store, for example, instructions that, when executed on the image processing apparatus 600, cause the image processing apparatus 600 to execute a method for determining a probability value indicating that an object captured in the stream of image frames belongs to an object type, and can include a computer-readable code unit. The image processing apparatus 600 can include a computer, in which case the computer-readable code unit can be executed on the computer to cause the computer to execute a method for determining a probability value indicating that an object captured in the stream of image frames belongs to an object type.

[0092] According to some embodiments of this specification, the image processing apparatus 600 and / or the processing module 601 includes a processing circuit 604 as an exemplary hardware module that may include one or more processors. Thus, the processing module 601 may be embodied in the form of the processing circuit 604 or may be "implemented by" the processing circuit 120. Instructions are executable by the processing circuit 604, whereby the image processing device 600 operates to execute the method of FIG. 5 as described above. As another example, when the instructions are executed by the image processing apparatus 600 and / or the processing circuit 604, the image processing apparatus 600 can be made to execute the method according to FIG. 5.

[0093] In view of the above, in one example, an image processing device 600 is provided for determining a probability value indicating that an object captured within a stream of image frames belongs to an object type.

[0094] Also in this case, the memory 602 includes instructions executable by the processing circuit 604, whereby the image processing apparatus 600 operates to execute the method according to FIG. 5.

[0095] FIG. 6 further shows a carrier 605 including the computing program 603 described immediately above, that is, a program carrier. The carrier 605 can be one of an electronic signal, an optical signal, a wireless signal, and a computer-readable medium.

[0096] In some embodiments, the image processing device 600 and / or the processing module 601 can include one or more of a detection module 610, a determination module 620, a masking module 630, and a counting module 640 as exemplary hardware modules. In other examples, one or more of the above-described exemplary hardware modules can be implemented as one or more software modules.

[0097] Furthermore, the processing module 601 may include an input / output unit 606. According to one embodiment, the input / output unit 606 may include an image sensor configured to capture the above-described raw image frames, such as the raw image frames included in the video stream 310 from the image sensor 301.

[0098] According to the various embodiments described above, the image processing device 600 and / or the processing module 601 and / or the detection module 610 are configured to receive the captured image frames 311, 312 of the image stream 310 from the image sensor 301 of the image processing device 600.

[0099] The image processing device 600 and / or the processing module 601 and / or the detection module 610 are configured to detect a first object 431 or a first portion 431a of the first object 431 within a first region 441 of a scene captured within a first stream 421 of image frames captured by the first camera 401 of the multi-camera system 400.

[0100] The image processing device 600 and / or the processing module 601 and / or the determination module 620 are further configured to determine a first probability value indicating a first probability that the detected first object 431 or the first portion 431a of the first object 431 belongs to an object type, based on the characteristics of the first object 431 or the first portion 431a of the first object 431.

[0101] The image processing device 600 and / or the processing module 601 and / or the detection module 610 are further configured to detect a second object 432 or a second portion 432a of the second object 432 within a second region 442 of a scene captured within a second stream 422 of image frames by the second camera 402 of the camera system 400. The second camera 402 is different from the first camera 401. The second region 442 at least partially overlaps with the first region 441.

[0102] The image processing apparatus 600 and / or the processing module 601 and / or the determination module 620 are further configured to determine a second probability value indicating a second probability that the detected second object 432 or the second part 432a of the second object 432 belongs to an object type, based on the characteristics of the second object 432 or the second part 432a of the second object 432.

[0103] The image processing apparatus 600 and / or the processing module 601 and / or the detection module 610 are further configured to determine an updated second probability value by increasing the second probability value when the second probability value is less than a second threshold and the first probability value is greater than a first threshold.

[0104] The image processing apparatus 600 and / or the processing module 601 and / or the determination module 620 can be further configured to increase the second probability value based on the first probability value.

[0105] The image processing apparatus 600 and / or the processing module 601 and / or the determination module 620 can be further configured to determine the difference between the first probability value and the first threshold, and increase the second probability value based on this difference, so as to increase the second probability value based on the first probability value.

[0106] The image processing device 600 and / or the processing module 601 and / or the masking module 630 can be further configured to apply a privacy mask to at least a part of the second object 432 in the second stream 422 of the image frame when the updated second probability value is greater than the second threshold.

[0107] The image processing apparatus 600 and / or the processing module 601 and / or the counting module 640 can be further configured to increase a counter value associated with the second stream 422 of the image frame when the updated second probability value is greater than the second threshold.

[0108] The image processing apparatus 600 and / or the processing module 601 and / or the determination module 620 can be further configured to increase the second probability value when the second probability value exceeds a third threshold in addition to being below the second threshold.

[0109] The image processing apparatus 600 and / or the processing module 601 and / or the determination module 620 determines that the second object 432 or the second part 432a of the second object and the first object 431 or the first part 431a of the first object are co-located within the overlapping region of the second region 442 and the first region 441, and is further configured to determine an updated second probability value in response to determining that the second object 432 or the second part 432a of the second object and the first object 431 or the first part 431a of the first object are co-located within the overlapping region.

[0110] As used herein, the term "module" may refer to one or more functional modules, each of which may be implemented as one or more hardware modules and / or one or more software modules and / or a combined software / hardware module. In some examples, a module may represent a functional unit implemented as software and / or hardware.

[0111] As used herein, the term "computer program carrier", "program carrier", or "carrier" can refer to one of an electronic signal, an optical signal, a wireless signal, and a computer-readable medium. In some examples, a computer program carrier may exclude transient propagation signals such as electronic signals, optical signals, and / or wireless signals. Thus, in these examples, a computer program carrier may be a non-transient carrier such as a non-transient computer-readable medium.

[0112] As used herein, the term "processing module" may include one or more hardware modules, one or more software modules, or a combination thereof. Any such module that is a module of hardware, software, or a combination of hardware and software may be a connection means, a providing means, a configuring means, a responding means, an invalidating means, etc. as disclosed herein. As an example, the expression "means" may be a module corresponding to the modules listed above together with the figures.

[0113] As used herein, the term "software module" may refer to a software application, a dynamic link library (DLL), a software component, a software object, an object according to the Component Object Model (registered trademark) (COM), a software component, a software function, a software engine, an executable binary software file, etc.

[0114] The term "processing module" or "processing circuit" may, herein, include, for example, a processing unit including one or more processors, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The processing circuit or the like may include one or more processor cores.

[0115] As used herein, the expression "configured / to be configured for" may mean that the processing circuit is configured, for example, adapted or operable to perform one or more of the actions described herein by a software configuration and / or a hardware configuration.

[0116] As used herein, the term "action" can refer to an action, step, operation, response, reaction, activity, etc. It should be noted that an action in this specification can be divided into two or more sub-actions where applicable. Further, it should be noted that two or more operations described in this specification may be merged into a single operation where applicable.

[0117] As used herein, the term "memory" can refer to a hard disk, magnetic storage medium, portable computer diskette or disk, flash memory, random access memory (RAM), etc. Memory may also refer to internal register memory of a processor, etc.

[0118] As used herein, the term "computer-readable medium" may be, for example, a Universal Serial Bus (USB) memory, a DVD disk, a Blu-ray disk, a software module received as a data stream, flash memory, a hard drive, a memory stick, a multimedia card (MMC), a secure digital (SD) card, or other memory cards. One or more of the foregoing examples of computer-readable media may be provided as one or more computer program products.

[0119] As used herein, the term "computer-readable code unit" may be the text of a computer program, a part or the whole of a computer program representing a binary file in a compiled format, or anything in between. As used herein, the term "number" and / or "value" may be any kind of number, such as a binary number, a real number, an imaginary number, or a rational number. Further, "number" and / or "value" may be one or more characters, such as a single character or a character string. "Number" and / or "value" may be represented by a bit string, i.e., by 0 and / or 1.

[0120] As used herein, the phrase "in some embodiments" is used to indicate that the features of the described embodiments can be combined with any other embodiments disclosed herein.

[0121] Embodiments in various aspects have been described, but many different changes, modifications, etc. will be apparent to those skilled in the art. Accordingly, the described embodiments are not intended to limit the scope of the present disclosure.

Claims

1. A method for determining a probability value indicating the probability that an object captured within a stream (422) of image frames, which is executed in a multi-camera system (400), belongs to an object type, comprising: detecting (501) a first object (431) or a first portion (431a) of the first object (431) within a first region (441) of a scene captured within a first stream (421) of image frames captured by a first camera (401) of the multi-camera system (400); determining (502) a first probability value indicating a first probability that the detected first object (431) or first portion (431a) belongs to the object type, based on characteristics of the first object (431) or the first portion (431a) of the first object (431); when the first probability value exceeds a first threshold, determining (503) that the detected first object (431) or first portion (431a) belongs to the object type; detecting (504) a second object (432) or a second portion (432a) of the second object (432) within a second region (442) of a scene captured in a second stream (422) of image frames by a second camera (402) of the camera system (400) different from the first camera (401), wherein the second region (442) at least partially overlaps with the first region (441); determining (505) a second probability value indicating a second probability that the detected second object (432) or second portion (432a) belongs to the object type, based on characteristics of the second object (432) or the second portion (432a) of the second object (432), and when the second probability value exceeds a second threshold, determining that the detected second object (432) or second portion (432a) belongs to the object type; when the second probability value is below the second threshold and the first probability value is above the first threshold, determining (507) an updated second probability value by increasing the second probability value, wherein increasing the second probability value is based on the first probability value; determining the difference between the first probability value and the first threshold; A method including increasing a second probability value based on a difference between a first probability value and a first threshold value. **Claim 2** The method according to claim 1, wherein increasing the second probability value based on a difference between the first probability value and the first threshold value includes increasing the second probability value according to the difference. **Claim 3** The method according to claim 1, wherein a second threshold value is lower than the first threshold value. **Claim 4** The method according to claim 1, further including determining that a second object belongs to an object type in response to the updated second probability value exceeding the second threshold value. **Claim 5** The object type is an object type to be masked out, the first and second threshold values are for masking an object or a part of an object of the object type to be masked out, and when the updated second probability value exceeds the second threshold value, the method according to claim 1 further includes applying (508) a privacy mask to at least a part of the second object (432) in a second stream (422) of an image frame. **Claim 6** The object type is an object type to be counted, the first and second threshold values are for counting an object or a part of an object of the object type to be counted, and when the updated second probability value exceeds the second threshold value, the method according to claim 1 further includes increasing a counter value associated with a second stream (422) of an image frame. **Claim 7** The method according to claim 1, wherein the first threshold value and the second threshold value are each specific to a type of an object or a part of an object. **Claim 8** The method according to claim 1, wherein a further condition for determining the updated second probability value by increasing the second probability value is that the second probability value exceeds a third threshold value in addition to being lower than the second threshold value. **Claim 9** Determining (506) that a second object (432) or a second part (432a) of the second object and a first object (431) or a first part (431a) of the first object are located at the same location within an overlapping region of a second region (442) and a first region (441), and determining a second probability value updated in response to determining that the second object (432) or the second part (432a) of the second object and the first object (431) or the first part (431a) of the first object are located at the same location within the overlapping region. The method according to claim 1 further includes this.

10. The method according to claim 1, wherein the second object part type of the second part (432a) is the same object part type as the first object part type of the first part (431a).

11. An image processing apparatus (402, 460) of a multi-camera system (400) configured to execute the method according to any one of claims 1 to 10.

12. The image processing apparatus (402, 460) according to claim 11, wherein the image processing apparatus (402, 460) is a camera (402) such as a surveillance camera or a video server (460).

13. A computer program (603) including a computer-readable code unit that causes the image processing apparatus (110) to execute the method according to any one of claims 1 to 10 when executed on the image processing apparatus (110).

14. A computer-readable medium (605) including the computer program according to claim 13.

Images