Method for reducing false positive identification during video conference tracking and detection

Abstract

This invention discloses methods for reducing false positives during video conferencing tracking and detection. One method includes detecting a set of sub-images in a digital image that match a selected object type. The method further includes generating a first confidence score for a first sub-image in the set of sub-images matching the selected object type. The method also includes generating a second confidence score for a second sub-image in the set of sub-images matching the selected object type. The method further includes generating a similarity metric by comparing the first and second sub-images. The method also includes removing a second sub-image from the set of sub-images in response to the similarity metric exceeding a similarity threshold and the first confidence score exceeding the second confidence score. The method further includes processing the digital image using the set of sub-images after removal.

Description

Background Technology

[0001] Video conferencing systems can use detection and tracking software to identify sub-images of objects displayed in an image or video stream. However, the tracking and detection software may unintentionally detect a sub-image of a person's reflection as a real person. Therefore, for example, if a camera is capturing images or video streams of a conference room with glass walls, windows, or any reflective surface, the tracking and detection software may unintentionally mistake a person's reflection in the glass for a real person. Summary of the Invention

[0002] One or more embodiments provide a method. The method includes detecting a set of sub-images in a digital image that match a selected object type. The method further includes generating a first confidence score for a first sub-image in the set of sub-images matching the selected object type. The method also includes generating a second confidence score for a second sub-image in the set of sub-images matching the selected object type. The method further includes generating a similarity measure by comparing the first and second sub-images. The method further includes removing a second sub-image from the set of sub-images in response to the similarity measure exceeding a similarity threshold and the first confidence score exceeding the second confidence score. The method also includes processing the digital image using the set of sub-images after removal.

[0003] One or more embodiments provide another method. The method includes detecting, at a first time, a sub-image of an object that matches an object type in a first digital image of a video stream. The method also includes determining, based on the detection, the presence of sub-images of the object in digital images following the first digital image in the video stream. The method further includes preventing modification of the video stream using the sub-images of the object, at least until a second time has elapsed after the first time.

[0004] One or more embodiments also provide a controller. The controller includes an image processing controller executable by a processor to detect sub-images corresponding to a selected object type in digital images of a video stream. The image processing controller may also be executable by a processor to assign a confidence score corresponding to a sub-image. The confidence score includes a measure that the sub-image is of the selected object type. The controller also includes a first filter executable by a processor to prevent the use of a first subgroup of sub-images when modifying the video stream. The first subgroup includes a first plurality of sub-images with confidence scores below a confidence threshold. The controller also includes a second filter executable by a processor to delay the use of a second subgroup of sub-images by a threshold time interval when modifying the video stream. The second subgroup includes a second plurality of sub-images detected before the threshold time interval. The controller also includes a third filter executable by a processor to prevent the use of a selected sub-image among the sub-images when modifying the video stream. The selected sub-image is selected from one of the first sub-images having a first similarity score within a similarity threshold of a second similarity score of a second sub-image. The selected sub-image includes lower confidence scores among the confidence scores. The controller also includes a video controller configured to modify the video stream using a first filter, a second filter, and a third filter.

[0005] Other aspects of one or more embodiments will become apparent from the following description and the appended claims. Attached Figure Description

[0006] Figure 1 A computing system according to one or more embodiments is shown.

[0007] Figure 2 and Figure 3 A flowchart illustrating a set of steps of a method for filtering a video stream according to one or more embodiments is shown.

[0008] Figure 4 , Figure 5 and Figure 6 An example of a filtered video stream according to one or more embodiments is shown.

[0009] Figure 7 This is another method for filtering video streams according to one or more embodiments.

[0010] Figure 8 and Figure 9 An example of a matrix used for filtering video streams is shown according to one or more embodiments. Detailed Implementation

[0011] Generally, one or more embodiments relate to filtering video streams. In particular, one or more embodiments are useful for preventing video software or image tracking and detection software from unintentionally detecting reflective images of people as images of natural persons. In one example, reflections of people on glass walls, windows, or other reflective surfaces may be common in an indoor video conferencing environment. Reflections may also be amplified, depending on camera placement and lighting conditions. These reflections can cause technical problems with tracking and detection software used to modify video streams or images or to track one or more objects displayed in the video stream.

[0012] For example, a video conferencing system can receive a video stream as input and then use audio and video tracking and detection data from the video stream to frame groups of people, automatically adjust the zoom level of the video stream, frame speakers in the video stream, identify individuals in an image or video stream, or perform other tracking and detection functions. However, if the tracking and detection software incorrectly identifies a person's reflection as a person, the output of the tracking and detection software may be undesirable. Examples of undesirable output include, but are not limited to, incorrectly calculating the number of people in a room, incorrectly identifying a person, counting the same person twice, incorrectly selecting the desired zoom level, incorrectly identifying a speaker, identifying two people as speaking simultaneously, and other possible undesirable outputs. While the tracking and detection software may not be characterized as malfunctioning, input caused by one or more reflections can confuse the tracking and detection software, leading to the various problems described above.

[0013] One or more embodiments provide one or more technical solutions to the above-described technical problems. One or more embodiments use one or more filters to automatically prevent false detections. True detection is the detection of an actual, existing object of interest (e.g., the detection of a person's head actually appearing in a room, captured in a video stream). False detection is the detection of an object of interest, but in which the detected object is not actually that object type (e.g., detecting a person's head reflected from a reflective object in a room).

[0014] Specifically, one or more embodiments may include three different filters, which may be used individually or in combination. If the detected object shown in the detected sub-image fails to meet a confidence threshold, the first filter removes the detected sub-image from further processing. For example, matching software assigns a probability that the detected sub-image is an object type. If the probability fails to meet the confidence threshold, the detected sub-image is discarded before further processing of the video stream or image.

[0015] The second filter compares the detected sub-images to each other and discards one or more detected sub-images when two or more are sufficiently similar. As explained further below, discarded images have low confidence scores, while retained images have high confidence scores. For example, the matching software assigns confidence scores to two detected sub-images of an object type present in the video stream. The similarity software assigns a similarity metric to the two detected sub-images being compared. If the similarity metric is higher than a similarity threshold, the detected sub-images with lower confidence scores are removed before further processing of the video stream or images.

[0016] The third filter is the delay filter. If a new sub-image of an object type is detected and then continues to be detected in the video stream, the newly detected sub-image is not used for further processing of the video stream or the image until a threshold time period has elapsed. The delay filter can be used to distinguish between sub-images created from reflective objects and sub-images created from physical objects, because reflective objects often appear flickering in digital images, while images of real people and objects show consistent and stable detection and tracking. For example, if a reflected head is initially detected when a natural person's head moves around in a room, the image of the reflected head will not undergo further processing in the video stream until after the threshold time period has elapsed.

[0017] Now let’s turn our attention to the attached figures. Figure 1 A computing system according to one or more embodiments is illustrated. The computing system includes a data storage library (100). In one or more embodiments, the data storage library (100) is a storage unit and / or device (e.g., a file system, database, table collection, or any other storage mechanism) for storing data. The data storage library (100) may be characterized as a non-transitory computer-readable storage medium. Furthermore, the data storage library (100) may include multiple different storage units and / or devices. These multiple different storage units and / or devices may or may not be of the same type and may or may not be located at the same physical site.

[0018] The data storage library (100) can at least temporarily store data used in one or more embodiments. For example, the data storage library (100) can store video streams (102). A video stream (102) can be considered as a series of digital images that can form a continuous video. The video stream (102) can take the form of a data structure, such as a video file, more specifically, it can be a ".mpg" file, etc. The video stream (102) can be stored in many different types of data structures.

[0019] The data storage library (100) can also store digital images (104). A digital image (104) can be one of many images captured sequentially as part of a video stream (102). A digital image (104) can also be captured individually as one or more digitized images. A digital image (104) is stored as a digital image file, such as a ".jpg" file, or it can be one of the frames of the video stream (102) (i.e., a frame in a ".mpg" file). Digital images (104) can be stored in many different types of data structures.

[0020] As used herein, the term “object” refers to a physical object. One or more sub-images (106) in a video stream (102) or a digital image (104) may represent a physical object in the video stream (102) or in the digital image (104).

[0021] Therefore, the data storage (100) stores or at least tracks sub-images (106). A sub-image (106) is a sub-part of a video stream (102) or a digital image (104). A sub-image (106) can be considered a sub-part of the data that forms the video stream (102) or the digital image (104). A sub-image (106) can be referred to as a detection bounding box, and its width and height are represented by its {x, y} coordinates in a predetermined or generated coordinate system.

[0022] In the video stream, each frame is extracted and then magnified into a smaller frame. Each magnified, smaller frame is then divided into sub-images called candidate detections. Each candidate detection is then fed through a trained head detection machine learning model, which in turn assigns a confidence score (as a head) in the range [0, 1] to each candidate detection.

[0023] The confidence score is the result of hierarchical mathematical calculations performed by a trained machine learning model. The higher the score, the higher the confidence level of the machine learning model in declaring the candidate as a detection head; the lower the score, the lower the confidence level.

[0024] The final detection is determined by setting a threshold for the score. For example, if the score is greater than the threshold, the candidate window is confirmed as the head. A confidence score is associated with each candidate detection (i.e., sub-image). Only detections with scores higher than the threshold are designated as confirmed head detections.

[0025] Each sub-image can be further subdivided into multiple additional sub-images of the sub-image (106). For example, a sub-image in sub-image (106) can be an image of a person shown in digital image (104), but the image of a person can be further subdivided into additional sub-images defined by the person's head.

[0026] In another example, a sub-image (106) may include a sub-image of a reflective object (108) among other physical objects represented in the video stream (102) or digital image (104). The reflective object (108) has reflective optical properties. Thus, the reflection of a physical object can be represented as an additional sub-image in the sub-image (106), as well as other sub-images (106) representing other physical objects in the video stream (102).

[0027] As indicated above, the sub-image (106) can be further subdivided. For example, one of the sub-images (106) is an image of a person. In this example, the portion of the digital image containing the person can be characterized as the first sub-image (110), and the portion of the digital image containing the head can be characterized as the second sub-image (112) within the first sub-image (110). The terms "first sub-image (110)" and "second sub-image (112)" refer only to a single sub-image within the sub-image (106) and do not necessarily refer to images of a person and a person's head.

[0028] For identification purposes, the sub-images (106) can be grouped. Thus, as used herein, a group of sub-images (114) is one or more sub-images (106) that are classified into a group. Specifically, the group of sub-images (114) are those sub-images that match the object type of the sub-images (106).

[0029] An object type is a classification of identifiable objects displayed in a digital image (104) or video stream (102). An identifiable object is an object that a machine learning model is trained to detect. In one example, if the machine learning model is trained to detect a person, a chair, a laptop, and a head, then all four are identifiable objects. In one embodiment, only a head constitutes an identifiable object. An object type has object instances. For example, if the object type is “head,” then an instance of the object type (i.e., a specific head detected in the digital image (104)) could be the head of a specific person shown in the digital image (104).

[0030] The data storage (100) also stores one or more selected object types (116). The selected object type is the type of object of interest for identification in the digital image (104) or video stream (102). Therefore, an instance of the selected object type (116) is one of the sub-images (106). For example, the selected object type (116) could be “head,” and the matching software is configured to identify head instances in the digital image (104) or video stream (102). Multiple selected object types may exist. For example, the software could be configured to detect both “head” and “table,” and both of these would then be examples of the selected object type (116).

[0031] The data repository (100) also stores one or more confidence scores (118). A confidence score (118) is a number assigned to the at least one set of sub-images (114), where the confidence score indicates the probability that a sub-image matches a selected object type. These confidence scores are calculated by a trained machine learning model through hierarchical mathematical operations. Thus, any given confidence score is a measure of how closely one of the sub-images (106) matches the selected object type (116), as determined by image recognition software. For example, one of the confidence scores (118) could be the number "0.92", which reflects a 92% probability that the first sub-image (110) is one of the selected object types (116). Different sub-images in the sub-images (106) can be assigned different confidence scores (118). Confidence scores can range from 0 to 1.

[0032] The data repository (100) also stores a confidence threshold (120). The confidence threshold (120) is a number indicating a limit on whether one of the sub-images (106) is one of the selected object types (116). In other words, the limit is the value at which one of the sub-images (106) is considered to be one of the selected object types (116). The limit can be inclusive (e.g., greater than or equal to the limit) or exclusive (greater than the limit).

[0033] For example, if the first sub-image (110) has a confidence score of 0.92 and if the confidence threshold (120) is 0.90, then the first sub-image (110) is determined to be one of the selected object types (116). In a more specific example, if the first sub-image (110) has a confidence score of 0.92 and if the confidence threshold (120) is 0.90, then the first sub-image (110) is determined to match the object type corresponding to the header (i.e., one of the selected object types (116)). Once determined, metadata can be assigned to the data file or data portion representing the sub-image in question, indicating that the sub-image is a member of the selected object type.

[0034] The data store (100) also stores one or more similarity measures (122). A similarity measure (122) is a number assigned to a pair of sub-images (106) that matches one of the selected object types (116) within a confidence threshold (120). Various methods exist for calculating the similarity measure (122). One method is to calculate the L2 (Euclidean) distance between features extracted from the detection. The smaller the distance, the greater the match. Calculating cosine similarity matching is another method for calculating one or more similarity measures (122). Calculating image hashing is yet another such method.

[0035] The similarity metric is a measure of how well the sub-image pairs (106) match each other. Thus, for example, a higher similarity metric indicates a higher probability that a pair of instances of the selected object type matches each other. In a particular example, if the first sub-image (110) and the second sub-image (112) are both in the selected object type (116) of the group (e.g., the first sub-image (110) and the second sub-image (112) are both “heads”), then the similarity metric (122) indicates how closely the first sub-image (110) and the second sub-image (112) match each other (e.g., whether the first sub-image (110) and the second sub-image (112) represent the natural head and the reflection of the natural head).

[0036] The data store (100) also stores a similarity threshold (124). The similarity threshold (124) is a number indicating a limit on whether a pair of sub-images (106) are detected as a match. In other words, in this case, the limit is the point at which one of the sub-images (106) is determined to match the other sub-image (106). The limit can be inclusive (e.g., greater than or equal to the limit) or exclusive (greater than the limit).

[0037] The similarity threshold (124) can be determined by balancing false positives and true positives. A threshold that is too low will result in too many candidates entering the final detection pool, leading to low recall, but also increasing false positives. A threshold that is too high can result in fewer false positives, but only highly accurate detections will enter the final detection pool, resulting in high precision but low recall, which may miss objects of actual interest. Testing can indicate a favorable similarity threshold (124).

[0038] For example, if the first sub-image (110) and the second sub-image (112) together have a similarity measure of 0.99, and if the similarity threshold (124) is 0.85, then the first sub-image (110) is determined to match the second sub-image (112). In a more specific example, both the first sub-image (110) and the second sub-image (112) are heads. As a pair, the first sub-image (110) and the second sub-image (112) have a similarity measure of 0.99. Therefore, in this example, it is determined that the first sub-image (110) and the second sub-image (112) are matching heads (e.g., twins are present in the room), or one of the first sub-image (110) and the second sub-image (112) is a sub-image of a natural person's head, and the other is a sub-image of a reflection of a natural person's head.

[0039] The data storage (100) also stores matrices (126). A matrix (126) is an array of numbers, such as in a table. A matrix can be characterized as having indices representing the values ​​of a row in the matrix. In a two-dimensional matrix, there are two indices, the first index can be called the row, and the second index can be called the column. The intersection of the indices is called a cell (e.g., the intersection of a row and a column in a two-dimensional matrix is ​​a cell). A number can be assigned to this cell, and this number is called the value of the cell. Figure 8 and Figure 9 Examples of matrices and their exemplary use in one or more embodiments are shown in the figure.

[0040] In one embodiment, the matrix consists of the scores of detections that match each other. In other words, the values ​​of the cells in the matrix represent how closely each detection matches each other. In this embodiment, the matrix can be a square matrix where the diagonal entries are identical, indicating the match of a detection with itself.

[0041] The data storage (100) also stores threshold time intervals (128). Threshold time intervals (128) are numerical representations of time limits allocated to measurements taken on a continuous series of digital images in the video stream (102). The time interval represents the elapsed time before decisions are made regarding framing, zooming, and the detection and tracking of people and objects. Regarding... Figure 7 The use of the threshold time interval (128) is described. This limit can be inclusive (e.g., greater than or equal to the limit) or exclusive (greater than the limit).

[0042] The data storage (100) also stores at least an indication of removed sub-images (130). Removed sub-images (130) are those sub-images removed from further consideration when the controller (132) (described below) processes the video stream (102) or digital image (104). Removed sub-images (130) may not be removed from the video stream (102) or from the digital image (104). Therefore, for example, when displayed, the video stream (102) or digital image (104) may still contain one or more removed sub-images (130); however, the removed sub-images (130) will not be used for further processing of the video stream (102) or digital image (104). Optionally, removed sub-images (130) may also be removed from the display of the video stream (102) or digital image (104).

[0043] Figure 1 The system shown may include other components. Therefore, for example, Figure 1 The system shown may include a controller (132). The controller (132) is software, hardware, or a combination thereof, programmed or configured to perform actions as described above. Figure 2 and Figure 3 One or more of the functions described above. The controller (132) may communicate with the data storage (100) via a network (156) (described below).

[0044] The controller (132) includes an image processing controller (134). The image processing controller (134) is software or dedicated hardware programmed to perform evaluation of a video stream (102) or a digital image (104). The image processing controller (134) can perform various functions, such as detecting that one or more sub-images (106) are of a selected object type (116), or assigning a confidence score to the sub-images (106) for matching the selected object type (116). The image processing controller (134) can perform other functions, such as evaluating confidence scores (118), similarity measures (122), and using confidence thresholds (120) and similarity thresholds (124). The controller (132) can also fill and use a matrix (126). The removed sub-images (130) are taken from the set of sub-images (106) processed by the image processing controller (134).

[0045] The controller (132) may also include a first filter (136). The first filter (136) is software or dedicated hardware programmed to determine whether one or more sub-images (106) are considered removed sub-images (130). Removed sub-images (or detections) are sub-images that are discarded, ignored, or otherwise disregarded during further processing. Regarding Figure 7 The operation of the first filter (136) is described.

[0046] The controller (132) may also include a second filter (138). The first filter (138) is also software or dedicated hardware, programmed to determine whether one or more sub-images (106) are considered to be removed sub-images (130). Regarding Figure 2 and Figure 7 The operation of the second filter (138) is described.

[0047] The controller (132) may also include a third filter (140). The third filter (140), also software or dedicated hardware, is programmed to determine whether one or more sub-images (106) are considered to be removed sub-images (130). Regarding... Figure 3 and Figure 7 The operation of the third filter (140) is described.

[0048] The controller (132) may also include a video controller (142). The video controller (142) is software or dedicated hardware programmed to manipulate the video stream (102) or digital image (104) in response to the output of the image processing controller (134). For example, if the image processing controller (134) determines that a first sub-image (110) should be framed, but a second sub-image (112) is a reflection, the video controller (142) may only frame the first sub-image (110). Regarding Figure 7 The operation of the video controller (142) is described, and... Figures 4 to 6 An example of the operation of the video controller (142) is shown.

[0049] The controller (132) may also include a tracking and detection controller (144). The tracking and detection controller (144) is software or dedicated hardware programmed to detect and track one or more instances of a selected object type (116) of a sub-image (106) in a digital image (104) or video stream (102). For example, the tracking and detection controller (144) can detect and track a head and its position in the video stream (102). Regarding... Figures 4 to 6 An operational example of the tracking and detection controller (144) is described.

[0050] The controller (132) may also include a communication device (146). The communication device (146) is hardware, software, or a combination thereof, configured to allow communication between the controller (132), the data storage (100), and a possible network (156). For example, the controller (132) may be a communication interface (1008).

[0051] The controller (132) may include, or be executed by, a computer (148). The computer (148) includes one or more processors, which are... Figure 1 The processor (150) in the computer (148) indicates that it may be in a distributed or cloud computing environment. The computer (148) also includes memory (152). The memory (152) may include a non-transitory computer-readable storage medium. The computer (148) may also include a user input device (154). The user input device (154) is operable to input user-provided instructions to the computer (148) and the controller (132).

[0052] Figure 1 The system shown may also include a network (156). A controller (132), a data storage unit (100), or both may communicate with the network (156). The network (156) is one or more networked computers or communication components that allow electronic communication between the computers or communication components. An example of a network is... Figure 1 The network described.

[0053] Figure 1 The system shown may optionally include one or more user devices (158) that can communicate with the controller (132), for example, via a network (156). The user device (158) is a computer, such as a desktop computer, laptop computer, tablet computer, mobile phone, etc.

[0054] Each user device may include a user input device (160). A user input device (160) is a device that allows a user to interact with the user device (158). Examples of user input devices (160) may include a keyboard, a mouse, gadgets on a graphical user interface (GUI), a microphone, etc.

[0055] Each user device may include a display device (162). The display device (162) is a screen that allows the user to see the GUI.

[0056] Each user device may include a camera (164). The camera (164) is another example of a user input device (160). The camera (164) may be used to generate a video stream (102) or a digital image (104), which may then be transmitted via a network (156) to a controller (132) for processing.

[0057] Figure 2 and Figure 3 A flowchart illustrating a set of steps of a method for filtering a video stream according to one or more embodiments is shown. Figure 2 It shows Figure 1 An example of the operation of the second filter (138) described in the text. Figure 3 It shows Figure 1 An operational example of the third filter (140) described herein. Therefore, Figure 3 The method shown can be used Figure 1 The system shown is used to execute this, possibly using data from... Figure 1 The components of the computer system and network shown.

[0058] Attention first shifts Figure 2 The method involves step 200, which includes detecting a set of sub-images in the digital image that match the selected object type. Image processing software can determine whether a given sub-image matches the selected object type. For example, a trained machine learning algorithm can determine that a particular sub-image is a head, or another selected object type.

[0059] Step 202 includes generating a first confidence score for the first sub-image in the set of sub-images to match the selected object type. Image processing software can detect the set of sub-images in step 200 and simultaneously generate the confidence score in step 202, or as part of the same detection process. The confidence score is the result of hierarchical mathematical calculations using a trained machine learning model. For example, a confidence score can be assigned to each sub-image, and sub-images with confidence scores higher than a threshold are labeled or marked as matching the selected object type.

[0060] Step 204 includes generating a second confidence score for matching the selected object type to the second sub-image in the set of sub-images. Similar to step 202, step 204 can be performed concurrently with step 200 or as part of step 200. However, the second sub-image differs from the first sub-image.

[0061] Step 206 involves generating a similarity metric by comparing the first and second sub-images. The first and second sub-images can be compared to each other using image recognition software that determines how closely they match. For example, a Siamese machine learning network can determine the probability that the first and second sub-images match each other. In this example, the probability is the similarity metric.

[0062] Generating similarity metrics can also be performed using other methods. For example, such as... Figure 8 and Figure 9 The example illustrates how a similarity matrix can be generated. The cells in the similarity matrix represent the similarity of sub-images relative to each other. The similarity measure between the first and second sub-images is the value of the corresponding cell.

[0063] Similarity measures in the matrix can be generated using machine learning (as indicated above) or other image processing software. Examples of other image processing techniques include using a cosine similarity index determined for a pair of sub-images. Another example of an image processing technique involves using the image hash values ​​of the first and second sub-images as the similarity values ​​between the first and second sub-images.

[0064] Step 208 includes removing a second sub-image from the set of sub-images in response to a similarity metric exceeding a similarity threshold and a first confidence score exceeding a second confidence score. In other words, if two sub-images match each other closely enough (i.e., the similarity metric meets the similarity threshold), the sub-image with the lower confidence score (e.g., the second sub-image in this case) is considered a removed sub-image. The second sub-image can be removed by setting a flag or some other indicator that the second sub-image should not be processed further, or by removing the second sub-image from a set of sub-images that match a selected object type.

[0065] Step 210 includes processing the digital image using the set of sub-images after removal. Processing may include taking an action regarding the digital image, which may be part of a video stream. For example, processing may include scaling or framing sub-images within the set of sub-images remaining after removal in step 208. Processing may include counting the number of sub-images remaining after removal in step 208. Processing the set of sub-images may include other actions, such as, but not limited to, tracking and detecting the set of sub-images, recognizing sub-images (e.g., using facial recognition software), modifying the digital image (e.g., by removing the removed sub-images), and combinations thereof.

[0066] The modified digital image or modified video stream can then be rendered on a display device. Therefore, as indicated above, modification could include scaling the digital image on the first sub-image to form a modified digital image, where the modified digital image excludes the second sub-image. In another example, the modified digital image could be displayed around a framing box added around a selected sub-image of the object.

[0067] Figure 2 The method can be used as a second filter to prevent a sub-image of a physical object's reflection from being detected as a sub-image of the physical object by the video controller. For example, the second sub-image could represent a reflection of the first sub-image. The reflection is caused by a reflective object displayed in the digital image. In this case, the second filter removes the second image from the set of sub-images used by the video controller to process the video stream or digital image.

[0068] In a more specific example, Figure 2 The method can further include receiving digital images from a video stream captured by a video conferencing camera. In this example, the selected object type is a head. The second sub-image includes reflections of the head of the reflective object from the digital image. In this example, the video controller will modify the video stream using only the detections from the first sub-image.

[0069] Now let's turn our attention to... Figure 3 . Figure 3 The method is about Figure 1 An example of the operation of the described third filter (140).

[0070] Step 300 includes detecting, at a first time, a sub-image representing an object that matches a selected object type in a first digital image of the video stream. The detection of the object sub-image can be performed using image recognition software, as described above. Figure 2 As described, when image recognition software determines the probability that a matching threshold is met (a sub-image of the object contains an instance of the selected object type), the sub-image of the object matches the selected object type.

[0071] Step 302 includes determining, based on detection, whether there are sub-images of an object in digital images following the first digital image in the video stream. The determination of continuous detection is "based" on detection, as detection is the initial time of step 302. Continuous detection exists when at least a threshold number of digital images in the video stream (e.g., half or more of the digital images) are included in the detection of sub-images of an object during a defined time period. The detection of sub-images of an object in digital images can be performed as described above. Figure 2 Perform as described in step 200.

[0072] Step 304 includes preventing the use of sub-images of the object to modify the video stream, at least until a second time has elapsed after the first time. This can be based on... Figure 2 Step 208 describes the process of preventing the use of sub-images of objects to modify the video stream. Therefore, for example, blocking could include preventing the scaling of the video stream from decreasing to include sub-images of the objects and second sub-images of detected second objects, at least until a threshold time interval has elapsed. Alternatively, blocking could include delaying the insertion of sub-images of the head and one or more of the body(s) attached to the head into the video stream. Many other examples are possible.

[0073] Figure 3 The method can be modified or further extended. For example, Figure 3 The method may also include generating a similarity score in the second time interval, representing the similarity between a sub-image of an object in the video stream and a third sub-image of a reflecting object. The similarity score can be generated using information about... Figure 2 The process described in step 206 is then executed. Figure 3 The method can further include preventing the use of a third sub-image of the reflecting object to modify the video stream in response to a similarity score exceeding a similarity threshold. In this example, the first confidence score of the third sub-image is lower than the second confidence score of the object's sub-image. Again, the blocking can be performed as described above.

[0074] In another variation, Figure 3 The method could also include modifying the video stream using the object after the second time period has elapsed. After the second time period has elapsed, it is possible to assume that the detected sub-image is a sub-image of the physical object, rather than a reflection sub-image. Therefore, once the second time period has passed, the method could include modifying the video stream in a manner that was previously prevented.

[0075] In a specific example, after the second time interval has elapsed, the newly detected sub-image of the head may be... Figure 1The video controller (142) is used to adjust the scaling level of the video stream to include sub-images of newly detected heads. Other examples of modifications include adjusting the scaling of the video stream; framing objects; adding text or images adjacent to sub-images of objects in the video stream; and recording the names and combinations of people whose heads are objects.

[0076] Figures 4 to 9 The above presents information about Figures 1 to 3 Specific examples of the described techniques are provided. The following examples are for illustrative purposes only and are not intended to limit the scope of one or more embodiments.

[0077] First, shift your attention Figures 4 to 6 . Figures 4 to 6 This refers to a series of digital images captured as part of a single video stream. Therefore, Figures 4 to 6 Shared reference numerals with common descriptions.

[0078] Figure 4 A representation of a digital image (400) captured by a camera is shown. The digital image (400) is received by a controller, for example, regarding... Figure 1 The controller (132) is described. In Figures 4 to 6 In the example, Figure 1 The controller (132) is programmed to automatically adjust the camera's zoom level to focus on the heads of people detected in the conference room.

[0079] In this example, the digital image (400) is the first image captured in the video stream. The digital image (400) includes a first sub-image (402) of a first person's first head, a second sub-image (404) of a second person's second head, and a third sub-image (406) that is a reflection of the second person's second head in the glass wall. Figure 4 In this example, scaling adjustments have not yet been made. However, in this case, it is not desirable to include reflections from a second person during the processing of the digital image (400).

[0080] Figure 5 The intermediate steps of processing a digital image (400) are shown, and thus the following are illustrated. Figure 1 An example of the operation of the image processing controller (134). In one embodiment, the digital image (500) is not displayed in the video stream. Instead, the digital image (500) may be used by the person responsible for maintenance. Figure 1 The controller (132) is visible to the programmer or technician. In one embodiment, the digital image (500) only shows how the controller (132) can process... Figure 4 Digital image (400).

[0081] A series of marks, such as mark (502), define the meaning of the marking. Figure 1The image processing controller (134) detects the contours of the person (i.e., sub-images). Furthermore, individual heads are detected as indicated by bounding boxes, such as the first bounding box (504), the second bounding box (510), and the third bounding box (508). In subsequent steps, as described below, Figure 1 The video controller (142) will automatically adjust the scaling based on the detected head.

[0082] Additionally, a confidence score, such as confidence score (506), is displayed above each head. The confidence score is a numerical value reflecting a certain probability that a sub-image is a head (e.g., the probability that the first sub-image (402), which is the portion of the digital image (500) within the first bounding box (504), is a head). Using the information about... Figure 2 Step 202 describes the process for determining a confidence score. Confidence scores below a threshold may not be displayed in the digital image (500). Therefore, for example, other sub-images within the digital image (500) may be represented by [the data]. Figure 1 The image processing controller (134) evaluates to determine whether a given sub-image contains a head. However, in Figure 5 In the examples, only those sub-images with confidence scores higher than 0.55 are shown.

[0083] exist Figure 5 In the example, the third bounding box (508) contains a third sub-image (406), which is a reflection of the second sub-image (404) of the second person's head. Because the reflection is not a perfect reflection, the confidence score (512) of the third sub-image (406) in the third bounding box (508) is only 0.58. A perfect reflection is a detection that has a high (or low, if distance-based matching is performed) match score with one of the other detections and has a confidence value that is the same as or lower than that of the matched detection. Conversely, the confidence score (514) of the second sub-image (404) within the second bounding box (510) of the second person is 0.98, therefore, the predicted probability that the second sub-image (404) is the head is 98%. This fact is useful for the second filter (138), as mentioned above regarding Figure 1 As stated above.

[0084] Figure 6 The image (600) is shown being scaled using... Figure 4 The digital image (400) was generated through automatic scaling and adjustment. Figure 6 In the example, Figure 1 The image processing controller (134) has been applied Figure 1 The first filter (136), the second filter (138), and the third filter (140). As a result, under one or more filters, it is determined that the third frame (508) should not be used during the processing of the digital image (400). Figure 5 The third sub-image (406) of the second person's head. In other words, the third sub-image (406) (the third sub-image (406) is a reflection of the second person's head) is... Figure 1 The video controller (142) blocks, excludes, or removes the video from processing.

[0085] therefore, Figure 1 The video controller (142) automatically adjusts the scaling level of the digital image (400) to include only the first sub-image (402) of the first person and the second sub-image (404) of the second person. In this way, the scaled image (600) is automatically scaled to the desired extent (i.e., the scaling level is not affected by the presence of the reflection of the second person in the glass wall).

[0086] Now let's turn our attention to... Figure 7 . Figure 7 It is relative to Figure 2 and Figure 3 An example of another method for filtering digital images or video streams. Figure 7 The method can be used Figure 1 The system shown is used to implement this.

[0087] Step 702 includes inputting an image frame. The image frame can be received from a camera, which may be a remote camera. The image frame may be part of a video stream. The image frame is input to an image processing controller, such as... Figure 1 Image processing controller (134).

[0088] Step 704 includes applying a head detector. The head detector can be an image recognition, classification, or detection machine learning model, or other software, such as [specific software name needed]. Figure 2 As explained in step 200, the head detector detects heads in the input image frame. The head detector can also count heads and determine the position of each head in the image plane coordinates.

[0089] Step 706 includes applying a first filter. The first filter may be... Figure 1 The first filter (136) discards heads whose detected confidence scores are below a confidence threshold. Discarded heads are considered as removed sub-images, as per the context of... Figure 1 As stated above.

[0090] Step 708 includes applying a second filter. The second filter can be... Figure 1 The second filter (138) finds head pairs that are similar to each other and removes the sub-images of the heads from the head pairs. The removed sub-images are those with lower confidence scores. Regarding... Figure 2An example of the operation of the second filter is described. However, steps 710, 712, and 714 are sub-steps in the process of applying the second filter in step 708.

[0091] Step 710 is a sub-step of step 708, which includes generating a similarity matrix. Figure 9 An example of a similarity matrix is ​​shown. A similarity matrix can be generated using image recognition algorithms to produce values ​​representing the degree of similarity between a pair of heads. (See also: Regarding...) Figure 9 The algorithm described above can identify duplicate head images for each pair of detected images.

[0092] Step 712 is a sub-step of step 708, involving identifying detected head pairs that match in the similarity matrix. Matching is performed by identifying two head groups with similarity scores above a similarity threshold. About Figure 2 Step 206 further describes the matching.

[0093] Step 714 is a sub-step of step 708, which involves retaining the detection of the head with the highest confidence score. Specifically, for each pair of matched heads, the head with the highest confidence score in a given pair is retained. The other head becomes the removed sub-image. Regarding... Figure 2 Step 208 describes the process of removing (and thus retaining) the detected header.

[0094] Step 716 includes applying a third filter. The third filter can be... Figure 1 A third filter (140). The third filter may include delaying the use of newly detected heads to further process the input image frame until i) a time threshold has passed, and ii) newly detected heads have been continuously detected during the time threshold period. In some embodiments, the term "continuously" may include detecting sub-images in fewer than all image frames of the video stream.

[0095] In other words, the third filter is a delay filter. Newly detected heads are defined as one of the sub-images to be removed before the time threshold has elapsed.

[0096] Step 718 includes merging the retained head detections. In other words, for the purpose of further processing the input image frames, all those heads that were not removed can be merged. Merging may include, for example, creating a file or data structure containing head sub-images that were not considered as removed sub-images.

[0097] Step 720 includes sending the detected header (i.e., a file or data structure containing a sub-image of the retained header) for further processing. For example, the detected header can be provided as input. Figure 1The video controller (142) or tracking and detection controller (144), or both. The detected head is then used to modify the input image frame or video stream, as per [the relevant parameters]. Figure 2 Step 210 or Figure 3 Step 304 is described. In one embodiment, Figure 7 This method can then be terminated.

[0098] Figure 8 and Figure 9 Examples of matrices that can be used in one or more of the above embodiments are shown. In particular, Figure 8 The diagram illustrates a detection matrix (800) data structure describing the detection of “n” sub-images (e.g., n heads). Each row of the matrix represents a distinct detected sub-image. The values ​​of “x” and “y” represent the XY coordinates of the sub-image in a coordinate system established for the digital image. “Width” and “Height” represent the width and height of the sub-image in the same coordinate system. “Score” is the confidence score of the sub-image in question, determined by, for example, a computer vision machine learning algorithm.

[0099] Figure 9 This diagram illustrates a similarity matrix (900) data structure describing the similarity between any two detected sub-images (e.g., heads) in a digital image. Both rows and columns reflect the sub-images in the array. The cells in the similarity matrix (900) store the similarity score between a pair of heads, represented by rows and intersecting columns. The diagonal cells of the matrix (from top left to bottom right) represent the similarity of a sub-image to itself and will have a value equal to or close to "1" (i.e., "1" reflects perfect similarity). If a cosine similarity measure is used as the matching technique, all diagonal elements will be "1".

[0100] Off-diagonal cell entries reflect the similarity between a sub-image and another sub-image detected in the digital image. Thus, for example, the top-left entry in the similarity matrix (900) is the similarity score of the sub-image pair formed by the first detected sub-image and the last detected sub-image.

[0101] In one or more embodiments, finding pairs of sub-images that are similar to each other includes identifying off-diagonal cells with similarity values ​​that satisfy a similarity threshold. In this way, similar sub-images (i.e., matching pairs of sub-images) can be found quickly. This process is computationally efficient because diagonal cells can be discarded before comparing the cell values ​​of the similarity matrix with the similarity threshold.

[0102] refer to Figure 1The system shown illustrates the configuration of the components. Other configurations may be used without departing from the scope of one or more embodiments. For example, various components may be combined to create a single component. As another example, the functionality performed by a single component may be performed by two or more components.

[0103] Referring to the flowcharts described herein, although the steps in the flowcharts are presented and described sequentially, those skilled in the art will understand that some or all steps may be performed in a different order, may be combined or omitted, and some or all steps may be performed in parallel. Furthermore, these steps may be performed actively or passively. For example, according to one or more embodiments, some steps may be performed using polling or be interrupt-driven. By way of example, according to one or more embodiments, the determining step may not require the processor to process instructions unless an interrupt is received to indicate that a condition exists. As another example, according to one or more embodiments, the determining step may be performed by performing a test, such as checking a data value to test whether the value is consistent with a test condition. Therefore, one or more embodiments are not necessarily limited to the examples provided herein.

[0104] Specific embodiments of the above-described identification are described in detail with reference to the accompanying drawings. For consistency, the same elements in each figure are denoted by the same reference numerals.

[0105] In the foregoing detailed description of the embodiments, numerous specific details have been set forth in order to provide a more thorough understanding of one or more embodiments. However, it will be apparent to those skilled in the art that one or more embodiments may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

[0106] While one or more embodiments have been described with respect to a limited number of examples, those skilled in the art who benefit from this disclosure will understand that other embodiments can be devised without departing from the scope of the one or more embodiments disclosed herein. Therefore, the scope of the one or more embodiments should be defined only by the appended claims.

Claims

1. A method for processing digital images, comprising: Detect a set of sub-images in a digital image that match a selected object type; Generate a first confidence score for matching the first sub-image in this set of sub-images to the selected object type; Generate a second confidence score for matching the selected object type in the second sub-image of this group of sub-images; A similarity metric is generated by comparing the first sub-image and the second sub-image. In response to a similarity metric exceeding a similarity threshold and a first confidence score exceeding a second confidence score, the second sub-image is removed from the group of sub-images; and After removal, use this set of sub-images to process the digital image.

2. The method of claim 1, wherein the processing includes modifying the digital image to generate the modified digital image.

3. The method of claim 2, wherein, The modification also includes scaling, framing, and tracking of the digital image on the first sub-image to form a modified digital image, wherein the modified digital image excludes the second sub-image.

4. The method of claim 1, wherein generating the similarity measure further comprises: Generate a matrix, wherein the cells in the matrix represent the similarity of a plurality of sub-images relative to each other, wherein the plurality of sub-images include at least a first sub-image and a second sub-image, and wherein the similarity measure includes cells in the matrix corresponding to the first sub-image and the second sub-image.

5. The method according to claim 1, wherein the similarity measure further includes one of the cosine similarity index of the first sub-image and the second sub-image and the image hash value.

6. The method according to claim 1, wherein: The second sub-image includes a reflection of the first sub-image, and Reflection is caused by reflective objects displayed in a digital image.

7. The method of claim 1, further comprising: Receive digital images from the video stream captured by the video conferencing camera.

8. The method of claim 1, wherein, The selected object type includes an image of a head, and the second sub-image includes a reflection of the head from a reflective object in a digital image.

9. A controller, comprising: Image processing controller, which can be executed by the processor to: Detecting multiple sub-images corresponding to a selected object type in the digital image of a video stream, and Assign multiple confidence scores corresponding to the multiple sub-images, wherein the multiple confidence scores include a measure that the multiple sub-images are of a selected object type; A first filter, which can be executed by a processor, prevents the use of a first subgroup of multiple sub-images when modifying the video stream. The first subgroup includes a first sub-image among the multiple sub-images with a confidence score below a confidence threshold. A second filter, which can be executed by the processor, is used to delay a threshold time interval for a second subgroup of the plurality of sub-images when modifying the video stream, wherein the second subgroup includes a second sub-image among the plurality of sub-images detected before the threshold time interval; A third filter, which can be executed by the processor, prevents the use of a selected sub-image among the plurality of sub-images when modifying the video stream, wherein the selected sub-image is selected from one of a first sub-images having a first similarity score within a similarity threshold of a second similarity score of a second sub-image, and wherein the selected sub-image includes a lower confidence score among the plurality of confidence scores. and The video controller is configured to modify the video stream using a first filter, a second filter, and a third filter.

10. The controller of claim 9, wherein, The selected object type includes headers, and the video controller is also configured to modify the video stream by performing amplification and framing of one of the detected headers after applying the first filter, the second filter, and the third filter.

11. The controller according to claim 10, wherein, The first subgroup, the second subgroup, and the selected sub-image include head reflections from reflective objects in the video stream.

12. The controller of claim 10, wherein the controller further comprises: The tracking and detection controller is configured to identify and track heads in the video stream.

13. The controller according to claim 9, further comprising: A communication device that communicates with the processor and is configured to receive the video stream.

Images