Classification of human types in visual media

Abstract

Disclosed herein are devices and methods for classifying a type of person in visual media. An implementation includes identifying a person in an image, generating a visual boundary around the identified person, and dividing the image into a plurality of image portions such that the plurality of image portions partially depict the identified person inside the visual boundary. The implementation includes generating the divided image on a user interface that prompts a user to select each portion that at least partially depicts a person of a predetermined type. The implementation includes receiving a selection of at least one portion. The implementation includes generating, for a machine learning model and in response to determining that the selection includes the identified person, a training input that includes a training image and a label that indicates that the identified person belongs to the predetermined type. Additionally, the implementation includes storing the training input in a memory.

Description

[0001] Cross-reference of related applications

[0002] This application claims the benefit of U.S. nonprovisional patent application No. 17 / 006,468, filed August 28, 2020, entitled “Classification of Person Type in a Visual Media,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The aspects described relate to image classification systems. Background Technology

[0004] The various aspects of this disclosure generally relate to image classification systems, and more specifically, to the classification of human types in visual media.

[0005] Modern image classification systems can distinguish between a wide variety of objects. For example, a machine learning model trained to differentiate between cats and dogs might succeed in doing so when given a sufficient database of images for training. However, for the average user, the accessibility of setting up a custom image classification system is limited—especially in niche cases.

[0006] For example, in a workplace context, suppose a department store owner wants to evaluate employee performance. The owner might set up cameras to detect when employees enter the frame and monitor their activities. Given that the department store may also have several customers, distinguishing between employees and non-employees becomes important. In this case, the owner cannot simply use any arbitrary employee image classification system, as such a system might be trained on a very different set of training images and / or might require specific types of input images. Setting up his / her own image classification system, generating a training database without knowing how many images are needed, model compatibility, and the type of scene to be captured, is cumbersome even for someone familiar with machine learning, let alone someone unfamiliar with it.

[0007] Therefore, there is a need to improve image classification systems. Summary of the Invention

[0008] The following is a simplified overview of one or more aspects to provide a basic understanding of such aspects. This invention is not a comprehensive review of all anticipated aspects, nor is it intended to identify key or essential elements of all aspects, nor to depict the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that follows.

[0009] An example implementation includes a method for classifying human types in visual media, the method comprising identifying people in an image using image classification. The method further includes generating a visual boundary around the identified person on the image. Additionally, the method further includes dividing the image into a plurality of image portions such that the plurality of image portions partially depict the identified person within the visual boundary. Furthermore, the method further includes generating the plurality of image portions for display on a user interface, the user interface prompting a user to select each of the plurality of image portions that at least partially depicts a person of a predetermined type. Furthermore, the method further includes receiving a selection of at least one portion of the divided image via the user interface. Furthermore, the method further includes generating training input for a machine learning model and in response to determining that the selection includes the identified person within the visual boundary, the training input including a training image depicting the content within the visual boundary and a label indicating that the identified person within the visual boundary belongs to the predetermined type. Furthermore, the method further includes storing the training input in memory.

[0010] Another example embodiment includes a device for classifying human types in visual media, the device including a memory and a processor in communication with the memory. The processor is configured to identify people in an image using image classification. The processor is further configured to generate visual boundaries around the identified people in the image. Additionally, the processor is further configured to divide the image into a plurality of image portions such that the plurality of image portions partially depict the identified people within the visual boundaries. Furthermore, the processor is further configured to generate the plurality of image portions for display on a user interface, the user interface prompting a user to select each portion of the plurality of image portions that at least partially depicts a person of a predetermined type. Furthermore, the processor is further configured to receive selections of at least a portion of the divided image portions through the user interface. Furthermore, the processor is further configured to generate training input for a machine learning model and in response to determining that the selection includes the identified person within the visual boundaries, the training input including a training image depicting the content within the visual boundaries and a label indicating that the identified person within the visual boundaries belongs to the predetermined type. Furthermore, the processor is further configured to store the training input in the memory.

[0011] Another example embodiment includes an apparatus for classifying human types in visual media, the apparatus including means for identifying humans in an image using image classification. The apparatus further includes means for generating a visual boundary around the identified human in the image. Additionally, the apparatus further includes means for dividing the image into a plurality of image portions such that the plurality of image portions partially depict the identified human within the visual boundary. Furthermore, the apparatus further includes means for generating the plurality of image portions for display on a user interface, the user interface prompting a user to select each of the plurality of image portions that at least partially depicts a human of a predetermined type. Additionally, the apparatus further includes means for receiving a selection of at least a portion of the divided image through the user interface. Furthermore, the apparatus further includes means for generating training input for a machine learning model and in response to determining that the selection includes the identified human within the visual boundary, the training input including a training image depicting the content within the visual boundary and a label indicating that the identified human within the visual boundary belongs to the predetermined type. Additionally, the apparatus further includes means for storing the training input in a memory.

[0012] Another example implementation includes a computer-readable medium for classifying human types in visual media, the computer-readable medium being executable by a processor to identify people in an image using image classification. Instructions can be further executed to generate visual boundaries around the identified people on the image. Additionally, instructions can be further executed to divide the image into a plurality of image portions, such that the plurality of image portions partially depict the identified people within the visual boundaries. Furthermore, instructions can be further executed to generate the plurality of image portions for display on a user interface, the user interface prompting a user to select each of the plurality of image portions that at least partially depicts a person of a predetermined type. Furthermore, instructions can be further executed to receive selections of at least a portion of the divided image through the user interface. Furthermore, instructions can be further executed to generate training input for a machine learning model and in response to determining that the selection includes the identified people within the visual boundaries, the training input including training images depicting the content within the visual boundaries and labels indicating that the identified people within the visual boundaries belong to the predetermined type. Furthermore, instructions can be further executed to store the training input in memory.

[0013] To achieve the foregoing and related objectives, one or more aspects include the features fully described below and specifically pointed out in the claims. The following description and drawings illustrate certain illustrative features of one or more aspects in detail. However, these features indicate only a few of the various ways in which the principles of each aspect can be employed, and this description is intended to encompass all such aspects and their equivalents. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate one or more exemplary aspects of this disclosure and, together with the detailed description, serve to explain the principles and implementations of these aspects.

[0015] Figure 1 This is a diagram of a user interface for classifying human types in visual media, based on an exemplary aspect of this disclosure.

[0016] Figure 2 It is a diagram depicting people in an image based on a type classification, according to an exemplary aspect of this disclosure.

[0017] Figure 3 This is a block diagram of a computing device for an executor type identifier component according to an exemplary aspect of this disclosure.

[0018] Figure 4 The flowchart illustrates a method for classifying human types in visual media, based on an exemplary aspect of this disclosure.

[0019] Figure 5 The flowchart illustrates a method for monitoring categorized types of people according to an exemplary aspect of this disclosure.

[0020] Figure 6 This is a flowchart illustrating a method for classifying verifier types according to an exemplary aspect of this disclosure.

[0021] Figure 7 The flowchart illustrates a method for filtering potential training images based on human visibility, according to an exemplary aspect of this disclosure. Detailed Implementation

[0022] Various aspects will now be described with reference to the accompanying drawings. In the following description, numerous specific details are set forth for illustrative purposes in order to provide a thorough understanding of one or more aspects. However, it will be apparent that such aspects can be practiced without these specific details.

[0023] This disclosure includes devices and methods that allow users to immediately set up custom image classification systems, particularly for niche scenarios such as classifying the types of people in images or videos. For example, the devices and methods implement a streamlined approach: generating a training dataset of images of any type of person via a user interface and implementing a machine learning model that can then classify people based on the generated training dataset.

[0024] Figure 1 This is a diagram of a user interface 100 for classifying human types in visual media, according to an exemplary aspect of this disclosure. User interface 100 provides an accessible method for generating a training database for a custom image classification system, particularly a system for classifying human types.

[0025] Revisiting the examples provided in the context of this disclosure, department store owners can install security cameras to collect videos and images, and upload said videos and images to a computing device (e.g., Figure 3 The computing device 300 described herein. In some aspects, the computing device may be an electronic device capable of processing images and transmitting information to other computing devices (e.g., servers, smartphones, laptops, desktops, etc.). The uploading process may be wireless (e.g., via Wi-Fi or Bluetooth) or wired (e.g., via Ethernet cable). The computing device may determine whether the received image contains a person (e.g., person 104) and, in response to detecting person 104, may generate a visual boundary 106 around person 104. The computing device may then divide the image 102 into multiple parts 102, such that the visual boundary 106 is segmented across the multiple parts. Figure 1 In the middle, the visual boundary 106 is visible in the lower right two parts.

[0026] Subsequently, the computing device can generate a user interface 100 for display. Generation can be performed on the computing device itself or on the display of a different computing device. For example, the computing device could be a server and a different computing device could be the owner's smartphone. Since person 104 has been categorized as "human," the purpose of the user interface 100 is to prompt the user to identify and select a subcategory of person 104. In this case, the user interface 100 prompts the user to select all portions of the segmented image 102 depicting person 110 (i.e., employee). It should be noted that the administrator of the computing device can change type 110. For example, type 110 can be changed to "customer," thus prompting the user to identify non-employees.

[0027] Visual boundary 106 allows users to easily identify points of interest, such as person 104 in image 102. This visual accessibility is further enhanced by dividing visual boundary 106 and its content (i.e., the image of person 104) across multiple parts. The user is thus informed that there may be a potential employee in image 102. A person standing in the background of the difficult-to-identify image 102 is therefore not defined, not selected, and consequently not removed from the machine learning model learned from the provided input data. When the user selects “Confirm” on user interface 100, the computing device receives the user’s selection of parts. For example, the user can select two parts that include visual boundary 106. It should be noted that even if the user selects only one of the two parts, the computing device can still identify person 104 as an employee, assuming visual boundary 106 spans both parts (i.e., the unselected part is automatically selected because the computing device recognizes person 104 as human).

[0028] User interface 100 thus generates training input characterized by the content of visual boundary 106 and the label indicating that person 104 is an employee. This training input can be used to train a machine learning model to distinguish between employees and non-employees. In some respects, the machine learning model can be a type of support vector machine (SVM). Therefore, starting a user's custom image classification system requires only one training input.

[0029] Figure 2 Image 200 is a depiction of people in a type-based classification image according to an exemplary aspect of this disclosure. Image 200 depicts customers and employees in a department store. Image 200 can be captured from the same security camera that captured image 102. In some aspects, image 200 can be captured from different cameras that can communicate with a computing device.

[0030] Suppose image 200 is input into a trained machine learning model (e.g., a class of SVMs). Based on the provided training input of person 104 (see... Figure 1 Knowing what the employees look like, the trained machine learning model can determine that person 204 is an employee and person 206 is not. Furthermore, the trained machine learning model can be retrained using the original training input and a new image of person 204. This allows the trained machine learning model to dynamically improve as more employee images become available and to adapt to changes in employee uniforms over time.

[0031] Figure 3 This is a block diagram of a computing device 300 with an executor type identifier component according to an exemplary aspect of this disclosure. Figure 4 The flowchart illustrates a method 400 for classifying human types in visual media, based on an exemplary aspect of this disclosure.

[0032] refer to Figure 3 and Figure 4 In operation, the computing device 300 may execute a method 400 for classifying human types in visual media by executing the human type identifier component 315 through the processor 305 and / or memory 310.

[0033] At box 402, method 400 includes identifying people in an image using image classification. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or recognition component 320 may be configured to or may include means for identifying people in an image using image classification.

[0034] For example, the recognition at box 402 could involve using a pre-trained machine learning model or computer vision techniques to determine whether the frame in a video or standalone image depicts a person. It should be noted that "person" is a class, while "employee" and "non-employee" are subclasses. Typically, this aspect first classifies people using any type of person classification model. One of the goals of this disclosure is to provide users with the ability to build training databases for custom machine learning models. Because the recognition component 320 automatically recognizes people, users do not need to manually evaluate whether an image correctly depicts a person for later classification into a subclass.

[0035] At box 404, method 400 includes generating a visual boundary around the identified person on the image. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315 and / or generation component 325 may be configured to or may include means for generating a visual boundary around the identified person on the image.

[0036] For example, generation at box 404 can generate shapes such as rectangles around person 104 in image 102. Generation component 325 can identify one or more boundaries of the image of a person, and then can create one or more boundary lines associated with such boundaries. For example, generation component 325 can attempt to generate the smallest possible visual boundary that completely outlines the body of person 104. In some optional aspects, the aspect ratio of the visual boundary can be constant for all identified people (e.g., 3:6). This allows person type identifier component 315 to be universal software compatible with any image. Suppose a department store owner has installed two cameras. Both cameras can communicate with person type identifier component 315 and provide images such as image 102. One camera can be configured to capture 720-pixel images, while the other camera can capture 4K images. Despite the different resolutions, the employee training images generated by person type identifier component 315 (containing only pixels inside the visual subject) will have the same aspect ratio and can be scaled to the same size. This results in a consistent training database in which all images have the same aspect ratio and size.

[0037] At box 406, method 400 includes dividing an image into multiple image portions such that the multiple image portions partially depict an identified person within a visual boundary. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or segmentation component 330 may be configured to, or may include, means for dividing an image into multiple image portions such that the multiple image portions partially depict an identified person within a visual boundary.

[0038] For example, the division at frame 406 may involve determining the size of image 102 and the size of visual boundary 106. To segment the content of visual boundary 106 across multiple sections 108, the division component 330 may determine the size of individual sections, (1) where the size of image 102 is divisible by the size of the individual section and (2) the size of the individual section is smaller than the size of visual boundary 106. For example, the size of image 102 may be 1200 × 700 pixels and the size of visual boundary 106 may be 300 × 600 pixels. The division component 330 determines sizes less than 300 (the shortest length of visual boundary 106) and divisible by 1200, 700, 600, and 300. In this particular example, the division component 330 may divide image 102 into multiple sections, each 100 × 100 pixels. It should be noted that this is merely an example. In some respects, the division component 330 may determine the maximum possible size of the sections (also satisfying the criteria discussed above) to reduce the number of choices the user must make.

[0039] At box 408, method 400 includes generating the plurality of image portions for display on a user interface that prompts a user to select at least partially depicting each part of a predetermined type of person. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or generation component 335 may be configured to, or may include, means for generating the plurality of image portions for display on a user interface that prompts a user to select at least partially depicting each part of a predetermined type of person.

[0040] For example, the generation of box 408 may include converting data representing the image, image portions, and each boundary into a user interface 100 on a computing device. The computing device may be device 300 or any device that can communicate with device 300. As previously discussed, user interface 100 provides a user with an accessible way to quickly set up a custom image recognition system by enabling manual verification of whether person 104 belongs to person type 110.

[0041] At box 410, method 400 includes receiving a selection of at least a portion of the segmented image via a user interface. For example, in one aspect, computer device 300, processor 305, memory 310, human type identifier component 315, and / or receiving component 340 may be configured to, or may include, means for receiving a selection of at least a portion of the segmented image via a user interface. Assume user interface 100 is generated on a touchscreen smartphone. A user can select two portions of image 102, including visual boundaries 106, via the touchscreen. In some aspects, user interface 100 may be generated on a desktop, and the user can select these two portions using mouse or keyboard input.

[0042] At box 412, method 400 includes determining whether the selection includes an identified person within a visual boundary. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or receiving component 340 may be configured to, or may include, means for determining whether the selection includes an identified person within a visual boundary.

[0043] For example, receiving component 340 may determine whether a person 104 is depicted in at least one portion of the selected portion (i.e., pixels marked as constituting the content contained in the visual boundary 106 of the selected portion).

[0044] In response to determining that the selection includes the identified person, method 400 proceeds to box 414, wherein method 400 includes generating training input for a machine learning model, the training input including training images depicting the content inside a visual boundary and labels indicating that the identified person inside the visual boundary belongs to a predetermined type. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or generation component 345 may be configured to or may include means for generating training input for a machine learning model, the training input including training images depicting the content inside a visual boundary and labels indicating that the identified person inside the visual boundary belongs to a predetermined type.

[0045] In some respects, the machine learning model is a type of support vector machine. Therefore, the model is trained on images depicting a single class of objects (e.g., images of people of type 110) and simply determines whether any arbitrary image depicts a person of type 110. The machine learning model can be configured to receive an input image and input labels that classify the image, and generate a function (e.g., a weighted dot product) that transforms the input image into the input labels. This function is then applied to other images to determine new labels indicating whether other images depict people of type 110. In this case, the generation component 345 identifies that person 104 belongs to type 110 (e.g., employee) and generates training input consisting of a cropped version of image 102 depicting only what is defined by visual boundary 106 and a label indicating that the cropped version of image 102 depicts an employee.

[0046] At block 416, method 400 includes storing training input in memory 310. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or storage component 350 may be configured to or may include means for storing training input in memory 310. Thus, person type identifier component 315 is capable of establishing a training image database in memory 310.

[0047] If, at box 412, the person type identifier component 315 and / or the receiving component 340 determine that the selection does not include the identified person, then method 400 proceeds to box 418, where the person type identifier component 315 retrieves a different image. For example, the user may not select any part of image 102 (because no employee is shown), or the user may select a part that the person type identifier component 315 determines does not contain a person based on its image classification system for recognizing humans. In response, the person type identifier component 315 generates a different image on user interface 100 for the user to interact with. More specifically, method 400 returns to box 402, where the recognition component 320 identifies another person in the different image. The loop between boxes 402 and 418 can continue until the user closes user interface 100. In some respects, method 400 also proceeds to 418 after executing box 416. This allows multiple training inputs to be stored in memory 310.

[0048] Figure 5 The flowchart illustrates a method 500 for monitoring the classification type of people according to an exemplary aspect of this disclosure. Alternatively, method 500 may be performed after method 400 is completed. Reference Figure 5At block 501, method 500 may include vectorizing the training input to create training vectors. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or training component 351 may be configured to or may include means for vectorizing the training input to create training vectors.

[0049] In instances where the predetermined type (i.e., type 110) is "employee", vectorizing the stored training input to create a training vector may include at least one feature that identifies the training input to be included in the training vector. For example, training component 351 may retrieve a training image from memory 310 and convert the pixel matrix into a vector that mathematically represents selected features of the training image. The at least one feature may be one or more of the following: the outline of an employee uniform, the color representation of the employee uniform, the presence of a logo or name tag on the employee uniform. For example, refer to... Figure 1 The training component 351 can capture the dominant color in each region of the visual boundary 106 (e.g., the black uniform of person 104, the yellow hat of person 104, etc.), or it can capture the shape of the employee uniform through edge detection. The captured information representing the selected features can be stored in the training vector (or any suitable data structure) to which a classification function can be applied. For simplicity, the classification function can be the dot product between the training vector and the learned weight vector. The result of the dot product can be a value such as 1 or -1, representing "employee" and "non-employee," respectively.

[0050] At box 502, method 500 may include training a machine learning model using training vectors to identify a predetermined type of person in an input image. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or training component 351 may be configured to or may include means for training a machine learning model using training vectors to identify a predetermined type of person in an input image.

[0051] For example, training component 351 can determine a learned weight vector that produces a known output. Through user interface 100, training component 351 receives labels indicating that the training input depicts an employee. If the subclass "employee" is represented by the value "1", then training component 351 learns a weight vector that produces a dot product of 1 when applied to the training vector. This weight vector is later used for new input images. It should be noted that the dot product is a simplified instance of a function used in machine learning models. Those skilled in the art will understand that functions and methods for reducing error (e.g., gradient descent) can vary depending on the machine learning model used.

[0052] At box 504, method 500 may further include identifying another person in a different image using image classification. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or recognition component 352 may be configured to, or may include, means for identifying another person in a different image using image classification. For example, the identification at box 504 may involve identifying person 204 in image 200.

[0053] At box 506, method 360 may further include using the machine learning model to determine whether another person belongs to a predetermined type. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or determination component 353 may be configured to, or may include, means for using the machine learning model to determine whether another person belongs to a predetermined type. For example, determination component 353 may generate visual boundaries around person 204, generate a cropped version of image 200 including only the content of the visual boundaries, vectorize the cropped version of image 200, and apply a classification function using learned weight vectors.

[0054] In response to determining that another person belongs to a predetermined type (e.g., type 110), method 500 may proceed to box 508, or optionally proceed to method 600.

[0055] At box 508, method 500 may further include using a camera to monitor another person until the other person is present in the camera's field of view. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or monitoring component 354 may be configured to, or may include, means for monitoring another person using a camera until the other person is present in the camera's field of view in response to determining that the other person belongs to a predetermined type.

[0056] For example, monitoring at frame 508 could include tracking, where person 204 moves within the field of view of a security camera installed by the department store owner.

[0057] In this optional aspect, at box 510, method 500 may further include generating statistical data associated with another person based on monitoring. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315 and / or generation component 355 may be configured to, or may include, means for generating statistical data associated with another person based on monitoring.

[0058] For example, the generation at box 510 can include determining how long person 204 remains idle, how long person 204 stays near non-employees such as person 206 (indicating customer interaction), and the location where person 204 is most frequently found in the environment.

[0059] In this optional aspect, at block 512, method 500 may further include transmitting statistical data to a device. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or transmission component 356 may be configured to or may include means for transmitting statistical data to a device.

[0060] For example, the transmission at box 512 could include sending various statistics from a modem or transmitter to the department store owner's device, for example, via a wired or wireless communication link. This allows the owner to evaluate employee performance through an objective lens.

[0061] Suppose that at 506, determining component 353 determines that another person does not belong to a predetermined type. In response, method 500 can proceed to 514, where person type identifier component 315 and / or recognition component 352 can determine whether another person exists in a different image. For example, recognition component 352 can first identify person 206 and can determine that person 206 does not belong to type 110 (i.e., is a non-employee).

[0062] If it is determined that another person exists in a different image (e.g., person 204), method 500 returns to 504, where the person type identifier component 315 and / or recognition component 352 identify the other person. Method 500 then loops between 504 and 514 until all persons in the different images (e.g., image 200) are classified as belonging to or not belonging to a predetermined type. When no other person remains in the different images, method 500 terminates at 516. In the context of the video, the person type identifier component 315 can determine whether a person of type 110 exists in every selected number of frames. For example, component 315 can evaluate every 300th frame in a video of 60 frames per second. The administrator can adjust the number of frames.

[0063] Figure 6 The flowchart illustrates a method 600 for classifying verifier types according to an exemplary aspect of this disclosure. This optional aspect may be implemented after determining in block 506 of method 500 that another person belongs to a predetermined type.

[0064] At box 602, method 600 may include a confidence score of the probability that the received indication output is correct. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or receiving component 357 may be configured to, or may include, means for receiving a confidence score of the probability that the indicated output is correct.

[0065] For example, a machine learning model might output a correct probability (e.g., 60%). In some cases, non-employees might look very similar to employees. For instance, a non-employee might wear clothing similar to an employee's uniform. To prevent misclassification, the machine learning model outputs a confidence score that can be used to determine if further training is needed. In some respects, the confidence score is a numerical representation of the deviation between the training images used to train the machine learning model and the images of the people being classified.

[0066] In this optional aspect, at block 604, method 360 may further include determining whether the confidence score is higher than a threshold confidence score. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or receiving component 357 may be configured to, or may include, means for determining whether the confidence score is higher than a threshold confidence score.

[0067] For example, the threshold confidence score can be a configurable value set by the administrator, such as, but not limited to, 80%. If the confidence score is below this value, the output may be unreliable. Therefore, in response to determining that the confidence score is not higher than the threshold confidence score, the person type identifier component 315 verifies the output from the machine learning model by implementing the steps in boxes 606, 608, and 610.

[0068] In this optional aspect, at block 606, method 600 may further include dividing different images into multiple different image portions. For example, in one aspect, computer device 300, processor 305, memory 310, human type identifier component 315 and / or partitioning component 330 may be configured to or may include means for dividing different images into multiple different image portions.

[0069] As an example, and return a reference. Figure 2The machine learning model can classify person 204 as an employee, but the associated confidence score could be 70%. This probability is determined based on the deviation between the training images used to train the machine learning model and the image of person 204. This deviation might be sufficient to guess that person 204 is not a non-employee, but it might not be enough to definitively confirm this. Because the confidence score is less than the 80% threshold confidence score, the person type identifier component 315 uses the same method as generating the training images for the machine learning model. For example, the segmentation component 330 can segment image 200 to be presented on user interface 100.

[0070] At box 608, method 600 may further include generating the plurality of different image portions on the user interface for display. For example, in one aspect, computer device 300, processor 305, memory 310, human type identifier component 315 and / or generation component 335 may be configured to, or may include, means for generating the plurality of different image portions on user interface 100 for display.

[0071] At box 610, method 600 may further include receiving, via a user interface, another selection of at least a portion of a plurality of distinctly divided image portions, wherein the other selection indicates that another person belongs to a predetermined type. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or receiving component 340 may be configured to, or may include, means for receiving, via user interface 100, another selection of at least a portion of a plurality of distinctly divided image portions (of image 200), wherein the other selection indicates that the other person (e.g., person 204) belongs to the predetermined type.

[0072] For example, a user can select a portion of image 200 that includes person 204 on user interface 100. Based on the confirmation that person 204 does indeed belong to type 110 (as determined by the same machine learning model), at box 612, method 600 may further include generating another training input in response to determining that another selection includes another person, said other training input including another training image depicting the other person and another label indicating that the other person belongs to the predetermined type. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315 and / or generation component 345 may be configured to or may include means for generating another training input in response to determining that the other selection includes the other person, said other training input including another training image depicting the other person and another label indicating that the other person belongs to the predetermined type.

[0073] If person 204 is excluded from the selection, the person type identifier component 315 determines that the machine learning model has been misclassified. In this case, another training input may still be generated, but the label will indicate that another person (i.e., person 204) does not belong to type 110. In this case, the machine learning model is retrained such that for this particular training input, the learned weight vector produces -1 when the classification function is applied, and for all other training inputs describing employees, the learned weight vector produces 1.

[0074] If at box 604, the person type identifier component 315 determines that the confidence score is higher than the threshold confidence score, then the person type identifier component 315 may not validate the output of the machine learning model and method 600 may proceed directly from box 604 to box 612.

[0075] Figure 7 This is a flowchart illustrating a method 700 for filtering potential training images based on human visibility, according to an exemplary aspect of this disclosure. This optional aspect may be implemented by the human type identifier component 315 after block 404 of implementing method 400.

[0076] At box 702, method 700 may further include determining whether at least a predetermined number of identified human bodies are depicted in the image. For example, in one aspect, computer device 300, processor 305, memory 310, human type identifier component 315, and / or determination component 358 may be configured to, or may include, means for determining whether at least a predetermined number of identified human bodies are depicted in the image before segmenting the image.

[0077] For example, the determination at box 702 could include determining whether a person's arms, legs, torso, and head are clearly visible in the image. Predetermined quantities could include at least one of the two arms, at least one of the two legs, the torso, and the head. If at least the predetermined quantities are not adequately depicted (e.g., the body is outside the frame, cropped, or behind an obstacle), using the image to train a machine learning model may produce undesirable results. Method 700 enables the person type identifier component 315 to filter out good training images from poor training images.

[0078] If, at box 702, the person type identifier component 315 determines that at least a predetermined threshold amount of body is not depicted (or is not visible), then method 700 proceeds to 704. In this alternative aspect, at box 704, method 700 may further include discarding the image. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315, and / or discarding component 359 may be configured to, or may include, means for discarding the image. This prevents the image from being used for training purposes. It should be noted that, in the context of this disclosure, discarding means not considering it as a training image. The image may not necessarily be deleted.

[0079] In this alternative aspect, at block 706, method 700 may further include identifying people in different images, said different images depicting at least a predetermined number of the bodies of the identified people. For example, in one aspect, computer device 300, processor 305, memory 310, person type identifier component 315 and / or identification component 320 may be configured to, or may include, means for identifying said people in different images, said different images depicting at least the bodies of said predetermined number of the identified people.

[0080] For example, recognition at box 706 could involve receiving a new image. The new image could be a subsequent frame in the input video. A discarded image could partially depict the person 204 entering the frame, while a subsequent frame could depict the entire body of the person 204. Therefore, the subsequent frame is used in box 402 to identify the person 204 because at least a predetermined amount of the person 204's body is shown.

[0081] After implementing box 706, method 700 proceeds to box 402 of method 400. If, at box 702, the human type identifier component 315 determines that a body is depicted (or visible) at least a predetermined threshold amount, then method 700 further proceeds to box 402 of method 400. In this case, the image can be used for training purposes and does not need to be discarded.

[0082] Although illustrative aspects and / or embodiments have been discussed above, it should be noted that various changes and modifications may be made herein without departing from the scope of the described aspects and / or embodiments as defined by the appended claims. Furthermore, while elements of the described aspects and / or embodiments may be described or claimed in the singular, the plural form is contemplated unless expressly specified as singular. Additionally, all or part of any aspect and / or embodiment may be used in conjunction with all or part of any other aspect and / or embodiment unless otherwise stated.

Claims

1. A method for classifying human types in visual media, the method comprising: Use image classification to identify people in images; Generate a visual boundary around the identified person on the image; The image is divided into multiple image parts, such that the multiple image parts partially depict the identified person inside the visual boundary; The plurality of image portions are generated on a user interface for display, the user interface prompting the user to select at least partially depicting each part of a predetermined type of person from the plurality of image portions; The user interface receives a selection of at least a portion of the segmented image. Training input is generated for the machine learning model and in response to determining that the selection includes the identified person inside the visual boundary, the training input including a training image depicting the content inside the visual boundary and a label indicating that the identified person inside the visual boundary belongs to the predetermined type; as well as The training input is stored in memory.

2. The method according to claim 1, wherein the machine learning model is a type of support vector machine.

3. The method according to claim 1, further comprising: Vectorize the training input to create training vectors; The machine learning model is trained using the training vectors to identify the person of the predetermined type in the input image.

4. The method of claim 3, further comprising: Use image classification to identify another person in different images; as well as The machine learning model is used to determine whether the other person belongs to the predetermined type.

5. The method of claim 4, wherein the different images are captured by a camera, the method further comprising: In response to determining that the other person belongs to the predetermined type, the camera is used to monitor the other person until the other person is in the field of view of the camera; Based on the monitoring, statistical data associated with the other person is generated; and The statistical data is transmitted to the device.

6. The method of claim 4, further comprising: Receive output from the machine learning model indicating that the other person belongs to the predetermined type; as well as Verify the output using the following methods: The different images are divided into multiple different image parts; The multiple different image portions are generated on the user interface for display. The user interface receives another selection for at least a portion of a plurality of distinctly divided image segments, wherein the other selection indicates that the other person belongs to the predetermined type; and In response to determining that the other selection includes the other person, another training input is generated, which includes another training image depicting the other person and another label indicating that the other person belongs to the predetermined type.

7. The method of claim 6, wherein the machine learning model further outputs a confidence score indicating the probability that the output is correct, and wherein the verification is performed in response to determining that the confidence score is below a threshold confidence score.

8. The method of claim 3, wherein the predetermined type is an employee, and wherein vectorizing the stored training input to create the training vector further comprises: Identify at least one feature of the training input to include in the training vector, wherein the at least one feature is one or more of the following: The outline of the employee uniform; The color of the employee uniform indicates; The markings on the employee uniforms; or The existence of brand names.

9. The method of claim 1, further comprising: Before segmenting the image, determine whether the image depicts at least a predetermined amount of the identified person's body; The image is discarded in response to the determination that at least the predetermined amount of the body is not depicted; and Identify the person in different images, wherein the different images depict at least a predetermined number of the bodies of the identified person.

10. An apparatus for classifying human types in visual media, the apparatus comprising: Memory; as well as A processor, which communicates with the memory and is configured to: Use image classification to identify people in images; Generate a visual boundary around the identified person on the image; The image is divided into multiple image parts, such that the multiple image parts partially depict the identified person inside the visual boundary; The plurality of image portions are generated on a user interface for display, the user interface prompting the user to select at least partially depicting each part of a predetermined type of person from the plurality of image portions; The user interface receives a selection of at least a portion of the segmented image. Training input is generated for the machine learning model and in response to determining that the selection includes the identified person inside the visual boundary. The training input includes training images depicting the content inside the visual boundary and labels indicating that the identified person inside the visual boundary belongs to the predetermined type. The training input is stored in the memory.

11. The device of claim 10, wherein the machine learning model is a type of support vector machine.

12. The device of claim 10, wherein the processor is further configured to: Vectorize the training input to create training vectors; The machine learning model is trained using the training vectors to identify the person of the predetermined type in the input image.

13. The device of claim 12, wherein the processor is further configured to: Use image classification to identify another person in different images; and The machine learning model is used to determine whether the other person belongs to the predetermined type.

14. The device of claim 13, wherein the different images are captured by a camera, wherein the processor is further configured to: In response to determining that the other person belongs to the predetermined type, the camera is used to monitor the other person until the other person is in the field of view of the camera; Based on the monitoring, statistical data associated with the other person is generated; and The statistical data is transmitted to the device.

15. The device of claim 13, wherein the processor is further configured to: Receive from the machine learning model an output indicating that the other person belongs to the predetermined type; and Verify the output using the following methods: The different images are divided into multiple different image parts; The multiple different image portions are generated on the user interface for display. The user interface receives another selection for at least a portion of a plurality of distinctly divided image segments, wherein the other selection indicates that the other person belongs to the predetermined type; and In response to determining that the other selection includes the other person, another training input is generated, which includes another training image depicting the other person and another label indicating that the other person belongs to the predetermined type.

16. The device of claim 15, wherein the machine learning model further outputs a confidence score indicating the probability that the output is correct, and wherein the verification is performed in response to determining that the confidence score is below a threshold confidence score.

17. The device of claim 12, wherein the predetermined type is an employee, and wherein the stored training input is vectorized to create the training vector, the processor is further configured to: Identify at least one feature of the training input to include in the training vector, wherein the at least one feature is one or more of the following: The outline of the employee uniform; The color of the employee uniform indicates; The markings on the employee uniforms; or The existence of brand names.

18. The device of claim 10, wherein the processor is further configured to: Before segmenting the image, determine whether the image depicts at least a predetermined amount of the identified person's body; The image is discarded in response to the determination that at least the predetermined amount of the body is not depicted; and Identify the person in different images, wherein the different images depict at least a predetermined number of the bodies of the identified person.

19. A computer-readable medium for classifying human types in visual media, the computer-readable medium being executable by a processor to: Use image classification to identify people in images; Generate a visual boundary around the identified person on the image; The image is divided into multiple image parts, such that the multiple image parts partially depict the identified person inside the visual boundary; The plurality of image portions are generated on a user interface for display, the user interface prompting the user to select at least partially depicting each part of a predetermined type of person from the plurality of image portions; The user interface receives a selection of at least a portion of the segmented image. Training input is generated for the machine learning model and in response to determining that the selection includes the identified person inside the visual boundary. The training input includes training images depicting the content inside the visual boundary and labels indicating that the identified person inside the visual boundary belongs to the predetermined type. The training input is stored in memory.

20. A method for classifying human types in visual media, the method comprising: Use image classification to detect people in images; Generate the visual boundary around the detected person on the image; The image is divided into multiple image parts, such that the multiple image parts partially depict the detected person inside the visual boundary; The plurality of image portions are generated on a user interface for display, the user interface prompting the user to select at least partially depicting each part of a predetermined type of person from the plurality of image portions; The user interface receives a selection of at least a portion of the segmented image. Training input is generated for the machine learning model and in response to determining that the selection includes the detected person inside the visual boundary, the training input including a training image depicting the content inside the visual boundary and a label indicating that the detected person inside the visual boundary belongs to the predetermined type; as well as The training input is stored in memory.

21. An apparatus for classifying human types in visual media, the apparatus comprising: Memory; as well as A processor, which communicates with the memory and is configured to: Use image classification to detect people in images; Generate the visual boundary around the detected person on the image; The image is divided into multiple image parts, such that the multiple image parts partially depict the detected person inside the visual boundary; The plurality of image portions are generated on a user interface for display, the user interface prompting the user to select at least partially depicting each part of a predetermined type of person from the plurality of image portions; The user interface receives a selection of at least a portion of the segmented image. Training input is generated for the machine learning model and in response to determining that the selection includes the detected person inside the visual boundary. The training input includes a training image depicting the content inside the visual boundary and a label indicating that the detected person inside the visual boundary belongs to the predetermined type. The training input is stored in the memory.

22. A method for classifying human types in visual media, comprising any combination of the technical features of claims 1-9.

23. An apparatus for classifying human types in visual media, comprising any combination of the technical features of claims 10-18.

24. A computer-readable medium for classifying human types in visual media, comprising any combination of the technical features of claim 19.

25. A method for classifying human types in visual media, comprising any combination of the technical features of claim 20.

26. An apparatus for classifying human types in visual media, comprising any combination of the technical features of claim 21.

Images