A groping method of the facial features

Abstract

A facial feature grouping method is applicable to grouping multiple facial feature objects and is implemented by a computing device including a processing unit and a storage unit. After the processing unit obtains multiple sets of facial feature object data using a facial feature object detection model, it obtains multiple sets of predicted facial feature object data corresponding to a person to be predicted using a facial feature object ratio prediction model. The processing unit classifies these facial feature object data into multiple groups and obtains a matching score for each group based on the facial feature object data and the predicted facial feature object data within each group. The processing unit then regroups the facial feature object data based on the matching scores.

Description

Technical Field

[0001] This invention relates to a grouping method, and more particularly to a facial feature grouping method for classifying multiple facial features. Prior Technology

[0002] Currently, with the increasing market demand for various image recognition technologies, there is often a need to find the face of a specific target in an image containing multiple people for analysis or image processing. However, due to the lack of distance and depth information in two-dimensional images, existing image analysis may confuse other targets with the specific target, leading to misidentification of different people's faces. If this situation occurs in the application of vehicle driving monitoring, it may cause serious driving safety problems.

[0003] Traditional in-vehicle driver monitoring systems use fixed cameras to capture two-dimensional images of the vehicle's interior and employ facial recognition technology to identify the driver's facial features and determine if the driver is distracted or fatigued. However, because two-dimensional images lack depth information, in situations with multiple passengers, facial recognition technology can easily confuse the driver's facial movements with those of passengers in the back seat or front passenger seat. This can lead to the system misjudging the driver's driving status, frequently triggering emergency alerts, and ultimately causing the driver to become distracted and cause accidents.

[0004] Current object grouping technologies on the market group objects by calculating the distance or similarity between multiple objects to be classified, or by calculating the intersection over union (IOU) ratio between objects to be classified. However, due to the limitation that in-vehicle camera lenses can only capture two-dimensional images, they lack third-dimensional depth information and therefore cannot distinguish whether the acquired facial information belongs to the driver in the front seat or a passenger in the back seat. Therefore, it is extremely important to develop a method that can group the facial features of multiple passengers and the driver in a vehicle to overcome the misjudgment caused by the driver's face being confused with the face of the front seat driver and passengers in the driving monitoring system. Summary of the Invention

[0005] Therefore, the purpose of this invention is to provide a facial feature grouping method suitable for grouping the facial features of the driver and multiple passengers located in the front and rear seats of a vehicle.

[0006] Therefore, the facial feature grouping method of the present invention is applicable to grouping all facial feature objects in an image containing K faces into K groups. Each face in the image contains N facial feature objects corresponding to N feature categories, and the process is implemented by a computing device. The computing device includes a storage unit storing a facial feature object detection model and a facial feature object ratio prediction model, an image acquisition unit, and a processing unit electrically connected to the storage unit and the image acquisition unit. The facial feature object detection model is used to obtain multiple facial feature object data related to multiple facial feature objects to be detected from an input image. The facial feature object ratio prediction model is used to predict N output facial feature object prediction data corresponding to N output facial feature objects indicating the input person based on input facial feature object data corresponding to one of the input facial feature objects of an input person. The facial feature grouping method includes steps (A), (B), (C), (D), and (E).

[0007] In step (A), the processing unit uses the facial feature object detection model to obtain K×N facial feature object data corresponding to the K×N facial feature objects based on the image.

[0008] In step (B), for each facial feature object, the processing unit treats the facial feature object as a facial feature object to be predicted for a person to be predicted, and uses the facial feature object ratio prediction model to obtain N predicted facial feature object data corresponding to the N predicted facial feature objects of the person to be predicted based on the data of the predicted facial feature object corresponding to the predicted facial feature object.

[0009] In step (C), the processing unit divides the K×N facial feature data corresponding to the K×N facial feature objects into K groups corresponding to the K faces.

[0010] In step (D), for each of the K×N facial feature objects, the processing unit uses the facial feature object as a reference facial feature object and the facial feature object data corresponding to the reference facial feature object as reference facial feature object data. Based on the N facial feature object data in the group corresponding to the reference facial feature object data and the N predicted facial feature object data obtained in step (B) based on the reference facial feature object data, a matching score indicating the probability that the reference facial feature object data is classified into the correct group is obtained.

[0011] In step (E), the processing unit takes the facial feature object corresponding to the target facial feature object data with the minimum matching score in all the baseline facial feature object data as a facial feature object to be re-segmented, and adjusts the facial feature object data corresponding to the facial feature object to be re-segmented in the K groups for the facial feature object to be re-segmented, so as to obtain K updated groups.

[0012] Another objective of this invention is to provide a facial feature grouping method suitable for grouping the facial features of multiple passengers in a vehicle.

[0013] Therefore, the facial feature grouping method of the present invention is applicable to grouping all facial feature objects in an image containing K faces into K groups. Each face in the image contains N facial feature objects corresponding to N feature categories. This is implemented by a computing device, which includes a storage unit storing a facial feature object detection model, a facial feature object proportion prediction model, and an object grouping model; an image acquisition unit; and a processing unit electrically connected to the storage unit and the image acquisition unit. The facial feature object detection model is used to obtain multiple target facial feature objects related to multiple target facial feature objects from an input image. The facial feature object detection model is used to predict N output facial feature object prediction data corresponding to N output facial feature objects that indicate the input person, based on the input facial feature object data corresponding to one of the input facial feature objects. The object grouping model is used to determine a correction method among the facial feature object data corresponding to the same feature category in multiple input groups based on an input state. The facial feature grouping method includes a step (A), a step (B), a step (C), a step (D), a step (E), and a step (F).

[0014] In step (A), the processing unit uses the facial feature object detection model to obtain K×N facial feature object data corresponding to the K×N facial feature objects based on the image.

[0015] In step (B), for each facial feature object, the processing unit treats the facial feature object as a facial feature object to be predicted for a person to be predicted, and uses the facial feature object ratio prediction model to obtain N predicted facial feature object data corresponding to the N predicted facial feature objects of the person to be predicted based on the data of the predicted facial feature object corresponding to the predicted facial feature object.

[0016] In step (C), the processing unit divides the K×N facial feature data corresponding to the K×N facial feature objects into K groups corresponding to the K faces.

[0017] In step (D), for each of the K×N facial feature objects, the processing unit uses the facial feature object as a reference facial feature object and the facial feature object data corresponding to the reference facial feature object as reference facial feature object data. Based on the N facial feature object data in the group corresponding to the reference facial feature object data and the N predicted facial feature object data obtained in step (B) based on the reference facial feature object data, a matching score indicating the probability that the reference facial feature object data is classified into the correct group is obtained.

[0018] In step (E), the processing unit takes the K×N facial feature objects as a current state and uses the object grouping model based on the current state to obtain an adjustment method related to the multiple facial feature object data corresponding to the same feature category in the K groups.

[0019] In step (F), the processing unit adjusts the K groups according to the adjustment method to obtain K updated groups.

[0020] Another objective of this invention is to provide a facial feature grouping method suitable for grouping the facial features of multiple passengers in a vehicle.

[0021] Therefore, the facial feature grouping method of the present invention is applicable to an in-vehicle monitoring system with an alarm function, which is used to issue an alarm when an abnormal event is detected in a driver. The facial feature grouping method is implemented by a computing device, which includes a storage unit storing a facial feature object detection model, a facial feature object ratio prediction model, and a storage unit that divides the K×N facial feature object data corresponding to K×N facial feature objects into K groups corresponding to K passenger faces, an image acquisition unit, and a processing unit electrically connected to the storage unit and the image acquisition unit. The feature grouping method includes a step (A), a step (B), a step (C), and a step (D).

[0022] In step (A), the processing unit uses the facial feature object detection model to obtain K×N facial feature object data corresponding to the K×N facial feature objects based on an in-vehicle image.

[0023] In step (B), for each facial feature object, the processing unit treats the facial feature object as a facial feature object to be predicted for a person to be predicted, and uses the facial feature object ratio prediction model to obtain N predicted facial feature object data corresponding to the N predicted facial feature objects of the person to be predicted based on the data of the predicted facial feature object corresponding to the predicted facial feature object.

[0024] In step (C), for each of the K×N facial feature objects, the processing unit uses the facial feature object as a reference facial feature object and the facial feature object data corresponding to the reference facial feature object as reference facial feature object data. Based on the N facial feature object data in the group corresponding to the reference facial feature object data and the N predicted facial feature object data obtained in step (B) based on the reference facial feature object data, a matching score indicating the probability that the reference facial feature object data is classified into the correct group is obtained.

[0025] In step (D), when the matching score is less than a threshold value, the in-vehicle monitoring system suspends an alarm function.

[0026] The advantages of this invention are as follows: The processing unit groups the facial feature data obtained by the facial feature detection model, and obtains matching scores using the predicted facial feature data and the facial feature data obtained by the facial feature ratio prediction model. The processing unit adjusts the facial feature data in the groups according to the matching scores to obtain updated groups, and correctly marks the facial features in the image as the corresponding faces. Furthermore, this invention can also be applied to detecting the facial features of a driver among multiple passengers in a vehicle, thereby reducing the problem of a vehicle-mounted driver monitoring system misjudging the driver's driving status and frequently triggering an emergency notification system. Simple Explanation of the Diagram

[0027] Other features and effects of the present invention will be clearly presented in the embodiments with reference to the drawings, wherein: Figure 1 is a block diagram illustrating a processing unit signal connection between an image acquisition unit and a storage unit in a first embodiment of the facial feature grouping method of the present invention; Figure 2 is a flowchart illustrating the first embodiment of the facial feature grouping method of the present invention; Figure 3 is a flowchart illustrating an initial clustering procedure of the first embodiment; Figure 4 is a flowchart illustrating an error level acquisition procedure of the first embodiment; Figure 5 is a flowchart illustrating a cluster update procedure of the first embodiment; Figure 6 is a flowchart illustrating a second embodiment of the facial feature grouping method of the present invention; Figure 7 is a flowchart illustrating the object grouping model training procedure of the second embodiment; Figure 8 is a schematic diagram illustrating an error matrix formed by multiple errors in each group; and Figure 9 is a flowchart illustrating the result of multiple iterations of the error matrix. Implementation

[0028] Before the invention is described in detail, it should be noted that similar elements are represented by the same numbers in the following description.

[0029] Referring to Figure 1, a first embodiment of the facial feature grouping method of the present invention is applicable to grouping all facial feature objects in an image containing K faces (i.e., the image contains a total of K faces) into K groups. Each face in the image contains N facial feature objects corresponding to N feature categories, wherein each face corresponds to the face of each person to be determined in the image, and the process is performed by a computing device 1. The computing device 1 includes a storage unit 11 storing a facial feature object detection model and a facial feature object ratio prediction model, an image acquisition unit 12, and a processing unit 13 electrically connected to the storage unit 11 and the image acquisition unit 12. The facial feature object detection model is used to obtain multiple facial feature object data related to multiple facial feature objects to be detected from an input image. The facial feature object ratio prediction model is used to predict N output facial feature object prediction data corresponding to N output facial feature objects that indicate the input person, based on one input facial feature object data corresponding to one of the input facial feature objects.

[0030] It is worth noting that, in this embodiment, the image acquisition unit 12 can be implemented as, for example, a vehicle-mounted camera, a surround-view camera, or a forward-facing camera to acquire the image. The processing unit 13 is, for example, a central processing unit, a microprocessor, a single-board computer, or a vehicle computer. The N feature categories are, for example, a nose, a left eye, a right eye, a face, a left ear, a right ear, or a mouth, so N=7, and each feature category includes K facial feature objects, but is not limited to this.

[0031] Referring to Figures 1 and 2, the first embodiment of the facial feature grouping method of the present invention includes the following steps.

[0032] In step 21, the processing unit 13 uses the facial feature object detection model to obtain K×N facial feature object data corresponding to the K×N facial feature objects based on the image acquired by the image acquisition unit 12. In this embodiment, each facial feature object data includes a value indicating the true horizontal coordinate of the facial feature object in the image, a value indicating the true vertical coordinate of the facial feature object in the image, a value indicating the true horizontal width of the facial feature object in the image, a value indicating the true vertical height of the facial feature object in the image, a value indicating the true area ratio of the facial feature object to the nose in the image, a value indicating the true area ratio of the facial feature object to the left eye in the image, and a value indicating the facial feature object... The data includes the following parameters: the actual area ratio of the facial feature object to the right eye in the image; the actual area ratio of the facial feature object to the face (outline) in the image; the actual area ratio of the facial feature object to the left ear in the image; the actual area ratio of the facial feature object to the right ear in the image; the actual area ratio of the facial feature object to the mouth in the image; and the feature category to which the facial feature object itself belongs. Therefore, each facial feature object data is a vector with the above 12 actual attribute values. It is worth mentioning that the position and area indicated by the bounding box of the facial feature object can be defined by the 12 real attribute values ​​contained in the aforementioned facial feature object data. The real horizontal coordinates and the real vertical coordinates of the facial feature object in the image can further define the position of the bounding box, and the real horizontal width and the real vertical height of the facial feature object in the image can jointly define the area of ​​the bounding box. In other words, the K×N facial feature object data can be used to completely describe the K×N facial features appearing in the image, but it is not limited to this.

[0033] In step 22, for each facial feature object, the processing unit 13 treats the facial feature object as a facial feature object to be predicted for a person to be predicted. Similarly, the facial feature object data is also treated as facial feature object data to be predicted. Using the facial feature object ratio prediction model, N predicted facial feature object data corresponding to N predicted facial feature objects for the person to be predicted are obtained based on the predicted facial feature object data corresponding to the predicted facial feature object. In this embodiment, the facial feature object ratio prediction model can be implemented using a multilayer perceptron (MLP) architecture, for example, with a rectified linear unit (ReLU) as an activation function. Since the features of this invention do not lie in the MLP architecture known to those skilled in the art, their details are omitted here for simplicity.

[0034] It should be noted that the facial feature object data to be predicted in this case is actually the vector of the true attribute values ​​of one of the N facial feature objects to be predicted, input into the facial feature object proportion prediction model as the input data for the facial feature object proportion prediction model to predict the corresponding predicted facial feature object data. The facial feature object proportion prediction model can predict the coordinates and size that the predicted facial feature object data should appear in the image based on the coordinates and size of the facial feature object data to be predicted in the image.

[0035] In this first embodiment, when one of the N predicted facial features is, for example, the left eye of the person to be predicted, and the predicted facial feature is, for example, the nose of the person to be predicted, the processing unit 13 inputs facial feature data corresponding to the nose in the image, including a data type indicating the true horizontal coordinate of the nose in the image, a data type indicating the true vertical coordinate of the nose in the image, etc., as described in step 21. This data is then input into the facial feature proportion prediction model, and the model obtains a predicted horizontal coordinate indicating the left eye in the image, a predicted vertical coordinate indicating the left eye in the image, a predicted horizontal width indicating the left eye in the image, and a predicted vertical width indicating the left eye in the image, based on the nose as the predicted facial feature. The processing unit 13 generates a total of 7 predicted facial feature object data corresponding to the N feature categories, including a predicted vertical height dimension, a predicted area ratio of the left eye and the nose in the image, a predicted area ratio of the left eye and the right eye in the image, a predicted area ratio of the left eye and the face in the image, a predicted area ratio of the left eye and the left ear in the image, a predicted area ratio of the left eye and the right ear in the image, and a predicted area ratio of the left eye and the mouth in the image. Therefore, for each facial feature object to be predicted, the processing unit 13 can generate a total of 7 predicted facial feature object data corresponding to the above N feature categories using the facial feature object ratio prediction model. Each predicted facial feature object data is a vector with the above 11 predicted attribute values, but is not limited to this.

[0036] In step 23, the processing unit 13 divides the K×N facial feature data corresponding to the K×N facial feature objects into K groups corresponding to the K faces.

[0037] Referring to Figure 3, step 23 also includes sub-steps 231 to 232, which are described in detail below.

[0038] In sub-step 231, the processing unit 13 selects a reference object from the N facial feature objects, takes the feature category to which the reference object belongs as a reference category, and divides the K facial feature object data belonging to the K reference objects in the K faces into the K groups corresponding to the K faces.

[0039] In sub-step 232, for each of the feature categories to be classified other than the reference category, the processing unit 13 assigns each of the K facial feature data belonging to the category to be classified to one of the K groups based on the K facial feature data of the K reference objects corresponding to the K faces. In this embodiment, for each of the K facial feature objects belonging to the category to be classified, the processing unit 13 calculates the K Euclidean distances between the facial feature object and the K reference objects, and assigns the facial feature data of the facial feature object to the group to which the reference object with the smallest Euclidean distance belongs (i.e., the group containing the facial feature data of the reference object).

[0040] It is worth noting that in this first embodiment, when the reference object is, for example, the nose, the processing unit 13 divides the K facial feature data of the K noses into K groups, each belonging to a different face. Since N-1 is 6, the remaining K×6 facial feature objects each belong to one of the following features in the feature category: left eye, right eye, face, left ear, right ear, or mouth. Therefore, each feature category includes K facial feature objects. The processing unit 13 classifies each of the K×6 facial feature data corresponding to the K×6 facial feature objects into one of the K groups. Thus, each group includes the nose, left eye, right eye, face, left ear, right ear, and mouth (that is, using the reference object—the nose—as the center, and using the minimum Euclidean distance as a condition, the remaining 6 facial feature objects, excluding the reference object, are grouped).

[0041] In step 24, for each of the K×N facial feature objects, the processing unit 13 uses the facial feature object as a reference facial feature object and the facial feature object data corresponding to the reference facial feature object as reference facial feature object data. Based on the N facial feature object data in the group corresponding to the reference facial feature object data and the N predicted facial feature object data obtained in step 22 based on the reference facial feature object data, a matching score indicating the probability that the reference facial feature object data is classified into the correct group is obtained.

[0042] Referring to Figures 4 and 8, for each reference facial feature object, step 24 also includes sub-steps 241 to 242, which are described in detail below.

[0043] In sub-step 241, for each of the N facial feature object data in the group corresponding to the reference facial feature object data, the processing unit 13 obtains an error related to a numerical difference based on the size of the facial feature object j in the facial feature object data and the size of the predicted facial feature object j corresponding to the predicted facial feature object data obtained from the reference facial feature object data of the reference facial feature object i. In this embodiment, the processing unit 13 generates the error Dij using the following formula: in, The size of facial feature object j, one of the facial feature object data; To determine the size of the predicted facial feature object j corresponding to the reference facial feature object i, the processing unit 13 inputs the facial feature object i's facial feature object data as the predicted facial feature object data into the facial feature object ratio prediction model, and generates the predicted facial feature object data for the predicted facial feature object j. The predicted facial feature object data indicates the predicted area ratio between the facial feature object i and the facial feature object j. The processing unit 13 then... The predicted area ratio of facial feature object i to facial feature object j is multiplied by the product of the actual horizontal width and actual vertical height of facial feature object i to obtain the predicted area size of facial feature object j. Then, the actual horizontal width of facial feature object j is multiplied by the actual vertical height of facial feature object j to obtain the actual area size of facial feature object j. The result of comparing the predicted area size of facial feature object j with the actual area size of facial feature object j is the error Dij. A larger Dij value indicates a greater numerical difference between the two, and the smaller the matching score calculated by superimposing multiple Dij values, the lower the probability that the face should be correctly assigned to one of the K groups corresponding to the aforementioned facial feature object data. Conversely, a smaller numerical difference results in a larger matching score, indicating a greater probability that the face should be correctly assigned to one of the K groups. In this embodiment, the error matrix 81 in Figure 8 is composed of 7x7 errors 82.

[0044] It is worth noting that, in this embodiment, when Dij is, for example, an error obtained from the size of the left eye based on the facial feature object data and the size of the predicted left eye based on the nose corresponding to the facial feature object data (i.e., facial feature object i is the nose, and facial feature object j is the left eye), the processing unit 13 inputs the facial feature object data of the nose as the facial feature object data to be predicted into the facial feature object ratio prediction model, and generates the predicted area ratio of the nose and the left eye in the predicted facial feature object data of the left eye. The processing unit 13 compares the predicted area ratio of the nose and the left eye obtained by multiplying the predicted area ratio of the nose and the left eye by the actual horizontal width and the actual vertical height of the nose, with the actual area size of the left eye obtained by multiplying the actual horizontal width and the actual vertical height of the left eye, but is not limited to this.

[0045] In sub-step 242, the processing unit 13 obtains the matching score Ei based on the N errors obtained in step 241 and using the following formula: Where Dij is the error obtained in step 241; N is the number of predicted facial feature objects, but is not limited to this.

[0046] In step 25, the processing unit 13 determines whether there are multiple candidate facial feature objects in all the reference facial feature object data that have a matching score less than a threshold value. When the processing unit determines that such candidate facial feature object data exists, the processing unit executes step 26. When the processing unit determines that such candidate facial feature object data does not exist, the processing unit executes step 27. It is worth noting that the matching score is the sum of the accumulated errors of Dij minus 1. The larger the value of Dij, the smaller the final calculated matching score will be, indicating a greater possibility that the facial feature object should not be classified into the correct group. This is because when facial feature object i is classified into an inappropriate group, the Dij calculated by the ratio of the predicted area of ​​facial feature object i to that of facial feature object j, obtained from facial feature object i, should be larger than the actual area size of facial feature object j. Consequently, Ei will naturally be smaller in the calculation. In this embodiment, the processing unit 13 uses the following formula to determine whether the matching score is less than the threshold value: Where Ei is the matching score generated by using the facial feature object i as the reference facial feature object; eth is the threshold value, but is not limited to it.

[0047] In step 26, the processing unit 13 takes the candidate facial feature data with the lowest matching score from all the reference facial feature data as a target facial feature data, and takes the corresponding facial feature data as a facial feature data to be re-divided. The processing unit 13 adjusts the facial feature data in the K groups corresponding to the facial feature data to be re-divided for the facial feature data to be re-divided, so as to obtain K updated groups. That is, the reference facial feature data corresponding to the group with the lowest matching score in the group is specifically redistributed to the K groups.

[0048] Referring to Figures 5 and 9, step 26 also includes sub-steps 261 to 263, which are described in detail below.

[0049] In sub-step 261, the processing unit 13 sorts the matching scores corresponding to the candidate facial feature object data and obtains the target facial feature object data with the smallest matching score from the candidate facial feature object data.

[0050] In sub-step 262, the processing unit 13, based on the target facial feature data and the corresponding adjusted feature category to which the facial feature to be reclassified belongs, obtains, from the remaining candidate facial feature data (excluding the target facial feature data), a corresponding adjusted facial feature object that belongs to the same adjusted feature category and has the lowest matching score. For example, when the adjusted feature category corresponding to the facial feature to be reclassified belongs to the nose, the processing unit 13 searches the remaining candidate facial feature data for at least one of the facial feature objects to be exchanged that also belongs to the adjusted feature category and is the nose. The processing unit 13 obtains the adjusted facial feature object with the lowest matching score as an adjusted facial feature object and obtains the adjusted facial feature object data corresponding to the adjusted facial feature object, but this is not limited to this.

[0051] In sub-step 263, the processing unit 13 exchanges the target facial feature data with the facial feature data to be adjusted to obtain the K updated groups. Then it returns to step 24. In this embodiment, after the processing unit 13 executes sub-step 263, K-2 of the K updated groups remain consistent with the K-2 groups. Only two updated groups exchange the target facial feature data originally in those two groups with the facial feature data to be adjusted, but this is not a limitation. By repeating steps 24-26, the number of Ei not greater than eth can be gradually reduced, eventually making all Ei greater than eth, thus achieving the goal of algorithm convergence. As shown in Figure 9, after 24 iterations, the error matrix 91 gradually conforms to the expected value for Ei.

[0052] In step 27, when the processing unit 13 determines that the at least one candidate facial feature object data does not exist, it uses the most recently obtained K groups as a grouping result. It is worth noting that in this embodiment, when the processing unit 13 determines that the at least one candidate facial feature object data does not exist during the first execution of step 25, the processing unit 13 uses the K groups obtained during the first execution of step 23 as the grouping result. If the processing unit 13 determines that the at least one candidate facial feature object data does not exist, for example, after the third execution of step 25, the processing unit 13 uses the K updated groups obtained during the second execution of step 26 as the grouping result, but this is not a limitation.

[0053] The facial feature grouping method of this invention can be applied to facial feature grouping applications involving multiple faces, such as an in-vehicle driver facial detection system and a criminal suspect image recognition system. Taking the application to the in-vehicle driver facial detection system as an example, the facial feature grouping method of this invention can be used to classify the facial features of multiple passengers in the vehicle. For ease of understanding, the following describes a method for grouping the facial features of passengers in the vehicle using the facial feature grouping method of this invention through a second embodiment.

[0054] Referring to Figures 1 and 6, the second embodiment of the facial feature grouping method is implemented using the same computing device 1 as the first embodiment in Figure 1. The only difference is that the storage unit 11 also stores an object grouping model. The object grouping model is used to determine a correction method among facial feature object data corresponding to the same feature category in multiple input groups based on an input state. In the second embodiment of the facial feature grouping method, steps 31-34 are the same as steps 21-24 in the first embodiment. The similarities will not be repeated. The difference is that in step 35, when the processing unit 13 determines that at least one candidate facial feature object data exists, the process proceeds to step 36; when the processing unit determines that at least one candidate facial feature object data does not exist, the process proceeds to step 38. The second embodiment of the facial feature grouping method adjusts the K groups through steps 36 and 37.

[0055] In step 36, the processing unit 13 takes the K×N matching scores corresponding to the K×N facial feature objects as a current state, and uses the object grouping model based on the current state to obtain an adjustment method among multiple facial feature object data corresponding to the same feature category in the K groups. In this embodiment, the facial feature object data to be adjusted consists of two pieces, and the adjustment method indicates that the facial feature object data to be adjusted is to be interchanged.

[0056] In step 37, the processing unit 13 adjusts the K groups according to the adjustment method to obtain K updated groups, and then returns to step 34.

[0057] In step 38, when the processing unit 13 determines that there is no candidate facial feature object data, the most recently obtained K groups are used as a grouping result.

[0058] Referring to Figure 7, the processing unit 13 of the computing device 1 obtains the object grouping model by performing an object grouping model training procedure. In this second embodiment, the object grouping model is implemented using a reinforcement learning (RL) algorithm, such as a deep Q-learning (DQN) model, and employs, for example, the Bellman function, but is not limited thereto. The object grouping model training procedure is described in detail below through steps 701-705. It is worth mentioning that the reinforcement learning algorithm can use an agent as a decision-making core and perform a specific action on an environment based on current information (state), and continuously update the decision based on a reward mechanism to achieve efficient and accurate grouping goals.

[0059] In step 701, the processing unit 13 takes the K×N matching scores corresponding to the K training groups at a current training time point as a current training state, and each facial feature training object corresponds to a facial feature training object data.

[0060] In step 702, the processing unit 13 obtains a training execution action based on the current training state using the object grouping model. The training execution action is an adjustment training method related to the facial feature training object data corresponding to the same training category of the K training groups.

[0061] In step 703, the processing unit 13 obtains the next training state and calculates a reward value based on the current training state and the training action. The current training state, the training action, the next training state, and the reward value are then stored as training data in an experience pool. In this embodiment, the processing unit 13 adds up the K×N matching scores in the current state to obtain a current sum, and adds up the other K×N matching scores in the next training state to obtain a next sum. When the next sum is greater than the current sum, the processing unit calculates the reward value as positive; when the next sum is not greater than the current sum, the processing unit calculates the reward value as negative, but this is not a limitation.

[0062] In step 704, the processing unit 13 obtains a training set containing multiple training data from the experience pool.

[0063] In step 705, the processing unit 13 updates the object grouping model based on the training set, and repeats steps 701-705 until the object grouping model converges. In this second embodiment, during each iteration, the processing unit 13 obtains the next training state and the reward value by updating the training execution action. When the reward value is less than a reward threshold, the object grouping model satisfies a convergence condition and stops iterating, but this is not a limitation.

[0064] It is worth mentioning that the facial feature grouping method of the present invention can also be applied to an in-vehicle monitoring system with an alarm function. The in-vehicle monitoring system includes a driver monitoring system (DMS) and an occupant monitoring system (OMS). The alarm function is used to issue an alarm when an abnormal event is detected in a driver. The facial feature grouping method is implemented by a computing device 1 as described in the first embodiment. The storage unit 11 stores a facial feature object detection model, a facial feature object ratio prediction model, and K×N facial feature object data corresponding to K×N facial feature objects into K groups corresponding to K passenger faces. After the processing unit 13 executes steps 21, 22, and 24, it determines whether the matching score is less than a threshold value. When it is determined that the matching score is less than the threshold value, the processing unit 13 controls the in-vehicle monitoring system to suspend the alarm function, but this is not a limitation.

[0065] In summary, the facial feature grouping method of the present invention involves processing unit 13 grouping facial feature data obtained by the facial feature object detection model, and obtaining matching scores using predicted facial feature object data and the facial feature object data obtained by the facial feature object ratio prediction model. Processing unit 13 adjusts the facial feature object data in the groups based on the matching scores to obtain updated groups. Furthermore, in the second embodiment, processing unit 13 can also efficiently adjust the facial feature object data in the groups using the object grouping model to obtain updated groups. The facial feature grouping method of the present invention can accurately label facial feature objects in an image as belonging to their respective faces, thereby detecting facial feature objects belonging to the driver and passengers in a vehicle, thus reducing the problem of a driving monitoring system confusing the driver's driving status and causing frequent triggering of an emergency notification system. Therefore, the objective of the present invention is indeed achieved.

[0066] However, the above description is merely an embodiment of the present invention and should not be construed as limiting the scope of the present invention. Any simple equivalent changes and modifications made in accordance with the scope of the patent application and the contents of the patent specification of the present invention shall still fall within the scope of the patent of the present invention.

[0067] 1: Arithmetic unit 11: Storage Unit 12: Image Acquisition Unit 13: Processing Unit Steps 21-27 231~232: Sub-steps 241~242: Sub-steps 261~263: Sub-steps Steps 31-38 701~705: Steps 81: Error Matrix 82: Error 91: Error Matrix

Claims

1. A facial feature grouping method, applicable to grouping all facial feature objects in an image containing K faces into K groups, wherein each face in the image contains N facial feature objects corresponding to N feature categories, and implemented by a computing device, the computing device comprising a storage unit storing a facial feature object detection model and a facial feature object proportion prediction model, an image acquisition unit, and a processing unit electrically connected to the storage unit and the image acquisition unit, wherein the facial feature object detection model is used to obtain multiple facial feature object data related to multiple facial feature objects to be detected from an input image, and the facial feature object proportion prediction model is used to predict N output facial feature object prediction data corresponding to N output facial feature objects indicating the input person based on an input facial feature object data corresponding to one of the input facial feature objects of an input person, the facial feature grouping method comprising the following steps: (A) The processing unit uses the facial feature object detection model to obtain K×N facial feature object data corresponding to the K×N facial feature objects based on the image; (B) For each facial feature object, the processing unit treats the facial feature object as a predicted facial feature object of a person to be predicted, and uses the facial feature object ratio prediction model to obtain N predicted facial feature object data corresponding to the N predicted facial feature objects of the person to be predicted based on the predicted facial feature object data corresponding to the predicted facial feature object; (C) The processing unit divides the K×N facial feature object data corresponding to the K facial feature objects into K groups corresponding to the K faces; (D) For each of the K×N facial feature objects, the processing unit uses the facial feature object as a reference facial feature object and the facial feature object data corresponding to the reference facial feature object as reference facial feature object data. Based on the N facial feature object data in the group corresponding to the reference facial feature object data and the N predicted facial feature object data obtained in step (B) based on the reference facial feature object data, the processing unit obtains a matching score indicating the probability that the reference facial feature object data is classified into the correct group; and (E) The processing unit uses the facial feature object corresponding to the target facial feature object data with the smallest matching score in all the reference facial feature object data as a facial feature object to be reclassified, and adjusts the facial feature object data in the K groups corresponding to the facial feature object to be reclassified for the facial feature object to be reclassified, so as to obtain K updated groups.

2. The facial feature grouping method as described in claim 1, wherein, Step (C) includes the following sub-steps: (C-1) The processing unit selects a reference object from the N facial feature objects, and takes the feature category to which the reference object belongs as a reference category, and divides the K facial feature object data belonging to the K reference objects in the K faces into the K groups corresponding to the K faces; and (C-2) For each of the feature categories other than the reference category to be classified, the processing unit assigns each of the K facial feature object data belonging to the category to be classified in the K faces to one of the K groups according to the K facial feature object data of the K reference objects corresponding to the K faces.

3. The facial feature grouping method as described in claim 1, wherein, In step (A), each piece of facial feature object data includes the size of the corresponding facial feature object, and in step (B), each piece of predicted facial feature object data includes the size of the corresponding predicted facial feature object.

4. The facial feature grouping method as described in claim 3, wherein, For each reference facial feature object, step (D) includes the following sub-steps: (D-1) For each of the N predicted facial feature object data in the group corresponding to the reference facial feature object data, the processing unit obtains an error based on the size of the facial feature object in the facial feature object data and the size of the predicted facial feature object corresponding to the facial feature object in the predicted facial feature object data; and (D-2) The processing unit obtains the matching score based on the N errors obtained in step (D-1).

5. The facial feature grouping method as described in claim 4, wherein, In step (D-1), the processing unit obtains the error Dij using the following formula: based on the size of one of the facial feature objects j in the facial feature object data and the size of one of the predicted facial feature objects j corresponding to the predicted facial feature object data obtained from the reference facial feature object data of a reference facial feature object i.

6. The facial feature grouping method as described in claim 1, further comprising the following steps before step (E): (F) the processing unit determines whether there exists at least one candidate facial feature object data in all reference facial feature object data that corresponds to a matching score less than a threshold value; and (G) when the processing unit determines that there is no candidate facial feature object data, the most recently obtained K groups are taken as a grouping result; wherein, When the processing unit determines that at least one candidate facial feature object data exists, step (E) is executed.

7. The facial feature grouping method as described in claim 6, wherein, Step (E) includes the following sub-steps: (E-1) The processing unit obtains the target facial feature object data with the minimum matching score from the at least one candidate facial feature object data; (E-2) The processing unit obtains a set of adjusted facial feature object data corresponding to the same adjusted feature category from the remaining candidate facial feature object data other than the target facial feature object data, based on the adjusted feature category to which the facial feature object to be reclassified belongs; and (E-3) The processing unit exchanges the target facial feature object data corresponding to the facial feature object to be reclassified with the adjusted facial feature object data to obtain the K updated groups, and returns to step (D).

8. A facial feature grouping method, applicable to grouping all facial feature objects in an image containing K faces into K groups, wherein each face in the image contains N facial feature objects corresponding to N feature categories, and is implemented by a computing device, the computing device comprising a storage unit storing a facial feature object detection model, a facial feature object proportion prediction model and an object grouping model, an image acquisition unit, and a processing unit electrically connected to the storage unit and the image acquisition unit, wherein the facial feature object detection model is used to obtain from an input image Multiple facial feature object data related to multiple facial feature objects to be detected; a facial feature object ratio prediction model is used to predict N output facial feature object prediction data corresponding to N output facial feature objects indicating the input person based on one input facial feature object data corresponding to one of the input facial feature objects; an object grouping model is used to determine a correction method among facial feature object data corresponding to the same feature category in multiple input groups based on an input state; the facial feature grouping method includes the following steps: (A) The processing unit uses the facial feature object detection model to obtain K×N facial feature object data corresponding to the K×N facial feature objects based on the image; (B) For each facial feature object, the processing unit treats the facial feature object as a predicted facial feature object of a person to be predicted, and uses the facial feature object ratio prediction model to obtain N predicted facial feature object data corresponding to the N predicted facial feature objects of the person to be predicted based on the predicted facial feature object data corresponding to the predicted facial feature object; (C) The processing unit divides the K×N facial feature object data corresponding to the K facial feature objects into K groups corresponding to the K faces; (D) For each of the K×N facial feature objects, the processing unit uses the facial feature object as a reference facial feature object and the facial feature object data corresponding to the reference facial feature object as reference facial feature object data. Based on the N facial feature object data in the group corresponding to the reference facial feature object data and the N predicted facial feature object data obtained in step (B) based on the reference facial feature object data, the processing unit obtains a matching score indicating the probability that the reference facial feature object data is classified into the correct group; (E) The processing unit uses the matching scores corresponding to the K×N facial feature objects as a current state and uses the object grouping model based on the current state to obtain an adjustment method related to the multiple facial feature object data corresponding to the same adjustment feature category in the K groups; and (F) The processing unit adjusts the K groups according to the adjustment method to obtain K updated groups.

9. The facial feature grouping method as described in claim 8, further comprising the following steps before step (E): (i) taking the K×N facial feature training objects corresponding to K training groups as a current training state, where each facial feature training object corresponds to a facial feature training object data; (ii) obtaining a training execution action based on the current training state using the object grouping model, where the training execution action is an adjustment training method related to the facial feature training object data corresponding to the same adjustment feature training category in the K training groups; (iii) obtaining the next training state and calculating a reward value based on the current training state and the training execution action, and storing the current training state, the training execution action, the next training state, and the reward value as a training data in an experience pool; (iv) obtaining a training set containing multiple training data from the experience pool; (v) updating the object grouping model based on the training set, and repeating steps (i) to (iv) until the object grouping model converges.

10. The facial feature grouping method as described in claim 9, wherein, In step (iii), the processing unit adds up the K×N matching scores in the current state to obtain a current sum, and adds up the other K×N matching scores in the next training state to obtain a next sum. When the next sum is greater than the current sum, the processing unit calculates a positive reward value. When the next sum is not greater than the current sum, the processing unit calculates a negative reward value.

11. The facial feature grouping method as described in claim 8, wherein, Step (C) includes the following sub-steps: (C-1) The processing unit selects a reference object from the N facial feature objects, and takes the feature category to which the reference object belongs as a reference category, and divides the K facial feature object data belonging to the K reference objects in the K faces into the K groups corresponding to the K faces; and (C-2) For each of the feature categories other than the reference category to be classified, the processing unit assigns each of the K facial feature object data belonging to the category to be classified in the K faces to one of the K groups according to the K facial feature object data of the K reference objects corresponding to the K faces.

12. The facial feature grouping method as described in claim 8, wherein, In step (A), each piece of facial feature object data includes the size of the corresponding facial feature object, and in step (B), each piece of predicted facial feature object data includes the size of the corresponding predicted facial feature object.

13. The facial feature grouping method as described in claim 12, wherein, For each reference facial feature object, step (D) includes the following sub-steps: (D-1) For each of the N predicted facial feature object data in the group corresponding to the reference facial feature object data, the processing unit obtains an error based on the size of the facial feature object in the facial feature object data and the size of the predicted facial feature object corresponding to the facial feature object in the predicted facial feature object data; and (D-2) The processing unit obtains the matching score based on the N errors obtained in step (D-1).

14. The facial feature grouping method as described in claim 13, wherein, In step (D-1), the processing unit obtains the error using the following formula based on the size of one of the facial feature objects j in the facial feature object data and the size of one of the predicted facial feature objects j corresponding to the predicted facial feature object data obtained from the reference facial feature object data of a reference facial feature object i: .

15. The facial feature grouping method as described in claim 8, further comprising the following steps before step (E): (G) the processing unit determines whether there exists at least one candidate facial feature object data in all reference facial feature object data that corresponds to a matching score less than a threshold value; and (H) when the processing unit determines that there is no candidate facial feature object data, the most recently obtained K groups are taken as a grouping result; wherein, When the processing unit determines that at least one candidate facial feature object data exists, step (E) is executed, and in step (F), after the processing unit obtains the K updated groups, it returns to step (D).

16. A facial feature grouping method applicable to an in-vehicle monitoring system with an alarm function, the alarm function being used to issue an alarm when an abnormal event is detected in a driver, the facial feature grouping method being implemented by a computing device, the computing device comprising a storage unit storing a facial feature object detection model, a facial feature object ratio prediction model, and dividing K×N facial feature object data corresponding to K×N facial feature objects into K groups corresponding to K passenger faces, an image acquisition unit, and a processing unit electrically connected to the storage unit and the image acquisition unit, the feature grouping method comprising the following steps: (A) the processing unit using the facial feature object detection model to obtain K×N facial feature object data corresponding to the K×N facial feature objects based on an in-vehicle image; (B) For each facial feature object, the processing unit takes the facial feature object as a facial feature object to be predicted for a person to be predicted, and uses the facial feature object ratio prediction model to obtain N predicted facial feature object data corresponding to the N predicted facial feature objects of the person to be predicted based on the data of the predicted facial feature object corresponding to the predicted facial feature object. (C) For each of the K×N facial feature objects, the processing unit uses the facial feature object as a reference facial feature object and the facial feature object data corresponding to the reference facial feature object as a reference facial feature object data. Based on the N facial feature object data in the group corresponding to the reference facial feature object data and the N predicted facial feature object data obtained in step (B) based on the reference facial feature object data, a matching score indicating the probability that the reference facial feature object data is classified into the correct group is obtained; and (D) When the matching score is less than a threshold value, the in-vehicle monitoring system suspends an alarm function.

Images