Image Processing Method and System

Abstract

The present application discloses an image processing method, including: obtaining a first image; obtaining a template image, where the template image has a health value, and the health value is used to measure whether the template image is valid; comparing the first image and the template image, if the first image and the template image match successfully, storing the first image as a template image in the template library, and deleting the template images in the template library whose health values are less than a set threshold. Through the method of the embodiment of the present application, by updating the template library every time image matching is performed and deleting the template images in the template library whose health values are less than the set threshold, the template images can be kept iteratively updated, and the accuracy of image recognition over a long time span can be achieved.

Description

Technical Field

[0001] This application relates to an image processing technology, and more specifically, to a technology for image recognition. Background Art

[0002] With the development of informatization and intelligence, more and more devices (such as mobile phones, IoT (Internet of Things), cars, etc.) are equipped with functions for identifying the identity of users. Facial recognition is widely used in fields such as mobile phones, IoT (Internet of Things), cars, etc. due to its non-contact and natural characteristics. Currently, facial recognition generally compares the face image frames obtained by the user through the camera during use with the face templates stored in the system (the face image templates obtained when the user initially registers) to determine whether they match. However, this solution often results in rejection when the user has a large span of usage years or wears accessories (such as hats, masks, scarves, glasses, etc.).

[0003] In the automotive field, the above problems are more obvious. First, the service life of cars is relatively long (usually several years or even more than a decade), and during this process, the appearance of users often changes significantly. Second, the space inside the car is small. Due to space limitations and the sitting posture habits of users, in many cases when identifying the user's identity, it is difficult for the user to directly face the camera, so the obtained face images are often not in the "front face" state. Finally, a considerable number of users may wear accessories (such as masks, hats, scarves, etc.) on the head and neck, which is particularly common in winter or when the weather conditions are poor. In these situations, if the user wears accessories or does not face the camera directly, they must wait for the user to remove accessories such as masks, hats, glasses, etc., or "sit upright" to face the camera before identification, which will result in a long identification time and a poor identification experience; and if the user's appearance changes significantly due to a large age span, rejection often occurs.

[0004] For the above problems, some solutions have been proposed in the prior art, such as providing auxiliary images, updating training data or models, etc. However, these methods still have a high dependence on hardware, cannot well solve user identification in the case of a large time span, and cannot identify users when the user wears accessories or the face is not directly facing the camera (in the "front face" state).

[0005] Based on the above, a new recognition method is needed that can overcome the above problems. Summary of the Invention

[0006] As one aspect of the present application, an embodiment of the present application provides an image processing method, which may specifically include: obtaining a first image and a template image, where the template image has a health value, and the health value is used to measure whether the template image is valid. Comparing whether the first image and the template image match, and if the match is successful, storing the first image as the template image in the template library. Through the method provided by the embodiment of the present application, a health value is assigned to the template image, and by using the newly obtained suitable image as the template image, the template library can be updated in a timely manner, and the user recognition process can be better completed.

[0007] In combination with a possible implementation manner of the first aspect, it further includes: obtaining the health value of the template image in the template library, and determining whether the health value of the template image is less than a set threshold. If the health value of the template image is less than or less than or equal to the set threshold, deleting the template image from the template library. For example, the template library can be updated each time image matching is performed, and the template image in the template library with a health value less than or less than or equal to the set threshold can be deleted, so that the template image can be iteratively updated, and the accuracy of image recognition can be improved over a long time span. Another example is that the template library can be updated regularly to delete the template image in the template library with a health value less than or less than or equal to the set threshold. Still another example is that based on event triggering (such as a user setting instruction), etc., the template library can be updated to delete the template image in the template library with a health value less than or less than or equal to the set threshold.

[0008] In combination with a possible implementation manner of the first aspect, both the template image and the first image can be face images. Applying the technical solution of the first aspect to the field of face recognition can improve the accuracy of face recognition over a long time span.

[0009] In combination with a possible implementation manner of the first aspect, the template image is divided into two categories: a first template image and a second template image. The first template image includes a master face image, and the second template image includes a special face image; the master face image includes an unobstructed face image within a restricted angle range. The special face image can be an image containing some facial features. The special face image includes at least one of the following: a face image with facial occlusion, a face image with accessories, or a face image with a large deviation angle. By making the above distinction of the template image, the technical solution of the present application can not only achieve traditional face recognition (master face image), but also achieve face recognition when the user has the above facial occlusion, wearing accessories, or large deviation angle, without the user removing the accessories or occlusions, so that face recognition can be quickly achieved.

[0010] In a possible implementation manner of the first aspect, recognition models such as Arcface and / or Facenet can be used to compare the template image and the first image. The above recognition models make judgments by obtaining the feature parameters of the template image and the first image. For example, the cosine distance or Euclidean distance of the feature parameters can be used to determine whether the template image and the first image match.

[0011] In a possible implementation manner of the first aspect, for a template image, the health value of the template image is dynamically changing. On the one hand, the health value of the template image decreases as its survival time increases. On the other hand, if the template image successfully matches a certain first image, the health value of the template image increases. This process has two implications: First, the health value of the template image will decrease / degrade over time. Second, if the template image successfully matches the first image, the health value of the template image will be "restored" to a certain extent; through the above process, the elimination and iteration rate of the template image can be better controlled.

[0012] In a possible implementation manner of the first aspect, the degradation method of the health value of the template image can be specifically expressed as:

[0013]

[0014] Where: S v represents the health value of the template image, t0 represents the survival time of the template image, e g represents the variance of similarity; s t represents the similarity value; τ A represents the age of the current face; s0 represents the similarity offset parameter. Through this model, the degradation process of the health value of the template image can be quantitatively measured.

[0015] In a possible implementation manner of the first aspect, the restoration method of the health value of the template image can be specifically expressed as:

[0016]

[0017] Where A is the health value of the template image when the life restoration starts according to the above formula, and v is the number of times the template image is successfully matched. Through this model, the restoration process of the health value of the template image can be quantitatively measured.

[0018] In a second aspect, there is provided an image processing system, including: a storage device for storing a template image, the template image having a health value that can be used to measure whether the template image is valid; a processing device communicatively connected to the storage device, the processing device being configured to: obtain a first image and the template image in the storage device, compare the first image and the template image, and if the template image and the first image match successfully, store the first image as a template image in a template library.

[0019] In a possible implementation manner in combination with the second aspect, the image processing system further includes an imaging device, and the first image is an image captured by the imaging device.

[0020] It can be understood that the system provided in the second aspect corresponds to the method provided in the first aspect. Therefore, for the relevant descriptions of the various implementation manners and the achieved technical effects in the second aspect, reference can be made to the relevant descriptions of the various implementation manners in the first aspect.

[0021] In a third aspect, there is provided a computer-readable storage medium storing an instruction set that can be executed by a processor to implement the method described in any one of the implementation manners of the foregoing first aspect.

[0022] In a fourth aspect, there is provided a device including the image processing system of the foregoing second aspect.

[0023] In a possible implementation manner in combination with the fourth aspect, the device may include: a vehicle, a smart home appliance, a user terminal device, a monitoring device, etc. By introducing the image processing system in the present application, the above Internet of Things devices can have a face recognition function and can complete face recognition quickly and accurately.

[0024] In various embodiments of the present application, on the one hand: By assigning a health value to the template image, using a degradation model and a restoration model to determine the health value, and polling to eliminate the template images with health values lower than the threshold, the template images are kept iteratively updated. Thus, when some template images are no longer suitable as template images due to being too old or having poor matching degrees, they can be deleted, which not only saves storage space but also improves the template search efficiency. In addition, factors such as registration time, variance, user age, and number of matches are fully considered in the degradation model and restoration model of the embodiments of the present application, so that it can be ensured that the template images retained in the system not only have a high health value but can also achieve a high matching degree over a long time span. Furthermore, the technical solution of the present application can continuously maintain a high recognition / matching accuracy during a long usage period. On the other hand, by distinguishing and storing the first template image (master face image) and the second template image (special face image), and further distinguishing two situations for the second template image (special face image), namely wearing accessories (such as hats, scarves, masks, glasses, etc.) and having a large deviation in the recognition angle. Thus, it not only accommodates the traditional "front face" matching situation but also enables user recognition when the user wears accessories or the initial pose of the user has a large deviation from the standard in the imaging angle in the camera (in these two situations, the prior art often cannot perform recognition), and the recognition process is rapid. That is, once the second template image is stored, the next time the user performs face recognition, there is no need to remove accessories such as masks and glasses, nor to change the head angle for recognition because the system already stores the second template image of the user with masks, glasses or a large angle deviation before. This can greatly reduce the time for face recognition and thus improve the user's convenience and satisfaction. Finally, the technical solution of the embodiments of the present application has low hardware requirements, good real-time performance, low cost, and good versatility. It can not only be applied to the automotive field but also be used for face recognition in fields such as the Internet of Things, having a wide range of application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of an image processing system provided by an embodiment of the present application;

[0026] Figure 2 Schematic diagram of an image processing system provided by an embodiment of the present application;

[0027] Figure 3-1 Schematic flowchart of an image processing method provided by an embodiment of the present application;

[0028] Figure 3-2 Schematic flowchart of an image processing method provided by an embodiment of the present application;

[0029] Figure 4 Schematic diagram of the face within the restricted angle range provided by the embodiments of the present application;

[0030] Figure 5 Schematic diagram of the attribute detection network provided by the embodiments of the present application;

[0031] Figure 6 Schematic diagram of the health value degradation and recovery of a template image provided by the embodiments of the present application;

[0032] Figure 7 Schematic diagram of an image processing system provided by the embodiments of the present application disposed in a vehicle and a cloud server;

[0033] Figure 8 Schematic diagram of an image processing system provided by the embodiments of the present application disposed in an IoT device and a cloud server. Detailed implementation manners

[0034] The embodiments of the present application provide an image processing method and system, which can quickly and accurately perform user identification, and still have high accurate recognition even in the case of a large usage time span.

[0035] Refer to Figure 1 , which shows an image processing system 100 provided by some embodiments of the present application. The system 100 includes: a camera device 101, a processing device 102, and a storage device 103. Among them, the processing device 102 includes a device or virtual machine with data processing capabilities, such as, but not limited to, a central processing unit (CPU: Central Process Unit), a graphics processing unit (GPU: Graphic Process Unit), a field programmable gate array (FPGA: Field Programmable Gate Array), a system on chip (SoC: Sysem on Chip), an application specific integrated circuit (ASIC: Application Specific Integrated Circuit), or a combination thereof, or a virtual machine architecture on a cloud server. The camera device 101 can be any device that can acquire images and / or videos, such as a camera, a webcam, or a combination thereof. The storage device 103 is used to store template images, and the template images have health values, which can be used to measure whether the template images are valid. The camera device 101 can obtain images or videos of a user, and the obtained image or video data can be sent to the processing device 102, so that the processing device 102 can determine whether the user identification passes according to the obtained images or videos of the user and the template images obtained from the storage device 103.

[0036] The above image processing system 100 can be set in various scenarios / devices that require user identification, such as vehicles, smart home appliances, user terminal devices, monitoring devices, etc. Smart home appliances can include, for example, TVs, refrigerators, washing machines, speakers, etc.; user terminal devices can include, for example, mobile phones, laptop computers, etc.; monitoring devices can include, for example, monitoring cameras and other devices. In this application, the vehicle scenario is mainly used as an example for illustration.

[0037] For example, in some embodiments, the above image processing system can be set on a vehicle. For example, the imaging device 101 and the processing device 102 can be integrated into the vehicle. Thus, when a user enters the vehicle, the imaging device 101 can be used to obtain the user's image or video, and the storage device 103 can be set on the vehicle side. The image or video data obtained by the imaging device 101 can be sent to the processing device 102 so that the processing device 102 can determine whether the user identification passes according to the obtained user's image or video and the template image obtained from the storage device 103.

[0038] In some other embodiments of the vehicle scenario, the imaging device 101 and the processing device 102 can also be integrated into the vehicle. And, considering the light environment inside the vehicle, the imaging device 101 is selected as a near-infrared camera.

[0039] In some other embodiments of the vehicle scenario, the imaging device 101 can also be set in the vehicle. The vehicle includes a communication unit for communicating with the cloud server. The processing device 102 is set on the cloud server, and the cloud server can include a physical machine and virtual machines obtained by virtualizing the physical machine. The cloud server can provide elastic cloud computing services. The vehicle sends the user's image or video obtained by the imaging device 101 to the processing device 102 located on the cloud server through the communication unit so that the processing device 102 can determine whether the user identification passes according to the obtained user's image or video and the template image.

[0040] In some embodiments, referring to Figure 7 As shown, the imaging device 701 can be set on the vehicle, while the storage device 7021 and the processing device (processor) 7022 can be set on the virtual machine 702 of the cloud server 703, and 704 is the physical machine at the bottom layer of the cloud server. The imaging device 701 can be communicatively connected to the storage device and the processing device through a network. In this case, the template image is stored on the cloud server, and the calculation process of face recognition is also completed on the cloud server. The recognition result can be transmitted to the vehicle side through the network. This architecture can achieve a large storage capacity and high computing power, so that the technical solution of the embodiments of this application can be more fully implemented.

[0041] In some embodiments, the image processing system (including a camera device, a storage device, and a processing device) of the embodiments of the present application can be disposed on a mobile terminal and / or an IoT (Internet of Things) device. The IoT device can include (but is not limited to) a television, a speaker, a refrigerator, etc.; the mobile terminal can include a mobile phone, etc. The camera device can include an ordinary optical camera or a near-infrared camera to adapt to different light environments. Thus, the mobile terminal and / or the IoT device has the user identification ability.

[0042] In some embodiments, only the camera device can be disposed on the mobile terminal and / or the IoT device, while the storage device and the processing device are disposed on a virtual machine of a cloud server. Figure 8 An exemplary architecture is shown. The camera device 800 can be disposed on the mobile terminal and / or the IoT device. The IoT device includes but is not limited to, for example, the illustrated television 8011, speaker 8012, refrigerator 8013, etc.; the mobile terminal can include a mobile phone 8014, while the storage device 8021 and the processing device 8022 can be disposed on the virtual machine 802 of the cloud server 803. 804 is the cloud server, and the cloud server can include a physical machine and a virtual machine obtained by virtualizing the physical machine. The cloud server can provide elastic cloud computing services. The camera device 800 can be communicatively connected to the storage device and the processing device through a network. In this case, the template image is stored on the cloud server, and the calculation process of face recognition is also completed on the cloud server. The recognition result can be transmitted to the vehicle terminal through the network. This architecture can achieve a large storage capacity and a high computing power, so that the technical solution of the embodiments of the present application can be more fully implemented.

[0043] It should be understood that the above application scenarios such as vehicles, smart home appliances, user terminal devices, monitoring devices, etc. are only shown for the convenience of understanding the present application, and the actual application of the technical solution of the present application is not limited in this regard.

[0044] Continue to refer to Figure 2, which shows an image processing system 200 provided by some embodiments of the present application. The system 200 includes a processing device 202, an imaging device 203, and a storage device 204. The processing device 202, the imaging device 203, and the storage device 204 are communicatively connected via a bus. It should be understood that the communication connection here can be a wired connection, a wireless connection, or a combination of wired and wireless connections. The processing device 202 and the storage device 204 can be separately provided or integrated together. The processing device 202 includes a device or virtual machine with data processing capabilities, such as, but not limited to, a central processing unit (CPU: Central Process Unit), a graphics processing unit (GPU: Graphic Process Unit), a field programmable gate array (FPGA: Field Programmable Gate Array), a system on chip (SoC: Sysem on Chip), an application specific integrated circuit (ASIC: Application Specific Integrated Circuit), or a combination thereof, or a virtual machine architecture on a cloud server. The processing device 202 includes module units or necessary circuits for performing various functions. The imaging device 203 can be any device that can acquire images and / or videos, such as a camera, a webcam, or a combination thereof. The storage device 204 stores a template image 2041 and an identification model 2042. The template image 2041 includes a first template image and a second template image. The template image 2041 can be understood as an image stored as a template to determine whether image recognition passes. More specifically, in the scenario of user face recognition, the template image 2041 refers to a face image stored as a template. The identification model 2042 can be a selected model suitable for feature recognition. The identification model 2042 can be implemented in a software, hardware, firmware, or a combination of software and hardware manner. For example, the identification model 2042 can be stored in the storage device 204. Another example is that the identification model can be implemented through a circuit and integrated in the processing device 202.

[0045] Similar to the description of the image processing system 100 above, the image processing system 200 can also be set in various scenarios / devices that require user identification, such as: vehicles, smart home appliances, user terminal devices, monitoring devices, etc.

[0046] For example, in some embodiments, the image processing system 200 may be disposed on a vehicle. For example, the image processing system may be set within the vehicle operating system. The storage device 204 is used to store the template image 2041 and / or the recognition model 2042. The imaging device 203 is used to acquire the user's image or video data. The processing device 202 is used to acquire the image or video data captured by the imaging device 203, and based on the obtained image or video data and the template image 2041, run the recognition model 2042 to perform user recognition.

[0047] It should be understood that the above vehicle application scenario is merely shown for the convenience of understanding the present application, and the actual application of the present application is not restricted in this regard.

[0048] At least based on the above scenario, an embodiment of the present application provides an image processing method. More specifically, this method can be used for face recognition. The following introduces this method.

[0049] See Figure 3-1 , which shows an image processing method provided by an embodiment of the present application. This method can be executed based on the foregoing image processing system structure. For example, it can be executed by the processing device 202. Specifically, it may include the following steps:

[0050] 311: The image processing method starts. In some embodiments, starting can be understood as preparing to perform user recognition;

[0051] 312: Obtain the template image. In some embodiments, in this step, the template image and the recognition model can be prepared (for example, but not limited to: loaded into the memory of the device) for subsequent use. The image template and / or the recognition model can be stored in a device (such as a vehicle, a user terminal, an Internet of Things device, a smart home appliance, or a monitoring device, etc.), or can be obtained by the device from a cloud server.

[0052] It should be understood that any known and suitable algorithm can be selected as the recognition model, such as Arcface or Facenet. And for the above algorithms, appropriate adjustments (such as adjusting the loss function) can be made to obtain the best effect. The present application does not limit this. The recognition model can be pre-stored on the device to be applied (for example: a vehicle, a terminal device, an Internet of Things device, etc.), or can also be a model downloaded or upgraded by the user. The recognition model extracts features from the image, digitizes the features, and performs comparison between images;

[0053] In some embodiments, the template image includes a first template image and a second template image. The first template image includes the master face image, that is, the unobstructed face image of the user within a limited angle range. See Figure 4, which shows a schematic diagram of the head pose of a human face in space, and restricts the pose of the human face in the camera through the rotation of three Cartesian coordinate axes (Yaw, Pitch, Roll), so as to obtain a master face image that meets the requirements. For example, it meets the following conditions:

[0054] Min yaw <Yaw<Max yaw ,Min pitch <Pitch<Max pitch ,Min roll <Roll<Max roll . (1)

[0055] Among them, [Min yaw Max yaw refers to the range of the angle of rotation of the user's head along the Z-axis; [Min pitch Max pitch refers to the range of the angle of rotation of the user's head along the X-axis; [Min roll Max roll refers to the range of the angle of rotation of the user's head along the Y-axis. It can be understood that different angle ranges can be set according to requirements. For example (but not limited to), the rotation angle ranges of the user's head along Yaw, Pitch, and Roll can be set to [-10, 10], [-5, 28], [-10, 10] (unit: degree) respectively. Only the face images within this range can be used as the master face images. It should be understood that the above ranges are not restrictive and can be adjusted adaptively according to the usage requirements.

[0056] In addition to meeting the above rotation angle conditions, the master face image generally requires that the user's face is not blocked, that is, the user is not wearing a mask, accessories, overly large glasses, or has overly long hair, beard, etc., because these behaviors (blocking) often cover the user's facial features, making face recognition impossible to implement.

[0057] Generally speaking, the master face image can be an image obtained when the user is using various devices (such as but not limited to: vehicles, terminal devices, Internet of Things devices, etc.) and facing the camera directly. This master face image can be stored as the initial first template image. However, in this application, the first template image is not limited to the user image obtained during registration. As the user uses the device for a longer time, the first template image will change and iterate. In some embodiments, when the user performs face recognition and the match is successful, if the face image that matches the template image meets the above requirements of the angle range limit and no facial occlusion, it will be stored as the first template image. Or, in some other cases, the user can take a face image by himself / herself and select an image that he / she thinks is suitable as a template. After the user completes the selection, he / she can upload the template image that he / she thinks is suitable to the system, and the system can confirm whether the image uploaded by the user can be used as the first template image according to the face recognition algorithm. The specific rules for the change and iteration of the first template image will be further elaborated in the subsequent process.

[0058] In some embodiments, the template image can also include a second template image. The technical solution of this application fully considers the situation of rejection due to the user wearing accessories such as glasses, hats, masks, etc., or due to a too large angle deviation when the user performs face recognition (one possible reason for a large angle deviation is that the face imaging in the camera deviates from the frontal face due to the user's height or habitual posture). That is, these situations generally result in the failure of face recognition. However, when the user removes the accessories or adjusts the angle and the face recognition is successful, the face images that could not be recognized due to wearing accessories or the angle will be automatically saved (for example: these face images can be the images of the user wearing a mask or glasses). Such face images are called second template images, or can be called special face images.

[0059] In the above process, if it is the case that the user wears accessories such as a mask or glasses, similar to the above first template image, in order to ensure that the second template image is a frontal face, the following restrictions are imposed on the angle:

[0060] Min yaw <Yaw<Max yaw ,Min pitch <Pitch<Max pitch ,Min roll <Roll<Max roll . (2)

[0061] Where, [Min yaw Max yaw refers to the angle range of the user's head rotation along the Z axis; [Min pitch Max pitchrefers to the angular range of the user's head rotation along the X-axis; [Min roll Max roll refers to the angular range of the user's head rotation along the Y-axis. It can be understood that different angular ranges can be set according to requirements. For example (but not limited to), the angular ranges of the user's head rotation along Yaw, Pitch, and Roll can be set as [-10, 10], [-5, 28], and [-10, 10] (unit: degree) respectively. Only the face images within this range can be used as the master face images. It should be understood that the above ranges are not restrictive and can be adaptively modified according to usage requirements.

[0062] In some embodiments, if it is only a type with a large angular deviation, that is, the user does not wear glasses, masks, hats, etc., and it is only due to a large angular deviation at the beginning of face recognition, then there is no need to limit the angle according to the above restrictions, and the face image at the moment when face recognition starts can be directly stored as the second template image.

[0063] In some embodiments, in order to ensure that the second template image is not an image in an intermediate state (such as the intermediate process of the user taking off a mask or glasses), an attribute detection network can be used to detect and classify the user's face image, such as Figure 5 shown, the attribute detection network 500 can use a convolutional neural network (CNN: Convolution Neural Network). An exemplary convolutional neural network can include an input layer, a convolutional layer, a pooling layer, and a fully connected layer (output layer). After the face image is input through the input layer, feature extraction is performed through the convolutional layer and the pooling layer, and attribute classification is performed to obtain the classification of the current face image. For example: whether it is black hair, whether wearing glasses or a mask, whether having bangs, whether using hands to take off the mask, etc. Through these classification information, it can be determined whether the user is in the "initial" state of face recognition or in the "intermediate" process of removing accessories or adjusting the posture. In addition, the attribute detection network can also be used to determine whether an image is a master face image or a special face image.

[0064] The significance of the above non-intermediate state detection is that in these "special" situations, such as when the user wears accessories (such as masks, glasses, etc.), or when the user's posture angle deviation is large, the face image at the moment when the user starts face recognition (this time is often when just entering the vehicle) is used as the second template image. In this way, when the user enters the vehicle in the same state next time, matching can be quickly performed without requiring the user to remove the accessories for recognition again, reducing the waiting time for face recognition.

[0065] It should be understood that appropriate adjustments can be made to the above-mentioned attribute detection network (for example, adjusting the layers of the convolutional neural network or the loss function) to achieve the best results, and this application does not impose any restrictions on this. The attribute detection network can be set to be built-in to the device at the factory or can be upgraded online by the user via the network.

[0066] In some embodiments of the present application, in some scenarios, for example, if a user initially wears a mask and passes face recognition after taking off the mask, the main face image frame that passes face recognition after taking off the mask (meets the above-mentioned angle range restriction) will be stored as the first template image, and the special face image of the user wearing a mask at the beginning (meets the above-mentioned angle range restriction) will be stored as the second template image.

[0067] In some embodiments of the present application, in some scenarios, for example, if a user passes the face recognition after adjusting the posture due to a large angle deviation (non-"frontal face") when starting face recognition, the main face image frame that passes the face recognition after the posture adjustment (meets the above-mentioned angle range limit) will be stored as the first template image; and the image frame with a large angle deviation when the recognition starts will be stored as the second template image.

[0068] 313: Acquire a first image. In some embodiments, the first image may be pre-stored on the device. The first image may also be acquired through a camera device. The camera device may be any device that can acquire images and / or videos, such as a camera, a webcam, or a combination thereof. In other embodiments, the camera device may be a near-infrared camera. When the user starts face recognition, the camera device acquires an image or video stream of the user's face, and then uses a face detection algorithm to perform face detection on the acquired image or video stream to acquire a face image, and uses the acquired face image as the first image. It should be understood that a suitable face detection algorithm (for example, including but not limited to: SSD, YOLO, etc.) can be selected as needed to acquire a face image from an image or video stream.

[0069] In some embodiments, the obtained facial image is preprocessed. Preprocessing refers to resizing and normalizing the facial image. Resizing refers to adjusting the image size according to the original scaling. Normalization refers to normalizing the data using the mean and standard deviation of the original data. The processed data conforms to the standard normal distribution, that is, the mean is 0 and the standard deviation is 1. The normalization function is:

[0070] x=(x-μ) / σ

[0071] Where μ is the mean of the sample data and σ is the standard deviation of the sample data.

[0072] By resizing and / or normalizing, the first image can be made more suitable for subsequent face recognition / matching.

[0073] 314: Determine whether face recognition is successful by using a recognition model to determine whether there is a match between the template image (e.g., including the first template image and / or the second template image) and the obtained first image. In some embodiments, it is determined whether face recognition is successful by calculating whether the similarity between the face image to be detected and the template image (e.g., including the first template image and the second template image) reaches a set threshold. In some embodiments, the determination is made by comparing the similarity of feature parameters. For example, the above feature determination is performed by using the cosine distance, and the determination method of the cosine distance can be expressed as:

[0074]

[0075] where A and B respectively represent the face feature values / feature parameters of the face image obtained from the template image and the video stream, and n represents the dimension of the features.

[0076] Those skilled in the art can understand that, in addition to using the above cosine distance to determine the similarity between two images, other conventional determination methods can also be used, such as using the Euclidean distance (the distance between two points in an n-dimensional vector space) to measure the similarity between two images; for another example, the cosine distance and the Euclidean distance can be combined for determination.

[0077] If the determination result in step 314 is negative, it means that the currently obtained first image fails face recognition, and then it continues to return upward to step 313, that is, continues to obtain a new next first image for comparison. It should be understood that, for security reasons, if face recognition fails within a set number of consecutive times (e.g., 5 consecutive times), the device can be locked for a period of time or other verification means are required.

[0078] If the determination result in step 314 is positive, it proceeds to step 315.

[0079] It should be noted that if the obtained first image is a "front face" image and the angle meets the above angle limit conditions (1) or (2), then if it can be successfully matched, it should be successfully matched with the first template image in the template image, that is, the master face image.

[0080] If the obtained first image shows that the user is wearing accessories (such as a scarf, hat, mask, glasses, etc.) or the user has a large angular deviation, and if it can be successfully matched in this case, then the first image and the second template image (i.e., the special face image) in the template image are successfully matched. This indicates that before this recognition, a face image of the user wearing accessories or with a large angular deviation has been stored as the second template image. Therefore, in this case, the user can directly complete the user recognition successfully while wearing accessories or with a large angular deviation. And if the first image fails to match in this case, as recorded in step 314 above, when the user removes the accessories or corrects the posture and passes the user recognition, the initial image will be stored as the second template image for subsequent user recognition.

[0081] 315: If the first image passes the face recognition, it will be stored as a template image. Specifically, if the first image belongs to the main face image, it will be stored as the first template image; if the first image belongs to the special face image, it will be stored as the second template image.

[0082] 316: After storing the first image as a template image, the health value of the stored template image will be judged to update and iterate the template image.

[0083] In some embodiments, when an image is created / saved as a template image, its registration time starts counting from the moment of its creation / saving. For the template image, its health value is partly controlled by the degradation model. The degradation model is used to represent the process of the health value decline of the template image.

[0084] In some embodiments, the degradation model is defined by the following formula:

[0085]

[0086] In the above degradation model: S v represents the health value of the face image. Generally, the maximum value of the health value can be set to one, and the minimum value to zero. The meanings of the parameters will be explained in detail below:

[0087] t0 represents the survival time of the template image, for example, the time from its creation / saving to the current moment. It should be understood that: the registration time can either be the time when the template image is created (i.e., the time when the image frame is obtained from the imaging device), or the time when the template image is saved (i.e., the time when the image is stored as a template image); or it can also be calculated starting from a certain fixed time point (such as but not limited to 5 pm every Saturday). This does not affect the technical solution of this application. The longer t0 is, the longer the survival time of the template image; it can be seen that S vIt decreases as the registration time t0 increases, which means that for a template image, without considering other influencing factors, the longer its survival time, the more likely it is to be replaced. Generally speaking, the health value of the template image decreases as time increases, that is: if a face image exists for a long time, then it may not accurately correspond / match the current user's appearance (the user's appearance may change over time).

[0088] e g It represents the variance of the face similarity during face recognition. It can be seen that the larger the variance, the lower the health value. For a template image, a similarity will be generated during face recognition / matching. After multiple matches, the numerical sequence of the generated similarities will have a variance. The mathematical meaning of variance is the degree of deviation between a variable and the mean. Specifically for the technical solution of this application, this variance e g measures the consistency of the template image as a template. The ideal situation is that under various conditions, the variance e g tends to zero, that is, the template image can give consistent results for face recognition / matching under different conditions; on the contrary, if the similarities generated by a template image during each face recognition / matching are inconsistent or the differences between them are large, then the variance e g will be large accordingly. Therefore, for a template image, without considering other influencing factors, the smaller the variance, the higher the health value, and the larger the variance, the lower the health value.

[0089] s t It represents the similarity value. The smaller the similarity value, the lower the matching success degree with the template image, and its health value is also lower. On the contrary, the higher the similarity, the higher the matching success degree of the current template image, so its health value is higher. Therefore, for a template image, without considering other influencing factors, s t the higher it is, the higher the health value, and s t the lower it is, the lower the health value. s0 represents the similarity offset parameter, which is a constant that can be specifically selected according to the actual situation.

[0090] τ A It represents the age of the user recognized based on the template image. For the template image, without considering other influencing factors, τ A the higher it is, the higher the health value, and τ A the lower it is, the lower the health value.

[0091] In some embodiments, in addition to the above degradation model, a recovery model is also considered. The health value of the template image is partially controlled by the recovery model, and the recovery model is used to represent the recovery process of the health value of the template image.

[0092] When a template image is successfully matched, it enters the recovery model, and the recovery model can be expressed as:

[0093]

[0094] where Sv represents the health value of the face image, A represents the health value of the template image when it enters the recovery model, and v represents the number of successful matches of the template image so far. It can be seen that the more times a template image is successfully matched, the faster its health value recovers. This means that if a template image has been successfully matched more frequently, it is more likely to be successfully matched next time.

[0095] Those skilled in the art should understand that in addition to the above recovery model, other suitable functional representations can also be used to describe the recovery process, not limited to the above representation.

[0096] See Figure 6 , which shows a schematic diagram of the change of the health value of the template image over time. The upper dotted line represents the maximum health value (i.e., 1), the lower dotted line represents the threshold (for example but not limited to: can be set to 0.2), the solid line part 61 represents the degradation process of the health value of the template image, and the health value decreases during this process. The dotted line part 62 represents the recovery process of the health value of the template image, and the health value increases during this process. Figure 6 shows two degradation processes and two recovery processes. It can be understood that in practice, for a given template image, there may be multiple degradation and / or recovery processes, or there may be only a degradation process without a recovery process.

[0097] In some embodiments, after each new template image is stored, the health values of each stored template image are calculated and updated through the above degradation model and recovery model to obtain the health value of each template image.

[0098] Figure 3-2 shows a schematic flow of the calculation and update of the health value of the template image. The calculation and update of the health value of the template image can be executed by the processing device 202.

[0099] 321: Start; for example, automatically start the update process or start the update process after receiving a user instruction.

[0100] 322: Obtain the health value of the template image;

[0101] 323: Determine the health value of each template image according to the set threshold. If the health values of all template images are higher than the set threshold, it means that all template images are qualified and can be retained, and the entire process directly enters step 325 to end; it should be understood that the above threshold can be set according to specific usage requirements. For example, the threshold can be set to 0.1 or 0.2, or other appropriate values, and this application does not make specific limitations on this.

[0102] And if it is determined through judgment that there is a template image with a health value S v lower than the set threshold, it indicates that this (these) template image(s) is no longer suitable as a template image, and enters step 324.

[0103] 324: In this step, update all template images. For example, you can delete those template images with health values lower than the set threshold, or remove the template images with health values lower than the set threshold from the template library and store them in other locations, and then enter step 325 to end.

[0104] It should be noted that: Although the entire recognition process is exemplified by face images in the above process, this application is not limited to face recognition. In fact, the technical solution of this application can be used to recognize any suitable image that can be used as an object of recognition.

[0105] In some embodiments, the technical solution of this application can be used together with other user authentication solutions to improve security. For example, in addition to using the image recognition (face recognition) technical solution of this application, biometric recognition methods such as voice recognition, fingerprint recognition, iris recognition, etc. can also be used for user authentication.

[0106] In various embodiments of the present application, on the one hand: By assigning a health value to the template image, using a degradation model and a restoration model to determine the health value, and polling to eliminate the template images with health values lower than the threshold, the template images are kept iteratively updated. Thus, when some template images are no longer suitable as template images due to being too old or having poor matching degrees, they can be deleted. In addition, factors such as registration time, variance, user age, and number of matches are fully considered in the degradation model and restoration model of the embodiments of the present application. Therefore, it can be ensured that the template images retained in the system not only have a high health value but can also achieve a high matching degree over a long time span. Furthermore, the technical solution of the present application can continuously maintain a high recognition / matching accuracy during a long usage cycle. On the other hand, by distinguishing and storing the first template image (master face image) and the second template image (special face image), and further distinguishing two situations of the second template image (special face image), namely wearing accessories (such as hats, scarves, masks, glasses, etc.) and having a large deviation in the recognition angle. Thus, it not only accommodates the traditional "front face" matching situation but also enables user recognition when the user wears accessories or the initial pose of the user is not correct and the imaging angle in the camera deviates greatly from the standard (in these two situations, the prior art often cannot perform recognition). And this recognition process is rapid. That is, once the second template image is stored, the next time the user performs face recognition, there is no need to remove accessories such as masks and glasses, nor to change the head angle for recognition because the system already stores the second template image of the user with masks, glasses or a large angle deviation before. This can greatly reduce the time for face recognition and thus improve the user's convenience and satisfaction. Finally, the technical solution of the embodiments of the present application has low hardware requirements, good real-time performance, and low cost.

[0107] The above storage device can be a random-access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), register, hard disk, mobile hard disk, CD-ROM, or any other form of storage medium known to those skilled in the art.

[0108] The above processing device may be, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of the present application. The processing device may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on.

[0109] In the specification, claims, and above-mentioned drawings of the present application, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments described here can be implemented in an order different from that illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0110] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0111] In the embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical service division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0112] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0113] In addition, each service unit in the various embodiments of the present application may be integrated into a processing unit, may exist physically separately for each unit, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software service units.

[0114] If the integrated unit is implemented in the form of a software service unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0115] Those skilled in the art should be able to realize that in the above one or more examples, the services described in the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these services can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. The computer-readable medium includes computer storage media and communication media, where the communication media includes any medium that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that can be accessed by a general or special computer.

[0116] The above specific implementation manners further elaborate on the purpose, technical solution, and beneficial effects of the present application. It should be understood that the above is only the specific implementation manners of the present application.

[0117] Above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, comprising: acquiring a first image; Acquire a template image, wherein the template image has a life value, and the life value is used to measure whether the template image is valid; as well as comparing the first image with the template image, and if the first image and the template image match successfully, storing the first image as the template image in a template library; The life value of the template image in the template library decreases as its life time increases; if the first image and the template image are successfully matched, the life value of the successfully matched template image is increased; The template library is updated each time an image match is performed, the life value of the template image increases after each successful image match, and template images in the template library whose life value is less than or equal to a set threshold are deleted.

2. The method according to claim 1, further comprising: The template library includes one or more template images, and template images in the template library whose life value is less than a set threshold are deleted.

3. The method according to claim 1, wherein: The template image includes a first template image and a second template image, the first template image includes a main face image, and the second template image includes a special face image.

4. The method according to claim 3, wherein: The main face image includes an unobstructed face image within a limited angle range, and the special face image includes at least one of the following: a face image with an obstructed face, a face image with accessories, and a face image with a large deviation angle.

5. The method according to any one of claims 1 to 4, wherein: A recognition model is used to compare the template image and the first image, the recognition model including at least one of Arcface and Facenet, the recognition model obtains feature parameters of the template image and the first image and makes a judgment based on the feature parameters.

6. The method according to claim 5, wherein: The recognition model determines whether the template image and the first image match by using the cosine distance and / or Euclidean distance of the feature parameters.

7. The method according to claim 1, wherein The degradation mode of the life value of the template image is expressed as follows: in: Represents the life value of the template image, Indicates the survival time of the template image, Indicates the variance of similarity; Represents the similarity value; Indicates the age of the current face; Represents the similarity offset parameter.

8. The method according to claim 1, wherein The life value recovery method of the template image is expressed as: Where A is the life value of the template image when life recovery starts according to the above formula, and v is the number of times the template image is successfully matched.

9. An image processing system comprising: A storage device, used for storing a template image, wherein the template image has a life value, and the life value is used to measure whether the template image is valid; a processing device configured to obtain a first image and a template image in the storage device, the processing device comparing the first image with the template image, and storing the first image as the template image in a template library if the first image and the template image successfully match; In which, the life value of the template image in the template library decreases as its survival time increases; if the first image and the template image are successfully matched, the life value of the successfully matched template image is increased; in which, the template library is updated each time an image match is performed, the life value of the template image is increased after each successful image match, and the template images in the template library whose life value is less than or equal to the set threshold are deleted.

10. The system according to claim 9, wherein: The processing device is configured to: Delete the template images whose health value is less than the set threshold in the template library.

11. The system according to claim 9 or 10, further comprising: A camera device, wherein the first image is an image captured by the camera device.

12. The system of claim 9, further comprising: The communication unit is used to obtain at least one template image from the cloud service end, and the storage device is used to store the at least one template image obtained by the communication unit from the cloud service end.

13. The system of claim 9, wherein: The template image includes a first template image and a second template image, the first template image includes a main face image, and the second template image includes a special face image.

14. The system according to claim 13, wherein: The main face image includes an unobstructed face image within a limited angle range, and the special face image includes at least one of the following: a face image with an obstructed face, a face image with accessories, and a face image with a large deviation angle.

15. The system of claim 9, wherein: A recognition model is used to compare the template image and the first image, the recognition model including at least one of Arcface and Facenet, the recognition model obtains feature parameters of the template image and the first image and makes a judgment based on the feature parameters.

16. The system of claim 15, wherein: The recognition model determines whether the template image and the first image match by using the cosine distance and / or Euclidean distance of the feature parameters.

17. The system according to claim 9, wherein: The degradation mode of the life value of the template image is expressed as follows: in: Represents the life value of the template image, Indicates the survival time of the template image, Indicates the variance of similarity; Represents the similarity value; Indicates the age of the current face; Represents the similarity offset parameter.

18. The system according to claim 9, wherein: The life value recovery method of the template image is expressed as: Where A is the life value of the template image when life recovery starts according to the above formula, and v is the number of times the template image is successfully matched.

19. A computer-readable storage medium storing an instruction set, wherein the instruction set can be executed by a processor to implement the method according to claim 1.

20. A device, characterized in that: It comprises the image processing system as claimed in claim 9.

21. The device according to claim 20, characterized in that: The device is at least one of the following: a vehicle, a smart home appliance, a user terminal device, and a monitoring device.

Images