Facial image inpainting method and system for assisting tracking

Abstract

The application provides a face image repairing method and system for assisting tracking, and the method comprises the following steps: step 1, training a face repairing model based on a deep learning algorithm; step 2, obtaining a captured original face image of a first person, and performing first restoration on the original face image based on the face repairing model; and step 3, obtaining description information of the first person from a second person, and performing second restoration on the first restored image. The face repairing model is trained by using the deep learning algorithm, and the face of the first person is restored by combining the description information of the first person from the second person, so that the identity recognition accuracy is improved, and the assisting efficiency is further improved.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, and in particular to a method and system for restoring human face images with assisted tracking. Background Technology

[0002] In existing technologies, the first person is often tracked by a description from a second person, by reviewing a large amount of surveillance video or by using facial detection and recognition technology. However, the first person's face is often obscured or the image is blurry, which makes identification difficult and ultimately leads to a low tracking success rate.

[0003] Therefore, there is an urgent need for an image restoration technology to assist in tracking. Thus, this invention proposes a face image restoration method and system for assisting in tracking. Summary of the Invention

[0004] This invention provides a method and system for assisted tracking of facial image restoration, which uses deep learning algorithms to train a facial restoration model and combines the description information of a second person to a first person to restore the face of the first person, thereby improving the accuracy of identity recognition and further improving the efficiency of assistance.

[0005] This invention provides a method for face image restoration with assisted tracking, comprising:

[0006] Step 1: Train a face restoration model based on deep learning algorithms;

[0007] Step 2: Obtain the original face image of the first captured person, and perform a first restoration on the original face image based on the face restoration model;

[0008] Step 3: Obtain the description information of the first person from the second person, and perform a second restoration on the first restored image.

[0009] Preferably, after performing a second restoration on the first restored image, the method further includes:

[0010] Based on the second restored image, the face of the first person is located;

[0011] Based on the face localization results, trackable information of the first person is obtained;

[0012] Based on the trackable information, the current location of the first person is searched and located in real time, and a reminder is issued.

[0013] Preferably, after acquiring the original facial image of the first captured person, the process further includes:

[0014] Occlusion analysis is performed on the original face image to obtain the facial occlusion information of the first person.

[0015] The difficulty of restoration is determined based on the facial occlusion information and standard contour information.

[0016] Based on the difficulty of restoration, the number of repair attempts is determined, and the face restoration model is controlled to perform the corresponding number of repair attempts on the original face image, thereby achieving the first restoration of the original face image.

[0017] Preferably, the process involves obtaining the second person's description of the first person and performing a second restoration on the first restored image, including:

[0018] Extract the first information related to the regional scene and the second information related to the brightness scene from the description information;

[0019] Based on the first information, determine the brightness-affecting factors present in the scene of the region;

[0020] Based on the second information, the bright areas and dark areas existing in the brightness scene are determined, and a first brightness positioning distribution for the bright areas and a second brightness positioning distribution for the dark areas are set.

[0021] Obtain the remaining information in the description information, parse the remaining information to obtain the parsed features, and fuse the parsed features with facial features to obtain a fused facial image;

[0022] Based on the first brightness positioning distribution, the second brightness positioning distribution, and the brightness influencing factor, the face fusion image is pre-adjusted to obtain the image to be adjusted.

[0023] The image to be adjusted is compared and analyzed with the first restored image. Based on the comparison and analysis results, the difference information is extracted to determine whether the conditions for reliable restoration are met.

[0024] If available, a second restoration is performed on the first restored image based on the difference information;

[0025] If not, extract key information from the difference information, and perform a second restoration on the first restored image based on the key information.

[0026] Preferably, after performing a second restoration on the first restored image, the method further includes:

[0027] Step A1: Obtain the RGB values ​​of the second restored image and the original face image, and use formula (1) to obtain the brightness conversion value of each pixel in the second restored image:

[0028] ΔY(i,j)=0.299×[R(i,j)-R0(i,j)]+0.587×[G(i,j)-G0(i,j)]+ 0.114×[B(i,j)-B0(i,j)](1)

[0029] Where ΔY(i,j) represents the brightness conversion value of the pixel in the i-th row and j-th column of the second restored image; [R(i,j), G(i,j), B(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image; [R0(i,j), G0(i,j), B0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the original face image;

[0030] Step A2: Using formula (2), based on the brightness conversion value of each pixel and the RGB value of the second restored image, obtain the RGB value of the second restored image that has the same brightness value as the original face image:

[0031]

[0032] Where [r0(i,j), g0(i,j), b0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image, which has the same brightness value as the original face image; This indicates the calculation of the intermediate matrix;

[0033] Step A3: Using formula (3), based on the RGB values ​​of the second restored image (which has the same brightness value as the original face image) and the original face image, obtain the difference between the second restored image (which has the same brightness value as the original face image) and the original face image, and mark the difference on the second restored image:

[0034]

[0035] Where μ represents the difference between the second restored image and the original face image, which have the same brightness value as the original face image; || represents the absolute value; n represents the total number of pixels in each column of the image; m represents the total number of pixels in each row of the image; max[,] represents the maximum value of the values ​​on both sides of the comma in the parentheses;

[0036] The μ value control is displayed on the second restored image.

[0037] This invention provides a face image restoration system with assisted tracking, comprising:

[0038] The model training module is used to train a face restoration model based on deep learning algorithms.

[0039] The first restoration module is used to acquire the original face image of the captured first person and perform a first restoration on the original face image based on the face restoration model.

[0040] The second restoration module is used to obtain the description information of the second person to the first person, and to perform a second restoration on the first restored image.

[0041] Preferred options also include:

[0042] A face localization module is used to locate the face of the first person based on the second restored image;

[0043] The information acquisition module is used to acquire trackable information of the first person based on the face positioning results;

[0044] The search and location module is used to search and locate the current location of the first person in real time based on the trackable information, and to provide a reminder.

[0045] Preferably, the first restoration module includes:

[0046] The occlusion analysis unit is used to perform occlusion analysis on the original face image to obtain the facial occlusion information of the first person.

[0047] The difficulty determination unit is used to determine the restoration difficulty based on the facial occlusion information and standard contour information.

[0048] The repair unit is used to determine the number of repair attempts based on the restoration difficulty, and control the face restoration model to perform the corresponding number of repair attempts on the original face image, thereby achieving the first restoration of the original face image.

[0049] Preferably, the second restoration module includes:

[0050] An information extraction unit is used to extract first information related to the regional scene and second information related to the brightness scene from the description information;

[0051] A factor determination unit is used to determine the brightness influence factors existing in the regional scene based on the first information;

[0052] The positioning distribution determination unit is used to determine the bright areas and dark areas existing in the brightness scene based on the second information, and to set a first brightness positioning distribution for the bright areas and a second brightness positioning distribution for the dark areas.

[0053] The feature fusion unit is used to obtain the remaining information in the description information, parse the remaining information to obtain the parsed features, and fuse the parsed features with facial features to obtain a face fusion image.

[0054] The pre-adjustment unit is used to pre-adjust the face fusion image based on the first brightness positioning distribution, the second brightness positioning distribution, and the brightness influencing factor to obtain the image to be adjusted;

[0055] The comparison analysis unit is used to compare and analyze the image to be adjusted with the first restored image, extract difference information based on the comparison analysis results, and determine whether the conditions for reliable restoration are met.

[0056] If available, a second restoration is performed on the first restored image based on the difference information;

[0057] If not, extract key information from the difference information, and perform a second restoration on the first restored image based on the key information.

[0058] Preferred options also include:

[0059] The first calculation module is used to obtain the RGB values ​​of the second restored image and the RGB values ​​of the original face image, and to obtain the brightness conversion value of each pixel in the second restored image using formula (1):

[0060] ΔY(i,j)=0.299×[R(i,j)-R0(i,j)]+0.587×[G(i,j)-G0(i,j)]+ 0.114×[B(i,j)-B0(i,j)] (1)

[0061] Where ΔY(i,j) represents the brightness conversion value of the pixel in the i-th row and j-th column of the second restored image; [R(i,j), G(i,j), B(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image; [R0(i,j), G0(i,j), B0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the original face image;

[0062] The second calculation module is used to obtain the RGB values ​​of the second restored image, which are the same as the brightness values ​​of the original face image, based on the brightness conversion value of each pixel and the RGB values ​​of the second restored image using formula (2):

[0063]

[0064] Where [r0(i,j),g0(i,j),b0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image, which has the same brightness value as the original face image; This indicates the calculation of the intermediate matrix;

[0065] The third calculation module is used to calculate the difference between the second restored image with the same brightness value as the original face image and the original face image using formula (3) based on the RGB values ​​of the second restored image with the same brightness value as the original face image and the RGB values ​​of the original face image, and to mark the difference on the second restored image:

[0066]

[0067] Where μ represents the difference between the second restored image and the original face image, which have the same brightness value as the original face image; || represents the absolute value; n represents the total number of pixels in each column of the image; m represents the total number of pixels in each row of the image; max[,] represents the maximum value of the values ​​on both sides of the comma in the parentheses;

[0068] The annotation module is used to annotate the μ value control on the second restored image for display.

[0069] Compared with the prior art, the beneficial effects of this application are as follows:

[0070] By training a face restoration model using deep learning algorithms and combining it with the description information of the first person by the second person, the face of the first person can be restored, thereby improving the accuracy of identity recognition and further enhancing the efficiency of assistance.

[0071] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0072] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0073] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0074] Figure 1 This is a flowchart of a face image restoration method for assisted tracking in an embodiment of the present invention;

[0075] Figure 2 This is a structural diagram of a face image restoration system for assisted tracking, as described in an embodiment of the present invention. Detailed Implementation

[0076] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0077] Example 1:

[0078] This invention provides a method for assisted tracking and face image restoration, such as... Figure 1 As shown, it includes:

[0079] Step 1: Train a face restoration model based on deep learning algorithms;

[0080] Step 2: Obtain the original face image of the first captured person, and perform a first restoration on the original face image based on the face restoration model;

[0081] Step 3: Obtain the description information of the first person from the second person, and perform a second restoration on the first restored image.

[0082] In this embodiment, deep learning algorithms such as convolutional neural networks are used.

[0083] In this embodiment, the original face image can be an image captured directly during the tracking process.

[0084] In this embodiment, the face restoration model is trained using several different faces and images of the same person with different degrees of occlusion as samples.

[0085] In this embodiment, the descriptive information can be a description of facial information of a person's face, such as facial features.

[0086] In this embodiment, the first restoration is achieved based on model training and recognition, and the second restoration is achieved by adjusting the first restoration.

[0087] The beneficial effects of the above technical solution are: by training a face restoration model through deep learning algorithms and combining it with the description information of the first person by the second person, the face of the first person can be restored, thereby improving the accuracy of identity recognition and further improving the efficiency of assistance.

[0088] Example 2:

[0089] Based on Example 1, after performing a second restoration on the first restored image, the method further includes:

[0090] Based on the second restored image, the face of the first person is located;

[0091] Based on the face localization results, trackable information of the first person is obtained;

[0092] Based on the trackable information, the current location of the first person is searched and located in real time, and a reminder is issued.

[0093] In this embodiment, face localization is used to track and identify faces, and the trackable information includes, for example, the acquired movement route and network connectivity information, to achieve real-time localization.

[0094] The beneficial effects of the above technical solution are: effective tracking is facilitated through face positioning, and real-time search and positioning can be achieved based on trackable information, which can further improve the auxiliary efficiency of capturing the first person.

[0095] Example 3:

[0096] Based on Example 1, after obtaining the captured original facial image of the first person, the method further includes:

[0097] Occlusion analysis is performed on the original face image to obtain the facial occlusion information of the first person.

[0098] The difficulty of restoration is determined based on the facial occlusion information and standard contour information.

[0099] Based on the difficulty of restoration, the number of repair attempts is determined, and the face restoration model is controlled to perform the corresponding number of repair attempts on the original face image, thereby achieving the first restoration of the original face image.

[0100] In this embodiment, facial occlusion information refers to the fact that the first person's facial features are not fully exposed and are occluded.

[0101] In this embodiment, generally speaking, the larger the occluded area, the greater the difficulty of restoration and the more times it needs to be repaired.

[0102] In this embodiment, the standard contour information is pre-set and serves only as a reference standard.

[0103] In this embodiment, the original face model is repaired a corresponding number of times according to the model in order to avoid incomplete repair in a single repair when the occlusion is severe.

[0104] The beneficial effects of the above technical solution are: by performing occlusion analysis and comparing with standard contour information, the number of repairs to be performed is determined, and then multiple repairs are performed according to the model to ensure the qualification of the original face image repair and further improve the accuracy of identity recognition.

[0105] Example 4:

[0106] Based on Example 1, the description information of the first person by the second person is obtained, and a second restoration is performed on the first restored image, including:

[0107] Extract the first information related to the regional scene and the second information related to the brightness scene from the description information;

[0108] Based on the first information, determine the brightness-affecting factors present in the scene of the region;

[0109] Based on the second information, the bright areas and dark areas existing in the brightness scene are determined, and a first brightness positioning distribution for the bright areas and a second brightness positioning distribution for the dark areas are set.

[0110] Obtain the remaining information in the description information, parse the remaining information to obtain the parsed features, and fuse the parsed features with facial features to obtain a fused facial image;

[0111] Based on the first brightness positioning distribution, the second brightness positioning distribution, and the brightness influencing factor, the face fusion image is pre-adjusted to obtain the image to be adjusted.

[0112] The image to be adjusted is compared and analyzed with the first restored image. Based on the comparison and analysis results, the difference information is extracted to determine whether the conditions for reliable restoration are met.

[0113] If available, a second restoration is performed on the first restored image based on the difference information;

[0114] If not, extract key information from the difference information, and perform a second restoration on the first restored image based on the key information.

[0115] In this embodiment, the remaining information is the facial structure information. By analyzing the facial structure information, facial features, or analytical features, such as a high nose and thick eyebrows, can be obtained, thereby obtaining a face fusion image. Furthermore, by adjusting the face fusion image based on different brightness factors, an image to be adjusted is obtained to ensure that the facial features are more saturated.

[0116] In this embodiment, the comparative analysis with the first restored image is to determine certain defects or missing features in the first restored image. Therefore, by extracting this information, it is determined whether the conditions for reliable restoration are met, that is, whether the condition of attaching all the difference information to the first restored image is satisfied.

[0117] In this embodiment, key information may refer to missing features, such as the eyebrow being completely obscured. In this case, the corresponding image to be adjusted includes the eyebrow, so the first restored image can be restored a second time.

[0118] In this embodiment, the descriptive information includes: a description of the facial structure, a description of the scene in which the face is located, and a description of the facial brightness and darkness.

[0119] In this embodiment, the area scene refers to the scene in which the face is located, and the first information is related to the description of the scene in which the face is located; the brightness scene refers to the brightness of the face, and the second information is related to the description of the brightness of the face.

[0120] In this embodiment, the first information is scene-related, so there will be factors that affect the brightness. For example, whether the environment is bright or dark will have a certain impact on the brightness description. The second information is related to the brightness of the face. The existing first brightness positioning distribution is related to brightness, and the existing second brightness positioning distribution is related to darkness.

[0121] In this embodiment, the bright area is called "light" and the dark area is called "dark".

[0122] In this embodiment, the brightness positioning distribution can be a positioning distribution achieved by determining the region for the same brightness as described.

[0123] The beneficial effects of the above technical solution are as follows: by extracting information from the description information according to the regional scene and brightness scene, and analyzing the first information and the second information, the brightness factor is obtained. Furthermore, by parsing and fusing the remaining information, a face fusion image is obtained. Then, through image adjustment based on the brightness factor and subsequent comparative analysis, the second restoration of the first restored image is achieved, ensuring a high degree of image restoration and improving recognition accuracy.

[0124] Example 5:

[0125] Based on Example 1, after performing a second restoration on the first restored image, the method further includes:

[0126] Step A1: Obtain the RGB values ​​of the second restored image and the original face image, and use formula (1) to obtain the brightness conversion value of each pixel in the second restored image:

[0127] ΔY(i,j)=0.299×[R(i,j)-R0(i,j)]+0.587×[G(i,j)-G0(i,j)]+ 0.114×[B(i,j)-B0(i,j)] (1)

[0128] Where ΔY(i,j) represents the brightness conversion value of the pixel in the i-th row and j-th column of the second restored image; [R(i,j), G(i,j), B(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image; [R0(i,j), G0(i,j), B0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the original face image;

[0129] Step A2: Using formula (2), based on the brightness conversion value of each pixel and the RGB value of the second restored image, obtain the RGB value of the second restored image that has the same brightness value as the original face image:

[0130]

[0131] Where [r0(i,j), g0(i,j), b0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image, which has the same brightness value as the original face image; This indicates the calculation of the intermediate matrix;

[0132] Step A3: Using formula (3), based on the RGB values ​​of the second restored image (which has the same brightness value as the original face image) and the original face image, obtain the difference between the second restored image (which has the same brightness value as the original face image) and the original face image, and mark the difference on the second restored image:

[0133]

[0134] Where μ represents the difference between the second restored image and the original face image, which have the same brightness value as the original face image; || represents the absolute value; n represents the total number of pixels in each column of the image; m represents the total number of pixels in each row of the image; max[,] represents the maximum value of the values ​​on both sides of the comma in the parentheses;

[0135] The μ value control is displayed on the second restored image.

[0136] The beneficial effects of the above technical solution are: using formula (1) in step A1, the luminance conversion value of each pixel in the second restored image is obtained based on the RGB values ​​of the second restored image and the RGB values ​​of the original face image, which facilitates the subsequent unification of the luminance values; then using formula (2) in step A2, the luminance conversion value of each pixel in the second restored image is obtained based on the RGB values ​​of the second restored image and the RGB values ​​of the original face image. The RGB values ​​of the second restored image with the same brightness value as the original face image are obtained, thereby unifying the brightness values ​​of the two images. This avoids the difference in brightness when calculating the difference in subsequent calculations, making the subsequent calculation results more accurate. In addition, the introduction of an intermediate matrix in the formula can simplify the calculation steps and improve the computational efficiency. Finally, using formula (3) in step A3, the difference between the second restored image with the same brightness value as the original face image and the original face image is obtained based on the RGB values ​​of the second restored image with the same brightness value as the original face image and the original face image. The difference control is marked on the second restored image, which is beneficial for the trackers to judge the credibility of the information in the second restored image and prevent the tracking of the first person from being missed due to the large difference in the second restored image.

[0137] Example 6:

[0138] This invention provides a face image restoration system for assisted tracking, such as... Figure 2 As shown, it includes:

[0139] The model training module is used to train a face restoration model based on deep learning algorithms.

[0140] The first restoration module is used to acquire the original face image of the captured first person and perform a first restoration on the original face image based on the face restoration model.

[0141] The second restoration module is used to obtain the description information of the second person to the first person, and to perform a second restoration on the first restored image.

[0142] The beneficial effects of the above technical solution are: by training a face restoration model through deep learning algorithms and combining it with the description information of the first person by the second person, the face of the first person can be restored, thereby improving the accuracy of identity recognition and further improving the efficiency of assistance.

[0143] Example 7:

[0144] Based on Example 6, it also includes:

[0145] A face localization module is used to locate the face of the first person based on the second restored image;

[0146] The information acquisition module is used to acquire trackable information of the first person based on the face positioning results;

[0147] The search and location module is used to search and locate the current location of the first person in real time based on the trackable information, and to provide a reminder.

[0148] The beneficial effects of the above technical solution are: effective tracking is facilitated through face positioning, and real-time search and positioning can be achieved based on trackable information, which can further improve the auxiliary efficiency of capturing the first person.

[0149] Example 8:

[0150] Based on Embodiment 6, the first restoration module includes:

[0151] The occlusion analysis unit is used to perform occlusion analysis on the original face image to obtain the facial occlusion information of the first person.

[0152] The difficulty determination unit is used to determine the restoration difficulty based on the facial occlusion information and standard contour information.

[0153] The repair unit is used to determine the number of repair attempts based on the restoration difficulty, and control the face restoration model to perform the corresponding number of repair attempts on the original face image, thereby achieving the first restoration of the original face image.

[0154] The beneficial effects of the above technical solution are: by performing occlusion analysis and comparing with standard contour information, the number of repairs to be performed is determined, and then multiple repairs are performed according to the model to ensure the qualification of the original face image repair and further improve the accuracy of identity recognition.

[0155] Example 9:

[0156] Based on Embodiment 6, the second restoration module includes:

[0157] An information extraction unit is used to extract first information related to the regional scene and second information related to the brightness scene from the description information;

[0158] A factor determination unit is used to determine the brightness influence factors existing in the regional scene based on the first information;

[0159] The positioning distribution determination unit is used to determine the bright areas and dark areas existing in the brightness scene based on the second information, and to set a first brightness positioning distribution for the bright areas and a second brightness positioning distribution for the dark areas.

[0160] The feature fusion unit is used to obtain the remaining information in the description information, parse the remaining information to obtain the parsed features, and fuse the parsed features with facial features to obtain a face fusion image.

[0161] The pre-adjustment unit is used to pre-adjust the face fusion image based on the first brightness positioning distribution, the second brightness positioning distribution, and the brightness influencing factor to obtain the image to be adjusted;

[0162] The comparison analysis unit is used to compare and analyze the image to be adjusted with the first restored image, extract difference information based on the comparison analysis results, and determine whether the conditions for reliable restoration are met.

[0163] If available, a second restoration is performed on the first restored image based on the difference information;

[0164] If not, extract key information from the difference information, and perform a second restoration on the first restored image based on the key information.

[0165] The beneficial effects of the above technical solution are as follows: by extracting information from the description information according to the regional scene and brightness scene, and analyzing the first information and the second information, the brightness factor is obtained. Furthermore, by parsing and fusing the remaining information, a face fusion image is obtained. Then, through image adjustment based on the brightness factor and subsequent comparative analysis, the second restoration of the first restored image is achieved, ensuring a high degree of image restoration and improving recognition accuracy.

[0166] Example 10:

[0167] Based on Example 6, it also includes:

[0168] The first calculation module is used to obtain the RGB values ​​of the second restored image and the RGB values ​​of the original face image, and to obtain the brightness conversion value of each pixel in the second restored image using formula (1):

[0169] ΔY(i,j)=0.299×[R(i,j)-R0(i,j)]+0.587×[G(i,j)-G0(i,j)]+ 0.114×[B(i,j)-B0(i,j)](1)

[0170] Where ΔY(i,j) represents the brightness conversion value of the pixel in the i-th row and j-th column of the second restored image; [R(i,j), G(i,j), B(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image; [R0(i,j), G0(i,j), B0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the original face image;

[0171] The second calculation module is used to obtain the RGB values ​​of the second restored image, which are the same as the brightness values ​​of the original face image, based on the brightness conversion value of each pixel and the RGB values ​​of the second restored image using formula (2):

[0172]

[0173] Where [r0(i,j), g0(i,j), b0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image, which has the same brightness value as the original face image; This indicates the calculation of the intermediate matrix;

[0174] The third calculation module is used to calculate the difference between the second restored image with the same brightness value as the original face image and the original face image using formula (3) based on the RGB values ​​of the second restored image with the same brightness value as the original face image and the RGB values ​​of the original face image, and to mark the difference on the second restored image:

[0175]

[0176] Where μ represents the difference between the second restored image and the original face image, which have the same brightness value as the original face image; || represents the absolute value; n represents the total number of pixels in each column of the image; m represents the total number of pixels in each row of the image; max[,] represents the maximum value of the values ​​on both sides of the comma in the parentheses;

[0177] The annotation module is used to annotate the μ value control on the second restored image for display.

[0178] The beneficial effects of the above technical solution are: using formula (1) in step A1, the luminance conversion value of each pixel in the second restored image is obtained based on the RGB values ​​of the second restored image and the RGB values ​​of the original face image, which facilitates the subsequent unification of the luminance values; then using formula (2) in step A2, the luminance conversion value of each pixel in the second restored image is obtained based on the RGB values ​​of the second restored image and the RGB values ​​of the original face image. The RGB values ​​of the second restored image with the same brightness value as the original face image are obtained, thereby unifying the brightness values ​​of the two images. This avoids the difference in brightness when calculating the difference in subsequent calculations, making the subsequent calculation results more accurate. In addition, the introduction of an intermediate matrix in the formula can simplify the calculation steps and improve the computational efficiency. Finally, using formula (3) in step A3, the difference between the second restored image with the same brightness value as the original face image and the original face image is obtained based on the RGB values ​​of the second restored image with the same brightness value as the original face image and the original face image. The difference control is marked on the second restored image, which is beneficial for the trackers to judge the credibility of the information in the second restored image and prevent the tracking of the first person from being missed due to the large difference in the second restored image.

[0179] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for face image restoration using assisted tracking, characterized in that, include: Step 1: Train a face restoration model based on deep learning algorithms; Step 2: Obtain the original face image of the first captured person, and perform a first restoration on the original face image based on the face restoration model; Step 3: Obtain the description information of the first person from the second person, and perform a second restoration on the first restored image; The process of obtaining the second person's description of the first person and performing a second restoration on the first restored image includes: Extract the first information related to the regional scene and the second information related to the brightness scene from the description information; Based on the first information, determine the brightness-affecting factors present in the scene of the region; Based on the second information, the bright areas and dark areas existing in the brightness scene are determined, and a first brightness positioning distribution for the bright areas and a second brightness positioning distribution for the dark areas are set. Obtain the remaining information in the description information, parse the remaining information to obtain the parsed features, and fuse the parsed features with facial features to obtain a fused facial image; Based on the first brightness positioning distribution, the second brightness positioning distribution, and the brightness influencing factor, the face fusion image is pre-adjusted to obtain the image to be adjusted. The image to be adjusted is compared and analyzed with the first restored image. Based on the comparison and analysis results, the difference information is extracted to determine whether the conditions for reliable restoration are met. If available, a second restoration is performed on the first restored image based on the difference information; If not, extract key information from the difference information, and perform a second restoration on the first restored image based on the key information.

2. The face image restoration method with assisted tracking as described in claim 1, characterized in that, After performing a second restoration on the first restored image, the process also includes: Based on the second restored image, the face of the first person is located; Based on the face localization results, trackable information of the first person is obtained; Based on the trackable information, the current location of the first person is searched and located in real time, and a reminder is issued.

3. The face image restoration method with assisted tracking as described in claim 1, characterized in that, After obtaining the original facial image of the first person captured, the process also includes: Occlusion analysis is performed on the original face image to obtain the facial occlusion information of the first person. The difficulty of restoration is determined based on the facial occlusion information and standard contour information. Based on the difficulty of restoration, the number of repair attempts is determined, and the face restoration model is controlled to perform the corresponding number of repair attempts on the original face image, thereby achieving the first restoration of the original face image.

4. The face image restoration method with assisted tracking as described in claim 1, characterized in that, After performing a second restoration on the first restored image, the process also includes: Step A1: Obtain the RGB values ​​of the second restored image and the original face image, and use formula (1) to obtain the brightness conversion value of each pixel in the second restored image: ΔY(i,j)=0.299×[R(i,j)-R0(i,j)]+0.587×([G(i,j)-G0(i,j)]+0.114×[B(i,j)-B0(i,j)](1) Where ΔY(i,j) represents the brightness conversion value of the pixel in the i-th row and j-th column of the second restored image; [R(i,j),G(i,j),B(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image; and [R0(i,j),G0(i,j),B0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the original face image. Step A2: Using formula (2), based on the brightness conversion value of each pixel and the RGB value of the second restored image, obtain the RGB value of the second restored image that has the same brightness value as the original face image: Where [r0(i,j),g0(i,j),b0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image, which has the same brightness value as the original face image; This indicates the calculation of the intermediate matrix; Step A3: Using formula (3), based on the RGB values ​​of the second restored image (which has the same brightness value as the original face image) and the original face image, obtain the difference between the second restored image (which has the same brightness value as the original face image) and the original face image, and mark the difference on the second restored image: Where μ represents the difference between the second restored image with the same brightness value as the original face image and the original face image; | represents the absolute value; n represents the total number of pixels in each column of the image; m represents the total number of pixels in each row of the image; max[,] represents the maximum value of the values ​​on both sides of the comma in the parentheses; The μ value control is displayed on the second restored image.

5. A face image restoration system with assisted tracking, characterized in that, include: The model training module is used to train a face restoration model based on deep learning algorithms. The first restoration module is used to acquire the original face image of the captured first person and perform a first restoration on the original face image based on the face restoration model. The second restoration module is used to obtain the description information of the second person to the first person, and to perform a second restoration on the first restored image; The second restoration module includes: An information extraction unit is used to extract first information related to the regional scene and second information related to the brightness scene from the description information; A factor determination unit is used to determine the brightness influence factors existing in the regional scene based on the first information; The positioning distribution determination unit is used to determine the bright areas and dark areas existing in the brightness scene based on the second information, and to set a first brightness positioning distribution for the bright areas and a second brightness positioning distribution for the dark areas. The feature fusion unit is used to obtain the remaining information in the description information, parse the remaining information to obtain the parsed features, and fuse the parsed features with facial features to obtain a face fusion image. The pre-adjustment unit is used to pre-adjust the face fusion image based on the first brightness positioning distribution, the second brightness positioning distribution, and the brightness influencing factor to obtain the image to be adjusted; The comparison analysis unit is used to compare and analyze the image to be adjusted with the first restored image, extract difference information based on the comparison analysis results, and determine whether the conditions for reliable restoration are met. If available, a second restoration is performed on the first restored image based on the difference information; If not, extract key information from the difference information, and perform a second restoration on the first restored image based on the key information.

6. The assisted tracking face image restoration system as described in claim 5, characterized in that, Also includes: A face localization module is used to locate the face of the first person based on the second restored image; The information acquisition module is used to acquire trackable information of the first person based on the face positioning results; The search and location module is used to search and locate the current location of the first person in real time based on the trackable information, and to provide a reminder.

7. The assisted tracking face image restoration system as described in claim 5, characterized in that, The first restoration module includes: The occlusion analysis unit is used to perform occlusion analysis on the original face image to obtain the facial occlusion information of the first person. The difficulty determination unit is used to determine the restoration difficulty based on the facial occlusion information and standard contour information. The repair unit is used to determine the number of repair attempts based on the restoration difficulty, and control the face restoration model to perform the corresponding number of repair attempts on the original face image, thereby achieving the first restoration of the original face image.

8. The assisted tracking face image restoration system as described in claim 5, characterized in that, Also includes: The first calculation module is used to obtain the RGB values ​​of the second restored image and the RGB values ​​of the original face image, and to obtain the brightness conversion value of each pixel in the second restored image using formula (1): ΔY(i,j)=0.299×[R(i,j)-R0(i,j)]+0.587×([G(i,j)-G0(i,j)]+0.114×[B(i,j)-B0(i,j)](1) Where ΔY(i,j) represents the brightness conversion value of the pixel in the i-th row and j-th column of the second restored image; [R(i,j),G(i,j),B(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image; and [R0(i,j),G0(i,j),B0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the original face image. The second calculation module is used to obtain the RGB values ​​of the second restored image, which are the same as the brightness values ​​of the original face image, based on the brightness conversion value of each pixel and the RGB values ​​of the second restored image using formula (2): Where [r0(i,j),g0(i,j),b0(i,j)] represents the RGB value of the pixel in the i-th row and j-th column of the second restored image, which has the same brightness value as the original face image; This indicates the calculation of the intermediate matrix; The third calculation module is used to calculate the difference between the second restored image with the same brightness value as the original face image and the original face image using formula (3) based on the RGB values ​​of the second restored image with the same brightness value as the original face image and the RGB values ​​of the original face image, and to mark the difference on the second restored image: Where μ represents the difference between the second restored image with the same brightness value as the original face image and the original face image; | represents the absolute value; n represents the total number of pixels in each column of the image; m represents the total number of pixels in each row of the image; max[,] represents the maximum value of the values ​​on both sides of the comma in the parentheses; The annotation module is used to annotate the μ value control on the second restored image for display.

Images