Construction of a face tracking network / Face tracking method, system, medium and terminal

Abstract

The present invention provides a face tracking network construction / face tracking method, system, medium, and terminal. The face tracking network is constructed using the face tracking network construction method. The face tracking method comprises: reading a current frame from facial data to be detected and inputting it into the face tracking network; extracting facial fusion features in the current frame using the face tracking network; comparing the facial fusion features in the current frame with the facial fusion features in the previous frame; and when the comparison results are consistent, annotating the face using a bounding box generated by the face tracking network; and looping through the above steps to track the face in the facial data to be detected using the bounding box. The present invention can accurately track faces in a specified scene using the constructed optimal face tracking network, eliminating the limitations of face tracking conditions such as changes in posture, proportion, expression, and lighting.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and relates to a construction / tracking method, and in particular to a face tracking network construction / face tracking method, system, medium and terminal. Background Art

[0002] Face tracking is the process of determining the correspondence between faces in frames of a dynamic image sequence. It is a crucial step in dynamic facial information processing and has significant applications in security surveillance, video conferencing, AI interaction, videophones, and identity verification. Therefore, it has become a highly active research area in computer vision. However, face tracking is a challenging task due to the uncertainty in the number and size of faces in dynamic image sequences, the diversity of facial poses and expressions, the variability of lighting conditions, and the complexity of the imaging environment.

[0003] Therefore, how to provide a face tracking network construction / face tracking method, system, medium and terminal to solve the defects of existing technologies such as the uncertainty of the number and size of faces, the diversity of facial postures and expressions, the variability of lighting conditions, the complexity of the imaging environment, etc., and the inability to meet real-time application requirements has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a face tracking network construction / face tracking method, system, medium and terminal, which are used to solve the problem that the prior art is unable to meet the needs of real-time applications due to limitations such as the uncertainty of the number and size of faces, the diversity of facial postures and expressions, the variability of lighting conditions, and the complexity of the imaging environment.

[0005] To achieve the above-mentioned objectives and other related objectives, the present invention provides, on the one hand, a method for building a face tracking network, comprising: obtaining a face tracking training dataset; the face tracking training dataset includes a plurality of face images; extracting low-level features and high-level features from the face images; the low-level features are used to characterize local features of the face; the high-level features are used to characterize detailed features of the face; based on the extracted low-level features and high-level features, a face fusion feature is formed, and the face tracking network is trained using the face fusion feature to build an optimal face tracking network; the optimal face tracking network is used to generate a bounding box for annotating a face.

[0006] In one embodiment of the present invention, before the step of extracting low-level features and high-level features from the facial image, the method for building a face tracking network further includes: preprocessing the facial image; the step of preprocessing the facial image includes: filtering out noise in the facial image; converting the facial image after noise filtering from RGB to HSV; and removing the background of the converted facial image to obtain a facial image after background removal.

[0007] In one embodiment of the present invention, a fusion feature of a face is formed according to the extracted low-level features and high-level features, and a face tracking network is trained using the fusion feature of the face to build an optimal face tracking network. The steps include: assigning an i-th first weight coefficient and an i-th second weight coefficient to the extracted low-level features and high-level features, respectively, to form an i-th face fusion feature; the i-th first weight coefficient is equal to the i-1-th first weight coefficient plus a preset step amount, and the i-th second weight coefficient is equal to the i-1-th second weight coefficient minus the preset step amount; or the i-th first weight coefficient is equal to the i-1-th first weight coefficient minus the preset step amount, and the i-th second The weight coefficient is equal to i-1 weight coefficients plus a preset step amount; the i-th first weight coefficient plus the i-th second weight coefficient is equal to 1; where i is greater than 1; the i-th face fusion feature is input into the face tracking network for face prediction, and the i-th face prediction result is obtained; the i-th face prediction result and the real face information are subjected to loss calculation; the above steps are executed repeatedly until the minimum loss is calculated, and the corresponding optimal first weight coefficient and optimal second weight coefficient are found through the minimum loss; the face tracking network is trained through the optimal face fusion feature formed by the optimal first weight coefficient and the optimal second weight coefficient to form an optimal face tracking network.

[0008] On the other hand, the present invention provides a face tracking method based on a face tracking network, wherein the face tracking network is constructed by the method for constructing a face tracking network according to any one of claims 1 to 3; the face tracking method comprises: reading a current frame from the face data to be detected and inputting it into the face tracking network; extracting face fusion features in the current frame through the face tracking network; comparing the face fusion features of the current frame with the face fusion features of the previous frame, and when the comparison results are consistent, marking the face with a bounding box generated by the face tracking network; and looping through the above steps so as to track the face in the face data to be detected through the bounding box.

[0009] In one embodiment of the present invention, the face fusion features include low-level features and high-level features; the low-level features are used to characterize local features of the face; and the high-level features are used to characterize detailed features of the face.

[0010] In one embodiment of the present invention, the step of comparing the face fusion features of the current frame with the face fusion features of the previous frame includes: comparing the low-level features of the current frame with the low-level features of the previous frame; comparing the high-level features of the current frame with the high-level features of the previous frame; when the comparison results of the low-level features of the current frame and the low-level features of the previous frame are consistent, and the comparison results of the high-level features of the current frame and the high-level features of the previous frame are consistent, it means that the face tracked in the current frame is consistent with the face tracked in the previous frame.

[0011] In one embodiment of the present invention, before reading the current frame from the facial data to be detected, the face tracking method further includes: preprocessing the facial data to be detected; the step of preprocessing the facial data to be detected includes: filtering out noise in the facial data to be detected; converting the facial data after noise filtering from RGB to HSV; and removing the background of the converted facial data to obtain facial data after background removal.

[0012] On the other hand, the present invention provides a system for building a face tracking network, including: a data acquisition module for acquiring a face tracking training data set; the face tracking training data set includes a plurality of face images; an extraction module for extracting low-level features and high-level features from the face images; the low-level features are used to characterize local features of the face; the high-level features are used to characterize detailed features of the face; a fusion training module for forming face fusion features based on the extracted low-level features and high-level features, and using the face fusion features to train a face tracking network to build an optimal face tracking network; the optimal face tracking network is used to generate a bounding box for annotating a face.

[0013] Another aspect of the present invention provides a face tracking system based on a face tracking network, wherein the face tracking network is constructed by the face tracking network construction system; the face tracking system includes: a reading module for reading a current frame from the face data to be detected and inputting it into the face tracking network; an extraction module for extracting face fusion features in the current frame through the face tracking network; a comparison module for comparing the face fusion features of the current frame with the face fusion features of the previous frame, and when the comparison results are consistent, annotating the face using the bounding box generated by the face tracking network; and a processing module for cyclically running the above modules so as to track the face in the face data to be detected through the bounding box.

[0014] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a face tracking network construction method and / or a face tracking method based on a face tracking network.

[0015] The last aspect of the present invention provides a terminal, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes a face tracking network construction method and / or a face tracking method based on a face tracking network.

[0016] As described above, the face tracking network construction / face tracking method, system, medium and terminal of the present invention have the following beneficial effects:

[0017] The face tracking network construction / face tracking method, system, medium, and terminal described in the present invention utilizes all low-level and high-level features, a channel attention mechanism, and a fusion network within the regression network. This significantly improves speed compared to the GOTURN tracker and meets the needs of real-time applications. Furthermore, the optimized face tracking network can accurately track faces in a given scene, freeing face tracking from constraints such as pose, proportion, expression, and lighting changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Shown is a schematic diagram of an application scenario of the present invention.

[0019] Figure 2 FIG. 1 is a flow chart of a method for building a face tracking network according to an embodiment of the present invention.

[0020] Figure 3 Shown is a flow chart of S22 in the method for building a face tracking network of the present invention.

[0021] Figure 4 FIG. 1 is a flow chart of a face tracking method based on a face tracking network according to an embodiment of the present invention.

[0022] Figure 5 Shown is a schematic diagram of the principle structure of a system for building a face tracking network in one embodiment of the present invention.

[0023] Figure 6 FIG. 1 is a schematic diagram showing the principle structure of a face tracking system based on a face tracking network in one embodiment of the present invention.

[0024] Component number description

[0025] 5. Building a Face Tracking Network System

[0026] 51 Data Acquisition Module

[0027] 52 Image Processing Module

[0028] 53 Extraction Module

[0029] 54 Fusion Training Module

[0030] 6 Face tracking network based face

[0031] Tracking system

[0032] 61 Image preprocessing module

[0033] 62 Read Module

[0034] 63 Extraction Module

[0035] 64 comparison module

[0036] 65 Processing Module

[0037] Steps S21 to S24

[0038] S41~S45 DETAILED DESCRIPTION

[0039] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0040] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0041] Example 1

[0042] This embodiment provides a method for building a face tracking network, including:

[0043] Obtaining a face tracking training dataset; the face tracking training dataset includes a plurality of face images;

[0044] Extracting low-level features and high-level features from a face image; the low-level features are used to characterize local features of the face; the high-level features are used to characterize detailed features of the face;

[0045] Based on the extracted low-level features and high-level features, a face fusion feature is formed, and the face tracking network is trained using the face fusion feature to build an optimal face tracking network; the optimal face tracking network is used to generate a bounding box for annotating the face.

[0046] The following is a detailed description of the face tracking network construction method provided by this embodiment with reference to the diagram. The face tracking network construction method of this embodiment can be applied to Figure 1 In the specific environment shown, for example, the company's attendance office, the company corridor, etc., the facial data entering and leaving the specific environment is trained to train a face tracking network that accurately locates the target.

[0047] See also Figure 2 , which is a flow chart of a method for building a face tracking network in one embodiment. Figure 2 As shown, the method for building the face tracking network specifically includes the following steps:

[0048] S21, obtaining a face tracking training dataset, wherein the face tracking training dataset includes a plurality of face images.

[0049] For example, a company clocking in and out video dataset and a corridor surveillance video dataset were obtained, named G1 and G2, respectively. This dataset contains 430 FPS sequences with a frame size of 800×600. Face locations in the frames of G1 and G2 were manually annotated with bounding boxes. Each video sequence consists of only one person and 120 frames at a given time. In this example, G1 was used to train a face tracking network, and G2 was used to evaluate the face tracking network.

[0050] S22, pre-processing the facial image. In this embodiment, since the facial image in the surveillance video data is distorted, it is necessary to perform image enhancement to obtain crucial image features.

[0051] See also Figure 3 , which is a flow chart of S22. Figure 3 As shown, the step of preprocessing the face image includes:

[0052] S221, filtering out noise in the face image.

[0053] Since surveillance video data often contains noise that reduces the quality of frame images, which can lead to false positive results centered on statistical assumptions, the Anisotropic Gaussian Edge Stop Function (GESFA) filters out unnecessary noise. However, this filter will have a blurring effect, so in order to avoid this problem, this application uses GESFA instead of Anisotropic Edge Stop Function as GESFA rendering. Because GESFA provides superior performance by limiting blur while retaining image details. The noise removal function of the GESFA filter is expressed as the following formula:

[0054]

[0055] Where i represents the intensity of the i-th pixel, w is a scalar related to the diffusion rate, ψ represents the set of adjacent pixels, E represents the edge stop function, g represents the Gaussian filter at scale s, and f i,j Represents the frame direction gradient magnitude from pixel i to pixel j.

[0056] S222: Convert the face image after noise is filtered out from RGB to HSV.

[0057] Typically, video data is stored in RGB format. The brightness of the video frame can be increased by converting RGB to HSV format, as HSV is more related to human experience.

[0058] S223: performing background removal on the converted facial image to obtain a facial image after background removal.

[0059] In this embodiment, the BGR technique uses a BG model of the scene to discover FG objects, using an incoming frame that is different from the model. The general requirements of the BGR algorithm are the accuracy of the MOD and the temporal stability of the detection. This application uses an improved Gaussian mixture model (IGMM) to support the BGR process. Because GMM uses DPHE to improve image enhancement quality. In GMM, each pixel in a frame is designed to be Gaussian distributed. Each pixel calculates its probability at once, whether it is included in FG or BG: The distribution D can be divided into the BG model or the FG model according to the relationship between w and the threshold T. Subsequently, each pixel is divided based on the following three cases:

[0060]

[0061] After dividing each pixel on the frame into BG and FG regions, the FG region in the frame is extracted by removing the BG region.

[0062] S23, extracting low-level features and high-level features from the face image. In this embodiment, the low-level features are used to characterize local features of the face, and the high-level features are used to characterize detailed features of the face.

[0063] For example, the low-level features include regional features such as eyes, nose, and mouth.

[0064] These high-level features include the relationship between facial features, facial color, and details like moles and scars. While these high-level features are very useful for tasks requiring semantic information, their receptive fields are very large, making them less accurate at locating objects. Therefore, this application utilizes both high-level and low-level features.

[0065] S24: Based on the extracted low-level and high-level features, a face fusion feature is formed. The face fusion feature is used to train a face tracking network to build an optimal face tracking network. The optimal face tracking network is used to generate a bounding box for annotating the face. The bounding box is a tuple consisting of three elements: the x-coordinate of the bounding box, the y-coordinate of the bounding box, and the width of the bounding box.

[0066] Specifically, the S24 includes the following steps:

[0067] S241, assigning the i-th first weight coefficient and the i-th second weight coefficient to the extracted low-level features and high-level features respectively to form the i-th face fusion feature; the i-th first weight coefficient is equal to the i-1-th first weight coefficient plus the preset step amount, and the i-th second weight coefficient is equal to the i-1-th second weight coefficient minus the preset step amount S, for example, S = 0.01; or the i-th first weight coefficient is equal to the i-1-th first weight coefficient minus the preset step amount, and the i-th second weight coefficient is equal to the i-1-th weight coefficient plus the preset step amount; the i-th first weight coefficient plus the i-th second weight coefficient is equal to 1; wherein, i is greater than 1.

[0068] In this embodiment, the first weight coefficient of the initial low-level feature may be 0, and the second weight coefficient of the initial high-level feature may be 1.

[0069] S242: Input the fusion feature of the i-th face into the face tracking network for face prediction, and obtain the i-th face prediction result.

[0070] S243, performing loss calculation on the predicted result of the i-th face and the real information of the face to obtain a loss L.

[0071] S244, loop through S241-S243 until the minimum loss Lmin is calculated, and find the corresponding optimal first weight coefficient and optimal second weight coefficient through the minimum loss Lmin.

[0072] In this embodiment, the above steps can be implemented using a channel attention module, a simple two-layer multilayer perceptron (MLP) used to weight input channels. Because the low-level and high-level features of the feature extraction module are 9×9 and 54×54 in size, respectively, they are converted to 3×3 using a 3×3 kernel and an 18×18 kernel with a stride of 3, respectively. That is, both the high-level feature channels and the low-level channels are 3×3 in size. The first fully connected (FC) layer has 9 Relu activation units, and each FC layer has a Sigmoid activation layer. The Sigmoid is used to ensure that no channel is suppressed to zero. The output of the activation layer corresponds to the weight coefficient of the input channel. These coefficients are then used to weight the channels according to their importance in tracking, that is, how to weight two features. Since six different VGG-Fac layers are used, there are also six different channel attention modules. The channel attention module is trained during the offline training process.

[0073] S245 , training the face tracking network using the optimal face fusion feature formed by the optimal first weight coefficient and the optimal second weight coefficient to form an optimal face tracking network.

[0074] In this embodiment, S245 is implemented by a regression network. The regression network is composed of a convolutional layer and two FC layers, and is used to regress the bounding box of the target from the input optimal face fusion.

[0075] The face tracking network constructed by the face tracking network construction method described in this embodiment utilizes all low-level and high-level features, channel attention mechanism and adopts a fusion network in the regression network, which has a significant speed improvement compared to the GOTURN tracker and can meet the needs of real-time applications.

[0076] This embodiment further provides a face tracking method based on a face tracking network, the face tracking method comprising:

[0077] Reading a current frame from the face data to be detected and inputting it into the face tracking network;

[0078] Extracting face fusion features in the current frame through the face tracking network;

[0079] Comparing the face fusion features of the current frame with the face fusion features of the previous frame, and when the comparison results are consistent, annotating the face using the bounding box generated by the face tracking network;

[0080] The above steps are executed in a loop so as to track the face in the face data to be detected through the bounding box.

[0081] The face tracking method based on the face tracking network provided by this embodiment will be described in detail below with reference to the figures. In this embodiment, the face tracking network is constructed by the above-mentioned face tracking network construction method.

[0082] See also Figure 4 , which is a flow chart of a face tracking method based on a face tracking network in one embodiment. Figure 4 As shown, the face tracking method specifically includes the following steps:

[0083] S41, preprocessing the face data to be detected.

[0084] S41 includes: filtering noise from the facial data to be detected; converting the noise-filtered facial data from RGB to HSV; and performing background removal on the converted facial data to obtain background-removed facial data. In this embodiment, noise processing, data conversion, and background removal are the same as those in the above-mentioned image preprocessing method and will not be further described here.

[0085] S42, reading the current frame from the face data to be detected and inputting it into the face tracking network.

[0086] S43, extracting face fusion features in the current frame through the face tracking network.

[0087] In this embodiment, the facial fusion features include low-level features and high-level features. The low-level features are used to characterize local features of a face, while the high-level features are used to characterize detailed features of a face. For example, low-level features include regional features such as the eyes, nose, and mouth. High-level features include the relationship between facial features, facial color, and detailed features such as moles and scars.

[0088] S44, comparing the face fusion features of the current frame with the face fusion features of the previous frame, and when the comparison results are consistent, annotating the face using the bounding box generated by the face tracking network.

[0089] In this embodiment, the S44 includes: comparing the low-level features of the current frame with the low-level features of the previous frame;

[0090] Compare the high-level features of the current frame with those of the previous frame;

[0091] When the comparison results of the low-level features of the current frame and the low-level features of the previous frame are consistent, and the comparison results of the high-level features of the current frame and the high-level features of the previous frame are consistent, it means that the face tracked in the current frame is consistent with the face tracked in the previous frame.

[0092] S45 , looping through S42 to S44 to track the face in the face data to be detected through the bounding box until all frames in the face data to be detected are read.

[0093] In the loop of this embodiment, the current frame is used as the previous frame and the next frame is used as the current frame. The face tracking network is used to continue comparing facial features and read the bounding box of the next frame. The face is tracked by sequentially marking the bounding boxes.

[0094] The face tracking method based on the face tracking network described in this embodiment can accurately track faces in a specified scene by building an optimal face tracking network, so that face tracking is no longer restricted by conditions such as changes in posture, proportion, expression and lighting, and meets real-time application requirements.

[0095] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following is achieved: Figure 2 and / or Figure 4 The method.

[0096] At any possible level of technical detail combination, the present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0097] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0098] The computer-readable program described herein can be downloaded to each computing / processing device from a computer-readable storage medium, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device. The computer program instructions for performing the operation of this application can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and procedural programming languages such as "C" language or similar programming languages. Computer readable program instructions can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partly on the user's computer, partly on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., utilizing an Internet service provider to connect through the Internet). In certain embodiments, personalized electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are customized by utilizing the state information of computer readable program instructions, which electronic circuits can execute computer readable program instructions, thereby realizing various aspects of the present application.

[0099] Example 2

[0100] This embodiment provides a system for building a face tracking network, including:

[0101] A data acquisition module is used to acquire a face tracking training data set; the face tracking training data set includes a plurality of face images;

[0102] An extraction module is used to extract low-level features and high-level features from a face image; the low-level features are used to characterize local features of the face; the high-level features are used to characterize detailed features of the face;

[0103] A fusion training module is used to form a face fusion feature based on the extracted low-level features and high-level features, and use the face fusion feature to train the face tracking network to build an optimal face tracking network; the optimal face tracking network is used to generate a bounding box for annotating the face.

[0104] The following will describe in detail the system for building a face tracking network provided by this embodiment with reference to the diagram. Figure 5 , which is a schematic diagram showing the principle structure of a system for building a face tracking network in one embodiment. Figure 5 As shown, the face tracking network construction system 5 includes a data acquisition module 51, an image processing module 52, an extraction module 53 and a fusion training module 54.

[0105] The data acquisition module 51 is used to acquire a face tracking training data set; the face tracking training data set includes a plurality of face images.

[0106] The image processing module 52 is used to pre-process the face image. In this embodiment, since the face image in the monitoring video data is distorted, it is necessary to perform image enhancement to obtain the crucial image features.

[0107] The image processing module 52 is specifically used to filter out noise in the facial image, perform RGB to HSV conversion on the facial image after noise filtering, and perform background removal on the converted facial image to obtain a facial image after background removal.

[0108] The extraction module 53 is used to extract low-level features and high-level features from the face image. In this embodiment, the low-level features are used to characterize local features of the face, and the high-level features are used to characterize detailed features of the face.

[0109] For example, the low-level features include regional features such as eyes, nose, and mouth.

[0110] These high-level features include the relationship between facial features, facial color, and details like moles and scars. While these high-level features are very useful for tasks requiring semantic information, their receptive fields are very large, making them less accurate at locating objects. Therefore, this application utilizes both high-level and low-level features.

[0111] The fusion training module 54 is used to generate a fusion face feature based on the extracted low-level and high-level features. The fusion face feature is then used to train the face tracking network to build an optimal face tracking network. The optimal face tracking network is used to generate a bounding box for annotating the face. The bounding box is a tuple consisting of three elements: the x-coordinate of the bounding box, the y-coordinate of the bounding box, and the width of the bounding box.

[0112] Specifically, the fusion training module 54 assigns the i-th first weight coefficient and the i-th second weight coefficient to the extracted low-level features and high-level features, respectively, to form the i-th face fusion feature. The i-th first weight coefficient is equal to the i-1-th first weight coefficient plus a preset step size, and the i-th second weight coefficient is equal to the i-1-th second weight coefficient minus a preset step size S, for example, S = 0.01. Alternatively, the i-th first weight coefficient is equal to the i-1-th first weight coefficient minus a preset step size, and the i-th second weight coefficient is equal to the i-1-th weight coefficient plus a preset step size. The i-th first weight coefficient plus the i-th second weight coefficient equals 1, where i is greater than 1. The i-th face fusion feature is input into the face tracking network for face prediction, obtaining the i-th face prediction result. The i-th face prediction result is compared with the real face information to calculate the loss, resulting in a loss L. The above process is repeated until the minimum loss Lmin is calculated. The optimal first weight coefficient and the optimal second weight coefficient corresponding to the minimum loss Lmin are found. The face tracking network is trained by using the optimal face fusion feature formed by the optimal first weight coefficient and the optimal second weight coefficient to form an optimal face tracking network.

[0113] This embodiment further provides a face tracking system based on a face tracking network, wherein the face tracking network is constructed by the above-mentioned face tracking network construction system; the face tracking system includes:

[0114] A reading module, configured to read the current frame from the face data to be detected and input it into the face tracking network;

[0115] An extraction module, configured to extract face fusion features in the current frame through the face tracking network;

[0116] A comparison module is used to compare the face fusion features of the current frame with the face fusion features of the previous frame. When the comparison results are consistent, the face is annotated using the bounding box generated by the face tracking network;

[0117] The processing module is used to run the above modules in a loop so as to track the face in the face data to be detected through the bounding box.

[0118] The following is a detailed description of the face tracking system based on the face tracking network provided by this embodiment with reference to the figures. Figure 6 , which is a flow chart of a face tracking system based on a face tracking network in one embodiment. Figure 6 As shown, the face tracking system 6 based on the face tracking network includes an image preprocessing module 61 , a reading module 62 , an extraction module 63 , a comparison module 64 and a processing module 65 .

[0119] The image preprocessing module 61 is used to preprocess the face data to be detected.

[0120] Specifically, the image preprocessing module 61 is used to filter out noise in the face data to be detected; convert the face data after the noise is filtered out from RGB to HSV; and remove the background of the converted face data to obtain face data after the background is removed.

[0121] The reading module 62 is used to read the current frame from the face data to be detected and input it into the face tracking network.

[0122] The extraction module 63 is used to extract the face fusion features in the current frame through the face tracking network.

[0123] In this embodiment, the facial fusion features include low-level features and high-level features. The low-level features are used to characterize local features of a face, while the high-level features are used to characterize detailed features of a face. For example, low-level features include regional features such as the eyes, nose, and mouth. High-level features include the relationship between facial features, facial color, and detailed features such as moles and scars.

[0124] The comparison module 64 is used to compare the face fusion features of the current frame with the face fusion features of the previous frame. When the comparison results are consistent, the face is annotated using the bounding box generated by the face tracking network.

[0125] In this embodiment, the comparison module 64 compares the low-level features of the current frame with the low-level features of the previous frame; and compares the high-level features of the current frame with the high-level features of the previous frame; when the comparison results of the low-level features of the current frame and the low-level features of the previous frame are consistent, and the comparison results of the high-level features of the current frame and the high-level features of the previous frame are consistent, it means that the face tracked in the current frame is consistent with the face tracked in the previous frame.

[0126] The processing module 65 is configured to cyclically run the reading module 62 , the extraction module 63 , and the comparison module 64 so as to track the face in the face data to be detected through the bounding box.

[0127] In the loop of this embodiment, the current frame is used as the previous frame and the next frame is used as the current frame. The face tracking network is used to continue comparing facial features and read the bounding box of the next frame. The face is tracked by sequentially marking the bounding boxes.

[0128] It should be understood that the division of the various modules in the above system is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a single physical entity or physically separated. Furthermore, these modules may be implemented entirely in software called by a processing element, entirely in hardware, or partially in software called by a processing element, while others may be implemented in hardware. For example, module x may be a separate processing element or integrated into a single chip in the above system. Furthermore, module x may be stored in the form of program code in the memory of the above system, called by a processing element in the system to perform the functions of module x. The implementation of other modules is similar. These modules may be fully or partially integrated or implemented independently. The processing element described herein may be an integrated circuit with signal processing capabilities. During implementation, the steps of the above method or the above modules may be performed by hardware integrated logic circuits in the processor element or by software instructions. The above modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), one or more field programmable gate arrays (FPGAs), etc. When a module is implemented by scheduling program code through a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. These modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0129] Example 3

[0130] This embodiment provides a terminal, including: a processor, a memory, a transceiver, a communication interface and / or a system bus; the memory and the communication interface are connected to the processor and the transceiver via the system bus and communicate with each other, the memory is used to store computer programs, the communication interface is used to communicate with other devices, and the processor and the transceiver are used to run the computer program, so that the terminal executes the above-mentioned method for building a face tracking network and / or each step of the face tracking method based on the face tracking network.

[0131] The system bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage.

[0132] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0133] The protection scope of the face tracking network construction / face tracking method described in the present invention is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing, or replacing steps in the prior art based on the principles of the present invention are included in the protection scope of the present invention.

[0134] The present invention also provides a system for building a face tracking network / face tracking method, which can implement the face tracking network building / face tracking method described in the present invention. However, the implementation device of the face tracking network building / face tracking method described in the present invention includes but is not limited to the structure of the face tracking network building / face tracking method system listed in this embodiment. All structural deformations and replacements of the existing technology made according to the principles of the present invention are included in the protection scope of the present invention.

[0135] In summary, the face tracking network construction / face tracking method, system, medium, and terminal described in the present invention utilize all low-level and high-level features, a channel attention mechanism, and a fusion network in the regression network. This significantly improves speed compared to the GOTURN tracker and can meet the needs of real-time applications. Furthermore, the optimized face tracking network can accurately track faces in a specified scene, freeing face tracking from being limited by conditions such as posture, proportion, expression, and lighting changes. Therefore, the present invention effectively overcomes the shortcomings of the prior art and has high industrial application value.

[0136] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A method for building a face tracking network, characterized in that: include: Get the face tracking training dataset; The face tracking training data set includes a number of face images; Extract low-level and high-level features from face images; The low-level features are used to characterize local features of a face; the high-level features are used to characterize detailed features of a face; the low-level features include regional features of the eyes, nose, and mouth; the high-level features include the relationship between facial features, facial color, and detailed features of moles and scars; Based on the extracted low-level features and high-level features, a face fusion feature is formed, and the face tracking network is trained using the face fusion feature to build an optimal face tracking network; the optimal face tracking network is used to generate a bounding box for annotating the face; The steps of forming a fusion feature of a face based on the extracted low-level features and high-level features, and using the fusion feature of the face to train a face tracking network to build an optimal face tracking network include: Assigning the i-th first weight coefficient and the i-th second weight coefficient to the extracted low-level features and high-level features respectively to form the i-th face fusion feature; the i-th first weight coefficient is equal to the i-1-th first weight coefficient plus a preset step amount, and the i-th second weight coefficient is equal to the i-1-th second weight coefficient minus the preset step amount; or the i-th first weight coefficient is equal to the i-1-th first weight coefficient minus the preset step amount, and the i-th second weight coefficient is equal to the i-1-th weight coefficient plus the preset step amount; the i-th first weight coefficient plus the i-th second weight coefficient is equal to 1; wherein i is greater than 1; Input the fusion features of the i-th face into the face tracking network for face prediction, and obtain the i-th face prediction result; Calculate the loss between the predicted result of the i-th face and the real information of the face; Repeat the above steps until the minimum loss is calculated, and find the corresponding optimal first weight coefficient and optimal second weight coefficient based on the minimum loss; The face tracking network is trained by using the optimal face fusion feature formed by the optimal first weight coefficient and the optimal second weight coefficient to form an optimal face tracking network.

2. The method for building a face tracking network according to claim 1, characterized in that: Before the step of extracting low-level features and high-level features from the face image, the method for building a face tracking network further includes: preprocessing the face image; the step of preprocessing the face image includes: Filter out noise from face images; Convert the face image after filtering out noise to HSV; The background of the converted face image is removed to obtain a face image after background removal.

3. A face tracking method based on a face tracking network, characterized in that: The face tracking network is constructed by the face tracking network construction method according to any one of claims 1 to 2; the face tracking method comprises: Reading a current frame from the face data to be detected, and inputting the current frame into the face tracking network; Extracting face fusion features in the current frame through the face tracking network; Comparing the face fusion features of the current frame with the face fusion features of the previous frame, and when the comparison results are consistent, annotating the face using the bounding box generated by the face tracking network; The above steps are executed in a loop to track the face in the face data to be detected using the bounding box.

4. The face tracking method based on the face tracking network according to claim 3, characterized in that: The face fusion features include low-level features and high-level features; the low-level features are used to characterize local features of the face; The high-level features are used to characterize detailed features of a human face.

5. The face tracking method based on the face tracking network according to claim 4, characterized in that: The steps of comparing the face fusion features of the current frame with the face fusion features of the previous frame include: Compare the low-level features of the current frame with the low-level features of the previous frame; Compare the high-level features of the current frame with those of the previous frame; If the comparison results of the low-level features of the current frame and the low-level features of the previous frame are consistent, and the comparison results of the high-level features of the current frame and the high-level features of the previous frame are consistent, it means that the face tracked in the current frame is consistent with the face tracked in the previous frame.

6. The face tracking method based on the face tracking network according to claim 3, characterized in that: Before reading the current frame from the face data to be detected, the face tracking method further includes: preprocessing the face data to be detected; The steps for preprocessing the face data to be detected include: Filter out noise in the face data to be detected; Convert the face data after filtering out noise to HSV; The background of the converted face data is removed to obtain face data after background removal.

7. A system for building a face tracking network, characterized in that: A method for building a face tracking network according to any one of claims 1 to 2, the system comprising: A data acquisition module is used to acquire a face tracking training data set; the face tracking training data set includes a plurality of face images; An extraction module is used to extract low-level features and high-level features from a face image; the low-level features are used to characterize local features of the face; the high-level features are used to characterize detailed features of the face; A fusion training module is used to form a face fusion feature based on the extracted low-level features and high-level features, and use the face fusion feature to train the face tracking network to build an optimal face tracking network; the optimal face tracking network is used to generate a bounding box for annotating the face.

8. A face tracking system based on a face tracking network, characterized in that: The face tracking network is constructed by the face tracking network construction system according to claim 7; the face tracking system comprises: A reading module, configured to read a current frame from the face data to be detected and input the current frame into the face tracking network; An extraction module, configured to extract face fusion features in the current frame through the face tracking network; A comparison module is used to compare the face fusion features of the current frame with the face fusion features of the previous frame. When the comparison results are consistent, the face is annotated using the bounding box generated by the face tracking network; The processing module is used to run the above modules in a loop and track the face in the face data to be detected through the bounding box.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the method for building a face tracking network as described in any one of claims 1 to 2 and / or the face tracking method based on a face tracking network as described in any one of claims 3 to 6.

10. A terminal, characterized in that: include: processor and memory; The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the terminal executes the method for building a face tracking network as described in any one of claims 1 to 2 and / or the face tracking method based on a face tracking network as described in any one of claims 3 to 6.

Images