Video image recognition method, device, terminal and storage medium

Abstract

The present disclosure provides a method, device, terminal and storage medium for recognizing video images, and belongs to the field of Internet technology. The method comprises: recognizing key points of faces in each frame of video images; if key points of faces in the video images are not recognized, recognizing key points of faces in the video images according to the face recognition results of designated pictures, and the designated pictures are reference pictures whose time interval between the acquisition time and the acquisition time of the video images is within a first preset time range; based on the recognized key points of faces, determining the face area of ​​the video images. When the present disclosure recognizes the face area in the video images, it first recognizes it based on the video recognition mode. For the video images whose face key points cannot be recognized by the video recognition mode, auxiliary recognition is performed in combination with the reference pictures. Since the reference pictures have relatively accurate face recognition results, auxiliary recognition based on the reference pictures can accurately recognize the face area.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a video image recognition method, device, terminal and storage medium. Background Art

[0002] With the development of Internet technology, live streaming has become more and more popular among users. During live streaming, the host usually turns on beauty features and stickers to beautify their appearance to attract more users to watch. Before implementing beauty features and stickers, the host's face area needs to be identified from the video image.

[0003] At present, the relevant technology mainly adopts the following methods when recognizing video images: real-time acquisition of user's video images, for the key frames in the video images, identifying the face key points in the key frames, and determining the face area in the key frames based on the identified face key points; for non-key frames of the video images, tracking and locating the face key points in the key frames by using a tracking algorithm, identifying the face key points in the non-key frames, and determining the face area in the non-key frames based on the identified face key points.

[0004] However, in the live video scenario, due to the distance and position between the host and the camera, the face area in the captured video image may be larger or smaller. When the face area accounts for a small proportion of the video image, the face area cannot be accurately identified, especially when the face area in the video image accounts for less than 20%, it may even be impossible to identify the face area in the video image. Summary of the invention

[0005] The embodiments of the present disclosure provide a video image recognition method, device, terminal and storage medium, which can accurately recognize the face area in the video image. The technical solution is as follows:

[0006] In one aspect, a method for recognizing a video image is provided, the method comprising:

[0007] During the video image acquisition process, identify the key points of the face in each frame of the video image;

[0008] For any frame of video image, if the key points of the face in the video image are not identified, obtain the face recognition result of a specified picture, where the specified picture is a reference picture whose time interval between the acquisition time and the acquisition time of the video image is within a first preset time range, and the reference picture is a picture used to assist in the recognition of the video image;

[0009] According to the face recognition result of the specified picture, identifying the key points of the face in the video image;

[0010] Based on the identified facial key points, a facial region of the video image is determined.

[0011] In another embodiment of the present disclosure, obtaining a face recognition result of a specified image includes:

[0012] Obtaining the acquisition time of the video image;

[0013] According to the acquisition time of the video image, a reference image with the shortest time interval between the acquisition time and the acquisition time of the video image is obtained from a reference image set, and the obtained reference image is used as the designated image, wherein the reference image set includes multiple reference images and face recognition results thereof;

[0014] Obtain a face recognition result of the specified picture from the reference picture set.

[0015] In another embodiment of the present disclosure, the method further includes:

[0016] During the video image acquisition process, a reference image is acquired every second preset time period;

[0017] Extract facial features from each reference image;

[0018] Based on the facial features of each reference image, each reference image is identified to obtain a facial recognition result of each reference image;

[0019] Based on each reference picture and its face recognition result, the reference picture set is constructed.

[0020] In another embodiment of the present disclosure, the step of identifying key points of a face in the video image according to the face recognition result of the designated image includes:

[0021] Cropping the video image according to the face recognition result of the specified image;

[0022] Identify facial landmarks in cropped video images.

[0023] In another embodiment of the present disclosure, after determining the face area of ​​the video image based on the identified face key points, the method further includes:

[0024] Detect the selected special effect option;

[0025] Based on the selected special effect option, a corresponding special effect is added to the face area.

[0026] In another aspect, a video image recognition device is provided, the device comprising:

[0027] A recognition module is used to identify key points of human faces in each frame of video images during the acquisition of video images;

[0028] an acquisition module, configured to acquire, for any frame of video image, a face recognition result of a specified picture if no face key points in the video image are recognized, wherein the specified picture is a reference picture whose time interval between the acquisition time and the acquisition time of the video image is within a first preset time range, and the reference picture is a picture used to assist in recognition of the video image;

[0029] The recognition module is further used to recognize key points of the face in the video image according to the face recognition result of the specified image;

[0030] The determination module is used to determine the face area of ​​the video image based on the identified face key points.

[0031] In another embodiment of the present disclosure, the acquisition module is used to obtain the acquisition time of the video image; according to the acquisition time of the video image, obtain a reference picture with the shortest time interval between the acquisition time and the acquisition time of the video image from a reference picture set, and use the obtained reference picture as the designated picture, the reference picture set including multiple reference pictures and their face recognition results; obtain the face recognition result of the designated picture from the reference picture set.

[0032] In another embodiment of the present disclosure, the device further comprises:

[0033] The acquisition module is used to acquire a reference picture every second preset time period during the video image acquisition process;

[0034] An extraction module, used to extract facial features of each reference image;

[0035] The recognition module is used to recognize each reference picture based on the facial features of each reference picture to obtain a facial recognition result of each reference picture;

[0036] The construction module is used to construct the reference picture set based on each reference picture and its face recognition result.

[0037] In another embodiment of the present disclosure, the recognition module is further used to crop the video image according to the face recognition result of the designated image; and recognize the key points of the face in the cropped video image.

[0038] In another embodiment of the present disclosure, the device further comprises:

[0039] A detection module, used to detect the selected special effect option;

[0040] The adding module is used to add corresponding special effects in the face area based on the selected special effects option.

[0041] On the other hand, a terminal is provided, comprising a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the video image recognition method described in one aspect.

[0042] On the other hand, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the video image recognition method described in one aspect.

[0043] The technical solution provided by the embodiments of the present disclosure brings the following beneficial effects:

[0044] When recognizing the face area in the video image, the recognition is first performed based on the video recognition mode. For the video image whose video recognition mode cannot recognize the key points of the face, auxiliary recognition is performed in combination with the reference picture. Since the reference picture has a more accurate face recognition result, auxiliary recognition based on the reference picture can accurately identify the face area. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 It is an implementation environment involved in a video image recognition method provided by an embodiment of the present disclosure;

[0047] Figure 2 is a flow chart of a video image recognition method provided by an embodiment of the present disclosure;

[0048] Figure 3 is a schematic diagram of key points of a face provided by an embodiment of the present disclosure;

[0049] Figure 4 is a flow chart of a facial image recognition process provided by an embodiment of the present disclosure;

[0050] Figure 5 is a schematic diagram of the structure of a video image recognition device provided by an embodiment of the present disclosure;

[0051] Figure 6 A structural block diagram of a terminal provided by an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0053] At present, there are mainly two recognition modes for recognizing face areas in video images:

[0054] The first is the video recognition mode, which mainly recognizes the key points of the face in the collected video images. This recognition method has a fast recognition speed, but a low recognition accuracy, and may fail to recognize video images with a small proportion of the face area.

[0055] The second is the picture recognition mode, which mainly recognizes facial images in a single picture. This recognition method can identify various characteristics of the face, such as facial features, face shape, age, gender, etc., so the recognition accuracy is higher. However, due to the use of sophisticated recognition algorithms, the recognition speed is slower.

[0056] However, in real-time scenarios such as live video, there are high requirements for the recognition accuracy and speed of facial images, and the two existing recognition modes cannot meet the requirements of real-time scenarios. To this end, the disclosed embodiment provides a video image recognition method, which, in the process of capturing video images, captures a frame of reference image every preset time, and recognizes the frame of reference image based on the image recognition mode, so that when each frame of the captured video image is recognized based on the video recognition mode, for video images in which facial key points cannot be recognized, the video image can be auxiliary recognized based on the image recognition result, thereby improving the recognition accuracy while ensuring the recognition speed.

[0057] Please refer to Figure 1 , which shows the implementation environment involved in the video image recognition method provided by the embodiment of the present disclosure, see Figure 1 , the implementation environment includes: a terminal 101 and a server 102.

[0058] Among them, a video live broadcast application or a photo taking application is installed in the terminal 101, and based on the installed video live broadcast application or photo taking application, a video shooting service can be provided to the user. In order to be able to shoot the user, the terminal 101 is also configured with a camera, which includes at least one of a front camera and a rear camera. The terminal 101 can be a device such as a smart phone, a tablet computer, a digital camera, etc., and the embodiment of the present disclosure does not specifically limit the product type of the terminal 101.

[0059] The server 102 is a background server of a live video application or a camera application. The server 102 can receive video data uploaded by a terminal based on the live video application or the camera application installed by the terminal, and forward the video data uploaded by the terminal to other users. The server 102 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers.

[0060] The terminal 101 and the server 102 may be connected via a wired network or a wireless network.

[0061] It can be understood that the terms "each", "multiple", and "any" used in the embodiments of the present disclosure include two or more, each refers to each of the corresponding multiple, and any refers to any one of the corresponding multiple. For example, the multiple words include 10 words, and each word refers to each word of the 10 words, and any word refers to any one of the 10 words.

[0062] based on Figure 1 The implementation environment shown in the figure, the embodiment of the present disclosure provides a method for recognizing a face image, see Figure 2 , the method flow provided by the embodiment of the present disclosure includes:

[0063] 201. During the video image acquisition process, the terminal identifies key points of the face in each frame of the video image.

[0064] In the video live broadcast or photo taking scenario, the terminal can automatically turn on the camera and call the installed camera to collect the user's video image. When the terminal calls the camera to collect video images, it usually uses a preset frame rate for collection. The preset frame rate can be determined according to the performance of the camera, and the performance of the camera is different, and the frame rate of collecting video images is also different. The preset frame rate can be 30 frames / second, 40 frames / second, etc. The embodiment of the present disclosure does not make a specific limitation on the preset frame rate. The preset frame rate can also reflect the speed of generating video images. The larger the frame rate, the faster the speed of generating video images, and the smaller the frame rate, the slower the speed of generating video images. When the preset frame rate is 30 frames / second, when collecting video images based on the frame rate, the speed of generating video images can be 30 milliseconds / frame; when the preset frame rate is 40 frames / second, when collecting video images based on the frame rate, the speed of generating video images can be 25 milliseconds / frame.

[0065] Among them, the key points of the face are key points that can represent the facial features and the shape of the face, including eyebrow key points, eye key points, nose key points, mouth key points and facial contour key points. The number of key points of the face is determined according to the recognition accuracy requirements of the video image and the computing power of the terminal, and can be 44, 88, 106, etc. If the recognition accuracy requirements for the video image are high and the computing power of the terminal is strong, a relatively large number of key points of the face can be selected; if the recognition accuracy requirements for the video image are low and the computing power of the terminal is weak, a relatively small number of key points of the face can be selected. See Figure 3 , which shows 44 labeled facial key points.

[0066] In live video or photo-taking scenarios, usually only simple processing such as beautification and mapping is performed on the face area in the video image, and there is no need to perform detailed recognition of each video image. Therefore, in live video or photo-taking scenarios, in order to improve the recognition speed of the face area in the video image, during the video image acquisition process, for each frame of the video image acquired, the terminal can identify the facial key points in each frame of the video image based on the video recognition mode.

[0067] Considering that the recognition accuracy of video images is low in video recognition mode, especially when the face area in the video image occupies a small proportion, it is even impossible to recognize the key points of the face in the video image. In order to improve the recognition accuracy of video images in video recognition mode, the method provided in the embodiment of the present disclosure will also collect multiple reference pictures during the video image acquisition process, and recognize the multiple reference pictures, and then construct a reference picture set based on the multiple reference pictures and their face recognition results, so that when recognizing the key points of the face in the video image, the reference picture set can be used for auxiliary recognition.

[0068] The reference picture set includes multiple reference pictures and their face recognition results. The following steps can be used to construct the reference picture set:

[0069] 2011. The terminal collects a frame of reference picture every second preset time period.

[0070] The reference image is a picture used to assist in the recognition of the video image. The second preset duration is longer than the acquisition speed of each frame of the picture, and can be determined according to the recognition accuracy of the video image. The higher the recognition accuracy of the video image, the shorter the preset duration, and the lower the recognition accuracy of the video image, the longer the preset duration. The second preset duration can be 1 second, 2 seconds, etc.

[0071] 2012. The terminal extracts facial features from each reference image.

[0072] Among them, facial features may include gender features, age features, facial features, etc. When extracting facial features from each reference image, the terminal may input each reference image into a facial feature extraction model, and extract facial features from each reference image based on the facial feature extraction model. The facial feature extraction model includes a gender feature extraction model, an age feature extraction model, and a facial feature extraction model. For example, the terminal may input each reference image into a gender feature extraction model, and output the gender features of each reference image; for another example, the terminal may input each reference image into an age feature extraction model, and output the age features of each reference image; for another example, the terminal may input each reference image into a facial feature extraction model, and output the facial features of each reference image.

[0073] For the gender feature extraction model, the age feature extraction model and the facial features extraction model, the initial gender feature extraction model, the initial age feature extraction model and the initial facial features extraction model can be trained based on the reference face image with the gender features, age features and the key points of the facial features marked. Taking the gender feature extraction model as an example, the training process of the gender feature extraction model can include: obtaining a preset number of facial images of different genders, marking the gender features of each facial image, extracting the Gabor features of each facial image marked with the gender features, and performing dimensionality reduction processing on the Gabor features of the preset number of facial images marked with the gender features, and training the initial gender feature extraction model based on the Gabor features after the dimensionality reduction processing to obtain the gender feature extraction model.

[0074] 2013. Based on the facial features of each reference image, the terminal identifies each reference image and obtains the facial recognition result of each reference image.

[0075] Based on the facial features of each reference image, the terminal inputs the facial features of each reference image into the facial recognition model and outputs the facial recognition results of each facial image. Among them, the facial recognition model includes an age recognition model, a gender recognition model, and a facial features recognition model, etc. Correspondingly, the facial recognition results include an age recognition result, a gender recognition result, and a facial features recognition result. The terminal outputs the age recognition result corresponding to each reference image by inputting the age features of each reference image into the age recognition model; the terminal outputs the gender recognition result corresponding to each reference image by inputting each reference image into the gender recognition model; the terminal outputs the facial features recognition result corresponding to each reference image by inputting each reference image into the facial features recognition model. For the age recognition model, the gender recognition model, and the facial features recognition model, the initial age recognition model, the initial gender recognition model, and the initial facial features recognition model can be trained based on the reference facial image with the age recognition result, the gender recognition result, and the facial features recognition result labeled.

[0076] 2014. Based on each reference image and its face recognition result, the terminal constructs a reference image set.

[0077] Based on each reference picture and the corresponding face recognition result, the terminal can construct a reference picture set by storing each reference picture and the corresponding face recognition result.

[0078] 202. For any frame of video image, if no facial key points in the video image are identified, the terminal obtains a face recognition result of a specified image.

[0079] For any frame of video image collected, when the terminal recognizes the frame of video image based on the video recognition mode, there are two recognition results, one is that the terminal recognizes the key points of the face in the frame of video image, and the other is that the terminal does not recognize the key points of the face in the frame of video image. If the terminal recognizes the key points of the face in the frame of video image, the following step 204 can be executed to determine the face area of ​​the frame of video image according to the recognized key points of the face; if the terminal does not recognize the key points of the face in the frame of video image, the terminal can obtain the face recognition result of the specified reference image, and further recognize the key points of the face in the frame of video image based on the face recognition result of the specified reference image.

[0080] The designated picture is a reference picture with the shortest time interval between its acquisition time and the acquisition time of the video image. When the terminal obtains the face recognition result of the designated reference picture, the following steps can be adopted:

[0081] 2021. The acquisition time of the video image obtained by the terminal.

[0082] Usually, when the terminal calls the camera to collect video images, it will record the collection time of each frame of video images. Based on the recorded collection time of each frame of video images, for any frame of video images in which no facial key points are recognized, the terminal can obtain the collection time of the frame of video images.

[0083] 2022. The terminal obtains, from a reference picture set according to the acquisition time of the video image, a reference picture whose acquisition time is within a first preset time range from the acquisition time of the video image, and uses the obtained reference picture as the designated picture.

[0084] The terminal compares the acquisition time of the frame of video image with the acquisition time of each reference image in the reference image set, and through the comparison, a reference image whose time interval between the acquisition time and the acquisition time of the video image is within a first preset time range can be obtained, and then the obtained reference image is used as the designated image. The first preset time can be 10 milliseconds, 20 milliseconds, etc.

[0085] 2023. The terminal obtains the face recognition result of the specified image from the reference image set.

[0086] Since the reference picture set stores multiple reference pictures and their face recognition results, when the reference picture with the shortest time interval between the acquisition time and the acquisition time of the video image is obtained, the terminal can obtain the face recognition result of the specified picture from the reference picture set.

[0087] 203. The terminal identifies key points of the face in the video image according to the face recognition result of the specified image.

[0088] When the terminal identifies the key points of the face in the video image based on the face recognition result of the specified picture, the following steps can be adopted:

[0089] 2031. The terminal crops the video image according to the face recognition result of the specified image.

[0090] Usually, in scenarios with high real-time requirements such as live video, the acquisition time of each frame of video image is generally tens of milliseconds. In order to ensure the recognition accuracy of the video image, the acquisition time interval of the reference image is very short. In a short time interval, it is generally difficult for the user to make large movements. Therefore, based on the face recognition result of the designated image with a short time interval between the acquisition time and the acquisition time of the frame of video image, the face area in the video image can be roughly determined, and then based on the roughly determined face area, the approximate position of the face area can be determined. The terminal can crop the video image according to the approximate position of the face area, crop the non-face area at the approximately same position of the video image, and retain the face area, so that the proportion of the face area in the video image is greatly increased, and the video image is changed from a "small face image" to a "large face image", which is convenient for identifying the key points of the face in the video image. For example, based on the face recognition result of the designated image, it is determined that the face area of ​​the designated image is 180 pixels * 180 pixels, then the video image can be cropped to 200 pixels * 200 pixels.

[0091] 2032. The terminal recognizes key points of the face in the cropped video image.

[0092] In the video recognition mode, the terminal recognizes the key points of the face in the cropped video image. If the key points of the face in the cropped video image are recognized, step 204 is executed to further determine the face area of ​​the video image; if the key points of the face in the cropped video image are not recognized, it is determined that the recognition of the video image has failed.

[0093] In another embodiment of the present disclosure, if the face area of ​​the specified picture cannot be identified based on the face recognition result of the specified picture, it is determined that the face key points of the video image cannot be identified and the recognition of the video image fails.

[0094] 204. The terminal determines a face region of the video image based on the identified face key points.

[0095] Based on the identified facial key points, the terminal can determine the facial area of ​​the frame video image by connecting the facial key points. The facial area includes the user's facial features, facial shape, etc.

[0096] Furthermore, in order to attract more users to watch during live video broadcasts, users usually beautify their facial features and add stickers to the face area based on special effect options such as beautification and stickers to beautify the user's appearance. In specific implementation, the terminal can detect the selected special effect option, and then add corresponding special effects to the face area based on the selected special effect option. For example, if it is detected that the sticker option is selected, the terminal will add stickers to the face area.

[0097] For the above video image recognition process, the following will be Figure 4 Take this as an example to illustrate.

[0098] See also Figure 4 After the live video broadcast starts, the terminal turns on the camera and collects video images based on the camera. For any frame of video image A collected, the terminal recognizes the key points of the face in the frame of video image A based on the video recognition mode. At the same time, during the video image collection process, the terminal collects a frame of picture C every 1 second, and recognizes the collected picture C based on the picture recognition mode to obtain the recognition result. For video image A, if the facial key points in the frame of video image are identified based on the video recognition mode, the beauty props are used to beautify the face area determined by the facial key points, and the beautified video image is pushed to other users; if the facial key points in the frame of video image cannot be identified based on the video recognition mode, the face recognition result of the most recently collected designated image is obtained, and then based on the face recognition result of the designated image, if the face area cannot be roughly determined, it is determined that the recognition of video image A has failed; if the face area can be roughly determined, the video image A is cropped to obtain video image B, and then, based on the video recognition mode, the facial key points in video image B are identified; if the facial key points in video image B are identified, the beauty props are used to beautify the face area determined by the facial key points, and the beautified video image is pushed to other users.

[0099] The method provided by the embodiments of the present disclosure first recognizes the face area in the video image based on the video recognition mode. For the video image whose face key points cannot be recognized by the video recognition mode, auxiliary recognition is performed in combination with a reference picture. Since the reference picture has a more accurate face recognition result, auxiliary recognition based on the reference picture can accurately recognize the face area.

[0100] See also Figure 5 , the present disclosure provides a video image recognition device, the device comprising:

[0101] The recognition module 501 is used to recognize the key points of the human face in each frame of the video image during the video image acquisition process;

[0102] The acquisition module 502 is used for acquiring the face recognition result of a specified picture for any frame of video image if the face key points in the video image are not recognized, where the specified picture is a reference picture whose time interval between the acquisition time and the acquisition time of the video image is within a first preset time range, and the reference picture is a picture used for assisting the recognition of the video image;

[0103] The recognition module 501 is also used to recognize key points of the face in the video image according to the face recognition result of the specified picture;

[0104] The determination module 503 is used to determine the face area of ​​the video image based on the identified face key points.

[0105] In another embodiment of the present disclosure, an acquisition module 502 is used to obtain the acquisition time of a video image; based on the acquisition time of the video image, a reference image with the shortest time interval between the acquisition time and the acquisition time of the video image is obtained from a reference image set, and the obtained reference image is used as a designated image, the reference image set includes multiple reference images and their face recognition results; the face recognition result of the designated image is obtained from the reference image set.

[0106] In another embodiment of the present disclosure, the device further comprises:

[0107] An acquisition module 502 is used to acquire a reference picture every second preset time period during the video image acquisition process;

[0108] An extraction module, used to extract facial features of each reference image;

[0109] A recognition module is used to recognize each reference image based on the facial features of each reference image, and obtain a facial recognition result of each reference image;

[0110] The construction module is used to construct a reference picture set based on each reference picture and its face recognition result.

[0111] In another embodiment of the present disclosure, the recognition module 501 is further used to crop the video image according to the face recognition result of the specified image; and recognize the key points of the face in the cropped video image.

[0112] In another embodiment of the present disclosure, the device further comprises:

[0113] A detection module, used to detect the selected special effect option;

[0114] Add a module for adding corresponding special effects in the face area based on the selected special effect option.

[0115] To sum up, the device provided by the embodiments of the present disclosure, when recognizing the face area in the video image, first performs recognition based on the video recognition mode. For the video image whose video recognition mode cannot recognize the key points of the face, auxiliary recognition is performed in combination with the reference picture. Since the reference picture has a more accurate face recognition result, auxiliary recognition based on the reference picture can accurately identify the face area.

[0116] Figure 6 The structure block diagram of a terminal 600 provided by an exemplary embodiment of the present disclosure is shown. The terminal 600 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 600 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0117] Typically, the terminal 600 includes a processor 601 and a memory 602 .

[0118] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0119] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the face image recognition method provided in the method embodiment of the present application.

[0120] In some embodiments, the terminal 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608 and a power supply 609.

[0121] The peripheral device interface 603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0122] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0123] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on the surface or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 605 can be one, and the front panel of the terminal 600 is set; in other embodiments, the display screen 605 can be at least two, which are respectively set on different surfaces of the terminal 600 or are folded; in some other embodiments, the display screen 605 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 600. Even, the display screen 605 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 605 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0124] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0125] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 601 for processing, or input them into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0126] Positioning component 608 is used to locate the current geographical location of terminal 600 to implement navigation or LBS (Location Based Service). Positioning component 608 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0127] The power supply 609 is used to power various components in the terminal 600. The power supply 609 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 609 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0128] In some embodiments, the terminal 600 further includes one or more sensors 610 , including but not limited to: an acceleration sensor 611 , a gyroscope sensor 612 , a pressure sensor 613 , a fingerprint sensor 614 , an optical sensor 615 , and a proximity sensor 616 .

[0129] The acceleration sensor 611 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 600. For example, the acceleration sensor 611 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 601 can control the display screen 605 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 611. The acceleration sensor 611 can also be used for collecting game or user motion data.

[0130] The gyro sensor 612 can detect the body direction and rotation angle of the terminal 600, and the gyro sensor 612 can cooperate with the acceleration sensor 611 to collect the user's 3D actions on the terminal 600. The processor 601 can implement the following functions based on the data collected by the gyro sensor 612: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0131] The pressure sensor 613 can be set on the side frame of the terminal 600 and / or the lower layer of the display screen 605. When the pressure sensor 613 is set on the side frame of the terminal 600, it can detect the user's holding signal of the terminal 600, and the processor 601 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 613. When the pressure sensor 613 is set on the lower layer of the display screen 605, the processor 601 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 605. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0132] The fingerprint sensor 614 is used to collect the user's fingerprint, and the processor 601 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 601 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 614 can be set on the front, back, or side of the terminal 600. When a physical button or a manufacturer logo is set on the terminal 600, the fingerprint sensor 614 can be integrated with the physical button or the manufacturer logo.

[0133] The optical sensor 615 is used to collect the ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 according to the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is reduced. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 according to the ambient light intensity collected by the optical sensor 615.

[0134] The proximity sensor 616, also called a distance sensor, is usually arranged on the front panel of the terminal 600. The proximity sensor 616 is used to collect the distance between the user and the front of the terminal 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from the screen-on state to the screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the terminal 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from the screen-off state to the screen-on state.

[0135] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the terminal 600, and the terminal 600 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0136] The terminal provided by the embodiment of the present disclosure first recognizes the face area in the video image based on the video recognition mode. For the video image whose video recognition mode cannot recognize the key points of the face, auxiliary recognition is performed in combination with the reference picture. Since the reference picture has a more accurate face recognition result, the auxiliary recognition based on the reference picture can accurately identify the face area.

[0137] The present disclosure provides a computer-readable storage medium, wherein the storage medium stores at least one program code, and the at least one program code is loaded and executed by a processor to implement Figure 2 The computer readable storage medium may be non-transitory. For example, the computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0138] The computer-readable storage medium provided by the embodiments of the present disclosure first performs recognition based on the video recognition mode when recognizing the face area in the video image. For the video image in which the video recognition mode cannot recognize the key points of the face, auxiliary recognition is performed in combination with a reference picture. Since the reference picture has a more accurate face recognition result, auxiliary recognition based on the reference picture can accurately identify the face area.

[0139] The above description is only an optional embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video image recognition method, characterized in that: The method comprises: During the video image acquisition process, the facial key points in each frame of the video image are identified based on the video recognition mode; For any frame of video image, if the key points of the face in the video image are not identified, the face recognition result of a specified picture is obtained from a reference picture set, the specified picture is a reference picture whose time interval between the acquisition time and the acquisition time of the video image is within a first preset time range, and the reference picture is a picture used to assist in the recognition of the video image; According to the face recognition result of the specified picture, identifying the key points of the face in the video image; Determining a face region of the video image based on the identified face key points; The process of obtaining the reference picture set includes: During the video image acquisition process, a reference image is acquired every second preset time period; Extract facial features from each reference image; Based on the facial features of each reference image, each reference image is identified to obtain a facial recognition result of each reference image; Based on each reference picture and its face recognition result, the reference picture set is constructed.

2. The method according to claim 1, characterized in that The step of obtaining the face recognition result of a specified picture from a reference picture set includes: Obtaining the acquisition time of the video image; According to the acquisition time of the video image, a reference image with the shortest time interval between the acquisition time and the acquisition time of the video image is obtained from a reference image set, and the obtained reference image is used as the designated image, wherein the reference image set includes multiple reference images and face recognition results thereof; Obtain a face recognition result of the specified picture from the reference picture set.

3. The method according to claim 1, characterized in that: The step of identifying key points of a face in the video image according to the face recognition result of the designated image includes: Cropping the video image according to the face recognition result of the specified image; Identify facial landmarks in cropped video images.

4. The method according to any one of claims 1 to 3, characterized in that After determining the face area of ​​the video image based on the identified face key points, the method further includes: Detect the selected special effect option; Based on the selected special effect option, a corresponding special effect is added to the face area.

5. A video image recognition device, characterized in that: The device comprises: A recognition module is used to recognize key points of human faces in each frame of video images based on a video recognition mode during the video image acquisition process; an acquisition module, configured to, for any frame of video image, if no facial key points in the video image are identified, acquire a face recognition result of a specified picture from a reference picture set, wherein the specified picture is a reference picture whose time interval between the acquisition time and the acquisition time of the video image is within a first preset time range, and the reference picture is a picture used to assist in the recognition of the video image; The recognition module is further used to recognize key points of the face in the video image according to the face recognition result of the specified image; A determination module, used to determine the face area of ​​the video image based on the identified face key points; The acquisition module is further used to acquire a reference picture every second preset time period during the video image acquisition process; An extraction module, used to extract facial features of each reference image; The recognition module is further used to recognize each reference picture based on the facial features of each reference picture to obtain a facial recognition result of each reference picture; The construction module is used to construct the reference picture set based on each reference picture and its face recognition result.

6. The device according to claim 5, characterized in that The acquisition module is used to acquire the acquisition time of the video image; according to the acquisition time of the video image, acquire a reference image with the shortest time interval between the acquisition time and the acquisition time of the video image from a reference image set, and use the acquired reference image as the designated image, wherein the reference image set includes multiple reference images and their face recognition results; Obtain a face recognition result of the specified picture from the reference picture set.

7. The device according to claim 5, characterized in that The recognition module is further used to crop the video image according to the face recognition result of the designated image; and to recognize key points of the face in the cropped video image.

8. The device according to any one of claims 5 to 7, characterized in that The device also includes: A detection module, used to detect the selected special effect option; The adding module is used to add corresponding special effects in the face area based on the selected special effects option.

9. A terminal, characterized in that: The terminal includes a processor and a memory, wherein at least one program code is stored in the memory, and the at least one program code is loaded and executed by the processor to implement the video image recognition method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: At least one program code is stored in the storage medium, and the at least one program code is loaded and executed by the processor to implement the video image recognition method according to any one of claims 1 to 4.

Images