A video identification method and system, an electronic device, and a storage medium

Abstract

Embodiments of the present application provide a video recognition method and device, electronic equipment and storage medium. The video signal collected is separated into audio and video signals, and the audio signal and the video signal are obtained. The video signal includes a to-be-verified object. The audio signal and the video signal are subjected to feature extraction, and the audio feature corresponding to the audio signal and the video feature corresponding to the video signal are obtained. The audio feature and the video feature are input into a pre-trained neural network, and a recognition result output by the neural network is obtained. The recognition result is used to represent whether the video signal is completed by the to-be-verified object under the guidance of an intermediary. The technical solution can determine whether the face recognition object is completed under the guidance of an intermediary.

Description

Technical Field

[0001] This application relates to the field of facial recognition, specifically, to the design of a video recognition method, system, electronic device, and storage medium. Background Technology

[0002] In the financial industry, the verification of personal information is frequently involved, hence the widespread application of facial recognition technology. Current technologies focus on whether the facial image is of a real person and whether it represents the same individual.

[0003] Current facial recognition technology does not consider whether the person being recognized is doing so voluntarily in a natural state. Natural facial recognition refers to recognition that is not performed under the inducement of interested parties. Existing technology makes it relatively simple to determine the authenticity of facial recognition results. Summary of the Invention

[0004] The purpose of this application is to provide a video recognition method to solve the technical problem in the prior art that face recognition cannot determine whether the person being identified is acting voluntarily or under the inducement of an intermediary.

[0005] To achieve the above objectives, the technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, embodiments of this application provide a video recognition method, which involves separating the acquired audio and video signals to obtain audio and video signals respectively; wherein the video signal includes an object to be verified; performing feature extraction on the audio and video signals to obtain audio features corresponding to the audio signal and video features corresponding to the video signal; inputting the audio features and video features into a pre-trained neural network to obtain a recognition result output by the neural network; wherein the recognition result is used to characterize whether the video signal represents the object to be verified completing the process under the guidance of an intermediary. This technical solution can determine whether the object undergoing face recognition has completed the process under the guidance of an intermediary.

[0007] In this embodiment, the acquired audio and video signals are first separated to obtain individual audio and video signals. The video signal mainly consists of the video recorded during face recognition, including the video of the face to be verified. Subsequently, feature extraction is performed on the audio and video signals. Specifically, audio features are extracted from the audio signal, and video features are extracted from the video signal. Then, a pre-trained neural network is used to input the audio and video features for recognition, ultimately determining whether the object to be verified in the video signal was completed under intermediary guidance.

[0008] Furthermore, the audio features include the number of speakers in the audio signal and / or whether it contains a first keyword; the video features include at least one of the following: shooting angle features, facial features, and shooting scene features.

[0009] In this embodiment, audio and video features are described. Audio features include determining the number of speakers to determine if multiple people are speaking, and whether the content of the speech contains a first keyword. This technical solution can extract both audio and video features.

[0010] Furthermore, the feature extraction of the audio signal includes: truncating the audio signal to obtain multiple audio segments; classifying the multiple audio segments and determining the number of speakers in the audio signal based on the classification results; and / or recognizing the speech in the audio signal and converting it into text, and performing keyword recognition on the text.

[0011] In this embodiment, the audio signal is first segmented into multiple audio segments. These segments are then classified, and the results are used to determine the number of speakers in the audio signal, thus identifying whether multiple people are speaking. Furthermore, the feature extraction step involves speech recognition, converting the speech into text, and then performing keyword recognition on the text. This technical solution enables the determination of the number of speakers and the presence of keywords in the audio signal during extraction.

[0012] Furthermore, the object to be verified is a face; the feature extraction of the video signal includes: dividing the video signal into frames to obtain multiple frame image signals; and extracting features from the image signals to obtain the shooting angle features, facial features, and shooting scene features.

[0013] In this embodiment, video features are extracted, with the specific object of detection being the human face. First, the video signal containing facial information is processed into frames to obtain multiple image signals. Then, feature extraction is performed on these image signals to obtain information features related to the shooting angle, facial features, and shooting scene. This technical solution can preprocess video signals and obtain multiple image signals.

[0014] Further, the step of extracting features from the image signal to obtain the shooting angle features includes: extracting facial angle features from the image signal to obtain the tilt angle, yaw angle, and roll angle of the face in the image signal; determining whether the face is facing the screen based on the tilt angle, the yaw angle, and the roll angle; and / or, identifying the shooting angle from the image signal to determine whether the shooting angle is a selfie angle.

[0015] In this embodiment, the step of extracting shooting angle features first requires extracting the angle features of the face in the image signal. Specifically, this involves first obtaining the face's tilt angle, yaw angle, and roll angle, and then determining whether the face is facing the screen directly based on these three angles. Furthermore, obtaining shooting angle features also includes identifying the shooting angle of the image signal to determine whether the shooting angle is a selfie or a shot taken by someone else. This technical solution can determine whether a person is facing the screen directly and whether the shooting angle is a selfie or a shot taken by someone else.

[0016] Further, the step of extracting features from the image signal to obtain the facial features includes: acquiring the eye region and mouth region in the image signal; extracting features from the eye region to determine whether the eye attention is focused on the screen; and / or extracting features from the mouth region to determine whether the mouth is moving; and / or using a micro-expression recognition model to recognize the facial image in the image signal to determine whether there are any abnormalities in the micro-expressions of the facial image.

[0017] In this embodiment, obtaining facial features first requires identifying the eye and mouth regions of the person in the image signal. Then, feature extraction is performed on the eye region to determine whether the person's eye attention is focused on the screen. Furthermore, feature extraction is performed on the mouth region to determine whether the person's mouth is moving in the video signal, thereby determining whether the person is speaking. Additionally, a pre-trained micro-expression recognition model is used to identify the facial image and determine if there are any abnormalities in the micro-expressions. This technical solution can determine whether a person's eye attention is focused on face recognition, whether the person is speaking during face recognition, and the micro-expressions during face recognition.

[0018] Furthermore, the step of extracting features from the image signal to obtain shooting scene features includes: acquiring the portrait background image of the image signal; identifying the portrait background image to determine whether the portrait background image is indoors; and / or identifying keywords in the portrait background image to determine whether keywords appear in the background.

[0019] In this embodiment, scene features are extracted. First, the background image of the person in the image signal is acquired. Then, the background image is identified to determine whether the scene is indoors or outdoors. Furthermore, keyword recognition can be performed on the background image to determine if a second keyword appears. This technical solution can determine whether the shooting scene is indoors or outdoors and whether there are keywords in the background.

[0020] Furthermore, the step of inputting the audio features and the video features into a pre-trained neural network to obtain the recognition result output by the neural network includes: generating corresponding labels for the audio features and the video features respectively; generating feature vectors based on the labels; inputting the feature vectors into the pre-trained neural network to obtain the recognition result output by the neural network.

[0021] In this embodiment, since audio and video features have already been acquired in the preceding steps, corresponding labels are generated for the audio and video features, and feature vectors are generated using these labels. These feature vectors are then input into a pre-trained neural network to obtain the recognition result output by the neural network. This technical solution enables comprehensive analysis using extracted audio and video features, leveraging the spatial relationships between features to improve the accuracy of determining whether there is intermediary involvement, thus more accurately identifying whether intermediary involvement exists in face recognition.

[0022] Secondly, embodiments of this application provide a video recognition system, comprising: a signal separation module, used to perform audio-video separation on the acquired audio and video signals to obtain audio signals and video signals respectively; wherein the video signal includes an object to be verified; a feature extraction module, used to extract features from the audio signal and the video signal to obtain audio features corresponding to the audio signal and video features corresponding to the video signal; and a judgment module, used to input the audio features and the video features into a pre-trained neural network to obtain a recognition result output by the neural network; wherein the recognition result is used to characterize whether the live video signal is the object to be verified under the guidance of an intermediary.

[0023] In this embodiment, the acquired audio and video signals are first separated to obtain separate audio signals and video signals excluding audio signals. The video signals mainly consist of the video recorded during face recognition, including the video of the face to be verified. Subsequently, feature extraction is performed on the audio and video signals. Specifically, audio features are extracted from the audio signals, and video features are extracted from the video signals. Then, a pre-trained neural network is used to input the audio and video features for recognition, ultimately determining whether the object to be verified in the video signal was completed under intermediary guidance.

[0024] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus; the processor and the memory communicate with each other via the bus; the memory stores program instructions executable by the processor, and the processor can execute the method as described in the first aspect by calling the program instructions.

[0025] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that cause the computer to perform the method as described in the first aspect.

[0026] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, embodiments of this application are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A schematic diagram illustrating the steps of the video recognition method provided in this application embodiment;

[0029] Figure 2 A schematic diagram of video feature extraction provided in an embodiment of this application;

[0030] Figure 3 A schematic diagram illustrating the steps for obtaining the recognition result of the neural network output provided in an embodiment of this application;

[0031] Figure 4 The extraction rules and tags for audio and video features provided in the embodiments of this application;

[0032] Figure 5This is a schematic diagram of the deep learning model structure provided in the embodiments of this application;

[0033] Figure 6 A schematic diagram of a video recognition system provided for embodiments of this application; and

[0034] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0036] Figure 1 This is a schematic diagram illustrating the steps of a video recognition method provided in an embodiment of this application.

[0037] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the steps of the video recognition method provided in the embodiments of this application. Figure 1 The specific execution steps of this method include:

[0038] Step 101: Perform audio-video separation on the acquired audio and video signals to obtain audio signals and video signals respectively; wherein the video signal includes the object to be verified.

[0039] In the specific implementation of step 101, the video recognition system first performs audio-video separation on the acquired audio and video signals. Audio-video separation mainly refers to separating the video and audio components of the original audio and video signals, thereby analyzing the audio and video signals separately. One scenario for audio-video acquisition is that during the execution of video recognition, the face recognition participant uses a mobile phone or other terminal device to record audio and video. The recorded audio and video are then uploaded to the video recognition system for processing. However, it should be noted that the face recognition participant may actively use the mobile phone or other terminal device to record audio and video, or they may pick up the phone or other terminal device themselves but not voluntarily, but under the guidance of an intermediary. Another scenario for audio-video acquisition is that the mobile phone or other terminal device is held by an intermediary or installed in a fixed position, taking advantage of the face recognition participant's inattention to record audio and video, and then uploading the recorded audio and video to the video recognition system for processing. The object to be verified is the face video information contained in the audio and video recorded by the face recognition participant during face recognition.

[0040] Step 102: Perform feature extraction on the audio signal and the video signal to obtain the audio features corresponding to the audio signal and the video features corresponding to the video signal.

[0041] In the specific implementation of step 102, the video recognition system performs feature extraction on the video signal obtained in step 101, extracting corresponding audio features from the audio signal and corresponding video features from the video signal.

[0042] Step 103: Input the audio features and the video features into a pre-trained neural network to obtain the recognition result output by the neural network.

[0043] In the specific implementation of step 103, since the relevant neural network has been pre-trained, in this step, the video recognition system can input the extracted audio features and video features into the relevant neural network. The neural network performs comprehensive analysis on the audio features and video features to obtain a comprehensive recognition result, thereby determining whether the relevant video recorded by the face recognition participant during face recognition was completed under the guidance of an intermediary, rather than on their own voluntary basis.

[0044] Furthermore, audio features include the number of speakers in the audio signal and / or whether it contains a primary keyword. Since intermediaries may be involved, instructing the facial recognition operator, multiple people may be speaking. Determining the number of speakers through audio features can help identify whether multiple people are speaking. Additionally, when an intermediary is involved, their instructions to the facial recognition operator may involve keywords such as blinking, speaking, or looking at the screen. Extracting these keywords during audio feature extraction can provide a preliminary assessment of whether intermediary involvement has occurred. Video features include shooting angle features, facial features, and shooting scene features; each feature can provide a basic preliminary assessment of whether intermediary involvement has occurred.

[0045] In a preferred embodiment, the identified features may include whether the audio or video recording is performed by a real person.

[0046] Furthermore, in the process of extracting audio signals, the video recognition system first segments the audio signal, that is, divides the speech data into segments. Segmentation requires determining the time interval for each segment. In a preferred embodiment, the speech data is segmented at 1-second intervals. After segmentation, multiple audio segments are obtained, each segment lasting 1 second. Specifically, the ffmpeg command can be used for segmentation. ffmpeg is an open-source computer program that can record, convert, and stream digital audio and video. For each segmented audio segment, an MFCC (Mel-scale Frequency Cepstral Coefficients) feature map is extracted using a trained neural network. Then, similarity calculations are performed on each audio feature, and the results are compared. Specifically, the audio features are arranged, and the similarity of later segments is compared with that of earlier segments. Segments with high similarity are grouped together. Audio segments belonging to the same category can be considered as belonging to the same speaker. The number of categories indicates the number of speakers, thus allowing for the determination of whether multiple speakers are involved. Then, different audio segments of the same category are spliced ​​together to form a new long audio file. Then, the speech-to-text function is used to perform speech recognition on the long audio file to determine whether the speech contains keywords.

[0047] In a preferred embodiment, the keywords in the speech include speech spoofing recognition for moving subjects, such as "nod", "blink", "look to the left", etc.; speech spoofing recognition for digital subjects, such as "read one, two, three, four" and "read five, six, seven, eight", etc.; and speech spoofing recognition for light subjects, such as "look at the screen and don't move" and "look at the screen quietly".

[0048] Furthermore, the video recognition system preprocesses the video signal. First, the video signal is divided into frames to obtain multi-frame image signals. Then, corresponding feature extraction is performed on the multi-frame image signals. Feature extraction is mainly processed from three aspects: shooting angle features, facial features, and shooting scene features.

[0049] Furthermore, the video recognition system acquires shooting angle features, starting with extracting the angle features of the face. The face angle primarily refers to determining whether the person is facing the screen directly during filming, preventing unauthorized filming by intermediaries and providing a preliminary assessment of potential intermediary involvement. In this process, after acquiring the image, the face image is extracted and input into a pre-trained face recognition model for angle identification. The angles are the estimated angles of the face: the tilt angle, the yaw angle, and the roll angle. If the tilt angle is greater than a first threshold, the yaw angle is greater than a second threshold, or the roll angle is greater than a third threshold, the face is determined not to be facing the screen directly. The first, second, and third thresholds can be the same or different. The condition for determining whether the face is facing the screen can also be: if the tilt angle is greater than the first threshold, the yaw angle is greater than the second threshold, and the roll angle is greater than the third threshold, the face is determined not to be facing the screen directly; or if any two of the tilt angle, yaw angle, and roll angle are greater than their corresponding thresholds, the face is determined not to be facing the screen directly.

[0050] In addition, it can also identify whether the user is taking a selfie or being photographed by someone else during facial recognition. When intermediaries induce users to undergo facial recognition, there is a high probability that they are taking photos of the user with a handheld camera. Therefore, the identification of selfies and photos taken by others can also make a preliminary judgment on whether there is intermediary involvement.

[0051] In a preferred embodiment, to determine whether a user is taking a selfie, the facial image can be input into a pre-set selfie recognition model for evaluation. The selfie recognition model is generated using deep learning training, which is performed in advance using multiple selfie and other photos to train the model, thereby enabling it to distinguish between selfies and other photos.

[0052] Furthermore, in facial feature extraction, it is first necessary to obtain the eye and mouth regions from the image signal. Generally, a partial image of the face is taken, with the upper half of the image identified as the eye region and the lower half as the mouth region. During the analysis of the eye region, a pre-trained eye recognition model is used. The image of the eye region is input into the eye recognition model to determine whether the eyes are focused on the screen.

[0053] During the analysis of the mouth region, it is determined whether the image of a person's mouth region includes both an open mouth and a closed mouth. In a preferred embodiment, a pre-trained mouth recognition model is used to input multiple frames of mouth region images into the model for recognition. If a portion of the multiple frames of mouth region images contains an open mouth and another portion contains a closed mouth, it can be determined that the user is speaking when recording the video.

[0054] In analyzing human micro-expressions, video recognition systems utilize pre-trained micro-expression recognition models. Facial images are input into the model to determine if the user's expression is abnormal. The generation of the micro-expression recognition model employs deep learning training, pre-training with multiple facial images to identify the emotional information represented by micro-expressions.

[0055] Furthermore, during the scene feature recognition process, the background image of the acquired image signal is first extracted. Then, the background image is analyzed, and a pre-trained background recognition model is used to determine whether the background represents an environment that may contain intermediaries.

[0056] In a preferred embodiment, it is determined whether there is an intermediary in the background. In background recognition, it is determined whether the background is indoors, whether there are white walls, and whether it is in a sealed environment.

[0057] In the process of keyword recognition of a portrait background image, the background image is first extracted from the acquired image signal. Then, a pre-trained OCR (Optical Character Recognition) model is used to recognize the fonts appearing in the background image. The recognition result is then input into a pre-trained background recognition model to determine whether keywords appear in the background image. The keywords in the background image include words such as "intermediary," "agency center," or "loan."

[0058] The following describes, with reference to the accompanying drawings, a preferred embodiment for feature extraction from video and fraud detection.

[0059] Please see Figure 2 , Figure 2 A schematic diagram of video feature extraction provided in an embodiment of this application;

[0060] like Figure 2 As shown, after identifying fraud based on audio data, the system then performs intermediary fraud identification based on video, that is, using user video data obtained from the video recognition system to identify whether a user is committing fraud. Specifically, this involves analyzing the acquired video data, extracting frames from the video, and performing various aspects of identification on each frame to determine if fraud exists. The identification process can be divided into face detection and OCR recognition. Face detection requires extracting features such as the shooting angle, facial features, and shooting scene features of the face in the image; face detection includes identification of proxy photography, shooting scene recognition, eye attention recognition, micro-expression recognition, mouth state recognition, and facing-to-screen recognition. OCR recognition requires extracting text information from the feature images.

[0061] Background recognition is crucial because the backgrounds of normal individuals undergoing liveness detection are typically complex. However, the backgrounds of intermediary venues are usually indoors and in a sealed environment, such as a white wall. Therefore, it's necessary to recognize the liveness background of the person's image to determine if fraud exists. Specific steps include: first, parsing and segmenting the acquired video to generate images; then, performing face detection on each frame; for images containing faces, filling in the detected face bounding boxes; and finally, inputting the filled image into a pre-trained background recognition model to perform background recognition and determine if the image contains intermediary fraud.

[0062] Proxy Photo Detection: Real users typically perform liveness detection themselves by taking a selfie; they don't have others do it on their behalf. To prevent intermediaries from intermediaries in the liveness detection process—for example, a real person standing in front of the camera while an intermediary holds the user's phone and guides them—this constitutes proxy operation. To address this, the system can identify whether the image of the person undergoing liveness detection is a selfie, thus determining if fraud is involved.

[0063] The specific steps include, for example, firstly, parsing and segmenting the acquired video to generate images; then, performing face detection on each frame; for images containing faces, inputting the image into a pre-trained selfie recognition model to perform selfie recognition and determine whether the image contains fraudulent intermediary behavior. The selfie recognition model is generated using deep learning training, pre-trained with multiple selfie and other-taken images to distinguish between selfies and other-taken images.

[0064] Scene recognition: When users perform liveness authentication, the shooting scene is usually indoors, rarely outdoors or inside a car. If it's inside a car, the possibility of it being a middleman is higher. To address this, the shooting scene of the image can be identified to determine if the user's liveness authentication involves middleman fraud. Specific steps include, for example, parsing and segmenting the acquired video into frames to generate images, performing face detection on each frame, and for images containing faces, inputting the image into a pre-trained scene recognition model to perform scene recognition and determine if middleman fraud exists.

[0065] Eye Attention Recognition: This involves recognizing the user's eye state based on their facial data. In some fraud scenarios, the person performing the liveness detection might not be a real person, but rather an intermediary who collects and captures previous videos of the user without their knowledge. This video data is then used for liveness detection to commit fraud. This type of operation usually occurs without the user's knowledge, as their eyes will not be looking at the camera. Otherwise, the user would easily notice that someone is illegally filming them. To detect this type of covert fraud, it's possible to determine whether the user's eye attention is on the screen, thus confirming the existence of fraud.

[0066] Specific steps include, for example, parsing and segmenting the acquired video into frames to generate images; performing face detection on each frame; for images containing faces, performing image matting to extract the image containing the face; obtaining the eye and mouth regions from the image; extracting the face portion image, identifying the upper half as the eye region and the lower half as the mouth region; and during the analysis of the eye region, using a pre-trained eye recognition model, inputting the eye region image into the eye recognition model to determine whether the eyes are focused on the screen.

[0067] Micro-expression recognition: Under normal circumstances, users' expressions are natural and relaxed when performing liveness detection. If the user's expression in front of the camera shows fear, grievance, etc., it is more likely that the user is being coerced by an intermediary to complete the process. Therefore, micro-expressions can be used to determine whether a user is committing fraud.

[0068] Specific steps include: parsing and segmenting the acquired video into frames to generate images; performing face detection on each frame; and inputting the faces into a pre-trained micro-expression recognition model within the video recognition system to determine if the user's expression is abnormal, thereby identifying potential fraud. The micro-expression recognition model utilizes deep learning training, pre-trained with multiple face images to analyze the emotional information represented by micro-expressions.

[0069] Mouth movement recognition: When identifying fraudulent behavior through audio, it is necessary to determine whether the user's speech contains keywords. At this time, it is also possible to combine the video image to determine whether the user's mouth is synchronized with the speech. If the user's mouth does not move and only the sound is produced, it may be that someone is guiding the user from the side. Therefore, it is necessary to recognize the user's mouth movement.

[0070] Specific steps include: parsing and segmenting the acquired video into frames to generate images; performing face detection on each frame; for images containing faces, performing face cutout processing on the faces in the images; and obtaining the portion of the image containing only the mouth. Specifically, the face image can be segmented into upper and lower parts according to a certain ratio to obtain the lower part containing the mouth. Using a pre-trained mouth recognition model, multiple frames of mouth region images are input into the model for recognition. If one part of the multiple frames of mouth region images shows an open mouth and another part shows a closed mouth, it can be determined that the user was speaking when recording the video.

[0071] Facing-the-screen recognition: The first step is to extract the angular features of the face. The angle of the face primarily determines whether the person is facing the screen directly during the recording, preventing unauthorized filming by intermediaries and providing a preliminary assessment of potential intermediary involvement. In this process, after acquiring the image, the facial image is extracted and input into a pre-trained facial recognition model for angle recognition. The angles are the estimated depression angle, yaw angle, and roll angle of the face, each with a corresponding threshold. These thresholds are used to determine whether the face is facing the screen directly.

[0072] OCR Recognition: This involves extracting text from images containing human faces obtained through liveness detection. The extracted text is then examined to see if it contains pre-defined keywords. If such keywords are detected, it suggests that an intermediary is directing the user to a specific location for liveness detection, indicating fraudulent activity. Specifically, a pre-trained OCR model is used to perform document detection and text recognition on each frame of the image to determine if pre-defined keywords are present.

[0073] By separating audio and video in liveness videos, and extracting features from the separated audio and video using algorithms, the audio and video features are finally encoded and assembled into a two-dimensional feature vector. A convolutional neural network is then used for fraud detection. This approach effectively detects fraud from multiple perspectives, fully exploring potential fraud scenarios within liveness videos, avoiding intermediary fraud during the liveness detection process, improving the accuracy of fraud detection, and providing guidance for fraud detection using unstructured audio and video features.

[0074] Figure 3 This is a schematic diagram illustrating the steps for obtaining the recognition result of the neural network output, as provided in an embodiment of this application.

[0075] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the steps for obtaining the recognition result from the neural network output, as provided in the embodiments of this application. Figure 3The specific execution steps of this method include:

[0076] Step 201: Generate corresponding labels for the audio features and the video features respectively.

[0077] Please see Figure 4 , Figure 4 The extraction rules and tags for audio and video features provided in the embodiments of this application;

[0078] In the specific implementation of step 201, the unstructured video data is separated into audio and video, and features are extracted from the separated audio and video respectively. Based on the extracted features, corresponding labels are generated.

[0079] In a preferred embodiment, for audio features, the labels are defined as follows: the extraction rule is whether there are multiple speakers, with the label being the number of speakers and -1 if there are no speakers; the extraction rule is whether a keyword is hit, with 1 for a hit and 0 for a miss; for video features, the extraction rule is whether the eyes are paying attention to the screen, with 1 for no attention and 0 for attention; the extraction rule is whether the head is facing the screen, with 1 for no head facing the screen and 0 for facing the screen; the extraction rule is a live video shooting scene, with 1 for a live shooting scene inside a vehicle and 0 for a live video shooting scene inside a vehicle. The label is 0 for indoor and 2 for outdoor. The extraction rule is based on whether the liveness video is a selfie; if it's a selfie, the label is 1, otherwise 0. The extraction rule is based on micro-expressions; the label is 0 for sadness, 1 for happiness, 2 for sadness, etc. The extraction rule is based on OCR text names; if the liveness video text OCR contains the keyword, the label is 1, otherwise 0. The extraction rule is based on whether the user's mouth moves; if the mouth moves, the label is 1, otherwise 0. The extraction rule is based on liveness video background recognition; if the background is a white wall, the label is 0, otherwise 1. Following these rules, specific labels are generated for audio and video features. These rules are just examples; the specific labels for each feature can be determined based on the actual situation.

[0080] Step 202: Generate a feature vector based on the label, input the feature vector into the pre-trained neural network, and obtain the recognition result output by the neural network.

[0081] In the specific execution of step 202, the features of the completed label are one-hot encoded to form a multi-dimensional feature code. Then, the encoded feature vectors are concatenated to form a two-dimensional image code. Subsequently, a pre-trained neural network is used to recognize the code and make the final judgment on whether there is any intermediary intervention.

[0082] Please see Figure 5 , Figure 5This is a schematic diagram of the deep learning model structure provided in an embodiment of this application.

[0083] In a preferred embodiment, the pre-training of the neural network involves building a deep learning model using PyTorch, employing four convolutional layers, varying the number of channels, downsampling, and receptive field to focus on extracting spatial features, and then using Softmax for classification. Specifically, for example... Figure 4 As shown, the model can be a 5-layer network with four convolutional kernels and one softmax layer. The input is a 10*10*1 two-dimensional feature map. After passing through the convolutional layer Conv1, the number of channels is changed without any other transformations, forming a 10*10*128 feature map. After passing through the convolutional layer Conv2, the number of channels is changed and downsampling is performed, forming a 5*5*256 feature map. After passing through the convolutional layer Conv3, spatial features are extracted and the receptive field is increased, forming a 3*3*512 feature map. After passing through the convolutional layer Conv4, the receptive field is increased further and the number of channels is increased to prepare for classification. Finally, softmax is used for classification.

[0084] Figure 6 This is a schematic diagram of a video recognition system 300 provided in an embodiment of this application. Figure 6 The signal separation module 301, feature extraction module 302, and judgment module 303 are shown.

[0085] Signal separation module 301: used to perform audio and video separation on the acquired face recognition liveness video signal, and obtain audio signal and video signal respectively; wherein the video signal includes the object to be verified.

[0086] In the specific implementation of the signal separation module 301, the acquired video signal is first subjected to audio-video separation. Audio-video separation mainly refers to separating the video part and the audio part in the original video signal, so as to analyze the audio signal of the audio part and the signal of the video part separately.

[0087] Feature extraction module 302: used to extract features from the audio signal and the video signal to obtain the audio features corresponding to the audio signal and the video features corresponding to the video signal.

[0088] In the specific implementation of the feature extraction module 302, feature extraction is performed on the video signal obtained in the signal separation module 301, extracting corresponding audio features from the audio signal and corresponding video features from the video signal.

[0089] Judgment module 303: used to input the audio features and the video features into a pre-trained neural network to obtain the recognition result output by the neural network; wherein, the recognition result is used to characterize whether the live video signal is completed under the guidance of the intermediary.

[0090] In the specific implementation of the judgment module 303, since the relevant neural network has been pre-trained, the extracted audio features and video features can be input into the relevant neural network in this module. The neural network performs comprehensive analysis on the audio features and video features to obtain a comprehensive recognition result, thereby determining whether the relevant video recorded by the user's face recognition was completed under the guidance of an intermediary rather than voluntarily.

[0091] Furthermore, in its specific execution, the feature extraction module 302 uses audio features including the number of speakers in the audio signal and / or whether it contains a first keyword. Since intermediaries may be involved, sometimes instructing the face recognition operator, multiple people may be speaking. Determining the number of speakers through audio features can confirm whether multiple people are speaking. Additionally, when an intermediary is involved, their instructions to the face recognition operator may involve keywords. Extracting these keywords during audio feature extraction can provide a preliminary assessment of whether intermediary involvement has occurred. Video features include shooting angle features, facial features, and shooting scene features; each feature can provide a basic preliminary assessment of whether intermediary involvement has occurred.

[0092] Furthermore, in the process of extracting the audio signal, the feature extraction module 302 first segments the audio signal, that is, divides the speech data into segments. First, it needs to determine the segmentation time interval. In a preferred embodiment, the speech data is segmented at 1-second intervals. After segmentation, multiple audio segments are obtained, each with a duration of 1 second. Then, a similarity calculation is performed on each audio segment, comparing the current speech features with previous speech features, and grouping audio segments with high similarity into one category. Since each category is similar, it can be considered as belonging to the same speaker. The number of categories indicates the number of speakers, thus allowing determination of whether multiple people are speaking. Next, different audio segments within the same category are concatenated to form a new long audio file. Then, the speech-to-text function is used to perform speech recognition on the long video, determining whether the speech contains keywords.

[0093] Furthermore, the feature extraction module 302 preprocesses the video signal. First, the video signal is divided into frames to obtain multi-frame image signals. Then, corresponding feature extraction is performed on the multi-frame image signals. Feature extraction is mainly processed from three aspects: shooting angle features, facial features, and shooting scene features.

[0094] Furthermore, the feature extraction module 302 acquires the shooting angle features. First, it extracts the angle features of the face, primarily determining whether the person is facing the screen directly during the shooting process. This helps prevent intermediaries from secretly taking photos when the person is not paying attention, providing a preliminary assessment of whether intermediaries are involved. In this process, the three angles of depression, tilt, and roll are estimated for the image. After acquiring the image, the face portion is extracted and input into a pre-trained face recognition model for angle recognition. If the three angles exceed a certain threshold, it is determined that the face is not facing the screen directly. Subsequently, it can identify whether the user is taking a selfie or being photographed by someone else during face recognition. Intermediaries are highly likely to be holding a camera and taking photos of the user when inducing them to undergo face recognition; therefore, the identification of selfies versus being photographed by someone else can also provide a preliminary assessment of whether intermediaries are involved.

[0095] Furthermore, in facial feature extraction, the feature extraction module 302 first needs to acquire the eye and mouth regions from the image signal. Generally, a partial image of the face is taken, with the upper half identified as the eye region and the lower half as the mouth region. During the analysis of the eye region, a pre-trained eye recognition model is used, inputting the eye region image into the model to determine if the eyes are focused on the screen. During the analysis of the mouth region, it is determined whether the mouth region image includes both open and closed mouth states. In a preferred embodiment, a pre-trained mouth recognition model is used, inputting the mouth region image into the model for recognition. If the mouth region image includes both open and closed mouth states, it can be determined that the user was speaking while recording the video. During the analysis of micro-expressions, a pre-trained micro-expression recognition model is used, inputting the facial image into the model to determine if the user's expression is abnormal.

[0096] Furthermore, during the scene feature recognition process, the feature extraction module 302 first extracts the background image from the acquired image signal. Then, it analyzes the background image and uses a pre-trained background recognition system to determine whether the background contains intermediaries.

[0097] Furthermore, the judgment module 303 generates corresponding labels based on the extracted features, encodes the features of the labels to form a multi-dimensional feature code, then concatenates the encoded feature vectors to form a two-dimensional image code, and then uses a pre-trained neural network to recognize the code to complete the final judgment on whether there is intermediary intervention.

[0098] Please see Figure 7 , Figure 7 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 400 provided in this application includes a processor 401 and a memory 402. The memory 402 stores machine-readable instructions executable by the processor 401. When the machine-readable instructions are executed by the processor 401, the method described above is performed.

[0099] For example, the processor 401 in this embodiment of the application can read a computer program from the memory 402 via a communication bus and execute the computer program to implement the following method: a data management method. In some examples, the processor 401 can also update configuration items, that is, it can perform the following steps: receiving input log data, storing the log data in a high-speed access module, and marking the storage time when storing the log data; determining whether the time length between the current time and the storage time of the log data is greater than a preset threshold, and if it is greater, migrating the log data to a normal-speed access module; and querying the existing log data, wherein the existing log data includes log data stored in the high-speed access module and the normal-speed access module.

[0100] Processor 401 can be an integrated circuit chip with signal processing capabilities. The processor 401 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0101] The memory 402 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0102] Understandable. Figure 7 The structure shown is for illustrative purposes only; the electronic device 400 may also include components that are more advanced than those shown. Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown. Figure 7 The components shown can be implemented using hardware, software, or a combination thereof. In the embodiments of this application, the electronic device 400 can be, but is not limited to, physical devices such as desktop computers, laptops, smartphones, smart wearable devices, and in-vehicle devices, or virtual devices such as virtual machines. Furthermore, the electronic device 400 is not necessarily a single device; it can be a combination of multiple devices, such as a server cluster, etc. In the embodiments of this application, the server in a method for photographing vehicles can be... Figure 7 The illustrated electronic device 400 is implemented.

[0103] This application also provides a computer-readable storage medium, including a computer program stored on the computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by the computer, the computer can perform the steps of a data management method described in the above embodiments, such as: receiving input log data, storing the log data in a high-speed access module, and marking the storage time when storing the log data; determining whether the time length between the current time and the storage time of the log data is greater than a preset threshold; if it is greater, migrating the log data to a normal-speed access module; and querying existing log data, wherein the existing log data includes log data stored in both the high-speed access module and the normal-speed access module.

[0104] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0105] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0107] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0108] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video recognition method, characterized in that, include: The acquired audio and video signals are separated to obtain the audio and video signals separately; The video signal mentioned above includes the object to be verified; Feature extraction is performed on the audio signal and the video signal to obtain the audio features corresponding to the audio signal and the video features corresponding to the video signal; The audio features and the video features are input into a pre-trained neural network to obtain the recognition result output by the neural network; wherein, the recognition result is used to characterize whether the video signal is completed under the guidance of the intermediary by the object to be verified; The audio features include the number of speakers in the audio signal and / or whether it contains a first keyword; the video features include at least one of the following: shooting angle features, facial features, and shooting scene features; The object to be verified is a face; the feature extraction of the video signal includes: dividing the video signal into frames to obtain multiple frame image signals; and extracting features from the image signals to obtain the shooting angle features, facial features, and shooting scene features. The step of extracting features from the image signal to obtain the shooting angle features includes: extracting facial angle features from the image signal to obtain the tilt angle, slant angle, and roll angle of the face in the image signal; determining whether the face is facing the screen based on the tilt angle, slant angle, and roll angle; and / or, identifying the shooting angle from the image signal to determine whether the shooting angle is a selfie angle.

2. The method according to claim 1, characterized in that, The feature extraction of the audio signal includes: The audio signal is segmented to obtain multiple audio segments; the multiple audio segments are classified, and the number of speakers in the audio signal is determined based on the classification results; and / or, The speech in the audio signal is recognized and converted into text, and the text is then subjected to keyword recognition.

3. The method according to claim 1, characterized in that, The step of extracting features from the image signal to obtain the facial features includes: Obtain the eye region and mouth region from the image signal; Feature extraction is performed on the eye region to determine whether the eye's attention is focused on the screen; and / or, Feature extraction is performed on the mouth region to determine whether the mouth is active; and / or, A micro-expression recognition model is used to identify facial images in the image signal in order to determine whether there are any abnormalities in the micro-expressions of the facial images.

4. The method according to claim 1, characterized in that, The step of extracting features from the image signal to obtain shooting scene features includes: Acquire the portrait background image of the image signal; The background image of the portrait is identified to determine whether the background image is indoors; and / or, The background image of the portrait is subjected to keyword recognition to determine whether a second keyword appears in the background.

5. The method according to any one of claims 1-4, characterized in that, The step of inputting the audio features and the video features into a pre-trained neural network to obtain the recognition result output by the neural network includes: Generate corresponding labels for the audio features and the video features respectively; A feature vector is generated based on the label, and the feature vector is input into the pre-trained neural network to obtain the recognition result output by the neural network.

6. A video recognition system, characterized in that, include: The signal separation module is used to separate the acquired audio and video signals, and obtain the audio signal and video signal respectively; The video signal mentioned above includes the object to be verified; The feature extraction module is used to extract features from the audio signal and the video signal to obtain the audio features corresponding to the audio signal and the video features corresponding to the video signal; The judgment module is used to input the audio features and the video features into a pre-trained neural network to obtain the recognition result output by the neural network; wherein, the recognition result is used to characterize whether the video signal is completed under the guidance of the intermediary of the object to be verified; The audio features include the number of speakers in the audio signal and / or whether it contains a first keyword; the video features include at least one of the following: shooting angle features, facial features, and shooting scene features; The object to be verified is a face; the feature extraction module is used to divide the video signal into frames to obtain multi-frame image signals; and to extract features from the image signals to obtain the shooting angle features, facial features, and shooting scene features. The feature extraction module is further configured to extract facial angle features from the image signal to obtain the tilt angle, slant angle, and roll angle of the face in the image signal; determine whether the face is facing the screen based on the tilt angle, slant angle, and roll angle; and / or, perform shooting angle recognition on the image signal to determine whether the shooting angle is a selfie angle.

7. An electronic device, characterized in that, include: Processor, memory, and bus; The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1-5 by calling the program instructions.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-5.

Images