Video processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a video processing method and device, a storage medium and electronic equipment. Relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the feature extraction of a target video collected in the financial business handling process of a target object through a feature extraction algorithm, and obtaining initial feature information; performing identity verification on the target object based on the initial feature information through a portrait comparison algorithm to obtain a verification result; identifying the authenticity of the target video based on the initial feature information through a counterfeiting discrimination algorithm to obtain a first identification result; and under the condition that the first recognition result represents that the target video is a non-forged video, gesture recognition is performed based on the initial feature information through a gesture recognition algorithm, and a second recognition result is obtained. Through the video gesture recognition method and device, the problem that the accuracy of video gesture recognition is low due to the fact that a video gesture recognition technology in related technologies is lack of effective resistance to AI generation video attacks is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a video processing method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the rapid development of multimedia and network technology, video has gradually become the most common information medium in human life. At present, in the field of financial services, the integration of video application and online business has become a development trend. For example, when a user handles online financial business, the user can prove the intention of personal service subscription through the way of recording sound and video, so as to realize the purpose of remote authorization, which takes into account convenience and security.

[0003] Among various video authorization mechanisms, a user-centered authorization method based on video gesture recognition technology captures and analyzes specific gestures or posture signals to interpret the user's authorization intention. This method overcomes the limitations of sound recognition, especially for the needs of elderly users and hearing or language impaired groups, providing an accessible video authorization experience. However, existing video gesture recognition technology lacks effective resistance to AI-generated video attacks.

[0004] In view of the problem that the video gesture recognition technology in the related art lacks effective resistance to AI-generated video attacks, resulting in relatively low accuracy of video gesture recognition, no effective solution has been proposed so far. SUMMARY

[0005] The main purpose of the present application is to provide a video processing method and device, a storage medium and an electronic device to solve the problem that the video gesture recognition technology in the related art lacks effective resistance to AI-generated video attacks, resulting in relatively low accuracy of video gesture recognition.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a video processing method is provided. The method comprises: performing feature extraction on a target video collected in a process of a target object handling financial business by a feature extraction algorithm to obtain initial feature information; performing identity verification on the target object based on the initial feature information by a portrait comparison algorithm to obtain a verification result; in a case where the verification result represents that the identity verification is passed, identifying the authenticity of the target video based on the initial feature information by a forgery discrimination algorithm to obtain a first identification result; in a case where the first identification result represents that the target video is a non-forged video, performing gesture recognition based on the initial feature information by a gesture recognition algorithm to obtain a second identification result, wherein whether there is an abnormality in the process of the target object handling financial business is determined by the second identification result.

[0007] Further, the feature extraction algorithm is used to extract features of the target video collected in the process of the target object handling the financial business, to obtain initial feature information, including: splitting the target video to obtain a plurality of video frames; grouping the plurality of video frames to obtain a plurality of video frame groups; extracting features of each video frame in each video frame group to obtain a tensor matrix corresponding to each video frame group; and obtaining the initial feature information according to the plurality of tensor matrices.

[0008] Further, the initial feature information is obtained according to the plurality of tensor matrices, including: performing dimension transformation on each tensor matrix to obtain a plurality of transformed tensor matrices; performing dimension reduction processing on the last dimension of each transformed tensor matrix to obtain a plurality of processed tensor matrices; and performing splicing processing on the plurality of processed tensor matrices according to a time sequence relationship to obtain the initial feature information.

[0009] Further, the portrait comparison algorithm is used to verify the identity of the target object based on the initial feature information, to obtain a verification result, including: performing dimension reduction processing on the time sequence dimension in the initial feature information to obtain processed initial feature information; extracting features of the processed initial feature information to obtain portrait feature information; obtaining real portrait feature information of the target object, and verifying the identity according to the portrait feature information and the real portrait feature information to obtain the verification result.

[0010] Further, the forgery discrimination algorithm is used to identify the authenticity of the target video based on the initial feature information, to obtain a first identification result, including: extracting features of the initial feature information according to a first number to obtain first feature information, wherein the first number is calculated based on a grouping number, and the first feature information is composed of feature information corresponding to a first number of first video frame groups; identifying the authenticity of each first video frame group according to the feature information corresponding to each first video frame group to obtain an identification result of each first video frame group; and obtaining the first identification result according to the identification result of each first video frame group.

[0011] Further, the feature extraction of the initial feature information according to the first number to obtain the first feature information includes: calculating the first number to obtain an initial seed; calculating the initial seed to obtain a current seed, and calculating the current seed to obtain a target serial number; determining the first number of first video frame groups from the plurality of video frame groups according to the target serial number; and obtaining the first feature information according to the feature information of the first video frame groups.

[0012] Further, the gesture recognition algorithm is used to recognize the gesture based on the initial feature information to obtain a second recognition result, including: uniformly sampling the initial feature information to obtain second feature information, wherein the second feature information is composed of feature information corresponding to a second number of second video frame groups; performing feature extraction on the contour information in the second feature information to obtain a human contour feature; performing feature extraction on the dynamic information in the second feature information to obtain a human dynamic feature; fusing the human contour feature and the human dynamic feature to obtain fused feature information; and performing gesture recognition according to the fused feature information to obtain the second recognition result.

[0013] Further, the feature extraction on the contour information in the second feature information to obtain a human contour feature includes: performing dimension reduction processing on the time sequence dimension of the second feature information to obtain third feature information; processing the third feature information through a plurality of linear network layers to obtain fourth feature information; performing frequency domain conversion on the fourth feature information, and performing high-frequency feature extraction on the converted feature information to obtain fifth feature information; and obtaining the human contour feature according to the fifth feature information and the fourth feature information.

[0014] Further, the feature extraction on the dynamic information in the second feature information to obtain a human dynamic feature includes: performing product calculation on a first matrix and the second feature information to obtain a first calculation result; performing product calculation on a second matrix and the second feature information to obtain a second calculation result; calculating the difference between the first calculation result and the second calculation result to obtain a posture information difference; and obtaining the human dynamic feature according to the posture information difference.

[0015] To achieve the above object, according to another aspect of the present application, a video processing device is provided. The device comprises: an extraction unit configured to perform feature extraction on a target video collected in a process of a target object conducting a financial business by a feature extraction algorithm to obtain initial feature information; a verification unit configured to perform identity verification on the target object based on the initial feature information by a portrait comparison algorithm to obtain a verification result; a first recognition unit configured to, in a case where the verification result represents that the identity verification is passed, perform identification on the authenticity of the target video based on the initial feature information by a forgery discrimination algorithm to obtain a first recognition result; and a second recognition unit configured to, in a case where the first recognition result represents that the target video is a non-forged video, perform gesture recognition based on the initial feature information by a gesture recognition algorithm to obtain a second recognition result, wherein whether there is an abnormality in the process of the target object conducting the financial business is determined by the second recognition result.

[0016] According to another aspect of the embodiments of the present application, an electronic device is also provided, comprising a memory storing an executable program, and a processor configured to execute the program, wherein the program, when executed, performs any of the video processing methods described above.

[0017] According to another aspect of the embodiments of the present application, a computer readable storage medium is also provided, the storage medium storing a program, wherein the program, when executed, controls a device in which the storage medium is located to perform any of the video processing methods described above.

[0018] According to another aspect of the embodiments of the present application, a computer program product is also provided, comprising a computer program or instructions, which, when executed by a processor, implement any of the video processing methods described above.

[0019] In the embodiments of the present application, the following steps are adopted: feature extraction is performed on a target video collected in a process of a target object handling a financial service by a feature extraction algorithm to obtain initial feature information; identity verification is performed on the target object based on the initial feature information by a portrait comparison algorithm to obtain a verification result; in a case where the verification result represents that the identity verification is passed, identification of authenticity of the target video is performed based on the initial feature information by a forgery discrimination algorithm to obtain a first identification result; in a case where the first identification result represents that the target video is a non-forged video, gesture recognition is performed based on the initial feature information by a gesture recognition algorithm to obtain a second identification result, wherein whether the process of the target object handling the financial service is abnormal is determined through the second identification result, and a technical problem that a video gesture recognition technology in the related art lacks effective resistance capability to AI generated video attacks, resulting in relatively low accuracy of video gesture recognition is solved.

[0020] In the present solution, after the initial feature information is extracted, portrait comparison is first performed to ensure that the target object in the video is consistent with the authenticated user. Only in the case where the identity of the target object is confirmed to be correct, subsequent forgery discrimination and gesture recognition will be continued. The introduction of the forgery discrimination algorithm can accurately identify the authenticity of the video. Even if the attacker uses a highly simulated AI generated video, abnormal features that may exist in the video can be identified, such as unnatural smoothness of the picture, disharmony of micro-expressions, or abnormal repetition of gestures, thereby preventing the false passage of forged content and improving the security of identification. By introducing the multi-stage process of identity verification, forgery video identification and gesture behavior pattern analysis, the security and accuracy of the video gesture recognition technology are significantly enhanced, especially the resistance capability to AI generated video attacks, which provides a more reliable and secure solution for online financial service authorization scenarios. Through the above technology, illegal transactions and operations can be effectively avoided, user fund safety is protected, and at the same time, barrier-free financial service experience is provided for the elderly and hearing or language impaired groups. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application and are incorporated herein in their entirety. The figures that constitute a part of this detailed description are used to explain the illustrative embodiments of the present application and to provide further understanding of the present application, and are not intended to limit the present application. In the drawings:

[0022] Figure 1 A hardware structure block diagram of a computer terminal for implementing a video processing method is shown;

[0023] Figure 2 A flow chart of a video processing method according to an embodiment of the present application is shown;

[0024] Figure 3 A schematic diagram of a video processing method according to an embodiment of the present application is shown Figure 1 ;

[0025] Figure 4 A schematic diagram of a video processing method according to an embodiment of the present application is shown Figure 2 ;

[0026] Figure 5 A schematic diagram of a video processing method according to an embodiment of the present application is shown Figure 3 ;

[0027] Figure 6 A schematic diagram of a video processing method according to an embodiment of the present application is shown Figure 4 ;

[0028] Figure 7 A schematic diagram of a video processing method according to an embodiment of the present application is shown Figure 5 ;

[0029] Figure 8 A schematic diagram of a video processing device according to an embodiment of the present application is shown;

[0030] Figure 9 A structure block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] In order to make the personnel in the technical field better understand the present application scheme, the technical scheme in the present application embodiment will be described clearly and completely in combination with the drawings in the present application embodiment. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor should belong to the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] It should be noted that the collected information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present application are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal. For example, the system and related users or institutions are provided with an interface to provide the user with a corresponding operation portal for the user to choose to agree or refuse the automatic decision result; if the user chooses to refuse, the expert decision process is entered.

[0034] Embodiment 1

[0035] According to the embodiments of the present application, a method for video processing is also provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0036] The method embodiment provided by the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing the video processing method is shown. As Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0037] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0038] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video processing method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned video processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0039] The transmission device 106 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module that is configured to communicate with the Internet wirelessly.

[0040] The display can be a touch screen liquid crystal display (LCD) that enables a user to interact with the user interface of the computer terminal 10 (or mobile device).

[0041] In the above operating environment, the present application provides a video processing method as shown in Figure 2 Figure 2 is a flowchart of a video processing method according to an embodiment of the present application. The video processing method includes:

[0042] At step S201, a feature extraction algorithm is used to extract features from a target video collected during the process of the target object conducting a financial business, to obtain initial feature information.

[0043] Optionally, a target video collected during the process of a user (i.e. the target object) conducting a financial business is obtained. The target video is first decoded into a series of video frames, and then each frame is converted into a set of feature vectors by a pre-trained deep learning model (i.e. the feature extraction algorithm), to obtain the initial feature information. These features can include color, texture, contour, dynamic change, and other visual information.

[0044] At step S202, a portrait comparison algorithm is used to verify the identity of the target object based on the initial feature information, to obtain a verification result.

[0045] Optionally, a video portrait feature of the target object can be extracted from the initial feature information by the portrait comparison algorithm. Then, according to the identity information of the target object, an authentication facial image of the user is retrieved from a user authentication information database. A large multi-modal pre-training model and a fine-tuned portrait feature extraction network are jointly used to convert the authentication facial image into an authentication portrait feature vector. This authentication portrait feature vector will be used as a reference for comparison with the video portrait feature.

[0046] ​In an optional embodiment, the cosine similarity is used as a measure to compare the similarity between the video portrait feature vector and the authentication portrait feature vector. If the cosine similarity between the video portrait feature vector and the authentication portrait feature vector is higher than a set threshold (e.g., 0.8), it is determined that the identity of the target object matches the authentication information, and the verification result is passed. Otherwise, the verification fails, indicating the risk of an unauthorized user or a tampered video.

[0047] The introduction of the portrait comparison algorithm enhances the verification capability of the identity of the target object, ensuring the security of subsequent financial business operations. By combining the feature extraction capability of the deep learning model and the comparison function of the user authentication information library, the system can efficiently and accurately determine whether the person in the video is a legitimate user, effectively resisting security threats such as identity fraud and video forgery, and providing a more secure and reliable online financial service environment for users.

[0048] Step S203, in the case where the verification result represents that the identity verification is passed, the authenticity of the target video is identified based on the initial feature information through the forgery discrimination algorithm, and a first identification result is obtained.

[0049] Optionally, a forged video is generated based on static portrait data, thereby serving as training data for the forgery discrimination algorithm. In the process of operation, the forgery discrimination algorithm takes multiple open-source portrait data sets as input and restricts the output of the generation model with instructions to generate specific actions. Artificially remove some undesirable results in the forged video, and the remaining results are used as forged video data. Through the feature extraction algorithm, the real video data and the forged video data will be processed into real video group features and forged video group features, respectively. Finally, the training of the forgery discrimination algorithm is completed based on the real video group features and the forged video group features. For the target video to be verified, the forgery discrimination algorithm evaluates the input initial feature information and outputs a true-false discrimination result (i.e., the first identification result mentioned above).

[0050] By analyzing the video features, it is possible to identify possible forgery traces in the video, such as unnatural motion fluency, abnormal facial expressions, or inconsistencies between frames, etc. This step ensures that only when the target video is confirmed to be real and not tampered with, will the gesture recognition proceed, further reducing the risk of illegal transactions.

[0051] Step S204, in the case where the first identification result represents that the target video is a non-forged video, gesture recognition is performed based on the initial feature information through the gesture recognition algorithm, and a second identification result is obtained, wherein the second identification result determines whether there is an abnormality in the process of the target object handling the financial business.

[0052] Optionally, after determining that the target video is a non-forged video, a gesture recognition algorithm is used to analyze the gestures of the target object in the video. Specifically, the gesture recognition algorithm can be used to extract the features of the hands and body contours of the target object from the initial feature information, and then perform gesture recognition based on the extracted features. The gesture recognition algorithm determines the category of the gestures contained in the target video (i.e., the second recognition result described above). By comparing the second recognition result with the gestures in the known normal financial business process, it can be determined whether the user's operation is consistent with the expectation. If the recognized gestures match the gestures in the normal business process, it can be considered that the target object is conducting normal financial business; otherwise, if the recognized gestures do not match the expectation or there are unrecognized gestures, it may indicate an anomaly in the operation process, which requires further manual review or security measures. Through the gesture recognition algorithm, the user's operation can be accurately understood, and only the user who has performed the expected gestures can complete the financial transaction, which effectively prevents unauthorized transaction activities.

[0053] In summary, after the initial feature information is extracted, portrait comparison is first performed to ensure that the target object in the video matches the authenticated user. Only when the target object's identity is confirmed, will the subsequent forgery discrimination and gesture recognition be continued. The introduction of the forgery discrimination algorithm can accurately identify the authenticity of the video. Even if the attacker uses a highly simulated AI-generated video, the possible abnormal features in the video, such as unnatural smoothness of the picture, inconsistency of micro-expressions, or abnormal repetition of gestures, can be identified, thereby preventing the false passage of forged content and improving the security of the identification. By introducing a multi-stage process of identity verification, forgery video recognition, and gesture behavior pattern analysis, the security and accuracy of the video gesture recognition technology are significantly enhanced, especially the ability to resist AI-generated video attacks, providing a more reliable and secure solution for online financial service authorization scenarios. Through the above technology, illegal transactions and operations can be effectively avoided, protecting user financial security, and also providing barrier-free financial service experience for elderly users and hearing or language impaired groups.

[0054] Optionally, in the video processing method provided in the embodiments of the present application, the feature extraction algorithm is used to extract features from the target video collected during the target object's financial business process to obtain initial feature information, including: splitting the target video to obtain a plurality of video frames; grouping the plurality of video frames to obtain a plurality of video frame groups; extracting features from each video frame in each video frame group to obtain a tensor matrix corresponding to each video frame group; and obtaining initial feature information based on the plurality of tensor matrices.

[0055] In an optional embodiment, the target video is first split into a series of independent video frames. For example, by decoding the video stream, its temporal continuity is divided into spatially independent static images for frame-by-frame analysis. Then, these video frames are grouped according to certain logic to form video frame groups. For example, in an embodiment of the present application, each group contains 20 video frames, which takes into account the continuity of gesture actions and facilitates subsequent processing to reduce computational complexity. If the number of video frames in the last group is less than 20, the last frame is copied to make up for it, ensuring that the number of video frames in each group is consistent.

[0056] For each video frame in each video frame group, the feature extraction algorithm extracts its visual features. These features can include color, edge, texture, and other information. After obtaining the feature information of each video frame, all feature information in the same group is further fused to generate a tensor matrix representing the group. This fusion process can be achieved by calculating the average value, performing maximum pooling, etc. After the fusion of the features in the group, each video frame group is represented as a three-dimensional tensor matrix, whose dimensions are determined by the sequence length of the video frame and the vector length. Finally, all tensor matrices of the video frame groups are integrated together in chronological order to form the initial feature information of the entire video.

[0057] By splitting the video into frames and grouping for feature extraction, the consumption of computing resources can be effectively reduced while maintaining the integrity and continuity of gesture actions.

[0058] Optionally, in the video processing method provided by the embodiments of the present application, obtaining the initial feature information according to the plurality of tensor matrices comprises: performing dimension transformation on each tensor matrix to obtain a plurality of transformed tensor matrices; performing dimension reduction processing on the last dimension of each transformed tensor matrix to obtain a plurality of processed tensor matrices; and performing splicing processing on the plurality of processed tensor matrices according to a time sequence relationship to obtain the initial feature information.

[0059] In an optional embodiment, there is information redundancy in each tensor matrix. To remove the redundant information and reduce the computational cost of subsequent components, a fusion operation is performed on the tensor matrices of multiple video frame groups. The size of all tensor matrices conforms to N x seq x dim. Among them, N is the number of video frames contained in the group, seq is the sequence length of each frame feature, and dim is the length of the sequence unit vector. In the present application, N is 20, seq is 64, and dim is 768. After dimension transformation of the tensor matrix, it is converted into a seq x dim x N pattern, and an average pooling layer is used to process the last dimension, that is, the last dimension of each transformed tensor matrix is reduced to obtain a seq x dim x 1 tensor matrix. Finally, according to the time sequence relationship of the video frames, B groups of features are spliced to obtain video group features (i.e. the initial feature information described above). The size of the video group feature is B x seq x dim.

[0060] In an optional embodiment, the video group feature (i.e. the initial feature information described above) can be obtained through the schematic diagram as shown in Figure 3 The user uploads a video, which can be divided into a series of independent video frames by a video frame grouping component. The video frame grouping component checks the encoding format (ensure AVI format) and frame rate (must be a multiple of 10) of the input video to ensure the uniformity of the data and the compatibility of the subsequent processing. The separation of video frames facilitates the analysis of key visual information in each frame.

[0061] Then, all video frames are evenly grouped into multiple subsets, each group containing a fixed number of video frames (such as 20 frames). The purpose of this is to consider the high similarity between adjacent video frames, and by grouping them, more efficient feature extraction and processing can be performed while reducing the consumption of computing resources. For each frame after grouping, the convolutional neural network can be used to capture key feature information such as texture, color, shape, etc. in the video frame. After completing the frame feature extraction, the group feature fusion is performed to obtain the video group feature. Through the fusion step, the complexity of the subsequent processing is reduced, while the temporal and spatial relationships between video frames are also preserved.

[0062] Through dimension transformation and dimension reduction processing, the computational efficiency and model performance of the video gesture security recognition technology are improved, while the dynamic and temporal information in the video data is effectively preserved, which can ensure that the subsequent components have high-quality and targeted feature representation when performing security judgment and gesture recognition.

[0063] Optionally, in the video processing method provided in the embodiments of the present application, the portrait comparison algorithm is used to perform identity verification on the target object based on the initial feature information, and the verification result is obtained by: performing dimension reduction processing on the time sequence dimension in the initial feature information to obtain processed initial feature information; performing feature extraction on the processed initial feature information to obtain portrait feature information; obtaining real portrait feature information of the target object, and performing identity verification according to the portrait feature information and the real portrait feature information to obtain the verification result.

[0064] In an optional embodiment, since the time sequence does not affect the facial features, in order to reduce the amount of calculation when performing identity verification, the time sequence dimension in the initial feature information can be processed by dimension reduction (for example, using a pooling layer such as maximum pooling or average pooling). In this way, the complex time sequence feature information can be simplified, and the key part can be retained, such as the most prominent facial features of each group of video frames. In this way, the burden of subsequent processing can be reduced, and it can be ensured that important information for identity verification is not lost.

[0065] After dimension reduction processing, the portrait comparison algorithm can be used to extract the facial features of the target object from the processed initial feature information, such as the features of key parts such as face contour, eyes, nose, and mouth, and further obtain portrait feature information. It should be noted that the portrait comparison algorithm can be a fine-tuned face recognition network. After obtaining the above-mentioned portrait feature information, the real portrait feature information left by the target object during registration or authorization is retrieved from the user authentication information library. The portrait comparison algorithm compares the extracted portrait feature information with the real portrait feature information, for example, using cosine similarity or Euclidean distance as an index to measure the similarity between the two. If the similarity is higher than a predetermined threshold, the algorithm will determine that the target object in the video matches the real user in the system, and the verification result is passed; otherwise, the verification fails.

[0066] By reducing data redundancy through dimension reduction processing, locking key visual elements through feature extraction, and ensuring the consistency of the target object and the authentication information through portrait comparison, the processing efficiency is improved, and the accuracy and security of user identity verification are ensured.

[0067] In an optional embodiment, the dimension reduction processing can be performed by, for example, Figure 4The schematic diagram shown implements authentication of a user, specifically including: obtaining video grouping features (i.e., the initial feature information described above), performing pooling fusion processing on the video grouping features to remove redundant information in the time dimension. Then, a fine-tuned portrait feature extraction network is used to extract facial features from each frame group to obtain a video portrait feature vector. Then, an authentication video of the user is obtained from a user authentication information library, and facial feature extraction is performed on the authentication video to obtain an authentication portrait feature vector. Finally, the similarity between the video portrait feature vector and the authentication portrait feature vector is calculated, and if the calculated cosine similarity is higher than a set threshold (such as 0.8), it is determined that the video subject and the authentication user are the same person; otherwise, it is considered that the two do not match, and thus a final portrait comparison result is obtained.

[0068] By confirming the identity consistency of the video subject and the authentication user, the reliability and security of the system for user identity authentication are enhanced, thereby laying a foundation for user identity confirmation for video gesture security recognition technology.

[0069] Optionally, in the video processing method provided in the embodiments of the present application, the authenticity of the target video is identified based on the initial feature information by the forgery discrimination algorithm, and the first identification result is obtained, including: performing feature extraction on the initial feature information according to a first number to obtain first feature information, wherein the first number is calculated based on the number of groups, and the first feature information is composed of feature information corresponding to the first number of first video frame groups; identifying the authenticity of each first video frame group according to the feature information corresponding to each first video frame group to obtain an identification result of each first video frame group; and obtaining the first identification result according to the identification result of each first video frame group.

[0070] In an optional embodiment, in order to improve the efficiency of authenticity identification, the first feature information can be obtained by extracting the features of the first number of first video frame groups from the initial feature information. The first number can be determined based on the number of video groups, for example, the first number is 50% of the number of video groups. It should be noted that the features of the first number of first video frame groups can be obtained by uniformly sampling the initial feature information.

[0071] In an optional embodiment, the first number described above can be calculated by the following formula, where the number of representative frame groups (i.e., the first number described above) and the number of video group feature groups are defined as G and B respectively, and the number relationship between the two can be represented by the following formula:

[0072]

[0073] After obtaining the first feature information composed of the feature information of the first number of first video frame groups, the forgery discrimination algorithm performs deep analysis on the feature information of each first video frame group to check whether there is a sign of AI forgery, such as checking the consistency of video frames, detecting abnormal visual texture, color or light change, etc. Each first video frame group will obtain an identification result indicating whether the segment is considered real. The identification result can be binary (true or false), or a probability value indicating the likelihood of the segment being real.

[0074] The first identification result of the target video is determined by the identification results of all first video frame groups, for example, if more than 75% of the representative video frame groups of the video to be predicted (i.e. the target video mentioned above) are determined to be real, the video to be predicted is finally determined to be a real video, otherwise it is a fake video.

[0075] Through uniform sampling and generation of first feature information, the input data amount of the model is effectively reduced, the demand for computing resources is reduced, and the identification speed is accelerated. By checking the authenticity of each first video frame group, combining the consistency of video frames and the abnormal detection of visual features, the accuracy of identification and the security of the system are improved.

[0076] Optionally, in the video processing method provided by the embodiments of the present application, the feature extraction is performed on the initial feature information according to the first number to obtain the first feature information, including: calculating the first number to obtain an initial seed; calculating the initial seed to obtain a current seed, and calculating the current seed to obtain a target serial number; determining the first number of first video frame groups from the plurality of video frame groups according to the target serial number; and obtaining the first feature information according to the feature information of the first video frame groups.

[0077] In an optional embodiment, in order to extract representative frame groups as evenly as possible while ensuring randomness, the application adopts the following steps for representative frame group extraction, specifically including: first, an initial seed needs to be calculated, which is the starting point of the entire feature extraction process. Starting from the initial seed, the algorithm will perform a series of calculations to generate the current seed. After each calculation, the current seed will be updated to guide the extraction of the subsequent representative frame group (i.e. the first video frame group). This process ensures that the extracted representative frame group is evenly distributed in the entire video frame group while maintaining a certain randomness, avoiding excessive dependence on specific areas and improving the comprehensiveness and accuracy of recognition. Through the calculation of the current seed, a target sequence number can be obtained. The target sequence number is used to determine which group to extract from the video frame group as the first video frame group. The generation of this sequence number is also based on the principles of randomness and uniform distribution, ensuring that each video frame group has a chance to be selected. According to the target sequence number, a specific group is selected from the video frame group for analysis. This process is repeated until the first number of first video frame groups is selected.

[0078] In an optional embodiment, the initial seed x0 is generated using G according to the following calculation formula:

[0079] x0 = 2039 G+1 mod(128·G+1)

[0080] When the ith representative frame group is taken, the seed x i needs to be calculated first. The seed is updated according to the following calculation formula:

[0081] x i = (2045·x i-1 +1) mod 2 20 ,G+1>i>0

[0082] The corresponding frame group position y i is calculated through the current seed x i , which is selected as the representative frame group:

[0083]

[0084] Repeat the above steps until G representative frame groups are obtained.

[0085] Through random and uniform sampling of video frames, key features of the video can be obtained at a lower computational cost, improving the efficiency of true and false identification. At the same time, since the sampling process takes into account randomness and representativeness, the extracted first feature information can more comprehensively reflect the characteristics of the video, thereby improving the accuracy of true and false identification.

[0086] In an optional embodiment, the first feature information can be obtained by Figure 5The schematic diagram shown realizes authenticity judgment of the video, and specifically includes: collecting real data not subjected to any tampering or forgery from an actual environment, generating a video containing a fake gesture by inputting static portrait data and a specific action instruction. Real video and fake video are subjected to video feature extraction to obtain real video group features and fake video group features, and then representative frame groups are extracted. Finally, a fake judgment algorithm is trained through a real representative frame group and a fake representative frame group to form a real-fake feature library. Each feature in the real-fake feature library has a real-fake label, and is the basis for the fake judgment algorithm to learn and identify the differential features of real video and fake video.

[0087] Video grouping features of the video to be verified are obtained, and representative frame group features representing video characteristics are extracted from each video frame group. This process is to reduce the amount of calculation and improve efficiency, while ensuring that key information of the video is captured. Finally, real-fake prediction is performed on the representative frame groups to obtain real-fake discrimination results.

[0088] Through the above steps, it can be effectively detected whether the video is tampered or forged, especially in the video authorization scene of financial services, which greatly improves the accuracy and security of identification, ensures that only real user operations can be accepted and processed by the system, thereby protecting the interests of users and financial security.

[0089] Optionally, in the video processing method provided in the embodiments of the present application, the gesture recognition algorithm performs gesture recognition based on the initial feature information to obtain a second recognition result, including: uniformly sampling the initial feature information to obtain second feature information, wherein the second feature information is composed of feature information corresponding to a second number of second video frame groups; performing feature extraction on the contour information in the second feature information to obtain a person contour feature; performing feature extraction on the dynamic information in the second feature information to obtain a person dynamic feature; fusing the person contour feature and the person dynamic feature to obtain fused feature information; and performing gesture recognition according to the fused feature information to obtain the second recognition result.

[0090] In an optional embodiment, the second feature information is obtained by uniformly sampling the initial feature information to extract features of a second number of second video frame groups. Uniform sampling ensures that the extracted video frame groups can cover the entire time sequence range of the video, thereby obtaining comprehensive video information. It should be noted that the number of second video frame groups (second number) is set according to the specific circumstances of the video (such as video length) and algorithm requirements (such as computing resources, recognition accuracy) to ensure the balance and efficiency of recognition.

[0091] Then, the edge, shape and structure information in the second feature information can be processed by a deep learning model, such as a convolutional neural network, to locate and identify the limb contour of the video subject, obtaining the person contour feature. In contrast to the contour information, the dynamic information feature extraction focuses on the dynamic changes of the gesture action, such as the capture of dynamic features such as object movement and posture conversion. The dynamic features of the gesture can be effectively extracted by mapping through a custom linear network layer and an activation function, obtaining the above-mentioned person dynamic feature. After the person contour feature and the person dynamic feature are extracted respectively, they need to be integrated through a fusion operation to form the fusion feature information. This fusion process can be a simple mathematical operation (such as addition), or an intelligent fusion through a complex neural network structure (such as an attention mechanism), the purpose being to integrate static and dynamic information to more comprehensively reflect the gesture features of the video subject.

[0092] Finally, the gesture is classified and recognized based on the fusion feature information through the gesture recognition algorithm, and the final output is the second recognition result, indicating the specific gesture action category performed by the video subject.

[0093] It should be noted that the above steps can be repeated for a predetermined number of times (for example, 4 times) to obtain multiple fusion feature information, and then the multiple fusion feature information is fused to obtain the final fusion feature information, and finally the gesture is classified and recognized according to the final fusion feature information.

[0094] Through the above steps, the gesture recognition algorithm can accurately recognize the gesture action from the video, even in the case of video frame rate changes, different lighting conditions and complex and variable backgrounds. This process not only improves the accuracy and robustness of the recognition, but also optimizes the computational efficiency through uniform sampling and feature fusion techniques, enabling the algorithm to run efficiently in resource-limited environments.

[0095] Optionally, in the video processing method provided by the embodiments of the present application, the feature extraction of the contour information in the second feature information to obtain the person contour feature comprises: performing dimensionality reduction processing on the time sequence dimension of the second feature information to obtain third feature information; processing the third feature information through multiple linear network layers to obtain fourth feature information; performing frequency domain conversion on the fourth feature information, and extracting high-frequency features from the converted feature information to obtain fifth feature information; obtaining the person contour feature according to the fifth feature information and the fourth feature information.

[0096] In an optional embodiment, first, dimension reduction in the time sequence dimension is performed on the second feature information (containing the features of the second number of second video frame groups) uniformly extracted from the video to obtain third feature information. The purpose of this operation is to simplify the data structure, reduce the computational burden of the subsequent processing stage, and at the same time retain sufficient information for subsequent feature extraction. Common dimension reduction methods include but are not limited to average pooling, in which the feature vector of each group of video frames is compressed into a single representative vector. In this way, not only is the relative temporal relationship between the video frame groups maintained, but also unnecessary inter-frame redundancy is eliminated, improving processing efficiency.

[0097] Then, the third feature information is processed through multiple linear network layers to further refine the features and reduce noise interference, obtaining fourth feature information. The core role of the linear network layer is to perform linear transformation on the input features, supplemented by an activation function (such as ReLU), to highlight important visual contour information while suppressing irrelevant information. The fourth feature information is then converted to the frequency domain, which helps to reveal periodic patterns and detail changes in the video frame groups, especially subtle features that are not easily detected in the time domain. High-frequency features are extracted in the frequency domain to obtain fifth feature information.

[0098] Finally, the fifth feature information (high-frequency features) and the fourth feature information (linear network processed features) are used to obtain the contour features of the person. The addition of high-frequency features enhances the detailed description of the contour features, while the fourth feature information provides more extensive framework information. The combination of the two can more comprehensively represent the contour characteristics of the video subject.

[0099] Through steps such as dimension reduction in the time sequence dimension, linear network layer processing, frequency domain conversion, and high-frequency feature extraction, the computational efficiency is effectively improved, while ensuring the accuracy and robustness of gesture recognition.

[0100] Optionally, in the video processing method provided in the embodiments of the present application, the feature extraction of the dynamic information in the second feature information to obtain the dynamic features of the person includes: performing product calculation on the first matrix and the second feature information to obtain a first calculation result; performing product calculation on the second matrix and the second feature information to obtain a second calculation result; calculating the difference between the first calculation result and the second calculation result to obtain the posture information difference; and obtaining the dynamic features of the person according to the posture information difference.

[0101] In an optional embodiment, the first matrix and the second feature information are multiplied to obtain a first calculation result, and the second matrix and the second feature information are multiplied to obtain a second calculation result. It should be noted that the first matrix and the second matrix are composed of a unit matrix (representing the invariant part of the video frame) and a zero vector (a feature used for misaligned video frames). The difference lies in the position of the zero vector splicing, which is located at the beginning or end of the unit matrix respectively. This design allows the algorithm to selectively compare different parts of adjacent frames when processing video data, thereby capturing the dynamic changes in the subject's posture. Two sets of calculation results can be obtained by multiplying the first matrix and the second matrix with the second feature information (composed of the features of the second number of second video frame groups) respectively. The calculation here is actually a feature dislocation, which allows the changes in timing between adjacent frames to be quantified.

[0102] Then, the difference between the first and second calculation results is calculated to obtain the pose information difference. This difference reflects the changes in the pose of the subject between different frames in the video. The pose information difference can reveal subtle changes in the subject's movements between video frames, even capturing rapid or subtle gestures. Finally, through the pose information difference, and subsequent linear network layer processing and the application of activation functions (such as ReLU), the dynamic characteristics of the person can be extracted.

[0103] By comparing and calculating the difference between the posture information of adjacent frames, the dynamic changes of the video subject's movements can be effectively captured, and the accuracy of obtaining the dynamic features of the characters is improved.

[0104] In an alternative embodiment, the Figure 6 The schematic diagram shown implements gesture recognition in a target video. Specifically, it involves selecting a certain number of video frame groups from the video grouping features based on the total number of video group features. These frame groups will be used for subsequent gesture feature extraction. A specific linear network layer mapping is performed on the selected video frame groups to compress the temporal dimension and extract key information about edges and shapes. High-frequency features are then extracted in the frequency domain using a Gaussian filter. These features are then restored to the real domain and fused with the previous information to obtain optimized person contour features. Two different fixed matrices are applied to a certain number of video frame groups, generating inter-frame feature misalignment. The difference between the two processed results is then calculated to obtain gesture information differences, which are then used to generate person dynamic features. After the contour features and dynamic features are extracted separately, the person contour features and dynamic features are fused to obtain fused features. Finally, the fused features are used for gesture discrimination to obtain gesture prediction results.

[0105] In an alternative embodiment, the Figure 7The overall architecture of the deep learning-based video gesture security recognition technology is shown to realize the processing of the video, specifically including a video feature extraction component, a portrait comparison component, a fake video discrimination component, and a gesture recognition component. The user shoots a video containing a specific gesture through a photographic terminal and uploads it to the system for subsequent processing and analysis. The video feature extraction component is responsible for converting the uploaded video into a computer-processable feature representation. This component includes video frame division, video frame grouping, frame feature extraction, and intra-group feature fusion steps, and finally outputs video group features to provide basic data for subsequent portrait comparison, fake video discrimination, and gesture recognition. The portrait comparison component identifies the portrait information of the video subject from the video group features and compares it with the authentication portrait data stored in the user authentication information library. This process ensures the consistency of the video subject and the authentication user identity, enhancing the security of user identity authentication. The fake video discrimination component evaluates the authenticity of the video, identifies whether the video may be forged or tampered with by AI tools through representative frame group extraction and authenticity prediction steps. This component ensures that only real and unaltered videos can enter the subsequent gesture recognition process, strengthening the security protection of video gesture recognition. The gesture recognition component focuses on identifying and classifying specific gestures from video group features. This component obtains gesture prediction results, i.e., the gesture action category performed by the video subject, through uniform frame extraction, contour feature extraction, dynamic feature extraction, feature fusion, and posture discrimination steps. The final gesture action discrimination result is uploaded to other applications of the system, such as a financial service authorization system, to determine whether to accept the specific gesture operation in the video, thereby realizing remote authorization or secure transactions.

[0106] In an optional embodiment, the video feature extraction component includes a video frame division component, a video frame grouping component, a frame feature extraction component, and an intra-group feature fusion component.

[0107] The video frame division component is used to divide the video into a series of individual frames in chronological order for further processing and analysis. To ensure the security of the input video, this component first detects the encoding format of the input video, limiting only to AVI format, and the video frame rate must be a multiple of 10. After receiving the input video data, the video frame division component decodes the video data and divides it into a series of individual frames based on the video frame rate, and finally temporarily saves it in memory.

[0108] The video frame grouping component is configured to evenly divide the video frame set into groups and ensure that the total number of video frames in each group is equal and maintains continuity in time sequence. In actual operation, each group contains 20 video frames. During the division process, if there are less than 20 video frames left, the remaining video frames are expanded into a group by copying. At the same time, since adjacent video frames have high content similarity in time sequence, the grouping operation can divide similar video frames into the same group. The number of output video frame groups is denoted as B.

[0109] The frame feature extraction component is configured to convert video frames into visual features and aggregate and save them in groups. This component uses a large multi-modal pre-trained model as the core, which can extract semantic features from a group of video frames at the same time and save them as a single tensor matrix. In order to facilitate the processing of subsequent components, the frame feature extraction component needs to unify the size of the video frames. Specifically, first, the video frames are processed into a style with equal length and width using the 0 padding method, and then the video frames are scaled based on the bilinear interpolation method to a size of 224x224. Through the processing of the frame feature extraction component, a single video will be converted into 10 tensor matrices. In this invention, the processed tensor matrices are arranged into a set according to the time sequence relationship of the video frames, denoted as the frame feature set.

[0110] The intra-group feature fusion component is configured to post-process the frame feature set to improve the processing efficiency of subsequent components. Since the video frames belonging to the same group are adjacent in time sequence and similar in content, there must be information redundancy in each tensor matrix in the frame feature set. To remove redundant information and reduce the computational cost of subsequent components, the intra-group feature fusion component performs fusion operations on each tensor matrix of the frame feature set. The size of all tensor matrices conforms to N x seq x dim. Among them, N is the number of video frames contained in the group, seq is the sequence length of each frame feature, and dim is the length of the sequence unit vector. In this invention, N is 20, seq is 64, and dim is 768. After dimension transformation, the tensor matrix is converted into a style of seq x dim x N, and then an average pooling layer is used to process the last dimension. After fusion, B groups of features are spliced according to the time sequence relationship of the video frames, denoted as video group features. The size of the video group features is B x seq x dim.

[0111] The portrait comparison component includes a pooling fusion component, a face feature extraction component, and a similarity calculation component.

[0112] The pooling fusion component is used to extract the portrait information of the video subject from the video group features, which is used to confirm the user authentication. This component is implemented by using a commonly used convolutional neural network. First, the B group features are reduced in dimension by using the max pooling layer. The purpose of this step is to eliminate the time dimension of the video group features and preliminarily condense the portrait information in the video. Second, the split operation is performed on the sequence dimension of the video group features, so that the size of the video group features becomes The split features are used to fine-tune the portrait feature extraction network. This portrait feature extraction network is constructed based on the main body of a commonly used face recognition network, and the face of the video subject is located as the target of network fine-tuning. Finally, the fine-tuned portrait feature extraction network is used to extract the video portrait feature vector from the video group features.

[0113] The face feature extraction component is used to process the user image returned from the user authentication information library and output the authentication portrait feature vector of the image. First, the authentication face image of a specific user is retrieved from the user authentication information library according to the user information. Second, the large multi-modal pre-training model and the fine-tuned portrait feature extraction network are used jointly to convert the authentication face image into the authentication portrait feature vector. Finally, the face feature extraction component outputs the authentication portrait feature vector, and the size of the vector is consistent with the video portrait feature vector output by the pooling fusion component.

[0114] The similarity calculation component is used to calculate the similarity between the video portrait feature vector and the authentication portrait feature vector, so as to measure whether the video subject and the authentication user belong to the same person. Specifically, the vector cosine similarity is used as the criterion for judgment. In the invention, when the cosine similarity between the video portrait feature vector and the authentication portrait feature vector is higher than 0.8, it is determined that the video subject and the authentication user are the same.

[0115] The fake video discrimination component includes a fake video generation component, a video feature extraction component, a representative frame group extraction component, and a true or false prediction component.

[0116] The function of the fake video generation component is to generate fake videos according to static portrait data, so as to serve as training data for the fake video discrimination component. This component uses the currently commonly used video generation model as the main body. In the operation process, the component uses multiple open source portrait data sets as input, and uses the instruction to generate a specific action to constrain the output of the generation model. The remaining results are used as fake video data after removing some bad results manually.

[0117] The video feature extraction component in the fake video discrimination component is the same as the video feature extraction component described above, and will not be repeated here. Through the video feature extraction component, the real video data and the fake video data will be processed into real video group features and fake video group features, respectively.

[0118] The function of the representative frame group extraction component is to extract frame features representing video characteristics from video group features, thereby constructing a real and fake feature library for real and fake discrimination. The frames containing video pixel characteristics and reflecting the authenticity of the video are defined as representative frame groups. The purpose of extracting representative frame group features is to reduce the cost of constructing a real and fake feature library and calculating video authenticity. The number of representative frame groups is related to the number of video group features. The number of representative frame groups and the number of video group features are defined as G and B respectively. The number relationship between the two can be expressed by the following formula:

[0119]

[0120] In order to extract representative frame group features as evenly as possible while ensuring randomness, the following steps are used to extract representative frame groups and construct a real and fake feature library:

[0121] Step (1), use G to generate an initial seed x0 according to the following calculation formula:

[0122] x0=2039 G+1 mod(128·G+1)

[0123] Step (2), when selecting the ith representative frame group, the seed x i needs to be calculated first. Update the seed according to the following calculation formula:

[0124] x i =(2045·x i-1 +1)mod 2 20 ,G+1>i>0

[0125] Step (3), calculate the corresponding frame position y i using the current seed x i , and select it as the representative frame group:

[0126]

[0127] Step (4), repeat steps (2) and (3) until G representative frame groups are obtained.

[0128] Step (5), save the features of the representative frame groups into the real and fake feature library, and use the authenticity of the source video as the label of the features of the representative frame groups.

[0129] The representative frame group extraction component will also process the grouping features of the video to be verified, output the features of the representative frame groups of the video to be verified for authenticity verification. The purpose of this approach is to avoid implementing authenticity verification on complete video data, thereby improving model running efficiency.

[0130] The authenticity prediction component is used to determine the authenticity of the video to be verified. The main body of the component is composed of a linear network layer and a sigmoid activation function, and is trained using representative frame group feature data in the authenticity feature library. The loss function used during training is the binary cross-entropy function. After training is complete, the authenticity prediction component takes the representative frame group features of the video to be verified as input and outputs the authenticity determination result of the video to be verified as real or fake. Specifically, each representative frame group feature of the video to be predicted will be mapped to a two-dimensional vector through the linear network layer and the activation function, representing the prediction probability of the frame being real or fake. The relatively larger one of the two probabilities determines the authenticity determination result of the frame. If more than 75% of the representative frame groups of the video to be predicted are determined to be real, the authenticity prediction component finally determines that the video to be predicted is a real video, otherwise it is a fake video.

[0131] The gesture recognition component includes a uniform frame extraction component, a contour feature extraction component, a dynamic feature extraction component, a feature fusion component, and a posture determination component. In the present application, the processing process of the contour feature extraction component, the dynamic feature extraction component, and the feature fusion component on the video group features is defined as the gesture feature extraction process. In the present application, the gesture feature extraction process will be repeated 4 times. After processing is complete, the fusion features will be output to the posture determination component to realize prediction of the video gesture.

[0132] The uniform frame extraction component is used to process the video group features, so that the processed video group features have uniform output specifications, facilitating the processing of subsequent components. Specifically, the uniform frame extraction component extracts frame features at uniform intervals according to the number of video group features. ′ , and outputs after reassembling the frame features.

[0133] The contour feature extraction component focuses on obtaining edge information in the video frame, thereby positioning the body of the video subject and improving the accuracy of gesture recognition. First, a linear network layer is used to map the time sequence dimension of the video group features, reducing it from B ′ to 1, and the result is denoted as V A . Second, the features are refined by the following calculation method to obtain the feature V B :

[0134]

[0135] where l1, l2, and l3 are linear network layer mappings, and T represents matrix transposition. Finally, the feature V B is mapped to the frequency domain, the high-frequency features are extracted using the Gaussian filtering method, then the high-frequency features are restored to the real domain and superimposed on V B , and the output is defined as the human contour feature.

[0136] The dynamic feature extraction component focuses on obtaining the action changes of the video subject, and realizes the identification of the subject gesture from the overall video perspective. First, two different fixed matrices W1 and W2 are used to transform the video grouping features, realize the dislocation of the video frame features, and record the results as H1 and H2. Among them, W1 and W2 are both B ′ order unit matrix and a B ′ dimensional zero vector are spliced, and the difference lies in the starting position or the ending position of the unit matrix. Then, H1 and H2 are subtracted, and a linear network layer and a ReLU activation function are used for mapping, so as to obtain the posture information difference of different frames. Finally, a linear network layer is used to map the posture information difference to B order, and is superimposed on the original features. The output is defined as the dynamic feature of the person.

[0137] The feature fusion component is used to fuse the person contour feature and the person dynamic feature, so as to extract information helpful for gesture judgment. Specifically, the component adds the person contour feature and the person dynamic feature. Then, a linear network layer is used to transform the above-mentioned features, and a ReLU activation function is used for processing. Finally, the output feature is sent to a new gesture feature extraction process for processing. The output of the component is defined as the fusion feature.

[0138] The posture judgment component is used to judge the gesture category according to the fusion feature. Specifically, the component takes all vectors of the fusion feature located at the first unit of the sequence, uses two linear network layers to perform linear transformation, and normalizes the calculation result through a softmax function. The above steps will output a probability vector with a length of 10, which is equal to the number of gesture categories in the training data. Finally, the gesture prediction result is equal to the gesture category corresponding to the maximum value in the probability vector.

[0139] The comprehensive judgment component integrates the portrait comparison result, the authenticity judgment result and the gesture prediction result, and outputs the judgment result of the final input video. Only when the portrait comparison result and the authenticity judgment result are true, the application accepts the input video and outputs the gesture prediction result. In other cases, the application will reject the input video.

[0140] A video gesture safety recognition technology based on deep learning, the steps are as follows:

[0141] Step (1), the user shoots a video containing a specific gesture through a photography terminal, and uploads the system;

[0142] Step (2), the video is converted into video grouping features by the video feature extraction component;

[0143] Step (3), the portrait comparison component pulls the user authentication portrait from the user authentication information library, judges the user's identity according to the video grouping features, and outputs the portrait comparison result;

[0144] Step (4), the fake video discrimination component discriminates the authenticity of the video according to the video grouping features, and outputs the authenticity discrimination result;

[0145] Step (5), the gesture recognition component recognizes specific gestures in the video according to the video grouping features, and outputs the gesture prediction result;

[0146] Step (6), the comprehensive discrimination component analyzes whether to accept the video uploaded by the user and the specific gesture contained in the video according to the portrait comparison result, the authenticity discrimination result and the gesture prediction result. The above discrimination results will be uploaded to other applications of the system.

[0147] The video processing method provided by the embodiment of the application extracts features from the target video collected during the target object handling financial business process through a feature extraction algorithm to obtain initial feature information; performs identity verification on the target object based on the initial feature information through a portrait comparison algorithm to obtain a verification result; in the case that the verification result represents that the identity verification is passed, identifies the authenticity of the target video based on the initial feature information through a forgery discrimination algorithm to obtain a first identification result; in the case that the first identification result represents that the target video is a non-forged video, performs gesture recognition based on the initial feature information through a gesture recognition algorithm to obtain a second identification result, wherein the second identification result is used to determine whether there is an abnormality in the process of the target object handling financial business, thereby solving the technical problem in the related art that the video gesture recognition technology lacks effective resistance to AI generated video attacks, resulting in relatively low accuracy of video gesture recognition.

[0148] In the present scheme, after the initial feature information is extracted, portrait comparison is first performed to ensure that the target object in the video is consistent with the authenticated user. Only in the case that the identity of the target object is confirmed to be correct, will the subsequent forgery discrimination and gesture recognition be continued. The introduction of the forgery discrimination algorithm can accurately identify the authenticity of the video. Even if the attacker uses a highly simulated AI generated video, abnormal features that may exist in the video can be identified, such as unnatural smoothness of the picture, incoordination of micro-expressions, or abnormal repetition of gestures, thereby preventing the false passage of forged content and improving the security of identification. By introducing the multi-stage process of identity verification, forgery video recognition and gesture behavior pattern analysis, the security and accuracy of the video gesture recognition technology are significantly enhanced, especially the resistance to AI generated video attacks, providing a more reliable and secure solution for online financial service authorization scenarios. Through the above technology, illegal transactions and operations can be effectively avoided, the safety of user funds is protected, and at the same time, barrier-free financial service experience is provided for the elderly and hearing or language impaired groups.

[0149] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0150] Example 2

[0151] The embodiment of the present application further provides a video processing device. It should be noted that the video processing device of the embodiment of the present application can be used to execute the video processing method provided in the embodiment of the present application. The video processing device provided in the embodiment of the present application is introduced below.

[0152] According to an embodiment of the present application, a video processing device for implementing the above-mentioned video processing method is also provided. Figure 8 As shown, the device includes: an extraction unit 801, a verification unit 802, a first recognition unit 803 and a second recognition unit 804.

[0153] The extraction unit 801 is configured to extract features from a target video collected during the target object's financial business processing using a feature extraction algorithm to obtain initial feature information;

[0154] The verification unit 802 is configured to authenticate the target object based on the initial feature information using a portrait comparison algorithm to obtain a verification result;

[0155] The first identification unit 803 is configured to identify the authenticity of the target video based on the initial feature information using a forgery discrimination algorithm when the verification result indicates that the identity authentication is passed, thereby obtaining a first identification result;

[0156] The second recognition unit 804 is used to perform gesture recognition based on the initial feature information using a gesture recognition algorithm to obtain a second recognition result when the first recognition result indicates that the target video is not a forged video, wherein the second recognition result is used to determine whether there is any abnormality in the process of the target object handling financial business.

[0157] The video processing device provided in the embodiment of the application extracts the target video collected in the process of the target object handling the financial business through a feature extraction algorithm by the extraction unit 801 to obtain initial feature information; the verification unit 802 performs identity verification on the target object based on the initial feature information through a portrait comparison algorithm to obtain a verification result; the first recognition unit 803 performs identification on the authenticity of the target video based on the initial feature information through a forgery discrimination algorithm in the case that the verification result represents that the identity verification is passed to obtain a first recognition result; and the second recognition unit 804 performs gesture recognition based on the initial feature information through a gesture recognition algorithm in the case that the first recognition result represents that the target video is a non-forged video to obtain a second recognition result, wherein whether the process of the target object handling the financial business is abnormal is determined through the second recognition result, thereby solving the technical problem that the video gesture recognition technology in the related art lacks effective resistance to AI generated video attacks, resulting in relatively low accuracy of video gesture recognition.

[0158] In the present scheme, after the initial feature information is extracted, portrait comparison is first performed to ensure that the target object in the video is consistent with the authenticated user. Only in the case that the identity of the target object is confirmed to be correct, the subsequent forgery discrimination and gesture recognition will be continued. The introduction of the forgery discrimination algorithm can accurately identify the authenticity of the video. Even if the attacker uses a highly simulated AI generated video, the abnormal features that may exist in the video can be identified, such as unnatural smoothness of the picture, incoordination of micro-expressions, or abnormal repetition of gestures, thereby preventing the false passage of the forged content and improving the security of the recognition. By introducing the multi-stage process of identity verification, forged video recognition and gesture behavior pattern analysis, the security and accuracy of the video gesture recognition technology are significantly enhanced, especially the resistance to AI generated video attacks, which provides a more reliable and secure solution for online financial service authorization scenarios. Through the above technology, illegal transactions and operations can be effectively avoided, the safety of user funds is protected, and at the same time, barrier-free financial service experience is provided for the elderly and hearing or language impaired groups.

[0159] Optionally, in the video processing device provided in the embodiment of the application, the extraction unit comprises: a splitting subunit, configured to split the target video to obtain a plurality of video frames; a grouping subunit, configured to group the plurality of video frames to obtain a plurality of video frame groups; a first extraction subunit, configured to extract features of each video frame in each video frame group to obtain a tensor matrix corresponding to each video frame group; and a first determination subunit, configured to obtain the initial feature information according to the plurality of tensor matrices.

[0160] Optionally, in the video processing apparatus provided in the embodiments of the present application, the determining subunit comprises: a transformation module, configured to perform dimension transformation on each tensor matrix to obtain a plurality of transformed tensor matrices; a first processing module, configured to perform dimension reduction processing on the last dimension of each transformed tensor matrix to obtain a plurality of processed tensor matrices; and a splicing module, configured to perform splicing processing on the plurality of processed tensor matrices according to a time sequence relationship to obtain initial feature information.

[0161] Optionally, in the video processing apparatus provided in the embodiments of the present application, the verifying unit comprises: a dimension reduction subunit, configured to perform dimension reduction processing on the time sequence dimension in the initial feature information to obtain processed initial feature information; a second extraction subunit, configured to perform feature extraction on the processed initial feature information to obtain portrait feature information; and an acquisition subunit, configured to acquire real portrait feature information of the target object, and perform identity verification according to the portrait feature information and the real portrait feature information to obtain a verification result.

[0162] Optionally, in the video processing apparatus provided in the embodiments of the present application, the first identifying unit comprises: a third extraction subunit, configured to perform feature extraction on the initial feature information according to a first quantity to obtain first feature information, wherein the first quantity is calculated based on the grouping quantity, and the first feature information is composed of feature information corresponding to the first quantity of first video frame groups; a first identifying subunit, configured to identify the authenticity of each first video frame group according to the feature information corresponding to each first video frame group to obtain an identification result of each first video frame group; and a second determining subunit, configured to obtain a first identification result according to the identification result of each first video frame group.

[0163] Optionally, in the video processing apparatus provided in the embodiments of the present application, the third extraction subunit comprises: a first calculation module, configured to calculate the first quantity to obtain an initial seed; a second calculation module, configured to calculate the initial seed to obtain a current seed, and calculate the current seed to obtain a target serial number; a first determining module, configured to determine the first quantity of first video frame groups from the plurality of video frame groups according to the target serial number; and a second determining module, configured to obtain the first feature information according to the feature information of the first video frame groups.

[0164] Optionally, in the video processing apparatus provided by the embodiment of the present application, the second identifying unit comprises: a sampling subunit, configured to uniformly sample the initial feature information to obtain second feature information, wherein the second feature information is composed of feature information corresponding to a second number of second video frame groups; a fourth extraction subunit, configured to perform feature extraction on the contour information in the second feature information to obtain a human contour feature; a fifth extraction subunit, configured to perform feature extraction on the dynamic information in the second feature information to obtain a human dynamic feature; a fusion subunit, configured to fuse the human contour feature and the human dynamic feature to obtain fused feature information; and a second identifying subunit, configured to perform gesture recognition according to the fused feature information to obtain a second recognition result.

[0165] Optionally, in the video processing apparatus provided by the embodiment of the present application, the fourth extraction subunit comprises: a dimension reduction module, configured to perform dimension reduction processing on a time sequence dimension of the second feature information to obtain third feature information; a second processing module, configured to process the third feature information through a plurality of linear network layers to obtain fourth feature information; a conversion module, configured to perform frequency domain conversion on the fourth feature information, and perform high-frequency feature extraction on the converted feature information to obtain fifth feature information; and a third determining module, configured to obtain the human contour feature according to the fifth feature information and the fourth feature information.

[0166] Optionally, in the video processing apparatus provided by the embodiment of the present application, the fifth extraction subunit comprises: a first calculation module, configured to perform product calculation on the first matrix and the second feature information to obtain a first calculation result; a second calculation module, configured to perform product calculation on the second matrix and the second feature information to obtain a second calculation result; a third calculation module, configured to calculate a difference between the first calculation result and the second calculation result to obtain a gesture information difference; and a fifth determining module, configured to obtain the human dynamic feature according to the gesture information difference.

[0167] It should be noted that the extraction unit 801, the verification unit 802, the first identifying unit 803 and the second identifying unit 804 correspond to steps S201 to S204 in Embodiment One, and the four units have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment One. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in the memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a, 102b, …, 102n), and the above-mentioned modules can also be a part of the device and can run in the computer terminal 10 provided in Embodiment One.

[0168] Embodiment 3

[0169] The embodiments of the present application can provide an electronic device, Figure 9is a structural block diagram of an electronic device according to an embodiment of the present application. As shown in Figure 9 the electronic device can include one or more (only one is shown in the figure) processors 902, a memory 904, a storage controller, and a peripheral interface, wherein the peripheral interface is connected with a radio frequency module, an audio module, and a display. Figure 9

[0170] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functions and data processing by running the software programs and modules stored in the memory, i.e., implements the above-mentioned methods. The memory can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the terminal through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0171] The processor can call information and application programs stored in the memory through the transmission device to execute the steps of the video processing method provided in Embodiment 1.

[0172] Those skilled in the art can understand that Figure 9 The structure shown in the figure is only schematic, and the electronic device can also be a terminal device such as a smart phone, a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, etc. Figure 9 It does not limit the structure of the above-mentioned electronic device. For example, the electronic device can further include more or less components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure. Figure 9 Figure 9 Those skilled in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by programs instructing the relevant hardware of the terminal device, and the programs can be stored in a computer readable storage medium, which can include a flash disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0173] Embodiment 4

[0174] Embodiment 4

[0175] ​​The embodiment of the present application further provides a computer readable storage medium. Optionally, in the embodiment, the storage medium can be used to save the program code executed by the video processing method provided in the first embodiment.

[0176] Optionally, in the embodiment, the storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0177] The present application further provides a computer program product, which is adapted to execute the steps of the video processing method when executed on a data processing device.

[0178] The serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0179] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0180] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit embodiment described above is only schematic. For example, the division of the units is only a logical function division. There can be another division manner for actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, units or modules, and can be electrical or other forms.

[0181] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment.

[0182] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of software functional unit.

[0183] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0184] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A video processing method, characterized in that: include: The feature extraction algorithm is used to extract features from the target video collected during the target object's financial business process to obtain initial feature information; Performing identity verification on the target object based on the initial feature information using a portrait comparison algorithm to obtain a verification result; If the verification result indicates that the identity authentication is passed, the authenticity of the target video is identified based on the initial feature information using a forgery discrimination algorithm to obtain a first identification result; When the first recognition result indicates that the target video is not a forged video, gesture recognition is performed based on the initial feature information using a gesture recognition algorithm to obtain a second recognition result, wherein the second recognition result is used to determine whether there is any abnormality in the process of the target object handling financial business.

2. The method according to claim 1, characterized in that The feature extraction algorithm is used to extract features from the target video collected during the target object's financial business process, and the initial feature information obtained includes: Splitting the target video to obtain multiple video frames; Grouping the multiple video frames to obtain multiple video frame groups; Perform feature extraction on each video frame in each video frame group to obtain a tensor matrix corresponding to each video frame group; The initial feature information is obtained according to a plurality of tensor matrices.

3. The method according to claim 2, characterized in that Obtaining the initial feature information based on multiple tensor matrices includes: Perform dimension transformation on each tensor matrix to obtain multiple transformed tensor matrices; Perform dimensionality reduction processing on the last dimension of each transformed tensor matrix to obtain multiple processed tensor matrices; The multiple processed tensor matrices are spliced ​​together according to a time sequence relationship to obtain the initial feature information.

4. The method according to claim 1, wherein The target object is authenticated based on the initial feature information using a portrait comparison algorithm, and the obtained verification result includes: Performing dimensionality reduction processing on the time series dimension in the initial feature information to obtain processed initial feature information; Performing feature extraction on the processed initial feature information to obtain portrait feature information; The real portrait feature information of the target object is obtained, and identity verification is performed based on the portrait feature information and the real portrait feature information to obtain the verification result.

5. The method according to claim 2, characterized in that The authenticity of the target video is identified based on the initial feature information using a forgery identification algorithm, and a first identification result is obtained, including: Extracting features from the initial feature information according to a first quantity to obtain first feature information, wherein the first quantity is calculated based on a number of groups, and the first feature information is composed of feature information corresponding to a first number of first video frame groups; Identifying the authenticity of each first video frame group based on feature information corresponding to each first video frame group to obtain an identification result for each first video frame group; The first recognition result is obtained according to the recognition result of each first video frame group.

6. The method according to claim 5, characterized in that Extracting features from the initial feature information according to the first quantity to obtain first feature information includes: Calculating the first quantity to obtain an initial seed; Calculating the initial seed to obtain a current seed, and calculating the current seed to obtain a target sequence number; Determining the first number of first video frame groups from the plurality of video frame groups according to the target sequence number; The first feature information is obtained according to the feature information of the first video frame group.

7. The method according to claim 2, characterized in that Performing gesture recognition based on the initial feature information using a gesture recognition algorithm, obtaining a second recognition result includes: uniformly sampling the initial feature information to obtain second feature information, wherein the second feature information is composed of feature information corresponding to a second number of second video frame groups; Extracting features from the contour information in the second feature information to obtain a character contour feature; Extracting features from the dynamic information in the second feature information to obtain dynamic features of the character; fusing the character outline features and the character dynamic features to obtain fused feature information; Perform gesture recognition based on the fused feature information to obtain the second recognition result.

8. The method according to claim 7, characterized in that Extracting the contour information in the second feature information to obtain the character contour features includes: Performing dimensionality reduction processing on the time series dimension of the second feature information to obtain third feature information; Processing the third feature information through multiple linear network layers to obtain fourth feature information; Performing frequency domain conversion on the fourth feature information, and performing high-frequency feature extraction on the converted feature information to obtain fifth feature information; The character outline feature is obtained based on the fifth feature information and the fourth feature information.

9. The method according to claim 7, characterized in that Extracting the dynamic information in the second feature information to obtain the character dynamic features includes: Performing a product calculation on the first matrix and the second characteristic information to obtain a first calculation result; Performing a product calculation on the second matrix and the second characteristic information to obtain a second calculation result; Calculating a difference between the first calculation result and the second calculation result to obtain a posture information difference; The character dynamic feature is obtained according to the posture information difference.

10. A video processing device, characterized in that: include: An extraction unit is used to extract features from a target video collected during the target object's financial business processing using a feature extraction algorithm to obtain initial feature information; A verification unit, configured to authenticate the target object based on the initial feature information using a portrait comparison algorithm to obtain a verification result; a first identification unit configured to, when the verification result indicates that the identity authentication is passed, identify the authenticity of the target video based on the initial feature information using a forgery discrimination algorithm to obtain a first identification result; A second recognition unit is configured to, when the first recognition result indicates that the target video is not a forged video, perform gesture recognition based on the initial feature information through a gesture recognition algorithm to obtain a second recognition result, wherein the second recognition result is used to determine whether there is any abnormality in the process of the target object handling financial business.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the video processing method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: include: a memory storing an executable program; A processor is used to run the program, wherein the program executes the video processing method according to any one of claims 1 to 9 when running.

13. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the video processing method according to any one of claims 1 to 9 are implemented.

Images