Image processing method, device, electronic device and storage medium

Abstract

The present disclosure provides an image processing method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology, specifically to the technical fields of deep learning, computer vision, etc., and can be applied to scenarios such as face image processing. The method includes: acquiring multiple frames of video, and determining multiple spatial related information corresponding to the multiple frames of images, and then determining multiple time series information corresponding to the multiple frames of images, and generating multiple predicted depth images corresponding to the corresponding multiple frames of images according to the multiple time series information, and performing liveness detection according to the multiple spatial related information and the multiple predicted depth images. As a result, the accuracy and liveness detection effect of image-based liveness detection can be effectively improved, and the generalization of the image processing method can be effectively improved, which is conducive to the deployment and application of the image processing method in liveness detection scenarios.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as deep learning and computer vision, and can be applied to face image processing scenarios, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] In the related art, the image-based liveness detection method has poor resolution when distinguishing between living and non-living objects, resulting in poor liveness detection effect, which is not conducive to the deployment and application of the liveness detection method. Summary of the invention

[0004] The present disclosure provides an image processing method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0005] According to a first aspect of the present disclosure, there is provided an image processing method, comprising: acquiring multiple frames of images of a video; determining multiple spatially related information respectively corresponding to the multiple frames of images; determining multiple timing information corresponding to the multiple frames of images; generating multiple predicted depth images corresponding to the multiple frames of images respectively according to the multiple timing information; and performing living body detection according to the multiple spatially related information and the multiple predicted depth images.

[0006] According to a second aspect of the present disclosure, an image processing device is provided, including: an acquisition module for acquiring multiple frames of images of a video; a first determination module for determining multiple spatial related information respectively corresponding to the multiple frames of images; a second determination module for determining multiple timing information corresponding to the multiple frames of images; a generation module for generating multiple predicted depth images corresponding to the corresponding multiple frames of images according to the multiple timing information; and a detection module for performing liveness detection based on the multiple spatial related information and the multiple predicted depth images.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image processing method of the first aspect of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the image processing method of the first aspect of the present disclosure.

[0009] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the image processing method of the first aspect of the present disclosure when executed by a processor.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0013] Figure 2 is a schematic diagram of the difference in spatially related information between living body and non-living body images according to an embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;

[0015] Figure 4 is a schematic diagram of a process of feature extraction based on a Transformer network according to an embodiment of the present disclosure;

[0016] Figure 5 is a structural schematic diagram of an image processing model according to an embodiment of the present disclosure;

[0017] Figure 6 is a schematic diagram of dynamic window drift of a PTA module according to an embodiment of the present disclosure;

[0018] Figure 7 is a schematic structural diagram of a TDA module according to an embodiment of the present disclosure;

[0019] Figure 8 is a schematic diagram according to a third embodiment of the present disclosure;

[0020] Fig. 9 is a schematic diagram of a liveness detection score of an abnormal frame image implemented according to the present disclosure;

[0021] Fig.10 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0022] Fig.11 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0023] Fig.12 A schematic block diagram of an example electronic device for implementing the image processing method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure.

[0026] It should be noted that the executor of the image processing method of this embodiment is an image processing device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include but is not limited to a terminal, a server, etc.

[0027] The embodiments of the present disclosure relate to the field of artificial intelligence technology, specifically to technical fields such as computer vision and deep learning, and can be applied to scenarios such as face image processing.

[0028] Among them, Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0029] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images, and sounds.

[0030] Computer vision refers to the use of cameras and computers to replace the human eye to identify, track and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection.

[0031] Face image processing refers to the use of computer technology to process input face images or video streams, extract face image information contained in the image, automatically detect and track faces in the image, and then process the detected face images using a series of related technologies.

[0032] When the image processing method provided by the present invention is applied to scenarios such as face image processing, it can effectively improve the accuracy and liveness detection effect in scenarios such as face image processing, effectively improve the generalization of the face image processing method, and effectively improve the application effect of liveness detection based on face images.

[0033] Among them, the above-mentioned facial image data, including video streams of facial images, and facial image information are all obtained in compliance with relevant laws and regulations. For example, the aforementioned data may come from public data sets, or may be obtained from authorized agencies after authorization by relevant agencies. The acquisition process complies with the relevant laws and regulations and does not violate public order and good customs.

[0034] It should be noted that the above-mentioned facial data, including the video stream of facial images, and the facial image information are not data obtained for a specific user and cannot reflect the personal information of a specific user.

[0035] In the embodiments of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the process are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0036] like Figure 1 As shown, the image processing method comprises:

[0037] S101: Acquire multiple frames of video images.

[0038] The video may include multiple video frames, which may be referred to as multiple-frame images. The multiple-frame images constitute a video segment or a complete video.

[0039] When acquiring multiple frames of images of a video, the embodiments of the present disclosure may use a camera device with a shooting function, such as a mobile phone or a camera, to acquire a video stream, and parse and process the acquired video stream to acquire multiple frames of images of the video, and then execute the image processing method described in the embodiments of the present disclosure for the multiple frames of images of the video. For details, please refer to the subsequent embodiments.

[0040] It should be noted that the multiple frames of images acquired in the embodiments of the present disclosure are not images acquired for a specific user, and they cannot reflect the personal information of a specific user. Moreover, the aforementioned images are acquired after authorization by the relevant user, and the acquisition process complies with the relevant laws and regulations and does not violate public order and good morals.

[0041] S102: Determine a plurality of space-related information respectively corresponding to a plurality of frames of images.

[0042] Among them, multiple frame images can have multiple information related to spatial dimensions, which can be called spatial related information, and the spatial related information can be specifically, for example, spatial features (for example, depth information, position information, etc. of each pixel in the image in the spatial dimension, or it can also be the spatial information of the scene objects represented by the image in the scene, etc.), and there is no limitation on this.

[0043] In some embodiments, a convolutional neural network can be combined to determine multiple spatially related information corresponding to multiple frames of images. For example, multiple frames of images can be input into the convolutional neural network respectively to obtain multiple feature maps corresponding to the multiple frames of images, and the multiple feature maps can be used as multiple spatially related information corresponding to the multiple frames of images.

[0044] In other embodiments, the multiple spatially related information corresponding to the multiple frames of images can be determined by using a feature extraction algorithm to determine the multiple spatial features corresponding to the multiple frames of images, and using the aforementioned multiple spatial features as the multiple spatial features corresponding to the multiple frames of images. Alternatively, any other possible method can be used to determine the multiple spatially related information corresponding to the multiple frames of images, without limitation.

[0045] S103: Determine a plurality of timing information corresponding to the plurality of frames of images.

[0046] Among them, multiple frame images can have multiple information related to the time dimension, which can be called timing information, and the timing information can be specifically, for example, a time feature (for example, the frame time of each frame image in an image sequence composed of multiple frame images, or the acquisition time corresponding to each frame image, or the time difference between the acquisition time corresponding to each frame image and a marked time, etc.), and there is no limitation on this.

[0047] In some embodiments, a recurrent neural network can be combined to determine multiple timing information corresponding to multiple frame images. For example, multiple frame images can be input into the recurrent neural network separately to perform timing operations on the multiple frame images to extract the time features of the multiple frame images, and the extracted time features can be used as multiple timing information corresponding to the multiple frame images. There is no limitation to this.

[0048] In other embodiments, a pre-trained timing information extraction model can be combined to determine multiple timing information corresponding to multiple frames of images. For example, multiple frames of images can be input into the pre-trained timing information extraction model respectively to obtain multiple timing information corresponding to the multiple frames of images output by the pre-trained timing information extraction model. Alternatively, any other possible method can be used to determine multiple timing information corresponding to the multiple frames of images, without limitation.

[0049] S104: Generate a plurality of predicted depth images corresponding to the corresponding plurality of frame images respectively according to the plurality of time series information.

[0050] After determining a plurality of timing information corresponding to a plurality of frame images, the embodiment of the present disclosure can generate a plurality of depth images corresponding to the corresponding plurality of frame images according to the plurality of timing information, and the depth images can be referred to as predicted depth images.

[0051] In some embodiments, based on multiple timing information, multiple predicted depth images corresponding to corresponding multiple frame images are generated respectively. Depth recognition may be performed on the corresponding multiple frame images respectively based on multiple timing information, and a depth image is formed according to the recognized depth. The depth image can be called a predicted depth image.

[0052] In other embodiments, multiple predicted depth images corresponding to the corresponding multi-frame images are generated respectively based on multiple time series information. It is also possible that multiple predicted depth images corresponding to the corresponding multi-frame images are generated based on multiple time series information in combination with a convolutional neural network. For example, multiple time series information and multi-frame images can be used as inputs to a convolutional neural network to obtain multiple predicted depth images output by the convolutional neural network. Alternatively, any other possible method can be used to execute the step of generating multiple predicted depth images corresponding to the corresponding multi-frame images respectively based on multiple time series information, without limitation.

[0053] S105: Perform living body detection according to the multiple spatial related information and the multiple predicted depth images.

[0054] Among them, liveness detection can be used to distinguish between living and non-living objects. It is a detection method that determines the true physiological characteristics of an object in some identity authentication scenarios.

[0055] In some embodiments, liveness detection is performed based on multiple spatially related information and multiple predicted depth images. The multiple spatially related information and multiple predicted depth images can be input into a pre-trained liveness detection model to obtain the liveness detection result output by the pre-trained liveness detection model. There is no limitation on this.

[0056] In the present disclosure, Figure 2 As shown, Figure 21 is a schematic diagram of the difference in spatially related information between living and non-living images according to an embodiment of the present disclosure. There are differences in spatially related information between living and non-living images. For example, the way of presenting non-living objects often exists on the same medium. The presentation medium of non-living objects presented in a printed manner is paper, the medium presented in a video playback manner is a screen, and the presentation medium of non-living objects with a three-dimensional headgear is a silicone headgear. These same presentation media have stronger spatial correlation with respect to the skin of living objects. In addition, non-living objects presented in a two-dimensional form account for a large proportion of non-living objects, and do not have additional depth information compared to living objects presented in a three-dimensional form.

[0057] It should be noted that Figure 2 The facial images involved are not facial images obtained for a specific user, and they cannot reflect the personal information of a specific user. In addition, the aforementioned facial images are obtained after authorization by the relevant user, and the acquisition process complies with the relevant laws and regulations and does not violate public order and good customs.

[0058] Therefore, in some embodiments, liveness detection is performed based on multiple spatially related information and multiple predicted depth images, which can be to determine the spatially related information of the liveness image and the predicted depth image corresponding to the liveness image, and then compare the multiple spatially related information and the multiple predicted depth images corresponding to the multiple frame images with the spatially related information of the liveness image and the predicted depth image corresponding to the liveness image, and combine the multiple spatially related information and the multiple predicted depth images corresponding to the multiple frame images, respectively, with the set conditions between the spatially related information of the liveness image and the predicted depth image corresponding to the liveness image (the set conditions can be adaptively configured according to the actual liveness detection scenario, and there is no restriction on this), to obtain the corresponding liveness detection result.

[0059] Alternatively, any other possible manner may be used to perform living body detection based on multiple spatial related information and multiple predicted depth images, and there is no limitation to this.

[0060] In this embodiment, multiple frames of video are acquired, multiple spatial related information corresponding to the multiple frames are determined, multiple time series information corresponding to the multiple frames are determined, and multiple predicted depth images corresponding to the corresponding multiple frames are generated according to the multiple time series information, and liveness detection is performed according to the multiple spatial related information and the multiple predicted depth images. This can effectively improve the accuracy and effect of liveness detection, effectively improve the generalization of the image processing method, and thus facilitate the deployment and application of the image processing method in liveness detection scenarios.

[0061] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure.

[0062] like Figure 3As shown, the image processing method comprises:

[0063] S301: Acquire multiple frames of video images.

[0064] The description of S301 can be specifically referred to the above embodiment, which will not be repeated here.

[0065] S302: Determine a plurality of correlation features corresponding to the plurality of frames of images respectively.

[0066] Among them, the features used to describe multiple correlations corresponding to multiple frames of images can be called correlation features, and the correlations corresponding to the images can be specifically, for example, image similarity, image vector cosine, etc., without limitation.

[0067] In some embodiments, determining multiple correlation features corresponding to multiple image frames can be performed by analyzing the multiple image frames to determine image representation vectors corresponding to the multiple image frames, and then determining vector cosines of the image representation vectors corresponding to the multiple image frames, and using them as multiple correlation features corresponding to the multiple image frames. Alternatively, the similarity between the multiple image frames can be determined, and the similarity can be used as multiple correlation features corresponding to the multiple image frames. Alternatively, any other possible method can be used to determine multiple correlation features corresponding to the multiple image frames, and there is no limitation on this.

[0068] Optionally, in some embodiments, determining multiple correlation features corresponding to multiple frames of images may be by segmenting the image to obtain multiple image segments, determining multiple encoding information corresponding to the multiple image segments, and determining multiple correlations corresponding to the multiple encoding information, and using the multiple correlations as multiple correlation features of the corresponding multiple frames of images, wherein the correlation describes the correlation between the corresponding encoding information and other encoding information. Since the image is segmented and then the multiple encoding information corresponding to the multiple image segments are determined, the contextual semantic information of the image can be fully represented. In addition, by encoding the multiple image segments, image segment serialization can be achieved, so that when determining the correlation feature based on the encoding information, the influence of image noise on the correlation feature can be avoided, thereby effectively improving the determination effect and efficiency of the correlation feature.

[0069] In the embodiment of the present disclosure, multiple frames of images may be segmented to obtain multiple segmented images, which may be referred to as image segments.

[0070] For example, assuming that the pixel value of one frame of the image is 224x224, the image can be segmented by dividing the image according to the pixel value. For example, the image can be divided into 196 image segments of size 16x16, and there is no limitation to this.

[0071] In the embodiment of the present disclosure, after obtaining a plurality of image segments, the plurality of image segments may be respectively coded, and information of the plurality of image segments after the coding process may be referred to as coding information.

[0072] Among the multiple coding information, the coding information whose relevance is currently to be determined can be called corresponding coding information. Correspondingly, among the multiple coding information, the coding information other than the corresponding coding information can be called other coding information.

[0073] In some embodiments, determining multiple encoding information corresponding to multiple image fragments can be done by inputting the multiple image fragments into a mapping network, encoding the multiple image fragments through the mapping network, and outputting multiple encoding information corresponding to the multiple image fragments, thereby encoding the multiple image fragments into a sequence.

[0074] For example, after dividing a frame of image with a pixel value of 224x224 into 196 image segments of size 16x16, the 196 image segments can be input into the mapping network, and each segment can be mapped into a 384-dimensional vector, which can be used as multiple encoding information corresponding to multiple image segments, without any limitation.

[0075] Among them, the multiple coding information corresponding to the image segment can respectively correspond to multiple correlations, and the correlation can be used to describe the correlation between the corresponding coding information and other coding information. The correlation can be specifically, for example, the similarity between the multiple coding information, the Euclidean distance between the multiple coding information, or any other possible form of correlation, which is not limited.

[0076] In the disclosed embodiment, multiple correlations corresponding to multiple coding information can be determined by combining the self-attention structure of the Transformer network (the self-attention structure is an important component of the Transformer network), continuously calculating the correlations between the corresponding coding information of different image segments, and using the multiple correlations as multiple correlation features of the corresponding multi-frame images.

[0077] S303: Extract features from multiple frames of images respectively to obtain multiple initial image features.

[0078] Among them, the multiple frames of images may have corresponding multiple image features, and among the multiple image features, the unprocessed image features may be referred to as initial image features.

[0079] For example, a Transformer network can be used to extract features from multiple frames of images to obtain multiple initial image features, such as Figure 4As shown, Figure 4 is a schematic diagram of the feature extraction process based on the Transformer network according to an embodiment of the present disclosure, that is, multiple frame images can be input into the following Figure 4 In the Transformer network shown, multiple initial image features are obtained as output by the Transformer network.

[0080] It should be noted that Figure 4 The facial images involved are not facial images obtained for a specific user, and they cannot reflect the personal information of a specific user. In addition, the aforementioned facial images are obtained after authorization by the relevant user, and the acquisition process complies with the relevant laws and regulations and does not violate public order and good customs.

[0081] In the embodiment of the present disclosure, the multiple frames of images may include multiple image segments. Accordingly, feature extraction is performed on the multiple frames of images respectively to obtain multiple initial image features. Feature extraction may be performed on multiple image segments of the multiple frames of images respectively. For example, the multiple image segments corresponding to the multiple frames of images may be input into the following example: Figure 4 In the Transformer network shown, multiple initial image features output by the Transformer network are obtained, and there is no restriction on this.

[0082] S304: Processing the corresponding multiple initial image features respectively according to the multiple correlation features to obtain multiple spatial correlation information.

[0083] After performing feature extraction on multiple frames of images respectively to obtain multiple initial image features, the disclosed embodiment can process the corresponding multiple initial image features according to multiple correlation features to obtain multiple spatially related information. Since the Transformer network is combined to determine the initial image features of the multiple frames of images, and the corresponding multiple initial image features are optimized using multiple correlation features corresponding to the multiple frames of images, not only can the comprehensiveness and reliability of the identified initial image features be effectively improved, but also the spatially related information obtained by processing the initial image features based on the correlation features can be more accurate and have a better representation effect, and can represent more complete video semantic information carried in the video, thereby ensuring the accuracy of subsequent living body detection.

[0084] For example, the corresponding multiple initial image features can be processed separately according to multiple correlation features. The corresponding multiple initial image features can be enhanced according to the multiple correlation features, and the features obtained by the enhancement processing can be used as spatial correlation information. Alternatively, the multiple correlation features and the corresponding multiple initial image features can be fused using a flexible maximum transfer function softmax, and the features obtained by the fusion processing can be used as spatial correlation information.

[0085] Alternatively, any other possible manner may be used to execute the step of processing the corresponding multiple initial image features according to the multiple correlation features to obtain multiple spatial correlation information, and there is no limitation to this.

[0086] Optionally, in some embodiments, corresponding multiple initial image features are processed separately according to multiple correlation features to obtain multiple spatially related information. The multiple correlation features may be weightedly fused with the corresponding multiple initial image features to obtain multiple spatially related information. Since the multiple correlation features are weightedly fused with the corresponding multiple initial image features, the spatially related information can comprehensively characterize the features of multiple dimensions of the image, thereby effectively improving the feature characterization effect of the spatial information.

[0087] For example, weighted fusion processing may be performed on multiple correlation features corresponding to multiple initial image features respectively, and multiple correlation features and the corresponding multiple initial image features may be added together to obtain multiple spatially correlated information. Alternatively, any other possible method may be used to weighted fusion processing may be performed on multiple correlation features corresponding to multiple initial image features respectively, and there is no limitation to this.

[0088] S305: Determine the length information corresponding to the video.

[0089] The information used to describe the length of the video may be referred to as length information, and the length information may specifically be, for example, the number of video frames included in the video, which is not limited.

[0090] That is to say, in the embodiments of the present disclosure, the number of video frames contained in the video can be determined and used as the length information corresponding to the video, or the number of video frames containing the face part of the video can be determined as the length information, and there is no limitation on this.

[0091] S306: Generate multiple window sizes according to the length information.

[0092] In the embodiment of the present disclosure, it can be combined with Figure 5 This embodiment is explained in detail. Figure 5 is a structural diagram of an image processing model according to an embodiment of the present disclosure, such as Figure 5As shown in FIG. 1 , after the frame image fragments at time t-△t, the frame image fragments at time t, and the frame image fragments at time t+△t are input into the spatial Transformer network to obtain the global classification information and the local classification information of the image respectively, the temporal difference attention module (TDA) can be combined to enhance the temporal related part in the local classification information, and then the multiple attention windows of the multi-window temporal Transformer part (pyramid temporal aggregation, PTA) are used to capture the temporal information of multiple ranges respectively, that is, the original window of the PTA module (multi-head attention module (Multi-head On the basis of SelfAttention (MSA), multiple windows (for example: w1-MSA, w2-MSA, where w is the window size of MSA) are added, and then the average value (Average, Avg) of the timing information captured by the aforementioned multiple windows is determined, and the aforementioned average value and the multi-range timing information are subjected to feature addition processing, and then the aforementioned processing results are subjected to layer normalization (LayerNormalization, LN) processing and multi-layer perceptron (Multilayer Perceptron, MLP) processing, thereby effectively improving the extraction effect of timing information of the PTA module.

[0093] It should be noted that Figure 5 The facial images involved are not facial images obtained for a specific user, and they cannot reflect the personal information of a specific user. In addition, the aforementioned facial images are obtained after authorization by the relevant user, and the acquisition process complies with the relevant laws and regulations and does not violate public order and good customs.

[0094] It should be noted that the image processing model of the embodiment of the present disclosure can be specifically a temporal transformer network (TTN) model, or the image processing model can also be configured as any other possible artificial intelligence model for performing image processing tasks, without limitation.

[0095] In the embodiment of the present disclosure, the multiple windows of the PTA module may correspond to different sizes respectively, and the size may be referred to as the window size. The window size is a size parameter for the corresponding attention extraction model to perform feature division on the input features.

[0096] That is to say, in the embodiment of the present disclosure, multiple windows of the PTA module may have corresponding attention extraction models, and the attention extraction model may be used to extract multiple attention information corresponding to multiple window sizes.

[0097] In the embodiment of the present disclosure, videos with different length information may have different window sizes respectively, thereby, multiple window sizes corresponding to the length information may be generated according to the length information of the video.

[0098] For example, suppose the length of the video is N f , the window sizes of multiple windows can be determined according to the length information of the video, that is, the window size of MSA can be determined as N f , the w1-MSA window size is 2, and the w2-MSA window size is

[0099] S307: Input the global classification features into multiple attention extraction models respectively, and obtain multiple attention information corresponding to multiple window sizes respectively output by the multiple attention extraction models.

[0100] After generating multiple window sizes according to the length information, the disclosed embodiment can input the global classification features into multiple attention extraction models respectively to obtain multiple attention information corresponding to the multiple window sizes respectively output by the multiple attention extraction models, and extract multiple timing information corresponding to the multiple image segments according to the multiple attention information, wherein the timing information describes the timing difference between the corresponding image segment and the adjacent frame image segment. Since the timing Transformer part of multiple windows is combined, the multiple timing information corresponding to the multiple frame images is determined, thereby effectively improving the robustness of the Transformer network, and can also effectively improve the accuracy and efficiency of the extraction of timing information, thereby helping to improve the overall response efficiency of living body detection.

[0101] In the embodiment of the present disclosure, multiple PTA modules of the Transformer network are connected in series, and each window of the PTA module can perform a dynamic window offset of the interlayer, such as Figure 6 As shown, Figure 6 Schematic diagram of dynamic window drift of the PTA module according to an embodiment of the present disclosure. The length of each window offset (offset length is l) is half of the current window size, that is, (w is the window size of MSA), thereby facilitating information interaction between adjacent windows of the PTA module.

[0102] S308: Extracting a plurality of timing information corresponding to the plurality of image segments respectively according to the plurality of attention information, wherein the timing information describes the timing difference between the corresponding image segment and the adjacent frame image segment.

[0103] Among the multiple image segments, the image segment whose corresponding timing information is currently to be determined can be called a corresponding image segment, and the corresponding image segment adjacent to the corresponding image segment can be called an adjacent frame image segment.

[0104] The timing difference situation can be used to describe the timing difference between the corresponding image segment and the adjacent frame image segment, and the timing difference can be specifically, for example, a timing feature difference, which is not limited.

[0105] In the disclosed embodiment, after obtaining multiple attention information corresponding to multiple window sizes respectively output by multiple attention extraction models, the timing feature differences between the corresponding image segments and the adjacent frame image segments can be determined and used as multiple timing information corresponding to the multiple image segments respectively, without any limitation.

[0106] S309: Enhance the coded motion features in the corresponding local classification features according to the time sequence information to obtain target classification features.

[0107] After extracting multiple time series information corresponding to multiple image segments according to multiple attention information, the disclosed embodiment can enhance the encoded motion features in the corresponding local classification features according to the time series information to obtain the enhanced classification features, which can be called target classification features.

[0108] In the embodiment of the present disclosure, a temporal difference attention module (TDA) can be combined to enhance the encoded motion features in the corresponding local classification features according to the temporal information, such as Figure 7 As shown, Figure 7is a structural schematic diagram of a TDA module according to an embodiment of the present disclosure, wherein the input of the time sequence change attention module TDA may be the time sequence information of adjacent frame image segments, for example, including the time sequence information of the frame image segment at time t-△t, the time sequence information of the frame image segment at time t, and the time sequence information of the frame image segment at time t+△t, the aforementioned time sequence information may be presented in a matrix form (the matrix size may be represented by L×C), and then, the reshape function is used to process the time sequence information corresponding to different time frames respectively, and the time sequence information corresponding to the corresponding time frames in the matrix form is transformed into a set dimension matrix (for example, K1 dimension, K3 dimension, wherein K1 dimension represents 1×1 dimensional convolution, K3 dimension represents 3×3×3 dimensional convolution, where N can take the value of dimension 1 in 1×1 dimensional convolution, or the value of dimension 3 in 3×3×3 dimensional convolution, and then, the temporal information of the frame image fragments at different times obtained by the transformed dimension can be differentially processed respectively, and the temporal information obtained by the differential processing can be subjected to information linking, average pooling, and Softmax function processing operations to realize channel enhancement processing of the encoded motion features in the corresponding local classification features according to the temporal information of the frame image fragment at time t-△t, the temporal information of the frame image fragment at time t+△t, and the temporal information of the frame image fragment at time t, so as to obtain the target classification features.

[0109] S310: Predicting a plurality of predicted depth images corresponding to the plurality of frame images respectively according to the plurality of target classification features.

[0110] After the embodiment of the present disclosure enhances the encoded motion features in the corresponding local classification features according to the timing information to obtain the target classification features, it can perform prediction processing on multiple frame images according to multiple target classification features and use the depth image obtained by the prediction processing as the predicted depth image.

[0111] For example, in order to generate a more accurate depth map in depth map supervision. The basic idea of ​​depth map supervision is to use the three-dimensional modeling algorithm to generate depth maps with living images during training, and at the same time, the pixel value of the depth map of non-living objects is 0, which is used as the depth map (pseudo-label) during training depth supervision, and then predict multiple predicted depth maps corresponding to multiple frames of images based on the target classification features. In this way, the depth information of multiple frames of images can be paid more attention to during the network forward transmission process, so as to achieve the purpose of using depth information to distinguish between living objects and attacks, and can also effectively improve the generation effect of predicted depth maps.

[0112] Optionally, in some embodiments, based on multiple target classification features, multiple predicted depth images corresponding to multiple frame images are predicted. Multiple initial depth images corresponding to the multiple frame images are generated based on multiple target classification features, and then multiple depth change information corresponding to the multiple initial depth images are determined, and according to the depth change situation, the loss value corresponding to the corresponding initial depth image is determined, and when the loss value meets the set conditions, the initial depth image is used as the predicted depth image. Since the depth change information of multiple frame images in the video is measured based on the loss function, the depth change information of the multiple frame images can be accurately and comprehensively grasped, so that when the multiple predicted depth images corresponding to the multiple frame images are determined based on the depth change information of the multiple frame images, more accurate depth map prediction can be achieved, and the prediction effect of the depth map can be effectively improved.

[0113] Among them, the unprocessed depth images corresponding to the multiple frame images can be called initial depth images, and correspondingly, the depth images of adjacent frame images can be called adjacent depth images.

[0114] The depth change information describes the depth change between the corresponding initial depth image and the adjacent initial depth image, and the depth change may specifically be, for example, a change in the depth value, which is not limited thereto.

[0115] That is to say, in the disclosed embodiment, based on the characteristics of the video task, the temporal depth change loss function can also be combined to better learn the changes in the depth map. For non-living videos, the depth value will not change, while for living videos, as the regional movement of the face, the depth values ​​corresponding to the multiple frames of the video are also changing. This change in depth value is a relative value change in time. Therefore, the temporal depth change loss function can be combined to implement the step of determining the loss value corresponding to the corresponding initial depth image according to the depth change. The expression of the loss function is:

[0116]

[0117] Among them, N f is the number of video frames, is the pseudo label corresponding to the t-th frame image of the video, is the pseudo label corresponding to the t-1th frame image of the video, is the predicted depth map corresponding to the t-th frame image of the video, is the predicted depth map corresponding to the t-1th frame of the video, L TDL is the temporal depth variation loss function.

[0118] The above is used to determine the time series depth change loss function L TDL After that, we can also introduce the binary classification loss function LBinary And the depth map loss function L Depth (L Depth for and The mean square error between them) is used to obtain the final loss function:

[0119] L overall =α·L Binary +(1-α)·L Depth ;

[0120] Among them, α is a preset hyperparameter, which is used to adjust the proportion of different loss functions in the total loss, L overall is the final loss function.

[0121] In the embodiment of the present disclosure, when determining the final loss function L overall After that, we can combine the loss function L overall Determine the loss value corresponding to the corresponding initial depth image, and when the loss value meets the set conditions (the set conditions can be adaptively configured according to the business needs of the actual image processing scenario, and there is no restriction on this), use the initial depth image as the predicted depth image.

[0122] In the embodiment of the present disclosure, the final loss function L is determined overall After that, we can combine the loss function L overall The image processing model (TTN model) is trained until the image processing model (TTN model) converges, and then a subsequent image processing method can be performed based on the image processing model (TTN model). For details, please refer to the subsequent embodiments.

[0123] S311: Perform living body detection according to multiple spatial related information and multiple predicted depth images.

[0124] For the description of S311, please refer to the above embodiment, which will not be repeated here.

[0125] In this embodiment, by acquiring multiple frames of video and determining multiple correlation features corresponding to the multiple frames of images, feature extraction is performed on the multiple frames to obtain multiple initial image features, and the corresponding multiple initial image features are processed according to the multiple correlation features to obtain multiple spatially related information. This can make the spatially related information obtained by processing the initial image features based on the correlation features more accurate and have a better representation effect. It can represent more complete video semantic information carried in the video to ensure the accuracy of subsequent liveness detection. The length information corresponding to the video is determined, and multiple window sizes are generated based on the length information. The global classification features are then input into multiple attention extraction models to obtain multiple attention information corresponding to multiple window sizes, thereby effectively improving the robustness of the Transformer network, and effectively improving the accuracy and efficiency of extracting temporal information, thereby helping to improve the overall response efficiency of liveness detection. Then, based on the multiple attention information, multiple time series information corresponding to the multiple image segments are extracted, and the encoded motion features in the corresponding local classification features are enhanced according to the time series information to obtain the target classification features. Then, based on the multiple target classification features, multiple predicted depth images corresponding to the multiple frame images are predicted, and then liveness detection is performed based on the multiple spatial related information and the multiple predicted depth images, thereby effectively improving the accuracy and effect of liveness detection.

[0126] Figure 8 is a schematic diagram according to a third embodiment of the present disclosure.

[0127] like Figure 8 As shown, the image processing method comprises:

[0128] S801: Acquire multiple frames of video images.

[0129] S802: Determine a plurality of correlation features corresponding to the plurality of frames of images respectively.

[0130] The description of S801 - S802 can be specifically referred to the above embodiment, which will not be repeated here.

[0131] S803: Determine the classification label.

[0132] Among them, the label information used for classification can be called a classification label (classification token). The classification token is a network parameter in the Transformer network. Accordingly, determining the classification label can be initializing the Transformer network to determine the network parameters of the Transformer network, and using the parameters as the classification token. The classification label (classification token) can be specifically expressed in the form of a 384-dimensional vector, or the classification label (classification token) can also be expressed in any other possible form, without limitation.

[0133] S804: Extracting a plurality of segment classification features corresponding to a plurality of image segments of the image, respectively, according to the classification label.

[0134] Among them, the feature information used for classification can be called classification features. Accordingly, multiple image fragments can have respectively corresponding classification features, which can be called fragment classification features. The fragment classification features can be specifically, for example, the feature representation vector corresponding to the image fragment, which is not limited.

[0135] In some embodiments, based on the classification label, multiple fragment classification features corresponding to multiple image fragments of an image are extracted. The classification feature information related to the image fragment can be projected into the vector space based on the classification label to obtain a feature representation vector, and the feature representation vector is used as the multiple fragment classification features corresponding to multiple image fragments of multiple frames of images. Alternatively, any other possible method can be used to extract multiple fragment classification features corresponding to multiple image fragments of an image based on the classification label, such as a multi-layer perceptron method, a recurrent neural network method, etc., without limitation.

[0136] Optionally, in some embodiments, based on the classification label, extracting multiple fragment classification features corresponding to multiple image fragments of the image respectively may be to extract multiple local classification features corresponding to the classification labels for the multiple image fragments respectively, and extracting multiple global classification features corresponding to the classification labels for the multiple image fragments respectively, and using the multiple local classification features and the multiple global classification features as multiple fragment classification features respectively, wherein the global classification features are local classification features corresponding to other image fragments related to the corresponding image fragment, and the other image fragments belong to multiple image fragments. Since the multiple local classification features corresponding to the multiple image fragments respectively and the multiple global classification features corresponding to the classification labels for the multiple image fragments are used together as multiple fragment classification features, the fragment classification features can be based on the features that comprehensively characterize multiple dimensions of the image, so that the fragment classification features can correspondingly characterize the differences in local classification features between different image fragments, thereby effectively improving the feature characterization effect of the fragment classification features.

[0137] Among them, the features with local information formed by the self-attention superposition effect and used to describe multiple image segments after encoding processing can be called local classification features.

[0138] The global classification feature is a local classification feature corresponding to other image segments related to the corresponding image segment, that is, the global classification feature can be a feature that can be used for classification in multiple image segments of an image captured by a classification label.

[0139] That is to say, in the embodiments of the present disclosure, multiple image fragments can be analyzed and processed to extract multiple local classification features corresponding to the classification labels of the multiple image fragments, and extract multiple global classification features corresponding to the classification labels of the multiple image fragments, and use the multiple local classification features and the multiple global classification features as multiple fragment classification features, respectively. Then, subsequent image processing methods can be performed based on the multiple fragment classification features. For details, please refer to the subsequent embodiments.

[0140] S805: Using the multiple segment classification features together as initial image features of the corresponding image.

[0141] After extracting multiple fragment classification features corresponding to multiple image fragments of an image according to the classification labels, the disclosed embodiment can use the multiple fragment classification features together as the initial image features of the corresponding image. Since the multiple fragment classification features corresponding to the multiple image fragments of an image are extracted according to the classification labels, it is possible to more comprehensively capture information that can be used for classification from the multiple image fragments, effectively improve the representation effect of the initial image features, and thus effectively ensure the smooth execution of subsequent image processing methods.

[0142] S806: Process the corresponding multiple initial image features respectively according to the multiple correlation features to obtain multiple spatial correlation information.

[0143] S807: Determine multiple timing information corresponding to the multiple frames of images.

[0144] S808: Generate a plurality of predicted depth images corresponding to the corresponding plurality of frame images respectively according to the plurality of time sequence information.

[0145] The description of S806-S808 can be specifically referred to the above embodiment, which will not be repeated here.

[0146] S809: Perform living body detection according to the multiple spatial related information and the multiple predicted depth images.

[0147] Optionally, in some embodiments, liveness detection is performed based on multiple spatially related information and multiple predicted depth images. This may be to determine, based on the multiple spatially related information, multiple description information corresponding to the predicted liveness positions in the corresponding multiple images, and determine multiple depth description information corresponding to the multiple predicted depth images, and then determine multiple liveness detection scores corresponding to the corresponding multiple images based on the multiple description information and the multiple depth description information, and perform liveness detection based on the multiple liveness detection scores. Since the liveness detection score is determined based on multiple description information and multiple depth description information, the accuracy of the liveness detection score can be effectively improved. In addition, performing liveness detection in combination with the liveness detection score can also improve the operability of liveness detection, and can also avoid introducing other subjective detection factors in the liveness detection process, thereby effectively improving the liveness detection effect.

[0148] The multiple description information may be used to describe the predicted living body positions in the corresponding multiple images, and the multiple description information may specifically be, for example, logarithmic values ​​of the predicted living body positions, which is not limited.

[0149] The information used to describe multiple predicted depth images may be referred to as depth description information. The depth description information may specifically be, for example, an average depth value of the predicted depth images, and there is no limitation on this.

[0150] Among them, the liveness detection score can be used to determine whether the detected image is alive. For example, when the liveness detection score is close to 0, the detected image can be judged as a non-liveness image, and when the liveness detection score is close to 1, the detected image can be judged as a liveness image. There is no limitation on this.

[0151] In the embodiment of the present disclosure, liveness detection is performed based on multiple spatial related information and multiple predicted depth images. A score calculation formula may be combined to determine multiple liveness detection scores corresponding to the corresponding multiple images, and then the corresponding liveness detection result is determined based on the liveness detection score. The calculation formula is:

[0152]

[0153] Among them, Score is the score of liveness detection, N f is the number of image frames in the video, α is a preset hyperparameter, is the average depth value of the predicted depth image at time t, b l The plurality of description information respectively corresponding to the living body positions.

[0154] In the embodiment of the present disclosure, when there are abnormal frames in the multiple frames of the detected video (for example, all zero frames, noise frames, which are not limited to this), the liveness detection score will change accordingly, such as Fig. 9 As shown, Fig. 9: is a schematic diagram of the liveness detection score of abnormal frame images implemented according to the present disclosure, wherein there are all-0 frame images in (a) and (b), and there are noise frame images in (c) and (d), w / 0PTA is the weight assigned by the PTA module to abnormal frame images (for example: all-0 frame images and noise frame images), w / PTA is the weight assigned by the PTA module to images other than abnormal frames, and the histogram shows the weights assigned by multiple PTA modules of the Transformer network to each frame image. The table shows the liveness detection scores corresponding to multiple frame images, and then, liveness detection can be performed in combination with the liveness detection scores in the table.

[0155] It should be noted that Fig. 9 The facial images involved are not facial images obtained for a specific user, and they cannot reflect the personal information of a specific user. In addition, the aforementioned facial images are obtained after authorization by the relevant user, and the acquisition process complies with the relevant laws and regulations and does not violate public order and good customs.

[0156] In this embodiment, multiple frames of video are acquired, and multiple correlation features corresponding to the multiple frames are determined, and then classification labels are determined. Based on the classification labels, multiple fragment classification features corresponding to multiple image fragments of the image are extracted, and the multiple fragment classification features are used together as initial image features of the corresponding image. Then, the corresponding multiple initial image features are processed respectively according to the multiple correlation features to obtain multiple spatial related information, and multiple time series information corresponding to the multiple frames are determined. Then, based on the multiple time series information, multiple predicted depth images corresponding to the corresponding multiple frames are generated respectively. Then, liveness detection is performed based on the multiple spatial related information and the multiple predicted depth images, and liveness detection is performed in combination with the liveness detection score. The operability of liveness detection can also be improved, and other subjective detection factors can be avoided in the liveness detection process, thereby effectively improving the liveness detection effect.

[0157] Fig.10 is a schematic diagram according to a fourth embodiment of the present disclosure.

[0158] like Fig.10 As shown, the image processing device 10 includes:

[0159] An acquisition module 101 is used to acquire multiple frames of video images;

[0160] A first determining module 102 is used to determine a plurality of spatial related information corresponding to the plurality of frames of images respectively;

[0161] A second determining module 103, used to determine a plurality of time sequence information corresponding to a plurality of frames of images;

[0162] A generating module 104, configured to generate a plurality of predicted depth images corresponding to the corresponding plurality of frames of images according to the plurality of time series information; and

[0163] The detection module 105 is used to perform living body detection based on multiple spatial related information and multiple predicted depth images.

[0164] In some embodiments of the present disclosure, Fig.11 As shown, Fig.11 is a schematic diagram according to the fifth embodiment of the present disclosure, the image processing device 11 comprises: an acquisition module 111, a first determination module 112, a second determination module 113, a generation module 114, and a detection module 115, wherein the first determination module 112 comprises:

[0165] A determination submodule 1121 is used to determine a plurality of correlation features corresponding to the plurality of frames of images respectively;

[0166] The extraction submodule 1122 is used to extract features from multiple frames of images to obtain multiple initial image features; and

[0167] The first processing submodule 1123 is used to process the corresponding multiple initial image features respectively according to the multiple correlation features to obtain multiple spatial correlation information.

[0168] In some embodiments of the present disclosure, the determining submodule 1121 is specifically configured to:

[0169] Segment the image to obtain multiple image segments;

[0170] Determine a plurality of encoding information respectively corresponding to a plurality of image segments; and

[0171] Determine a plurality of correlations respectively corresponding to the plurality of coding information, and use the plurality of correlations as a plurality of correlation features of the corresponding plurality of frames of images, wherein the correlations describe the correlation between the corresponding coding information and other coding information.

[0172] In some embodiments of the present disclosure, the extraction submodule 1122 includes:

[0173] A determination unit 11221, used to determine a classification label;

[0174] An extraction unit 11222 is used to extract a plurality of segment classification features corresponding to a plurality of image segments of the image according to the classification label; and

[0175] The processing unit 11223 is used to use the multiple segment classification features together as the initial image features of the corresponding image.

[0176] In some embodiments of the present disclosure, the extraction unit 11222 is specifically used to:

[0177] Extracting a plurality of local classification features corresponding to the classification labels of the plurality of image segments;

[0178] Extracting a plurality of global classification features corresponding to the classification labels of the plurality of image segments respectively, wherein the global classification features are local classification features corresponding to other image segments related to the corresponding image segment, and the other image segments belong to the plurality of image segments;

[0179] The multiple local classification features and the multiple global classification features are respectively used as multiple fragment classification features.

[0180] In some embodiments of the present disclosure, the first processing submodule 1123 is specifically configured to:

[0181] A plurality of correlation features are respectively subjected to weighted fusion processing with a plurality of corresponding initial image features to obtain a plurality of spatial correlation information.

[0182] In some embodiments of the present disclosure, the second determining module 113 is specifically configured to:

[0183] Determine the length information corresponding to the video;

[0184] Generate multiple window sizes according to the length information, where the multiple window sizes are different, and the window size is a size parameter for the corresponding attention extraction model to divide the input features;

[0185] The global classification features are respectively input into multiple attention extraction models to obtain multiple attention information corresponding to multiple window sizes output by the multiple attention extraction models;

[0186] Based on the multiple attention information, multiple timing information corresponding to the multiple image segments are extracted, wherein the timing information describes the timing difference between the corresponding image segment and the adjacent frame image segment.

[0187] In some embodiments of the present disclosure, the generating module 114 includes:

[0188] The second processing submodule 1141 is used to enhance the coded motion features in the corresponding local classification features according to the time sequence information to obtain the target classification features;

[0189] The prediction submodule 1142 is used to predict a plurality of predicted depth images corresponding to the plurality of frame images according to a plurality of target classification features.

[0190] In some embodiments of the present disclosure, the prediction submodule 1142 is specifically used to:

[0191] Generate multiple initial depth images corresponding to the multiple frame images respectively according to multiple target classification features;

[0192] Determine a plurality of depth change information corresponding to the plurality of initial depth images, wherein the depth change information describes a depth change between a corresponding initial depth image and an adjacent initial depth image;

[0193] According to the depth change, determine the loss value corresponding to the corresponding initial depth image;

[0194] If the loss value satisfies the set conditions, the initial depth image is used as the predicted depth image.

[0195] In some embodiments of the present disclosure, the detection module is specifically used to:

[0196] Determine, based on the plurality of spatial related information, a plurality of description information corresponding to the predicted living body positions in the corresponding plurality of images;

[0197] Determine a plurality of depth description information corresponding to the plurality of predicted depth images respectively;

[0198] Determine, according to the plurality of description information and the plurality of depth description information, a plurality of living body detection scores corresponding to the corresponding plurality of images respectively;

[0199] Liveness detection is performed based on multiple liveness detection scores.

[0200] It is understandable that the present embodiment Fig.11 The image processing device 11 in the embodiment may have the same function and structure as the image processing device 10 in the above embodiment, the acquisition module 111 may have the same function and structure as the acquisition module 101 in the above embodiment, the first determination module 112 may have the same function and structure as the first determination module 102 in the above embodiment, the second determination module 113 may have the same function and structure as the second determination module 103 in the above embodiment, the generation module 114 may have the same function and structure as the generation module 104 in the above embodiment, and the detection module 115 may have the same function and structure as the detection module 105 in the above embodiment.

[0201] It should be noted that the above explanation of the image processing method is also applicable to the image processing device of this embodiment and will not be repeated here.

[0202] In this embodiment, multiple frames of video are acquired, multiple spatial related information corresponding to the multiple frames are determined, multiple time series information corresponding to the multiple frames are determined, and multiple predicted depth images corresponding to the corresponding multiple frames are generated according to the multiple time series information, and liveness detection is performed according to the multiple spatial related information and the multiple predicted depth images. Thus, the accuracy and effect of liveness detection can be effectively improved, and the generalization of the image processing method can be effectively improved, which is conducive to the deployment and application of the image processing method in liveness detection scenarios.

[0203] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0204] Fig.12 A schematic block diagram of an example electronic device for implementing the image processing method of an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0205] like Fig.12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0206] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0207] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0208] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0209] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0210] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0211] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0212] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0213] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0214] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0215] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, comprising: Get multiple frames of video; Determine a plurality of space-related information respectively corresponding to the plurality of frames of images; Determine a plurality of time series information corresponding to the plurality of frames of images, wherein the plurality of frames of images are respectively input into a recurrent neural network to perform time series operations on the plurality of frames of images to extract time features of the plurality of frames of images, and the extracted time features are used as a plurality of time series information corresponding to the plurality of frames of images; According to the plurality of time series information, a plurality of predicted depth images corresponding to the corresponding plurality of frame images are respectively generated, wherein, according to the plurality of time series information, depth recognition is respectively performed on the corresponding plurality of frame images, and the depth images are formed according to the recognized depths; and Liveness detection is performed according to the multiple spatial related information and the multiple predicted depth images.

2. The method according to claim 1, wherein: The determining of a plurality of space-related information respectively corresponding to the plurality of frames of images comprises: Determine a plurality of correlation features respectively corresponding to the plurality of frames of images; Extracting features from the multiple frames of images respectively to obtain multiple initial image features; and The multiple initial image features are processed respectively according to the multiple correlation features to obtain the multiple spatial correlation information.

3. The method according to claim 2, wherein: The determining of a plurality of correlation features respectively corresponding to the plurality of frames of images comprises: Segmenting the image to obtain a plurality of image segments; Determine a plurality of encoding information respectively corresponding to the plurality of image segments; and Determine a plurality of correlations respectively corresponding to the plurality of encoding information, and use the plurality of correlations as a plurality of correlation features corresponding to the plurality of frames of images, wherein the correlations describe the correlation between the corresponding encoding information and other encoding information.

4. The method according to claim 3, wherein: The extracting features of the multiple frames of images respectively to obtain multiple initial image features includes: Determine the classification label; extracting a plurality of segment classification features respectively corresponding to a plurality of image segments of the image according to the classification label; and The multiple segment classification features are collectively used as initial image features of the corresponding image.

5. The method according to claim 4, wherein: The step of extracting a plurality of segment classification features respectively corresponding to a plurality of image segments of the image according to the classification label comprises: Extracting a plurality of local classification features corresponding to the classification labels respectively from the plurality of image segments; Extracting a plurality of global classification features corresponding to the classification labels respectively from the plurality of image segments, wherein the global classification features are local classification features corresponding to other image segments related to the corresponding image segments, and the other image segments belong to the plurality of image segments; The multiple local classification features and the multiple global classification features are respectively used as the multiple segment classification features.

6. The method according to claim 5, wherein: The step of respectively processing the corresponding multiple initial image features according to the multiple correlation features to obtain the multiple spatial correlation information includes: The plurality of correlation features are respectively subjected to weighted fusion processing with the corresponding plurality of initial image features to obtain the plurality of spatial correlation information.

7. The method according to claim 5, wherein: The determining of a plurality of timing information corresponding to the plurality of frames of images comprises: Determining length information corresponding to the video; Generate multiple window sizes according to the length information, wherein the multiple window sizes are different, and the window size is a size parameter for the corresponding attention extraction model to perform feature division on the input feature; Inputting the global classification features into the plurality of attention extraction models respectively, to obtain a plurality of attention information corresponding to the plurality of window sizes respectively output by the plurality of attention extraction models; Based on the multiple attention information, multiple timing information corresponding to the multiple image segments are extracted, wherein the timing information describes the timing difference between the corresponding image segment and the adjacent frame image segment.

8. The method according to claim 7, wherein: The step of respectively generating a plurality of predicted depth images corresponding to the plurality of frames of images according to the plurality of time series information comprises: According to the time sequence information, the coded motion features in the corresponding local classification features are enhanced to obtain target classification features; Based on the multiple target classification features, multiple predicted depth images corresponding to the multiple frame images are predicted.

9. The method according to claim 8, wherein: The step of predicting a plurality of predicted depth images respectively corresponding to the plurality of frames of images according to the plurality of target classification features comprises: Generating a plurality of initial depth images corresponding to the plurality of frames of images respectively according to the plurality of target classification features; Determine a plurality of depth change information corresponding to the plurality of initial depth images, wherein the depth change information describes a depth change between a corresponding initial depth image and an adjacent initial depth image; Determine, according to the depth change, a loss value corresponding to the initial depth image; If the loss value satisfies a set condition, the initial depth image is used as the predicted depth image.

10. The method according to any one of claims 1 to 9, wherein: The performing living body detection according to the multiple spatial related information and the multiple predicted depth images includes: Determine, according to the plurality of spatial related information, a plurality of description information corresponding to the predicted living body positions in the plurality of images respectively; Determine a plurality of depth description information respectively corresponding to the plurality of predicted depth images; Determine, according to the plurality of description information and the plurality of depth description information, a plurality of living body detection scores corresponding to the plurality of images respectively; The liveness detection is performed according to the multiple liveness detection scores.

11. An image processing device, comprising: An acquisition module, used to acquire multiple frames of video images; A first determining module, used to determine a plurality of space related information respectively corresponding to the plurality of frames of images; A second determination module is used to determine a plurality of time series information corresponding to the plurality of frames of images, wherein the plurality of frames of images are respectively input into a recurrent neural network to perform time series operations on the plurality of frames of images to extract time features of the plurality of frames of images, and the extracted time features are used as a plurality of time series information corresponding to the plurality of frames of images; A generating module, configured to generate a plurality of predicted depth images corresponding to the corresponding multiple frames of images respectively according to the multiple time series information, wherein depth recognition is performed on the corresponding multiple frames of images respectively according to the multiple time series information, and the depth images are formed according to the recognized depths; and A detection module is used to perform living body detection based on the multiple spatial related information and the multiple predicted depth images.

12. The device according to claim 11, wherein The first determining module includes: A determination submodule, used to determine a plurality of correlation features respectively corresponding to the plurality of frames of images; An extraction submodule, used to extract features from the multiple frames of images respectively to obtain multiple initial image features; and The first processing submodule is used to process the corresponding multiple initial image features respectively according to the multiple correlation features to obtain the multiple spatial correlation information.

13. The device according to claim 12, wherein: The determination submodule is specifically used for: Segmenting the image to obtain a plurality of image segments; Determine a plurality of encoding information respectively corresponding to the plurality of image segments; and Determine a plurality of correlations respectively corresponding to the plurality of encoding information, and use the plurality of correlations as a plurality of correlation features corresponding to the plurality of frames of images, wherein the correlations describe the correlation between the corresponding encoding information and other encoding information.

14. The device according to claim 13, wherein: The extraction submodule comprises: A determination unit, used to determine a classification label; an extraction unit, configured to extract, according to the classification label, a plurality of segment classification features respectively corresponding to the plurality of image segments of the image; and A processing unit is used to use the multiple segment classification features together as initial image features corresponding to the image.

15. The device of claim 14, wherein: The extraction unit is specifically used for: Extracting a plurality of local classification features corresponding to the classification labels respectively from the plurality of image segments; Extracting a plurality of global classification features corresponding to the classification labels respectively from the plurality of image segments, wherein the global classification features are local classification features corresponding to other image segments related to the corresponding image segments, and the other image segments belong to the plurality of image segments; The multiple local classification features and the multiple global classification features are respectively used as the multiple segment classification features.

16. The device of claim 15, wherein: The first processing submodule is specifically used to: The plurality of correlation features are respectively subjected to weighted fusion processing with the corresponding plurality of initial image features to obtain the plurality of spatial correlation information.

17. The device of claim 15, wherein: The second determining module is specifically used to: Determining length information corresponding to the video; Generate multiple window sizes according to the length information, wherein the multiple window sizes are different, and the window size is a size parameter for the corresponding attention extraction model to perform feature division on the input feature; Inputting the global classification features into the plurality of attention extraction models respectively, to obtain a plurality of attention information corresponding to the plurality of window sizes respectively output by the plurality of attention extraction models; Based on the multiple attention information, multiple timing information corresponding to the multiple image segments are extracted, wherein the timing information describes the timing difference between the corresponding image segment and the adjacent frame image segment.

18. The device of claim 17, wherein: The generating module comprises: A second processing submodule is used to enhance the coded motion features in the corresponding local classification features according to the time sequence information to obtain target classification features; The prediction submodule is used to predict a plurality of predicted depth images corresponding to the plurality of frame images respectively according to the plurality of target classification features.

19. The device according to claim 18, wherein: The prediction submodule is specifically used for: Generating a plurality of initial depth images corresponding to the plurality of frames of images respectively according to the plurality of target classification features; Determine a plurality of depth change information corresponding to the plurality of initial depth images, wherein the depth change information describes a depth change between a corresponding initial depth image and an adjacent initial depth image; Determine, according to the depth change, a loss value corresponding to the initial depth image; If the loss value satisfies a set condition, the initial depth image is used as the predicted depth image.

20. The device according to any one of claims 11 to 19, wherein: The detection module is specifically used for: Determine, according to the plurality of spatial related information, a plurality of description information corresponding to the predicted living body positions in the plurality of images respectively; Determine a plurality of depth description information respectively corresponding to the plurality of predicted depth images; Determine, according to the plurality of description information and the plurality of depth description information, a plurality of living body detection scores corresponding to the plurality of images respectively; The liveness detection is performed according to the multiple liveness detection scores.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images