Image processing method and device based on adversarial learning

Abstract

The invention discloses an image processing method and device based on adversarial learning. The method comprises the following steps: acquiring to-be-processed image information; preprocessing the to-be-processed image information to obtain initial image processing information; and analyzing and processing the initial image processing information by using the target image processing model to obtain image analysis result information.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method and device based on adversarial learning. Background Art

[0002] Heart rate (HR) is an important physiological indicator that can be measured by contact instruments such as electrocardiogram (ECG) and photoplethysmography (PPG). ECG is the extraction of HR by sensing the human heart current through electrodes. PPG is a simple and low-cost method for obtaining HR, which uses optical technology to detect changes in blood volume in capillaries. Some convenient PPG detection methods use portable wearable devices (such as smart watches or bracelets) to obtain PPG signals. However, these PPG methods require contact with the human body, which may cause skin irritation or discomfort to some patients who need to monitor HR for a long time. Human-computer interaction has certain limitations (for example, the user needs to wear a measuring device for the computer to receive HR information). Therefore, ideal HR detection should be remote and contactless. Although infrared images can be used to remotely measure heart pulse, this method still requires special equipment. Therefore, an image processing method and device based on adversarial learning are provided to generate adversarial examples of rPPG signals of different qualities and feed them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an image processing method and device based on adversarial learning, which are conducive to generating adversarial examples of rPPG signals of different qualities and feeding them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0004] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses an image processing method based on adversarial learning, the method comprising:

[0005] Obtaining image information to be processed;

[0006] Pre-processing the image information to be processed to obtain initial image processing information;

[0007] The initial image processing information is analyzed and processed using a target image processing model to obtain image analysis result information.

[0008] A second aspect of an embodiment of the present invention discloses an image processing device based on adversarial learning, the device comprising:

[0009] An acquisition module, used for acquiring image information to be processed;

[0010] A first processing module, used for pre-processing the image information to be processed to obtain initial image processing information;

[0011] The second processing module is used to analyze and process the initial image processing information using a target image processing model to obtain image analysis result information.

[0012] The third aspect of the present invention discloses another image processing device based on adversarial learning, the device comprising:

[0013] A memory storing executable program code;

[0014] a processor coupled to the memory;

[0015] The processor calls the executable program code stored in the memory to execute part or all of the steps in the image processing method based on adversarial learning disclosed in the first aspect of the embodiment of the present invention.

[0016] The fourth aspect of the present invention discloses a computer-readable storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute part or all of the steps in the image processing method based on adversarial learning disclosed in the first aspect of an embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a schematic diagram of a scenario of an image processing system based on adversarial learning provided by an embodiment of the present invention;

[0019] Figure 2 It is a flowchart of an image processing method based on adversarial learning disclosed in an embodiment of the present invention;

[0020] Figure 3 is a structural schematic diagram of an image processing device based on adversarial learning disclosed in an embodiment of the present invention;

[0021] Figure 4 is a schematic structural diagram of another image processing device based on adversarial learning disclosed in an embodiment of the present invention;

[0022] Figure 5 is a structural schematic diagram of an initial image processing model disclosed in an embodiment of the present invention;

[0023] Figure 6 It is a structural schematic diagram of a basic image processing model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0026] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0027] In this application, the word "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0028] It should be noted that since the method of the embodiment of the present application is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, time is actually time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data for processing by the computer device. The details will not be repeated here.

[0029] The embodiments of the present application provide an image processing method, apparatus, computer device, and computer-readable storage medium based on adversarial learning, which are described in detail below.

[0030] See also Figure 1 , Figure 1 Schematic diagram of a scenario of an image processing system based on adversarial learning provided in an embodiment of the present application. The image processing system based on adversarial learning may include a computer device 100, in which an image processing device based on adversarial learning is integrated, such as Figure 1 Computer equipment in.

[0031] In the embodiment of the present application, the computer device 100 is mainly used to obtain image information to be processed; pre-process the image information to be processed to obtain initial image processing information; and analyze and process the initial image processing information using a target image processing model to obtain image analysis result information.

[0032] It can generate adversarial examples of rPPG signals of different qualities and feed them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0033] In the embodiment of the present application, the computer device 100 may be an independent server, or a server network or server cluster composed of servers. For example, the computer device 100 described in the embodiment of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0034] It is understandable that the computer device 100 used in the embodiments of the present application may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such a device may include: a cellular or other communication device having a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific computer device 100 may be a desktop terminal or a mobile terminal, and the computer device 100 may also be one of a mobile phone, a tablet computer, a laptop computer, etc.

[0035] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or less computer equipment as shown in Figure 1 Only one computer device is shown in the figure. It can be understood that the image processing system based on adversarial learning can also include one or more other services, which are not specifically limited here.

[0036] In addition, if Figure 1 As shown, the adversarial learning-based image processing system may further include a memory 200 for storing data, such as image data, position information, and the like.

[0037] It should be noted that Figure 1 The scenario diagram of the adversarial learning-based image processing system shown is merely an example. The adversarial learning-based image processing system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the adversarial learning-based image processing system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0038] The present invention discloses an image processing method and device based on adversarial learning, which is conducive to generating adversarial examples of rPPG signals of different qualities and feeding them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images. The following is a detailed description of each.

[0039] Embodiment 1

[0040] See also Figure 2 , Figure 2 is a flow chart of an image processing method based on adversarial learning disclosed in an embodiment of the present invention. Figure 2 The described adversarial learning-based image processing method is applied to a management system, such as a local server or a cloud server for management, and the embodiments of the present invention do not limit this. Figure 2 As shown, the image processing method based on adversarial learning may include the following operations:

[0041] 101. Obtain image information to be processed.

[0042] 102. Pre-process the image information to be processed to obtain initial image processing information.

[0043] 103. Analyze and process the initial image processing information using the target image processing model to obtain image analysis result information.

[0044] It should be noted that the above-mentioned pre-processing of the image information to be processed is to perform image detection and segmentation processing on the image to be processed in the image information to be processed to obtain the first processed image information; further, the image pixels of the first processed image information are randomly adjusted within the image specification range (image pixels are between 32×32 and 128×128) to obtain the initial image processing information. Furthermore, the above-mentioned image detection and segmentation processing can be implemented by any one of the artificial intelligence models based on the Cascade-CNN model, the model based on R-CNN and Faster-RCNN, and the model based on the feature pyramid network, or other artificial intelligence models in the prior art, and the embodiments of the present invention are not limited thereto.

[0045] It should be noted that the adversarial learning-based image processing method of the present application uses random cropping and random flipping to jointly perform data enhancement in the video pre-processing part, enriches data diversity, enhances model robustness, and converts the single-branch channel into a double-branch channel, which facilitates the subsequent adversarial training and learning of the model, and the embodiments of the present invention are not limited to this.

[0046] It can be seen that implementing the adversarial learning-based image processing method described in the embodiment of the present invention is conducive to generating adversarial examples of rPPG signals of different qualities and feeding them into the multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0047] In an optional embodiment, the target image processing model is obtained based on the following method:

[0048] Get video information;

[0049] Performing sample generation processing on the video information to obtain initial image sample information;

[0050] Based on the initial image sample information, determining the target image sample information;

[0051] Using the target image sample information to perform adversarial learning training on the initial image processing model, and obtaining model training result information and a training image processing model;

[0052] Determine whether the model training result information meets the model training termination condition and obtain the conditional judgment result;

[0053] When the result of the conditional judgment is no, the training image processing model is determined as a new initial image processing model, and the execution is triggered to determine the target image sample information based on the initial image sample information;

[0054] When the conditional judgment result is yes, the model parameters of the training image processing model are transferred to the basic image processing model to obtain the target image processing model; the model structure of the basic image processing model is partially consistent with that of the initial image processing model.

[0055] like Figure 6 As shown, the basic image processing model includes a basic feature extraction submodel and a basic feature fusion submodel, which is not limited in the embodiment of the present invention. Further, the model structures of the basic feature extraction submodel, the first feature extraction submodel, and the second feature extraction submodel are consistent; the model structures of the basic feature fusion submodel and the feature fusion submodel are consistent. Further, the above-mentioned migration of the model parameters of the training image processing model to the basic image processing model is to migrate the model parameters of the first feature extraction submodel to the basic feature extraction submodel, and to migrate the model parameters of the feature fusion submodel to the basic feature fusion submodel, which is not limited in the embodiment of the present invention. Further, the basic feature extraction submodel, the first feature extraction submodel, and the second feature extraction submodel mainly expand the receptive field by superimposing convolutional layers, extract rich signal features, that is, accurately capture the semantic information of the rPPG signal in the low-resolution face image, and splice it in the time dimension to obtain a more comprehensive feature representation, which is not limited in the embodiment of the present invention.

[0056] It should be noted that the training of the above-mentioned first feature extraction sub-model and the second feature extraction sub-model is used for adversarial learning, that is, the model training process is optimized by comparing the performance of the two models, so as to eventually migrate the parameters of the first feature extraction sub-model to the basic feature extraction sub-model, which is not limited in the embodiments of the present invention.

[0057] It should be noted that the above basic feature fusion sub-model can restore the degraded rPPG signal in the low-resolution image and minimize the quality gap of the rPPG signal in the low-resolution image.

[0058] It should be noted that the above determination of the target image sample information based on the initial image sample information is to randomly select an initial image sample set information from all the initial image sample set information in the initial image sample information as the target image sample information, which is not limited in the embodiment of the present invention. Further, the initial image sample set information includes the first initial image sample set information and the second initial image sample set information, which correspond to the first target image sample set information and the second target image sample set information in the target image sample information, which is not limited in the embodiment of the present invention.

[0059] It should be noted that the above video information represents a video of a specific object showing a human face for the first time, which is not limited in the embodiment of the present invention.

[0060] It should be noted that the above-mentioned use of the target image sample information to perform adversarial learning training on the initial image processing model is to input the first target image sample set information and the second target image sample set information in the target image sample information into the first feature extraction sub-model and the second feature extraction sub-model in sequence, and then the model performs training processing, which is not limited to the embodiments of the present invention.

[0061] It should be noted that the above-mentioned model training result information includes the first model training result information and the second model training result information, which is not limited in the embodiment of the present invention. Further, the above-mentioned first model training result information corresponds to the first target image sample set information, and the second model training result information corresponds to the second target image sample set information, which is not limited in the embodiment of the present invention.

[0062] It should be noted that the above-mentioned judgment of whether the model training result information meets the model training termination condition obtains the conditional judgment result by using the loss function to calculate the loss function value of the model training result information. If the loss function value is less than the loss function threshold, and the data composed of the loss function value and the historical loss function value are converged, then the conditional judgment result is yes, otherwise the conditional judgment result is no, and the embodiment of the present invention does not limit this.

[0063] It should be noted that the above loss function can be constructed based on the cross entropy loss function, or based on the KL divergence loss function, or based on the mean square error loss function, or based on a composite loss function of the above loss functions, which is not limited in the embodiments of the present invention. Furthermore, by using these different loss functions or their combinations, a more flexible and comprehensive optimization goal can be provided for adversarial learning training, thereby improving the performance and generalization ability of the model in data processing, which is not limited in the embodiments of the present invention.

[0064] It should be noted that the above-mentioned training method based on adversarial learning uses the adversarial learning training method framework to solve the problem of rPPG signal degradation in low-resolution images under long-distance conditions, so as to restore the degraded rPPG signal in the low-resolution image and minimize the gap between high-quality and low-quality rPPG signals, which is not limited in the embodiments of the present invention.

[0065] It can be seen that implementing the adversarial learning-based image processing method described in the embodiment of the present invention is conducive to generating adversarial examples of rPPG signals of different qualities and feeding them into the multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0066] In another optional embodiment, Figure 5As shown, the initial image processing model includes a first feature extraction sub-model, a second feature extraction sub-model, a first fusion module, a feature fusion sub-model and a segmentation model; wherein,

[0067] The input end of the first feature extraction sub-model is connected to the first model input of the initial image processing model, and the output end of the first feature extraction sub-model is connected to the input end of the first fusion module; the second feature extraction sub-model is connected to the second model input of the initial image processing model, and the output end of the second feature extraction sub-model is connected to the input end of the first fusion module; the output end of the first fusion module is connected to the input end of the feature fusion sub-model; the output end of the feature fusion sub-model is connected to the input end of the segmentation module; the output end of the segmentation module is connected to the first model output and the second model output of the initial image processing model.

[0068] It should be noted that the above-mentioned first fusion module is constructed based on the channel number addition operation, that is, the contact operation is used to merge the channel number, the feature number (channel number) describing the image itself is increased, and the information under each feature is not increased, and the embodiment of the present invention does not limit it. Further, the above-mentioned first fusion module splices the data output by the first feature extraction sub-model and the second feature extraction sub-model so as to be processed in the subsequent feature fusion sub-model, that is, a feature fusion sub-model is used to process the output information of the two feature extraction sub-models, thereby simplifying the model structure and improving the model processing efficiency. Further, the above-mentioned segmentation module is constructed based on the split segmentation function, so as to split the data spliced ​​in the first fusion module again after being processed by the feature fusion sub-model, so as to facilitate the subsequent loss function to independently evaluate the two feature extraction sub-models, thereby realizing the independent and comparative model performance evaluation of adversarial learning, which is not limited in the embodiment of the present invention.

[0069] It can be seen that implementing the adversarial learning-based image processing method described in the embodiment of the present invention is conducive to generating adversarial examples of rPPG signals of different qualities and feeding them into the multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0070] In another optional embodiment, the first feature extraction sub-model includes a first network module, a second network module, a third network module, a grid optimization module, a one-dimensional conversion module, a multi-layer perceptron and a reshaping module; wherein the first network module includes a first convolution unit, a first normalization unit, a first activation unit, and a first pooling unit;

[0071] The input end of the first convolution unit is connected to the first model input of the initial image processing model, and the output end of the first convolution unit is connected to the input end of the first normalization unit; the output end of the first normalization unit is connected to the input end of the first activation unit; the output end of the first activation unit is connected to the input end of the first pooling unit; the output end of the first pooling unit is connected to the input end of the second network module; the output end of the second network module is connected to the input end of the third network module; the output end of the third network module is connected to the input end of the grid optimization module; the output end of the grid optimization module is connected to the input end of the one-dimensional conversion module; the output end of the one-dimensional conversion module is connected to the input end of the multilayer perceptron; the output end of the multilayer perceptron is connected to the input end of the reshaping module; the output end of the reshaping module is connected to the input end of the first fusion module.

[0072] It should be noted that the kernel size of the first convolution unit is 3x3x3, which is not limited in the embodiment of the present invention.

[0073] It should be noted that the above-mentioned first normalization unit is constructed based on the batch normalization layer, which is not limited in the embodiment of the present invention.

[0074] It should be noted that the above-mentioned first activation unit is constructed based on the RELU activation function, which is not limited in the embodiment of the present invention.

[0075] It should be noted that the above-mentioned first pooling unit is constructed based on the maximum pooling layer, and its parameter specification can be [1,2,2], which is not limited in the embodiment of the present invention.

[0076] It should be noted that the above-mentioned grid optimization module is constructed based on Fixed Grid Interpolation to map the extracted features to a fixed grid and mine the semantic information of the rPPG signal, which is not limited in the embodiment of the present invention.

[0077] It should be noted that the above-mentioned one-dimensional conversion module is constructed based on the Flatten layer, and mainly converts the input multi-dimensional data into one dimension, which is not limited in the embodiment of the present invention.

[0078] It should be noted that the number of layers of the multilayer perceptron can be between 2-15 (preferably, 3-8), and the number of neurons in each layer can be between 20-100 (preferably, 50-80), which is not limited in the embodiment of the present invention.

[0079] It should be noted that the above-mentioned reshaping module provides an efficient and flexible way to change the shape of data by using the reshape function to adapt to different data processing and analysis needs. The reshaping module not only improves the flexibility of data operations, but also helps to optimize memory usage, enhance data visualization, and support multi-dimensional data processing, which is not limited in the embodiments of the present invention.

[0080] It can be seen that implementing the adversarial learning-based image processing method described in the embodiment of the present invention is conducive to generating adversarial examples of rPPG signals of different qualities and feeding them into the multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0081] In another optional embodiment, the feature fusion sub-model includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a word segmentation module, a multi-head attention module, a second fusion module, a third fusion module, a first normalization module, a second normalization module and a prediction module; wherein,

[0082] The input end of the first convolution module is connected to the output end of the first fusion module, and the output end of the first convolution module is connected to the input end of the second convolution module; the output end of the second convolution module is connected to the input end of the third convolution module; the output end of the third convolution module is respectively connected to the input end of the word segmentation module and the input end of the second fusion module; the output end of the word segmentation module is connected to the input end of the multi-head attention module; the output end of the multi-head attention module is connected to the input end of the second fusion module; the output end of the second fusion module is connected to the input end of the first normalization module; the output end of the first normalization module is respectively connected to the input end of the fourth convolution module and the input end of the third fusion module; the output end of the fourth convolution module is connected to the input end of the fifth convolution module; the output end of the fifth convolution module is connected to the input end of the sixth convolution module; the output end of the sixth convolution module is connected to the input end of the third fusion module; the output end of the third fusion module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the prediction module; the output end of the prediction module is connected to the input end of the segmentation module.

[0083] It should be noted that the above-mentioned first normalization module and the second normalization module are constructed based on the batch normalization layer, which is not limited in the embodiment of the present invention.

[0084] It should be noted that the kernel size of the above-mentioned first convolution module is 1x5x5; the kernel sizes of the second convolution module, the third convolution module and the fifth convolution module are all 3x3x3, and they are depth convolution blocks; the kernel size of the fourth convolution module and the sixth convolution module is 1x1x1, which is not limited in the embodiment of the present invention.

[0085] It should be noted that the above-mentioned multi-head attention module is constructed based on the multi-head self-attention mechanism, which is not limited in the embodiments of the present invention.

[0086] It should be noted that the second fusion module and the third fusion module are constructed based on the Add operation, which is not limited in the embodiment of the present invention.

[0087] It should be noted that the above prediction module is constructed based on the predictor operation, and the prediction module outputs one-dimensional feature information, which can represent the main features of the original data set and reduce the complexity of the data, facilitating subsequent processing and analysis, which is not limited in the embodiments of the present invention.

[0088] It should be noted that the above-mentioned word segmentation module uses natural language processing (NLP) technology, such as machine learning, deep learning or rule engine, to achieve accurate segmentation of text. For example, using Tube word segmenter as a word segmentation module, by using the tube word segmenter, text data can be effectively divided into tube tokens with spatiotemporal attributes, which not only improves the accuracy and efficiency of text processing, but also provides strong support for further data analysis and knowledge discovery, which is not limited in the embodiments of the present invention.

[0089] It can be seen that implementing the adversarial learning-based image processing method described in the embodiment of the present invention is conducive to generating adversarial examples of rPPG signals of different qualities and feeding them into the multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0090] In an optional embodiment, the above-mentioned sample generation processing of the video information to obtain the initial image sample information includes:

[0091] Perform segmentation and conversion processing on the video information to obtain basic image sample information; the basic image sample information includes first basic sample set information and second basic sample set information;

[0092] Based on the basic image sample information, initial image sample information is determined.

[0093] It should be noted that, based on the basic image sample information, the initial image sample information is determined by interpolating and performing time-frequency domain and calculation processing on the PPG value (pulse wave signal, which can be a separate input setting for each image by the user, or data collected by the system's photosensor, which is not limited in the embodiment of the present invention) of the first basic sample set information and the second basic sample set in the first basic sample set information and the second basic sample set information to obtain the real label value, and then associating the real label value with the basic image sample to obtain the initial image sample of the initial image sample information, which is not limited in the embodiment of the present invention. For example, for a basic sample set (the first basic sample set, or the second basic sample set), the PPG discrete values ​​of each basic image sample in the basic sample set are interpolated by cubic spline to obtain a continuous waveform of the PPG signal, and the PPG true value sequence of the corresponding time point is calculated according to the time point of each face image (time point = face image sequence number / frame rate). Similarly, the L groups of PPG true value sequences are obtained by dividing the sequence into a group of M PPG data points, and finally a fast Fourier transform is performed to convert the time domain into the frequency domain, find the horizontal axis frequency corresponding to the highest peak in the frequency domain, and then multiply the horizontal axis frequency corresponding to the highest peak in the frequency domain by 60 to obtain the true label value corresponding to this basic sample set, that is, the true label values ​​of the same basic sample set are consistent, which is not limited in the embodiment of the present invention. Further, the above-mentioned true label value is between 40 and 180, which is not limited in the embodiment of the present invention.

[0094] It can be seen that implementing the adversarial learning-based image processing method described in the embodiment of the present invention is conducive to generating adversarial examples of rPPG signals of different qualities and feeding them into the multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0095] In another optional embodiment, segmentation and conversion processing is performed on the video information to obtain basic image sample information, including:

[0096] Performing face recognition extraction processing on the video information to obtain face image information; the face image information includes M face images; M is a positive integer not less than 160;

[0097] Sequentially dividing and processing the face image information to obtain face image set information; the face image set information includes L face image sets; each face image set includes N face images;

[0098] Performing two random image size cropping processes on the face image set information within the image specification range to obtain cropped image set information; the cropped image set information includes first cropped image set information and second cropped image set information; the first cropped image set information includes L first cropped image sets; the second cropped image set information includes L second cropped image sets;

[0099] Performing random flipping processing on the cropped image set information within a flipping angle range to obtain flipped image set information;

[0100] The flipped image set information is normalized to obtain basic image sample information.

[0101] It should be noted that the above-mentioned face recognition and extraction processing of the video information uses the RetinaFace face detection algorithm to save the face video as a face image, which is not limited in the embodiment of the present invention. Furthermore, the face images in the above-mentioned face image information are arranged in sequence according to the time sequence in the video, which is not limited in the embodiment of the present invention.

[0102] It should be noted that the above sequential division of facial image information is to divide N facial images arranged in chronological order into one facial image set. Further, when the number of facial images finally divided is less than N, these facial images are discarded, which is not limited in the embodiment of the present invention.

[0103] It should be noted that the above-mentioned random image size cropping processing is performed twice on the face image set information within the image specification range. The face images in each face image set are randomly cropped within the image specification range (image pixels are between 32×32 and 128×128) with an adjustment accuracy of 0.1, thereby converting a single-branch channel into a dual-branch channel. The adjusted image sequences in the dual branches are the first cropped image set information and the second cropped image set information, respectively. Using this random cropping method, not only the sizes of the first cropped image set information and the second cropped image set information are different, but also the internal sizes of the first cropped image set information and the second cropped image set information are different. They are all randomly cropped to image pixels between 32×32 and 128×128. This data enhancement method greatly enriches the data diversity, and the embodiments of the present invention are not limited thereto.

[0104] It should be noted that the above-mentioned random flipping of the cropped image set information within the flipping angle range is to randomly flip the image between [-90°, 90°], which is not limited in the embodiment of the present invention. . Further, the normalization processing of the flipped image set information is to divide the pixel value of each pixel in each image by the sum of the pixel values ​​corresponding to the pixels of the entire image to normalize the image, which is not limited in the embodiment of the present invention. Furthermore, random flipping further performs data enhancement operations to enhance the robustness of the model, and normalization facilitates model training and learning, which is not limited in the embodiment of the present invention.

[0105] It can be seen that implementing the adversarial learning-based image processing method described in the embodiment of the present invention is conducive to generating adversarial examples of rPPG signals of different qualities and feeding them into the multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0106] Embodiment 2

[0107] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an image processing device based on adversarial learning disclosed in an embodiment of the present invention. Figure 3 The described device can be applied to a management system, such as a local server or a cloud server for management, etc., which is not limited in the embodiments of the present invention. Figure 3 As shown, the device may include:

[0108] An acquisition module 201 is used to acquire image information to be processed;

[0109] The first processing module 202 is used to pre-process the image information to be processed to obtain initial image processing information;

[0110] The second processing module 203 is used to analyze and process the initial image processing information using the target image processing model to obtain image analysis result information.

[0111] It can be seen that the implementation Figure 3 The described adversarial learning-based image processing apparatus is beneficial for generating adversarial examples of rPPG signals of different qualities and feeding them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0112] In another optional embodiment, Figure 3 As shown, the target image processing model is obtained based on the following method:

[0113] Get video information;

[0114] Performing sample generation processing on the video information to obtain initial image sample information;

[0115] Based on the initial image sample information, determining the target image sample information;

[0116] Using the target image sample information to perform adversarial learning training on the initial image processing model, and obtaining model training result information and a training image processing model;

[0117] Determine whether the model training result information meets the model training termination condition and obtain the conditional judgment result;

[0118] When the result of the conditional judgment is no, the training image processing model is determined as a new initial image processing model, and the execution is triggered to determine the target image sample information based on the initial image sample information;

[0119] When the conditional judgment result is yes, the model parameters of the training image processing model are transferred to the basic image processing model to obtain the target image processing model; the model structure of the basic image processing model is partially consistent with that of the initial image processing model.

[0120] It can be seen that the implementation Figure 3 The described adversarial learning-based image processing apparatus is beneficial for generating adversarial examples of rPPG signals of different qualities and feeding them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0121] In yet another optional embodiment, Figure 3 As shown, the initial image processing model includes a first feature extraction sub-model, a second feature extraction sub-model, a first fusion module, a feature fusion sub-model and a segmentation model; wherein,

[0122] The input end of the first feature extraction sub-model is connected to the first model input of the initial image processing model, and the output end of the first feature extraction sub-model is connected to the input end of the first fusion module; the second feature extraction sub-model is connected to the second model input of the initial image processing model, and the output end of the second feature extraction sub-model is connected to the input end of the first fusion module; the output end of the first fusion module is connected to the input end of the feature fusion sub-model; the output end of the feature fusion sub-model is connected to the input end of the segmentation module; the output end of the segmentation module is connected to the first model output and the second model output of the initial image processing model.

[0123] It can be seen that the implementation Figure 3 The described adversarial learning-based image processing apparatus is beneficial for generating adversarial examples of rPPG signals of different qualities and feeding them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0124] In yet another optional embodiment, Figure 3 As shown, the first feature extraction sub-model includes a first network module, a second network module, a third network module, a grid optimization module, a one-dimensional conversion module, a multi-layer perceptron and a reshaping module; wherein the first network module includes a first convolution unit, a first normalization unit, a first activation unit, and a first pooling unit;

[0125] The input end of the first convolution unit is connected to the first model input of the initial image processing model, and the output end of the first convolution unit is connected to the input end of the first normalization unit; the output end of the first normalization unit is connected to the input end of the first activation unit; the output end of the first activation unit is connected to the input end of the first pooling unit; the output end of the first pooling unit is connected to the input end of the second network module; the output end of the second network module is connected to the input end of the third network module; the output end of the third network module is connected to the input end of the grid optimization module; the output end of the grid optimization module is connected to the input end of the one-dimensional conversion module; the output end of the one-dimensional conversion module is connected to the input end of the multilayer perceptron; the output end of the multilayer perceptron is connected to the input end of the reshaping module; the output end of the reshaping module is connected to the input end of the first fusion module.

[0126] It can be seen that the implementation Figure 3 The described adversarial learning-based image processing apparatus is beneficial for generating adversarial examples of rPPG signals of different qualities and feeding them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0127] In yet another optional embodiment, Figure 3 As shown, the feature fusion sub-model includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a word segmentation module, a multi-head attention module, a second fusion module, a third fusion module, a first normalization module, a second normalization module and a prediction module; wherein,

[0128] The input end of the first convolution module is connected to the output end of the first fusion module, and the output end of the first convolution module is connected to the input end of the second convolution module; the output end of the second convolution module is connected to the input end of the third convolution module; the output end of the third convolution module is respectively connected to the input end of the word segmentation module and the input end of the second fusion module; the output end of the word segmentation module is connected to the input end of the multi-head attention module; the output end of the multi-head attention module is connected to the input end of the second fusion module; the output end of the second fusion module is connected to the input end of the first normalization module; the output end of the first normalization module is respectively connected to the input end of the fourth convolution module and the input end of the third fusion module; the output end of the fourth convolution module is connected to the input end of the fifth convolution module; the output end of the fifth convolution module is connected to the input end of the sixth convolution module; the output end of the sixth convolution module is connected to the input end of the third fusion module; the output end of the third fusion module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the prediction module; the output end of the prediction module is connected to the input end of the segmentation module.

[0129] It can be seen that the implementation Figure 3The described adversarial learning-based image processing apparatus is beneficial for generating adversarial examples of rPPG signals of different qualities and feeding them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0130] In yet another optional embodiment, Figure 3 As shown, the second processing module 203 performs sample generation processing on the video information to obtain initial image sample information, including:

[0131] Perform segmentation and conversion processing on the video information to obtain basic image sample information; the basic image sample information includes first basic sample set information and second basic sample set information;

[0132] Based on the basic image sample information, initial image sample information is determined.

[0133] It can be seen that implementation Figure 3 The described adversarial learning-based image processing apparatus is beneficial for generating adversarial examples of rPPG signals of different qualities and feeding them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0134] In yet another optional embodiment, Figure 3 As shown, the second processing module 203 performs segmentation and conversion processing on the video information to obtain basic image sample information, including:

[0135] Performing face recognition extraction processing on the video information to obtain face image information; the face image information includes M face images; M is a positive integer not less than 160;

[0136] Sequentially dividing and processing the face image information to obtain face image set information; the face image set information includes L face image sets; each face image set includes N face images;

[0137] Performing two random image size cropping processes on the face image set information within the image specification range to obtain cropped image set information; the cropped image set information includes first cropped image set information and second cropped image set information; the first cropped image set information includes L first cropped image sets; the second cropped image set information includes L second cropped image sets;

[0138] Performing random flipping processing on the cropped image set information within a flipping angle range to obtain flipped image set information;

[0139] The flipped image set information is normalized to obtain basic image sample information.

[0140] It can be seen that implementation Figure 3The described adversarial learning-based image processing apparatus is beneficial for generating adversarial examples of rPPG signals of different qualities and feeding them into a multi-channel signal extraction network to achieve more accurate HR measurement, thereby solving the problem of recovering degraded rPPG signals from low-resolution images.

[0141] Embodiment 3

[0142] See also Figure 4 , Figure 4 is a schematic diagram of the structure of another image processing device based on adversarial learning disclosed in an embodiment of the present invention. Figure 4 The described device can be applied to a management system, such as a local server or a cloud server for management, etc., which is not limited in the embodiments of the present invention. Figure 4 As shown, the device may include:

[0143] A memory 301 storing executable program codes;

[0144] a processor 302 coupled to the memory 301;

[0145] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the image processing method based on adversarial learning described in the first embodiment.

[0146] Embodiment 4

[0147] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the image processing method based on adversarial learning described in the first embodiment.

[0148] Embodiment 5

[0149] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the image processing method based on adversarial learning described in Embodiment 1.

[0150] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0151] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0152] Finally, it should be noted that the image processing method and device based on adversarial learning disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical scheme of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical schemes described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical schemes from the spirit and scope of the technical schemes of the embodiments of the present invention.

Claims

1. An image processing method based on adversarial learning, characterized in that: The method comprises: Obtaining image information to be processed; Pre-processing the image information to be processed to obtain initial image processing information; The initial image processing information is analyzed and processed using a target image processing model to obtain image analysis result information.

2. The image processing method based on adversarial learning according to claim 1, characterized in that: The target image processing model is obtained based on the following method: Get video information; Performing sample generation processing on the video information to obtain initial image sample information; Based on the initial image sample information, determining target image sample information; Using the target image sample information to perform adversarial learning training on the initial image processing model to obtain model training result information and a training image processing model; Determine whether the model training result information meets the model training termination condition, and obtain a conditional judgment result; When the result of the conditional judgment is no, determining the training image processing model as a new initial image processing model, and triggering the execution of determining the target image sample information based on the initial image sample information; When the conditional judgment result is yes, the model parameters of the training image processing model are transferred to the basic image processing model to obtain the target image processing model; the model structure of the basic image processing model is partially consistent with that of the initial image processing model.

3. The image processing method based on adversarial learning according to claim 2, characterized in that: The initial image processing model includes a first feature extraction sub-model, a second feature extraction sub-model, a first fusion module, a feature fusion sub-model and a segmentation model; wherein, The input end of the first feature extraction sub-model is connected to the first model input of the initial image processing model, and the output end of the first feature extraction sub-model is connected to the input end of the first fusion module; the second feature extraction sub-model is connected to the second model input of the initial image processing model, and the output end of the second feature extraction sub-model is connected to the input end of the first fusion module; the output end of the first fusion module is connected to the input end of the feature fusion sub-model; the output end of the feature fusion sub-model is connected to the input end of the segmentation module; the output end of the segmentation module is connected to the first model output and the second model output of the initial image processing model.

4. The image processing method based on adversarial learning according to claim 3, characterized in that: The first feature extraction sub-model includes a first network module, a second network module, a third network module, a grid optimization module, a one-dimensional conversion module, a multi-layer perceptron and a reshaping module; wherein the first network module includes a first convolution unit, a first normalization unit, a first activation unit, and a first pooling unit; The input end of the first convolution unit is connected to the first model input of the initial image processing model, and the output end of the first convolution unit is connected to the input end of the first normalization unit; the output end of the first normalization unit is connected to the input end of the first activation unit; the output end of the first activation unit is connected to the input end of the first pooling unit; the output end of the first pooling unit is connected to the input end of the second network module; the output end of the second network module is connected to the input end of the third network module; the output end of the third network module is connected to the input end of the grid optimization module; the output end of the grid optimization module is connected to the input end of the one-dimensional conversion module; the output end of the one-dimensional conversion module is connected to the input end of the multi-layer perceptron; the output end of the multi-layer perceptron is connected to the input end of the reshaping module; the output end of the reshaping module is connected to the input end of the first fusion module.

5. The image processing method based on adversarial learning according to claim 3, characterized in that: The feature fusion sub-model includes a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a sixth convolution module, a word segmentation module, a multi-head attention module, a second fusion module, a third fusion module, a first normalization module, a second normalization module and a prediction module; wherein, The input end of the first convolution module is connected to the output end of the first fusion module, and the output end of the first convolution module is connected to the input end of the second convolution module; the output end of the second convolution module is connected to the input end of the third convolution module; the output end of the third convolution module is respectively connected to the input end of the word segmentation module and the input end of the second fusion module; the output end of the word segmentation module is connected to the input end of the multi-head attention module; the output end of the multi-head attention module is connected to the input end of the second fusion module; the output end of the second fusion module is connected to the input end of the first normalization module; the output end of the first normalization module is respectively connected to the input end of the fourth convolution module and the input end of the third fusion module; the output end of the fourth convolution module is connected to the input end of the fifth convolution module; the output end of the fifth convolution module is connected to the input end of the sixth convolution module; the output end of the sixth convolution module is connected to the input end of the third fusion module; the output end of the third fusion module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the prediction module; the output end of the prediction module is connected to the input end of the segmentation module.

6. The image processing method based on adversarial learning according to claim 2, characterized in that: The performing sample generation processing on the video information to obtain initial image sample information includes: Performing segmentation and conversion processing on the video information to obtain basic image sample information; the basic image sample information includes first basic sample set information and second basic sample set information; Based on the basic image sample information, initial image sample information is determined.

7. The image processing method based on adversarial learning according to claim 6, characterized in that: The segmentation and conversion processing of the video information to obtain basic image sample information includes: Performing face recognition and extraction processing on the video information to obtain face image information; the face image information includes M face images; M is a positive integer not less than 160; Sequentially dividing and processing the face image information to obtain face image set information; the face image set information includes L face image sets; each face image set includes N face images; Performing two random image size cropping processes on the face image set information within the image specification range to obtain cropped image set information; the cropped image set information includes first cropped image set information and second cropped image set information; the first cropped image set information includes the L first cropped image sets; the second cropped image set information includes the L second cropped image sets; Performing random flipping processing on the cropped image set information within a flipping angle range to obtain flipped image set information; The flipped image set information is normalized to obtain basic image sample information.

8. An image processing device based on adversarial learning, characterized in that: The device comprises: An acquisition module, used for acquiring image information to be processed; A first processing module, used for pre-processing the image information to be processed to obtain initial image processing information; The second processing module is used to analyze and process the initial image processing information using a target image processing model to obtain image analysis result information.

9. An image processing device based on adversarial learning, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the image processing method based on adversarial learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when called, are used to execute the image processing method based on adversarial learning as described in any one of claims 1 to 7.

Images