Pain level identification method, device, server and storage medium

Abstract

The present invention provides a method, device, server and storage medium for identifying pain levels, including: obtaining a first face image of a target object to be evaluated; performing pain evaluation on the target object based on the first face image through a pre-trained pain evaluation model to obtain the pain level of the target object; wherein the pain evaluation model is obtained by training multiple two-dimensional mapping images corresponding to a three-dimensional virtual pain expression, and the three-dimensional virtual pain expression is generated by a pre-trained data set generation network based on a second face image of the target object. The present invention can significantly improve the real-time performance of identifying pain levels.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, server and storage medium for identifying pain levels. Background Art

[0002] Pain is defined as unpleasant sensations and emotional experiences associated with actual or potential tissue damage or descriptions of such damage. Medical research has shown that extensive interactions between pain perception and autonomous control neural structures lead to increased sympathetic outflow, so most related technologies use physiological features such as EEG (Electroencephalogram), ECG (electrocardiogram), and EDA (Electrodermal activity) to assess pain. However, the extraction and use of physiological features are relatively complex. Specifically, because physiological signals are easily interfered by noise, they need to be denoised before extracting physiological features, which greatly increases the amount of computation in the system, resulting in poor real-time performance of pain assessment. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a method, device, server and storage medium for identifying pain levels, which can significantly improve the real-time performance of identifying pain levels.

[0004] In a first aspect, an embodiment of the present invention provides a method for identifying pain levels, comprising: obtaining a first facial image of a target object to be evaluated; performing pain evaluation on the target object based on the first facial image using a pre-trained pain evaluation model to obtain the pain level of the target object; wherein the pain evaluation model is trained based on multiple two-dimensional mapping images corresponding to a three-dimensional virtual pain expression, and the three-dimensional virtual pain expression is generated based on a second facial image of the target object using a pre-trained data set generation network.

[0005] In one embodiment, the step of performing pain assessment on the target object based on the first facial image using a pre-trained pain assessment model to obtain the pain level of the target object includes: extracting color texture features and facial shape features of the first facial image through the pain assessment model, and determining the pain level of the target object based on the color texture features and the facial shape features.

[0006] In one embodiment, the pain assessment model includes a first feature extraction unit, a second feature extraction unit and a multilayer perceptron; the step of extracting color texture features and facial shape features of the first facial image through the pain assessment model, and determining the pain level of the target object based on the color texture features and the facial shape features, includes: extracting the color texture features of the first facial image through the first feature extraction unit, and extracting the facial shape features of the first facial image through the second feature extraction unit; performing feature fusion on the color texture features and the facial shape features to obtain fused features, and determining the pain level of the target object based on the fused features through the multilayer perceptron.

[0007] In one embodiment, the training step of the pain assessment model includes: obtaining a second facial image of the target object; wherein the first facial image and the second facial image are both two-dimensional facial images; generating a plurality of three-dimensional virtual pain expressions corresponding to the second facial image through a pre-trained data set generation network; mapping each of the three-dimensional virtual pain expressions according to a plurality of preset mapping angles to obtain a plurality of two-dimensional mapping images corresponding to each of the three-dimensional virtual pain expressions; constructing a training image set based on each of the two-dimensional mapping images, and using the training image set to train the pain assessment model; wherein the training image set includes a plurality of two-dimensional mapping images and a pain label annotated on each of the two-dimensional mapping images.

[0008] In one embodiment, the data set generation network includes a pain expression detection subnetwork and a pain expression generation subnetwork, and the output end of the pain expression detection subnetwork is connected to the input end of the pain expression generation subnetwork; the step of generating multiple three-dimensional virtual pain expressions corresponding to the second facial image through the pre-trained data set generation network includes: detecting at least one facial AU value corresponding to the second facial image through the pain expression detection subnetwork; wherein the facial AU value is used to characterize the pain expression contained in the second facial image; and generating a three-dimensional virtual pain expression corresponding to the second facial image through the pain expression generation subnetwork based on a preset three-dimensional facial model and each of the facial AU values.

[0009] In one embodiment, the step of detecting at least one facial AU value corresponding to the second facial image through the pain expression detection subnetwork includes: detecting key points of the two-dimensional facial image through the pain expression detection subnetwork, dividing the two-dimensional facial image into multiple regions of interest based on the key points, and determining the facial AU value corresponding to each region of interest; the pain expression generation subnetwork adopts a generative adversarial network, and the generative adversarial network includes a generator; the step of generating a three-dimensional virtual pain expression corresponding to the second facial image based on a preset three-dimensional facial model and each facial AU value through the pain expression generation subnetwork includes: determining the expression adjustment action corresponding to each facial AU value through the generator, and adjusting the current expression of the three-dimensional facial model based on the expression adjustment action to obtain the three-dimensional virtual pain expression corresponding to the two-dimensional facial image.

[0010] In one embodiment, the pain expression generation subnetwork also includes a discriminator; the training step of the pain expression generation subnetwork includes: obtaining the three-dimensional face model and at least one training AU value; wherein the training AU value is obtained by detecting the training face image by the pain expression detection subnetwork; generating at least one training virtual pain expression based on the three-dimensional face model and each training AU value by the generator; wherein the pain level corresponding to each training virtual pain expression is the same; generating an orthogonal mapping image corresponding to each training virtual pain expression; determining the data discrimination result of the orthogonal mapping image corresponding to each training virtual pain expression based on the training face image by the discriminator; and training the generator and the discriminator based on the data discrimination result.

[0011] In a second aspect, an embodiment of the present invention further provides a pain level identification device, comprising: an image acquisition module, used to acquire a first facial image of a target object to be evaluated; a pain assessment module, used to perform pain assessment on the target object based on the first facial image through a pre-trained pain assessment model, to obtain the pain level of the target object; wherein the pain assessment model is trained based on multiple two-dimensional mapping images corresponding to a three-dimensional virtual pain expression, and the three-dimensional virtual pain expression is generated based on a second facial image of the target object through a pre-trained data set generation network.

[0012] In a third aspect, an embodiment of the present invention further provides a server, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement any one of the methods provided in the first aspect.

[0013] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0014] The pain level recognition method, device, server and storage medium provided in the embodiments of the present invention first obtain a first facial image of a target object to be evaluated, and then perform a pain evaluation on the target object based on the first facial image using a pre-trained pain evaluation model to obtain the target object's pain level. The pain evaluation model is obtained based on three-dimensional virtual pain expression training, and the three-dimensional virtual pain expression is generated based on the second facial image of the target object through a pre-trained data set generation network. The above method uses a data set generation network to generate multiple three-dimensional virtual pain expressions corresponding to the second facial image, and then trains the pain evaluation model based on the three-dimensional virtual pain expression to improve the problem that the pain evaluation model cannot be effectively trained due to the small amount of training data, and the trained pain evaluation model can directly determine the target object's pain level based on the first facial image, thereby significantly improving the real-time performance of identifying the pain level.

[0015] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A framework diagram of an automatic pain recognition system provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a flow chart of a method for identifying pain levels provided by an embodiment of the present invention;

[0020] Figure 3 An overall framework diagram of a pain assessment model provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the structure of a pain expression detection subnetwork provided by an embodiment of the present invention;

[0022] Figure 5 A schematic diagram of the structure of a pain expression generation subnetwork provided by an embodiment of the present invention;

[0023] Figure 6 A schematic diagram of the structure of a pain assessment model provided by an embodiment of the present invention;

[0024] Figure 7 A schematic diagram of the structure of a pain level recognition device provided by an embodiment of the present invention;

[0025] Figure 8 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] At present, pain is one of the symptoms that should be addressed first in critically ill patients. Literature reports that 70% of critically ill patients are in moderate / severe pain. For example, Europe reported that more than 50% of ICU (Intensive Care Unit) patients in 28 countries experienced moderate to severe pain, and more than 60% of ICU patients in the United States still had unrelieved pain when they were discharged. Pain that is not treated in time will have a serious negative impact on critically ill patients. Pain-induced increased catecholamine release, immunosuppression, increased metabolic rate and other mechanisms lead to serious clinical consequences such as hemodynamic instability (increased blood pressure, tachycardia), shortness of breath, delirium, hyperglycemia, and urinary retention in clinically critically ill patients.

[0028] Pain management driven by "assessment" is believed to improve the clinical prognosis of critically ill patients, such as shortening the duration of mechanical ventilation, reducing the length of ICU stay, and reducing complications. Accurate and timely pain assessment is a prerequisite for accurate titration of analgesic and sedative doses. Pain assessment methods mainly include self-assessment and behavioral assessment. For critically ill patients who can communicate reliably (such as verbally, in writing, on communication boards, or through gestures such as nodding and shaking their heads), self-assessment scales can be used for pain assessment. The 0-10 numeric rating scale (NRS) is recommended. Behavioral assessment tools are often used in critically ill patients who cannot communicate or self-report pain. The Behavioral Pain Scale (BPS) shown in Table 1 and the Critical-Care Pain Observation Tool (CPOT) shown in Table 2 are recommended.

[0029] Table 1

[0030]

[0031]

[0032] Table 2

[0033]

[0034]

[0035] Although pain in the ICU has been given full attention in the guidelines in the past decade, and the validity and reliability of current pain assessment methods have been verified in many studies, the proportion of critically ill patients whose pain cannot be assessed and treated in a timely manner is still higher than 20%. In clinical practice, there are many obstacles to the assessment and management of pain, which come from many aspects. First, the differences in the knowledge level and enthusiasm of medical staff in the assessment and management of pain in critically ill patients lead to deviations in the accuracy of the assessment. Secondly, the ICU has a high workload and a lack of nursing human resources, and accurate pain assessment will bring a lot of workload to medical staff. Thirdly, good pain management depends on repeated dynamic assessments before and after treatment. However, due to the workload, it is difficult for medical staff to conduct assessments at any time, and clinical staff often adopt the scheme of assessment at certain time intervals. Therefore, the demand for automatic pain recognition systems is increasing. Automatic pain recognition systems can reduce the workload of medical staff, achieve accurate and dynamic pain assessments, and provide objective references for clinicians.

[0036] The features used by the automatic pain recognition system mainly include facial expression features, body movement features, voice features, physiological features, etc. Figure 1The framework diagram of an automatic pain recognition system is shown in Figure 1, where facial expression is one of the most common indicators of pain, and facial expression is an essential part of all currently publicly released pain-related databases. Facial expression is defined as the movement and distortion of facial muscles associated with painful stimuli, and facial movement can be described by activity units (AUs).

[0037] Related technologies have proposed exploring deep visual features and facial geometric features, including head movement, facial key points and facial activity units, and combining attention mechanisms and long short-term memory recurrent neural networks to achieve pain recognition on the EmoPain database. Related technologies have also proposed a multi-stream convolutional neural network that extracts corresponding features for several areas most relevant to pain, and combines these features through adaptive weights to predict pain intensity.

[0038] Medical research has shown that the extensive interaction between pain perception and the neural structures of autonomic control leads to increased sympathetic outflow, which causes sweat to be discharged into the pores on the surface of the skin. The secretion of sweat changes the electrical properties of the skin; cardiovascular changes affect heart rate; pain processing involves a complex network of cerebral cortical areas, which affects the electrical and metabolic activities of these areas. Therefore, there are many studies that use physiological features such as EEG, ECG, and EDA to assess pain. Specifically, the amplitude, frequency, stationarity, entropy, and linear features of physiological signals sEMG (surface electromyography, which measures the movement of corrugator muscles, zygomatic muscles, etc.) and EDA can be explored. In addition, similarity features can also be considered, which are used to measure the similarity between the current signal and the average baseline signal of a specific person. Many subsequent works are based on the above-mentioned signal features or subsets thereof, or combine these signal feature subsets with additional features, such as EDA signals being decomposed into phase and tension components. Different from extracting the heart rate variability features of ECG signals, such as extracting general statistical features from ECG signals without considering any frequency information. In another technology, physiological features are binarized into normal and abnormal states and then input into the classifier for classification. However, the extraction and utilization of physiological features are relatively complicated, because physiological signals are easily interfered by noise, so noise reduction processing is required before extraction, which greatly increases the amount of calculation of the system, and thus cannot achieve the real-time performance of the system well.

[0039] Based on the features extracted above, a single feature can be used for decision making, or a corresponding fusion strategy can be used for decision making. Feature-level fusion connects the features of all modalities into a single high-dimensional feature vector, and then uses the cascaded feature vector to train a single classifier for classification; decision-level fusion aims to merge the decision results of multiple classifiers into a single decision. The Hammal team proposed a method for pain classification using a transferable belief model (TBM) to fuse fixed facial features (such as eyes, eyebrows, etc.), instantaneous features (such as frowns at the root of the nose), and background information.

[0040] However, the above studies were all conducted on public standard data sets, and the experimenters were mostly recruited adult men and women, or special groups (such as infants). They cannot meet the differences and particularities of patients in real hospitals, and cannot solve the problems of small data volume or data imbalance. For example, elderly people with wrinkles on their faces, patients with facial injuries, patients with minor pain, etc. Moreover, these methods are all studied from an experimental perspective and cannot cope with real complex scenarios, and therefore cannot be truly applied in clinical practice.

[0041] Based on this, the present invention provides a method, device, server and storage medium for identifying pain levels, which can significantly improve the real-time performance of identifying pain levels.

[0042] To facilitate understanding of this embodiment, a pain level recognition method disclosed in an embodiment of the present invention is first described in detail. Figure 2 The flowchart of a method for identifying pain levels is shown in FIG. 1 , and the method mainly includes the following steps S202 to S204:

[0043] Step S202, obtaining a first facial image of the target object to be evaluated. The first facial image may be a two-dimensional facial image. In one embodiment, a two-dimensional facial image of the target object's facial region acquired by an image acquisition device (such as a mobile phone or a camera device) may be obtained, and the target object may be a patient.

[0044] Step S204, performing pain assessment on the target object based on the first facial image using a pre-trained pain assessment model to obtain the target object's pain level. The pain assessment model is trained based on multiple two-dimensional mapping images corresponding to the three-dimensional virtual pain expression, and the three-dimensional virtual pain expression is generated by a pre-trained data set generation network based on the second facial image of the target object. The second facial image may also be a two-dimensional facial image of the target object, the input of the pain assessment model is a two-dimensional facial image, and the output is a pain level.

[0045] In one embodiment, a plurality of three-dimensional virtual pain expressions corresponding to the second face image can be generated by a data set generation network, and the three-dimensional virtual pain expressions are used to construct a training data set for training a pain assessment model, the training data set including a plurality of two-dimensional mapping images corresponding to each three-dimensional virtual pain expression and a pain label corresponding to each two-dimensional mapping image, the pain label being the real pain level. In a specific implementation, the data set generation network includes a pain expression detection subnetwork and a pain expression generation subnetwork, the output end of the pain expression detection subnetwork is connected to the input end of the pain expression generation subnetwork, the pain expression detection subnetwork is used to detect a plurality of face AU values ​​corresponding to the second face image, and the pain expression generation subnetwork is used to generate a three-dimensional virtual pain expression corresponding to the second face image.

[0046] The pain level recognition method provided in an embodiment of the present invention utilizes a data set generation network to generate multiple three-dimensional virtual pain expressions corresponding to a second facial image, thereby training a pain assessment model based on the three-dimensional virtual pain expressions to improve the problem of being unable to effectively train the pain assessment model due to a small amount of training data. Moreover, the trained pain assessment model can directly determine the pain level of the target object based on the first facial image, thereby significantly improving the real-time performance of identifying the pain level.

[0047] To facilitate understanding of the above embodiments, the present invention also provides an overall framework of a pain assessment model, see Figure 3 The overall framework diagram of a pain assessment model is shown in Figure 3 The diagram shows the connection between the pain assessment model and the data set generation network. The data set generation network includes a pain expression detection subnetwork and a pain expression generation subnetwork. The pain expression generation subnetwork includes a generator and a discriminator. The output of the pain expression detection subnetwork is connected to the input of the generator in the pain expression generation subnetwork. The output of the generator in the pain expression generation subnetwork is connected to the input of the pain assessment model. In addition, the output of the generator in the pain expression generation subnetwork is also connected to the input of the discriminator. The input of the pain expression detection subnetwork is the second face image, and the output is at least one AU value; the input of the pain expression generation subnetwork is the AU value and the three-dimensional face model, and the output is a three-dimensional virtual pain expression; in the training stage, the input of the pain assessment model is multiple two-dimensional mapping images corresponding to the three-dimensional virtual pain expression, and the output is the pain level; in the application stage, the input of the pain assessment model is the first face image, and the output is the pain level.

[0048] In the aforementioned Figure 3 Based on this, an embodiment of the present invention provides an implementation method for training a pain assessment model, see steps 1 to 4 below:

[0049] Step 1, obtaining a second facial image of the target object; wherein the first facial image and the second facial image are both two-dimensional facial images.

[0050] Step 2: Generate a network using a pre-trained dataset to generate multiple three-dimensional virtual pain expressions corresponding to the second face image. Figure 3 ,exist Figure 3 Based on this, an embodiment of the present invention provides an implementation method for generating multiple three-dimensional virtual pain expressions corresponding to a two-dimensional face image, which can be specifically referred to in the following steps 2.1 to 2.2:

[0051] Step 2.1, detecting at least one face AU value corresponding to the second face image through the pain expression detection subnetwork. The face AU value is used to characterize the pain expression contained in the second face image, and the pain expression detection subnetwork can use an AUR-CNN (Region-CNN) network. Exemplarily, an embodiment of the present invention provides a mapping relationship between AU values ​​and expressions as shown in Table 3 below, wherein AU4, AU6, AU7, AU8, AU9, AU10, AU12, AU20, AU25, AU26, and AU46 in Table 3 are AU values ​​related to pain expressions.

[0052] Table 3

[0053] AU1 Inner eyebrow lift AU13 Pull the corners of your mouth upward AU25 Lips parted to show teeth AU2 External eyebrow lift AU14 The corners of the mouth are downward and the teeth are retracted AU26 Lips parted to see tongue AU4 Eyebrows drooping overall AU15 Pull the corners of your mouth vertically downward AU27 The lips parted and the throat was revealed AU5 Lift the upper eyelid AU16 Pull the lower lip down AU28 Sucking lips over teeth AU6 Lift your cheeks AU17 Squeeze the lower lip upward AU41 Slightly drooping upper eyelids AU7 Eye contraction AU18 Pucker your mouth in the middle AU42 Drooping upper eyelids AU9 Retract and lift the nose AU20 Lips pulled back AU43 Close your eyes AU10 Lift your upper lip AU22 Pout your lips into a funnel shape AU44 Lower eyelid pushes up AU11 Deepen the middle nasolabial area AU23 Tighten your lips into a straight line AU45 Blink AU12 Raise the corners of your mouth AU24 Squeeze your lips together AU46 Blink

[0054] Step 2.2, generate a three-dimensional virtual pain expression corresponding to the second face image through the pain expression generation subnetwork based on the preset three-dimensional face model and each face AU value. Among them, the pain expression generation subnetwork can adopt a generative adversarial network, and the pain expression generation subnetwork includes a generator and a discriminator. The input of the generator is the three-dimensional face model and each face AU value, and the output is a plurality of three-dimensional virtual pain expressions. The input of the discriminator is the orthogonal mapping image and the two-dimensional face image corresponding to each three-dimensional virtual pain expression, and the output is a data discrimination result, which includes the true and false probability p of the orthogonal mapping image corresponding to each three-dimensional virtual pain expression. In the specific implementation, the generator and the discriminator are constantly confronting each other, and finally the generator and the discriminator reach a dynamic equilibrium, the three-dimensional virtual pain expression generated by the generator is close to the real face distribution, and the discriminator cannot recognize the true and false of the orthogonal mapping image corresponding to the three-dimensional virtual pain expression.

[0055] Step 3, mapping each three-dimensional virtual pain expression according to a plurality of preset mapping angles, and obtaining a plurality of two-dimensional mapping images corresponding to each three-dimensional virtual pain expression. In practical applications, for each three-dimensional virtual pain expression, by mapping the three-dimensional virtual pain expression to a plurality of mapping angles, a two-dimensional mapping image corresponding to each mapping angle can be obtained, and the pain label corresponding to each two-dimensional mapping image is the same (that is, the real pain level is the same), thereby significantly enriching the training data set used to train the personalized pain assessment model of the target object, so that the trained pain assessment model has higher accuracy.

[0056] Step 4, construct a training image set based on each two-dimensional mapping image, and use the training image set to train the pain assessment model; wherein the training image set includes multiple two-dimensional mapping images and pain labels annotated on each two-dimensional mapping image. In one embodiment, the pain label can be determined from self-reports or observer pain reports provided by doctors and patients' families to determine the pain label corresponding to each two-dimensional mapping image, and on this basis, the pain assessment model is trained until the preset requirements are met, wherein the preset requirements can be reaching a preset number of iterations, or the loss value of the pain assessment model converges, etc.

[0057] The training method of the pain assessment model proposed in the embodiment of the present invention is the first to reconstruct the three-dimensional virtual pain expression of the target object from the two-dimensional face image based on the generative adversarial network (that is, the above-mentioned pain expression generation subnetwork). The pain expression generation subnetwork is composed of a generator and a discriminator. The generator learns the distribution of real two-dimensional face images based on reinforcement learning, so as to make the three-dimensional virtual pain expression generated by itself more realistic. The discriminator distinguishes the true and false of the orthogonal mapping images of the generated three-dimensional virtual pain expression. By training the generator and the discriminator, the generator and the discriminator are constantly confronting each other, and finally reach a dynamic equilibrium. The three-dimensional virtual pain expression generated by the generator is close to the real face distribution, while the discriminator cannot recognize the true and false of the orthogonal mapping image corresponding to the three-dimensional virtual pain expression.

[0058] For ease of understanding, the embodiment of the present invention also provides an implementation method for detecting at least one facial AU value corresponding to a two-dimensional facial image through a pain expression detection subnetwork. Specifically, the key points of the two-dimensional facial image can be detected through the pain expression detection subnetwork, and the two-dimensional facial image can be divided into multiple regions of interest (ROIs) based on the key points, and the facial AU value corresponding to each region of interest is determined. Among them, the number of key points can be 68, and the regions of interest can include the mouth region, nose region, eye region, eyebrow region, etc. The two-dimensional facial image can be divided into multiple regions of interest based on actual needs. For example, see Figure 4The structure diagram of a pain expression detection subnetwork is shown, and the pain expression detection subnetwork can divide the two-dimensional face image into multiple regions of interest according to 68 key points and generate a bounding box corresponding to each region of interest, and use the right detection head to detect the face AU value corresponding to each region of interest. The embodiment of the present invention uses the above-mentioned AU R-CNN network and the key points of the face to eliminate the differences in facial features as much as possible, so that the face AU value can be detected more subtly and accurately.

[0059] In practical applications, in order to improve the accuracy and training efficiency of the face AU value output by the pain expression detection subnetwork, the pain expression detection subnetwork can be pre-trained using the EmotioNet and GFT (Google flu trends) databases to improve the training efficiency of the subnetwork. Further, self-reports or observer pain reports provided by doctors and patients' families can be obtained, and the true AU value can be determined from the above report, and the true AU value can be used as a training label (ground-truth label). It should be noted that the training label also needs to be divided according to the region of interest, and since there may be multiple pain expressions in the same region of interest, that is, corresponding to multiple face AU values, each region of interest can be marked with multiple training labels. The embodiment of the present invention can use the patient's two-dimensional face image, self-report or observer pain report provided by doctors and patients' families to generate different three-dimensional virtual pain expressions corresponding to the patient's pain intensity, and then use the three-dimensional pain expression to orthogonally map the patient's two-dimensional facial data to train a personalized pain assessment model.

[0060] Preferably, the sigmoid cross entropy loss function can be used to calculate the loss value and back propagate to optimize the network parameters of the pain expression detection sub-network, wherein the cross entropy loss function is as follows:

[0061]

[0062] Among them, y is the training AU value, which is also the face AU value output by the expression detection sub-network during the training phase. is the true AU value, R is the total number of training face images, which are the two-dimensional face images input to the expression detection subnetwork during the training phase, L is the total number of regions of interest in the training face images, and y r,l That is, the training AU value corresponding to the lth region of interest in the rth training face image.

[0063] For ease of understanding, an embodiment of the present invention also provides an implementation method for generating a three-dimensional virtual pain expression corresponding to a second face image through a pain expression generation subnetwork based on a preset three-dimensional face model and each face AU value. Specifically, the generator can determine the expression adjustment action corresponding to each face AU value, and adjust the current expression of the three-dimensional face model (also referred to as the current three-dimensional face expression) based on the expression adjustment action to obtain a three-dimensional virtual pain expression corresponding to the two-dimensional face image. Among them, the three-dimensional face model can adopt 3DMM (3D facial deformation statistical model). In one implementation, the generator is the Agent in reinforcement learning, with the corresponding face AU value as action a t , the current 3D facial expression of the 3D face model is taken as state s t Different expression adjustment actions can be obtained according to different states, and the expression adjustment action is recorded as τ={s1,a1,s2,a2...s T ,a T}, adjust the current three-dimensional facial expression of the three-dimensional facial model according to the expression adjustment action to obtain a corresponding three-dimensional virtual pain expression.

[0064] In practical applications, in order to make the simulation of pain more standardized and accurate, we first use the 3D face model and expression database for pre-training, and then migrate to use the UNBC shoulder pain dataset for training. Figure 5 The structure diagram of a pain expression generation sub-network is shown. On this basis, the embodiment of the present invention also provides an implementation method for training the pain expression generation sub-network, see the following steps a to e:

[0065] Step a, obtaining a three-dimensional face model and at least one training AU value, wherein the training AU value is obtained by detecting the training face image through the pain expression detection sub-network, and the training AU value is also the face AU value outputted by the expression detection sub-network during the training phase.

[0066] Step b, generating at least one training virtual pain expression based on the three-dimensional face model and each training AU value through a generator. The pain level corresponding to each training virtual pain expression is the same. Considering that in actual applications, a patient has less training data, in order to customize a personalized pain assessment model for the patient, at least one training virtual pain expression is generated by a generator in an embodiment of the present invention, that is, different three-dimensional virtual pain expressions of corresponding pain levels are reconstructed from the patient's two-dimensional face image to enrich the training samples of the pain expression generation subnetwork. The specific process of generating a training virtual pain expression can refer to the aforementioned process of generating a three-dimensional virtual pain expression, which will not be repeated in the embodiment of the present invention.

[0067] Step c, generating an orthogonal mapping image corresponding to each training virtual pain expression. In one embodiment, for each training virtual pain expression, the front side of the training virtual pain expression may be mapped to obtain an orthogonal mapping image corresponding to the training virtual pain expression.

[0068] Step d, using a discriminator to determine the data discrimination result of the orthogonal mapping image corresponding to each training virtual pain expression based on the training face image. In one embodiment, the discriminator is composed of a ResNet-50 network, which converts the current Agent's action a t And orthogonal mapping pictures, as well as training face images and real AU values ​​as input, the discriminator strives to distinguish whether the orthogonal mapping picture is a training face image or an orthogonal mapping picture of training virtual pain expression, and outputs the true or false probability p, which is the data discrimination result.

[0069] Step e: training the generator and the discriminator based on the data discrimination result. In one embodiment, the reward value r can be calculated using the following formula: t :

[0070] Among them, ε is a constant, which can be set manually based on actual needs. The role of ε is to make the denominator of the reward value calculation formula not equal to 0.

[0071] Furthermore, according to the above reward value r t Calculate the cumulative average return D corresponding to compensation T (τ) , the cumulative average return D (τ) As shown below:

[0072]

[0073] Furthermore, the network parameters of the pain expression generation subnetwork are updated according to the above-mentioned cumulative average return and policy gradient, wherein the updating process is as follows:

[0074] Among them, π is the policy gradient, θ is the network parameter, η is the hyperparameter, is the gradient.

[0075] Through the training method provided in the above-mentioned embodiment, a pain assessment model with high accuracy can be obtained. On this basis, the embodiment of the present invention also provides an implementation method for evaluating the pain of a target object based on a first face image through a pre-trained pain assessment model to obtain the target object's pain level. The color texture features and facial shape features of each first mapping image can be extracted through the pain assessment model, and the target object's pain level can be determined based on the color texture features and facial shape features. The pain recognition subnetwork provided in the embodiment of the present invention not only uses expression information (that is, multiple two-dimensional mapping images corresponding to the above-mentioned three-dimensional virtual pain expression), but also uses specific patient color texture information (that is, the above-mentioned color texture features) and facial geometric contour information (that is, the above-mentioned facial shape features) to better evaluate the pain level of a specific patient.

[0076] In one embodiment, the pain assessment model may include a first feature extraction unit, a second feature extraction unit and a multilayer perceptron (MLP), the inputs of the first feature extraction unit and the second feature extraction unit are both first face images, the output of the first feature extraction unit is color and texture features, the output of the second feature extraction unit is facial shape features, the input of the multilayer perceptron is a fusion feature of color and texture features and facial shape features, and the output is a pain level.

[0077] The embodiment of the present invention not only uses RGB (optical three primary colors) two-dimensional mapping images to capture the color and texture features of the face, but also extracts facial landmark data to describe the facial shape features (also known as facial geometric relationships). These two descriptors (i.e., color and texture features and facial shape features) have different statistical characteristics. Therefore, the fusion of different descriptors can jointly train the pain assessment model through the extracted potential representations to complement each other and improve the accuracy of the pain assessment model recognition.

[0078] In practical applications, the pain assessment model includes a first feature extraction unit, a second feature extraction unit and a multi-layer perceptron, such as Figure 6 A schematic diagram of the structure of a pain assessment model is shown in FIG. 1 , wherein the first feature extraction unit can adopt a SANET network and the second feature extraction unit can adopt an SDNET network. Studies have shown that a single feature has a weak ability to describe pain and will cause interference when the scene changes. RGB face image data has rich facial appearance information, including color and texture. However, the image data is easily affected by various lighting conditions. The facial landmark data extracted from the face conveys information describing the geometric relationship in the face shape, but the landmarks do not contain any information about texture and color. The two descriptors have different statistical properties. Therefore, the fusion of different descriptors can achieve better recognition results by complementing each other through the extracted latent representations.

[0079] exist Figure 6 Based on the above, the embodiment of the present invention also provides an implementation method for extracting the color texture features and facial shape features of the first face image through a pain assessment model, and determining the pain level of the target object based on the color texture features and facial shape features, see the following (1) to (4):

[0080] (1) The color and texture features of the first face image are extracted by the first feature extraction unit. In one embodiment, Dlib is used to capture the feature points of the two-dimensional mapping image, and face alignment and masking are performed. The face area is extracted and background information is removed using a convex hull algorithm, which is then input into a SANET network. The SANET network is composed of a two-dimensional convolutional layer, batch normalization (BN), a rectified linear unit (ReLU) activation function, and Maxpooling. The SANET network will output 256-dimensional color and texture features.

[0081] (2) extracting facial shape features of the first face image by a second feature extraction unit. In one embodiment, SDNET is a multi-layer perceptron (MLP) composed of a series of fully connected layers, which maps the extracted facial shape features to the same dimension as the color and texture features, that is, SDNET will output 256-dimensional facial shape features.

[0082] (3) Perform feature fusion on the color texture features and the facial shape features to obtain fused features. In one embodiment, the color texture features and the facial shape features can be fused into 512-dimensional features using a feature-level tandem fusion strategy.

[0083] (4) Determining the pain level of the target object based on the fusion features by using a multi-layer perceptron In one embodiment, the PSPI value of the target object can be predicted by using a multi-layer perceptron to assess the pain level.

[0084] In practical applications, two-dimensional facial images of the corresponding patients can be collected first, and doctors and patients' families can mark the pain. Different levels of painful expressions are generated based on the collected images and the extracted facial AU values ​​to expand the pain data information of specific patients. Finally, the expression data generated by the patients is used to train the pain assessment model to generate a pain assessment model that can personalize the recognition of the patient's painful expression. When applied, the camera captures the facial video information of the corresponding patient, and the input into the pain model can evaluate the patient's pain level in real time.

[0085] In summary, the pain level recognition method provided by the embodiment of the present invention has at least the following characteristics:

[0086] (1) For the first time, we reconstruct different three-dimensional virtual pain facial expressions of corresponding pain intensity from patients’ two-dimensional pain pictures based on generative adversarial imitation learning.

[0087] (2) Based on the AU R-CNN network, the facial key points are used to eliminate the differences in facial features as much as possible, and facial AUs are detected more subtly and accurately.

[0088] (3) The pain recognition model not only uses facial image expression information, but also uses the facial geometric contour information of specific patients to better assess the pain level of specific patients.

[0089] (4) The patient's two-dimensional pain expression, self-report, or observer pain report provided by doctors and patients' families can be used to generate different three-dimensional virtual facial expressions corresponding to the patient's pain intensity. The three-dimensional facial expression is then orthogonally mapped to the patient's two-dimensional facial data to train a personalized pain assessment model, which solves the problems of unbalanced patient data and individual differences, and has better accuracy and generalization.

[0090] For the pain level recognition method provided in the above embodiment, the present invention provides a pain level recognition device, see Figure 7 The structure diagram of a pain level recognition device shown in FIG. 1 mainly includes the following parts:

[0091] An image acquisition module 702 is used to acquire a first face image of a target object to be evaluated;

[0092] A pain assessment module 704 is used to perform pain assessment on the target object based on the first face image using a pre-trained pain assessment model to obtain a pain level of the target object;

[0093] Among them, the pain assessment model is trained based on multiple two-dimensional mapping images corresponding to the three-dimensional virtual pain expression, and the three-dimensional virtual pain expression is generated by a pre-trained data set generation network based on the second face image of the target object.

[0094] The pain level recognition device provided in an embodiment of the present invention utilizes a data set generation network to generate multiple three-dimensional virtual pain expressions corresponding to a second facial image, thereby training a pain assessment model based on the three-dimensional virtual pain expressions to improve the problem of being unable to effectively train the pain assessment model due to a small amount of training data. Moreover, the trained pain assessment model can directly determine the pain level of the target object based on the first facial image, thereby significantly improving the real-time performance of identifying the pain level.

[0095] In one embodiment, the pain assessment module 704 is further used to: extract color texture features and facial shape features of the first facial image through the pain assessment model, and determine the pain level of the target object based on the color texture features and the facial shape features.

[0096] In one embodiment, the pain assessment model includes a first feature extraction unit, a second feature extraction unit and a multi-layer perceptron; the pain assessment module 704 is also used to: extract the color and texture features of the first facial image through the first feature extraction unit, and extract the facial shape features of the first facial image through the second feature extraction unit; perform feature fusion on the color and texture features and the facial shape features to obtain fused features, and determine the pain level of the target object based on the fused features through the multi-layer perceptron.

[0097] In one embodiment, the above-mentioned device also includes a training module, which is used to: obtain a second facial image of the target object; wherein the first facial image and the second facial image are both two-dimensional facial images; generate multiple three-dimensional virtual pain expressions corresponding to the second facial image through a pre-trained data set generation network; map each of the three-dimensional virtual pain expressions according to a plurality of preset mapping angles to obtain multiple two-dimensional mapping images corresponding to each of the three-dimensional virtual pain expressions; construct a training image set based on each of the two-dimensional mapping images, and use the training image set to train the pain assessment model; wherein the training image set includes multiple two-dimensional mapping images and a pain label annotated on each of the two-dimensional mapping images.

[0098] In one embodiment, the data set generation network includes a pain expression detection subnetwork and a pain expression generation subnetwork, and the output end of the pain expression detection subnetwork is connected to the input end of the pain expression generation subnetwork; the training module is also used to: detect at least one face AU value corresponding to the second face image through the pain expression detection subnetwork; wherein the face AU value is used to characterize the pain expression contained in the second face image; and generate a three-dimensional virtual pain expression corresponding to the second face image through the pain expression generation subnetwork based on a preset three-dimensional face model and each of the face AU values.

[0099] In one embodiment, the training module is also used to: detect the key points of the two-dimensional facial image through the pain expression detection subnetwork, divide the two-dimensional facial image into multiple regions of interest based on the key points, and determine the facial AU value corresponding to each region of interest; the pain expression generation subnetwork adopts a generative adversarial network, and the generative adversarial network includes a generator; the training module is also used to: determine the expression adjustment action corresponding to each facial AU value through the generator, and adjust the current expression of the three-dimensional facial model based on the expression adjustment action to obtain a three-dimensional virtual pain expression corresponding to the two-dimensional facial image.

[0100] In one embodiment, the training module is also used to: obtain the three-dimensional face model and at least one training AU value; wherein the training AU value is obtained by detecting the training face image through the pain expression detection subnetwork; generate at least one training virtual pain expression based on the three-dimensional face model and each of the training AU values ​​by the generator; wherein the pain level corresponding to each of the training virtual pain expressions is the same; generate an orthogonal mapping image corresponding to each of the training virtual pain expressions; determine the data discrimination result of the orthogonal mapping image corresponding to each of the training virtual pain expressions based on the training face image by the discriminator; and train the generator and the discriminator based on the data discrimination result.

[0101] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0102] An embodiment of the present invention provides a server. Specifically, the server includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0103] Figure 8 A structural diagram of a server provided in an embodiment of the present invention, the server 100 includes: a processor 80, a memory 81, a bus 82 and a communication interface 83, wherein the processor 80, the communication interface 83 and the memory 81 are connected via the bus 82; the processor 80 is used to execute an executable module stored in the memory 81, such as a computer program.

[0104] The memory 81 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 83 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0105] The bus 82 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0106] Among them, the memory 81 is used to store programs, and the processor 80 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiments of the present invention can be applied to the processor 80 or implemented by the processor 80.

[0107] The processor 80 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 80. The above processor 80 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 81, and the processor 80 reads the information in the memory 81 and completes the steps of the above method in combination with its hardware.

[0108] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0109] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0110] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for identifying pain levels, characterized in that: include: Obtain a first face image of a target object to be evaluated; Performing pain assessment on the target object based on the first face image through a pre-trained pain assessment model to obtain the pain level of the target object, including: extracting color texture features and facial shape features of the first face image through the pain assessment model, and determining the pain level of the target object based on the color texture features and the facial shape features; Wherein, the pain assessment model is obtained by training based on multiple two-dimensional mapping images corresponding to three-dimensional virtual pain expressions, and the three-dimensional virtual pain expressions are generated by a pre-trained data set generation network based on a second facial image of the target object; the training steps of the pain assessment model include: obtaining the second facial image of the target object, the first facial image and the second facial image are both two-dimensional facial images; generating multiple three-dimensional virtual pain expressions corresponding to the second facial image through a pre-trained data set generation network; mapping each of the three-dimensional virtual pain expressions according to multiple preset mapping angles, and obtaining multiple two-dimensional mapping images corresponding to each of the three-dimensional virtual pain expressions; constructing a training image set based on each of the two-dimensional mapping images, and using the training image set to train the pain assessment model, the training image set including multiple two-dimensional mapping images and pain labels annotated on each of the two-dimensional mapping images; The data set generation network includes a pain expression detection subnetwork and a pain expression generation subnetwork, the pain expression generation subnetwork adopts a generative adversarial network, and the generative adversarial network includes a generator and a discriminator; the training step of the pain expression generation subnetwork includes: obtaining a three-dimensional face model and at least one training AU value, the training AU value is obtained by detecting a training face image through the pain expression detection subnetwork; generating at least one training virtual pain expression based on the three-dimensional face model and each training AU value by the generator, and each training virtual pain expression corresponds to the same pain level; generating an orthogonal mapping image corresponding to each training virtual pain expression; determining a data discrimination result of the orthogonal mapping image corresponding to each training virtual pain expression based on the training face image by the discriminator; training the generator and the discriminator based on the data discrimination result, including: The reward value is calculated using the following formula : ;in, is the data identification result, is a constant, The function is to make the denominator of the reward value calculation formula not equal to 0; According to the reward value Calculate the cumulative average return corresponding to compensation T , the cumulative average return As shown below: ; According to the accumulated average reward and the policy gradient, the network parameters of the pain expression generation subnetwork are updated.

2. The method according to claim 1, characterized in that The pain assessment model includes a first feature extraction unit, a second feature extraction unit and a multi-layer perceptron; the step of extracting the color texture features and the facial shape features of the first face image through the pain assessment model, and determining the pain level of the target object based on the color texture features and the facial shape features includes: Extracting color and texture features of the first face image by the first feature extraction unit, and extracting facial shape features of the first face image by the second feature extraction unit; The color texture feature and the facial shape feature are subjected to feature fusion to obtain a fusion feature, and the pain level of the target object is determined based on the fusion feature by the multi-layer perceptron.

3. The method according to claim 2, characterized in that The output end of the pain expression detection sub-network is connected to the input end of the pain expression generation sub-network; The step of generating a plurality of three-dimensional virtual pain expressions corresponding to the second face image by generating a network through a pre-trained data set comprises: Detecting at least one face AU value corresponding to the second face image through the pain expression detection subnetwork; wherein the face AU value is used to characterize the pain expression contained in the second face image; The pain expression generation subnetwork generates a three-dimensional virtual pain expression corresponding to the second face image based on a preset three-dimensional face model and each face AU value.

4. The method according to claim 3, characterized in that The step of generating a three-dimensional virtual pain expression corresponding to the second face image based on a preset three-dimensional face model and each face AU value through the pain expression generation subnetwork includes: The generator determines the expression adjustment action corresponding to each facial AU value, and adjusts the current expression of the three-dimensional face model based on the expression adjustment action to obtain the three-dimensional virtual pain expression corresponding to the two-dimensional face image.

5. A pain level recognition device, characterized in that: include: An image acquisition module, used to acquire a first face image of a target object to be evaluated; a pain assessment module, configured to perform pain assessment on the target object based on the first face image using a pre-trained pain assessment model to obtain a pain level of the target object; The pain assessment module is specifically used to: extract the color texture features and facial shape features of the first face image through the pain assessment model, and determine the pain level of the target object based on the color texture features and the facial shape features; Wherein, the pain assessment model is obtained by training based on multiple two-dimensional mapping images corresponding to three-dimensional virtual pain expressions, and the three-dimensional virtual pain expressions are generated based on the second facial image of the target object through a pre-trained data set generation network; it also includes a training module for: obtaining the second facial image of the target object, the first facial image and the second facial image are both two-dimensional facial images; generating multiple three-dimensional virtual pain expressions corresponding to the second facial image through a pre-trained data set generation network; mapping each of the three-dimensional virtual pain expressions according to multiple preset mapping angles, and obtaining multiple two-dimensional mapping images corresponding to each of the three-dimensional virtual pain expressions; constructing a training image set based on each of the two-dimensional mapping images, and using the training image set to train the pain assessment model, the training image set including multiple two-dimensional mapping images and pain labels annotated on each of the two-dimensional mapping images; The data set generation network includes a pain expression detection subnetwork and a pain expression generation subnetwork, the pain expression generation subnetwork adopts a generative adversarial network, and the generative adversarial network includes a generator and a discriminator; the training module is specifically used to: obtain a three-dimensional face model and at least one training AU value, the training AU value is obtained by detecting a training face image through the pain expression detection subnetwork; generate at least one training virtual pain expression based on the three-dimensional face model and each training AU value by the generator, and each training virtual pain expression corresponds to the same pain level; generate an orthogonal mapping image corresponding to each training virtual pain expression; determine the data discrimination result of the orthogonal mapping image corresponding to each training virtual pain expression based on the training face image by the discriminator; and train the generator and the discriminator based on the data discrimination result; The training module is specifically used for: The reward value is calculated using the following formula : ;in, is the data identification result, is a constant, The function is to make the denominator of the reward value calculation formula not equal to 0; According to the reward value Calculate the cumulative average return corresponding to compensation T , the cumulative average return As shown below: ; According to the accumulated average reward and the policy gradient, the network parameters of the pain expression generation subnetwork are updated.

6. A server, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 4.

Images