Expression recognition method and device based on cross-modal reconstruction, equipment and medium

Abstract

The invention is suitable for the field of financial science and technology, and particularly relates to an expression recognition method and device based on cross-modal reconstruction, equipment and a medium. According to the invention, a trained image encoder is used to encode a target expression sequence to obtain target expression features, a trained text encoder is used to encode target text description information to obtain target text features, and a trained image decoder is used to decode the target text features to obtain first target reconstruction features; using a trained text decoder to decode the target expression feature to obtain a second target reconstruction feature, performing classification to obtain an expression category of the target face image, and extracting a target expression feature in an image mode and a target text feature in a text mode to obtain a target facial image; and cross-modal decoding is carried out to obtain the first target reconstruction feature and the second target reconstruction feature, so that the expression information representation capability and the expression recognition accuracy are improved, and the service efficiency and quality of customer service in the financial science and technology field are effectively improved.

Description

Technical Field

[0001] The present invention is applicable to the field of financial technology, and in particular relates to an expression recognition method, device, equipment and medium based on cross-modal reconstruction. Background Art

[0002] Expression recognition is a research direction of artificial intelligence. With the development of artificial intelligence, expression recognition technology has become a channel for identifying, interpreting and responding to more emotional exchanges between people, and is widely used in various industries such as finance, medical care, and education. For example, in the customer service scenario in the field of financial technology, with the assistance of expression recognition technology that can instantly feedback user satisfaction, customer service can add the real-time results of expression recognition to the consideration scope of the recommendation model, which is extremely necessary and practical in solving customer questions, guiding customer transactions, and providing after-sales service, and can effectively improve service efficiency and service quality in the financial field.

[0003] There are two main types of expression recognition methods at present: those based on manual design and those based on deep learning. Among them, the classification method based on manual features has low stability in the recognition process and is difficult to accurately and reliably recognize customer expressions; while when performing expression recognition based on deep features and face images, due to the small amount of expression data in actual applications, it is difficult to obtain robust feature representation using only a single image modality information under limited samples, resulting in low accuracy of expression recognition, which is difficult to meet the requirements for expression recognition accuracy in the financial field.

[0004] Therefore, in customer service scenarios in the field of financial technology, how to improve the accuracy of facial expression recognition has become an urgent problem to be solved. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide an expression recognition method, apparatus, device and medium based on cross-modal reconstruction to solve the problem of low recognition accuracy of existing expression recognition technologies.

[0006] In a first aspect, an embodiment of the present invention provides an expression recognition method based on cross-modal reconstruction, the expression recognition method comprising: Obtain a target expression sequence and corresponding target text description information of a target face image to be recognized; Using a trained image encoder to perform feature encoding on the target expression sequence to obtain target expression features, and using a trained text encoder to perform feature encoding on the target text description information to obtain target text features; Decoding the target text feature using a trained image decoder to obtain a first target reconstruction feature, and decoding the target expression feature using a trained text decoder to obtain a second target reconstruction feature; The target expression feature, the target text feature, the first target reconstruction feature and the second target reconstruction feature are classified using a preset classifier to obtain the expression category of the target facial image.

[0007] In a second aspect, an embodiment of the present invention provides an expression recognition device based on cross-modal reconstruction, the expression recognition device comprising: A data acquisition module, used to acquire a target expression sequence and corresponding target text description information of a target face image to be recognized; A feature encoding module is used to perform feature encoding on the target expression sequence using a trained image encoder to obtain target expression features, and to perform feature encoding on the target text description information using a trained text encoder to obtain target text features; A feature decoding module, used to decode the target text feature using a trained image decoder to obtain a first target reconstruction feature, and to decode the target expression feature using a trained text decoder to obtain a second target reconstruction feature; The expression recognition module is used to classify the target expression feature, the target text feature, the first target reconstruction feature and the second target reconstruction feature using a preset classifier to obtain the expression category of the target facial image.

[0008] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the expression recognition method as described in the first aspect when executing the computer program.

[0009] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the expression recognition method as described in the first aspect is implemented.

[0010] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: obtaining a target expression sequence and corresponding target text description information of a target facial image to be identified, using a trained image encoder to perform feature encoding on the target expression sequence to obtain a target expression feature, using a trained text encoder to perform feature encoding on the target text description information to obtain a target text feature, using a trained image decoder to decode the target text feature to obtain a first target reconstruction feature, using a trained text decoder to decode the target expression feature to obtain a second target reconstruction feature, using a preset classifier to classify the target expression feature, the target text feature, the first target reconstruction feature and the second target reconstruction feature to obtain the expression category of the target facial image, by respectively extracting the target expression feature of the target facial image in the image modality and the target text feature in the text modality, and performing cross-modal decoding on the target expression feature and the target text feature, the interactivity of the features in the image modality and the text modality is improved, thereby improving the representation capability of the expression information in the facial image, thereby improving the accuracy of expression recognition, and effectively improving the service efficiency and service quality of customer service in the field of financial technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0012] Figure 1 Schematic diagram of an application environment of an expression recognition method based on cross-modal reconstruction provided by the first embodiment of the present invention; Figure 2 It is a flowchart of an expression recognition method based on cross-modal reconstruction provided by the first embodiment of the present invention; Figure 3 Schematic diagram of a model of an expression recognition method based on cross-modal reconstruction provided in the first embodiment of the present invention; Figure 4 is a structural schematic diagram of an expression recognition device based on cross-modal reconstruction provided by Embodiment 2 of the present invention; Figure 5 It is a structural diagram of a computer device provided in Embodiment 3 of the present invention.

[0013] In the figure, CNN Enc is the image encoder, RNN Enc For text encoder, CNN Dnc For the image decoder, RNN Dncis the text decoder, Class1 is the preset image classifier, Class2 is the preset text classifier, v is the expression sequence, t is the text description information, and f v is the facial expression feature, f t is the text feature, k t is the first cross-modal reconstruction feature, k v is the second cross-modal reconstruction feature, x1 is the first category probability sequence, and x2 is the second category probability sequence. DETAILED DESCRIPTION

[0014] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0015] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0016] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0017] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0018] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0019] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0020] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0021] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0022] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0023] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0024] The expression recognition method based on cross-modal reconstruction provided by the first embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs), and other computer devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0025] See also Figure 2, is a flow chart of an expression recognition method based on cross-modal reconstruction provided by the first embodiment of the present invention. The expression recognition method can be applied to Figure 1 The client in the expression recognition method may include the following steps: Step S201, obtaining a target expression sequence and corresponding target text description information of a target face image to be recognized.

[0026] The facial expression recognition task of this embodiment is to extract and analyze the features of the target face image to be recognized, identify the facial expression category of the target face image, and become a channel for identifying, interpreting and responding to more people's emotional communication, which is widely used in various industries such as finance, medical care, and education. The expression category can be set according to the actual situation. For example, the expression category in this embodiment can be at least one of excitement, happiness, surprise, sadness, fear, shyness, contempt, and anger.

[0027] For example, in the customer service scenario in the financial technology field, with the assistance of the expression recognition model that can provide instant feedback on user satisfaction, the customer service can collect the target customer's target face image, use the trained expression recognition model to extract and analyze the target customer's target face image, identify the target customer's expression category, and add the identified expression category to the consideration of the recommendation model. For example, when the target customer's expression category is identified as happy, the current product can be further recommended to the customer; when the target customer's expression category is identified as contempt, the current recommended product can be abandoned and other products can be recommended to the customer. Therefore, expression recognition of target customers is extremely necessary and practical in solving customer questions, guiding customer transactions, and providing after-sales service, which can effectively improve service efficiency and service quality in the financial field.

[0028] The target expression sequence is a sequence of target face images containing expression information, and the target text description information is information obtained by performing text description on the target face images containing expression information. For example, the target text description information may be "lift up the cheeks and raise the corners of the mouth", which describes the expression information in the target face image through text information.

[0029] This embodiment regards the target expression sequence as the expression information of the image modality and the target text description information as the expression information of the text modality, and performs expression recognition on the target facial image by integrating the expression information of the image modality and the text modality to improve the accuracy of the expression recognition result.

[0030] The above steps of obtaining the target expression sequence of the target facial image to be identified and the corresponding target text description information, integrate the expression information of the image modality and the text modality as the basis for expression recognition of the target facial image, and improve the accuracy of the expression recognition results.

[0031] Step S202, using the trained image encoder to perform feature encoding on the target expression sequence to obtain target expression features, and using the trained text encoder to perform feature encoding on the target text description information to obtain target text features.

[0032] Among them, the target expression sequence can be regarded as the expression information of the image modality, the target text description information can be regarded as the expression information of the text modality, the image encoder is used to extract the features of the face image in the image modality, and the text encoder is used to extract the features of the face image in the text modality. Therefore, this embodiment uses the trained image encoder to perform feature encoding on the target expression sequence, and the target expression features belonging to the image modality can be obtained. The trained text encoder is used to perform feature encoding on the target text description information, and the target text features belonging to the text modality can be obtained, so that the target expression features and the target text features can be used together as the basis for identifying the expression category of the target face image.

[0033] The above steps of using a trained image encoder to feature encode the target expression sequence to obtain the target expression features, using a trained text encoder to feature encode the target text description information to obtain the target text features, and using the trained image encoder and the trained text encoder to extract feature information of the target facial image in image modality and text modality as the basis for identifying the expression category of the target facial image, thereby improving the accuracy of expression category recognition of the target facial image.

[0034] Step S203, using the trained image decoder to decode the target text feature to obtain a first target reconstruction feature, and using the trained text decoder to decode the target expression feature to obtain a second target reconstruction feature.

[0035] Among them, generally, the image decoder is used to decode the expression features, and the text decoder is used to decode the text features. In this embodiment, since the target expression features in the image modality and the target text features in the text modality represent the expression information of the same facial image, in order to improve the interactivity of the features in the two modalities and the ability to represent expression information, this embodiment uses the image decoder to decode the target text features to obtain the first target reconstruction features, and uses the text decoder to decode the expression features to obtain the second target reconstruction features, and uses the first target reconstruction features and the second target reconstruction features as the basis for expression recognition.

[0036] The above-mentioned steps of using a trained image decoder to decode the target text features to obtain the first target reconstruction features, and using a trained text decoder to decode the target expression features to obtain the second target reconstruction features, respectively perform cross-modal decoding on the target text features and the target expression features, thereby improving the interactivity of the features under the image modality and the text modality, and thereby improving the ability to represent the expression information in the facial image.

[0037] Step S204, using a preset classifier to classify the target expression feature, the target text feature, the first target reconstruction feature and the second target reconstruction feature to obtain the expression category of the target facial image.

[0038] The preset classifier is used to classify the target expression feature, the target text feature, the first target reconstruction feature and the second target reconstruction feature to obtain the expression category of the target face image. In this embodiment, the target expression feature and the second target reconstruction feature of the image modality, as well as the target text feature and the first target reconstruction feature of the text modality, can be classified using corresponding classifiers respectively to improve the accuracy of the classification results.

[0039] Optionally, the preset classifier includes a preset image classifier and a preset text classifier; The target expression features, target text features, first target reconstruction features and second target reconstruction features are classified using a preset classifier, and the expression categories of the target face image are obtained, including: Use a preset image classifier to classify the target expression features and the second target reconstruction features to obtain a first target category probability sequence; Using a preset text classifier to classify the target text features and the first target reconstruction features to obtain a second target category probability sequence; The sum of the first target category probability sequence and the second target category probability sequence is calculated to obtain the target category probability sequence, and the preset expression category corresponding to the maximum probability value in the target category probability sequence is determined as the expression category of the target face image.

[0040] Among them, a preset image classifier is used to classify the target expression features and the second target reconstruction features to obtain a first target category probability sequence, which is used to represent the first probability that the target face image belongs to each preset expression category, and a preset text classifier is used to classify the text features and the first target reconstruction features to obtain a second target category probability sequence, which is used to represent the second probability that the target face image belongs to each preset expression category.

[0041] Therefore, the sum of the first target category probability sequence and the second target category probability sequence is calculated to obtain the target category probability sequence, which is used to represent the target probability that the target facial image belongs to each preset expression category. The preset expression category corresponding to the maximum probability value in the target category probability sequence can be determined as the expression category of the target facial image.

[0042] The above-mentioned step of using a preset classifier to classify the target expression features, target text features, first target reconstruction features and second target reconstruction features to obtain the expression category of the target facial image, combines the target expression features, target text features, first target reconstruction features and second target reconstruction features to obtain the expression classification result of the target facial image, thereby improving the accuracy of the expression classification result.

[0043] Optionally, the training method of the trained image encoder, the trained text encoder, the trained image decoder, and the trained text decoder includes: Obtain expression sequences and corresponding text description information of facial images for training; Use an image encoder to encode the expression sequence to obtain expression features, and use a text encoder to encode the text description information to obtain text features; The text feature is decoded using an image decoder to obtain a first cross-modal reconstruction feature, and the expression feature is decoded using a text decoder to obtain a second cross-modal reconstruction feature; The total model loss is calculated according to the text features, the first cross-modal reconstruction features, the expression features and the second cross-modal reconstruction features. The parameters of the image encoder, text encoder, image decoder and text decoder are modified according to the total model loss to obtain the trained image encoder, trained text encoder, trained image decoder and trained text decoder.

[0044] Among them, the image encoder is used to extract the features of the face image in the image modality, and the text encoder is used to extract the features of the face image in the text modality. In order to improve the accuracy of the expression recognition results, it is necessary to train the image encoder and the text encoder to obtain the trained image encoder and the trained text encoder for feature extraction.

[0045] Specifically, this embodiment obtains an expression sequence and corresponding text description information of a facial image for training, and uses an image encoder to perform feature encoding on the expression sequence to obtain expression features in image mode, and uses a text encoder to perform feature encoding on the text description information to obtain text features in text mode, wherein both the expression features and the text features can be used to represent the expression information in the corresponding facial image.

[0046] Since the expression features in the image modality and the text features in the text modality represent the expression information of the same facial image, in order to improve the interactivity of the features in the two modalities and the ability to represent expression information, this embodiment uses an image decoder to decode the text features to obtain a first cross-modal reconstruction feature, and uses a text decoder to decode the expression features to obtain a second cross-modal reconstruction feature, and uses the first cross-modal reconstruction feature and the second cross-modal reconstruction feature as the basis for model training, thereby improving the interactivity of the features in the image modality and text modality, and thereby improving the ability to represent expression information in facial images.

[0047] As the accuracy of the text encoder and the image decoder is higher, the text features and the first cross-modal reconstruction features are more similar; the accuracy of the image encoder and the text decoder is higher, the expression features and the second cross-modal reconstruction features are more similar.

[0048] Therefore, this embodiment calculates the total model loss based on the distance between the text feature and the first cross-modal reconstruction feature, and the distance between the expression feature and the second cross-modal reconstruction feature, and corrects the parameters of the image encoder, text encoder, image decoder and text decoder according to the total model loss until the total model loss converges, thereby obtaining a trained image encoder, a trained text encoder, a trained image decoder and a trained text decoder.

[0049] Optionally, obtaining text description information includes: Obtain N frames of face images, use the trained image description model to describe each frame of face image, and obtain the description text of the corresponding frame of face image; The N description texts are sorted according to the arrangement order of each frame of the face image to obtain text description information composed of the N description texts.

[0050] Among them, the text description information is information obtained by textually describing a facial image containing expression information. Specifically, N frames of facial images and a trained image description model are obtained, and for any frame of facial image, the trained image description model is used to describe it to obtain the corresponding description text. Then, by traversing all frames of facial images, N description texts are obtained, and the N description texts are sorted according to the arrangement order of each frame of facial images to obtain text description information composed of N description texts.

[0051] Optionally, calculating the total model loss according to the text feature, the first cross-modal reconstruction feature, the expression feature, and the second cross-modal reconstruction feature includes: Calculating a first reconstruction loss based on the text feature and the first cross-modal reconstruction feature; Calculate a second reconstruction loss based on the expression feature and the second cross-modal reconstruction feature; Calculate the sum of the first reconstruction loss and the second reconstruction loss to get the total model loss.

[0052] Among them, the first reconstruction loss is used to characterize the accuracy of the text encoder and image decoder, and the second reconstruction loss is used to characterize the accuracy of the image encoder and text decoder. The sum of the first reconstruction loss and the second reconstruction loss can be calculated to obtain the total model loss, so as to correct the parameters of the image encoder, text encoder, image decoder and text decoder according to the total model loss until the total model loss converges, and obtain the trained image encoder, trained text encoder, trained image decoder and trained text decoder, thereby improving the feature extraction accuracy of the image encoder and text encoder.

[0053] Optionally, the expression recognition method further includes: Obtain the true category corresponding to each face image, use a preset image classifier to classify the expression features to obtain a first category probability sequence, and use a preset text classifier to classify the text features to obtain a second category probability sequence; The first classification loss is calculated based on the first category probability sequence and the true category, and the second classification loss is calculated based on the second category probability sequence and the true category; Correspondingly, the total model loss is calculated based on the text feature, the first cross-modal reconstruction feature, the expression feature, and the second cross-modal reconstruction feature, including: The total loss of the model is calculated based on text features, first cross-modal reconstruction features, expression features, second cross-modal reconstruction features, first classification loss, and second classification loss.

[0054] In this embodiment, a preset image classifier and a preset text classifier are obtained to train the image encoder and the text encoder to improve the reliability of the trained image encoder and the trained text encoder.

[0055] Specifically, the true category corresponding to each face image is obtained, and the preset image classifier is used to classify the expression features to obtain a first category probability sequence. The first classification loss can be calculated based on the first category probability sequence and the true category, and the first classification loss is used to characterize the accuracy of the image encoder; the text features are classified using a preset text classifier to obtain a second category probability sequence, and the second classification loss is calculated based on the second category probability sequence and the true category. The second classification loss is used to represent the accuracy of the text encoder.

[0056] A first reconstruction loss is calculated based on the text features and the first cross-modal reconstruction features, a second reconstruction loss is calculated based on the expression features and the second cross-modal reconstruction features, and the sum of the first reconstruction loss, the second reconstruction loss, the first classification loss and the second classification loss is calculated to obtain the total model loss, which is used to correct the parameters of the image encoder, the text encoder, the image decoder and the text decoder to obtain a trained image encoder and a trained text encoder.

[0057] Optionally, calculating the first classification loss according to the first category probability sequence and the true category, and calculating the second classification loss according to the second category probability sequence and the true category includes: Determine a true category probability sequence of the face image according to the true category; Calculate a first distance between a first category probability sequence and a true category probability sequence, and determine the first distance as a first classification loss; A second distance between the second category probability sequence and the true category probability sequence is calculated, and the second distance is determined as a second classification loss.

[0058] Among them, in order to correspond to the first category probability sequence and the first category probability sequence, thereby improving the calculation accuracy and reliability of the classification loss, this embodiment determines the true category probability sequence of the face image according to the true category of the face image. Specifically, in the true category probability sequence, the probability value corresponding to the true category is 1, and the probability value corresponding to other preset expression categories is 0.

[0059] Then, a first distance between the first category probability sequence and the true category probability sequence can be calculated, and the first distance can be determined as the first classification loss, and a second distance between the second category probability sequence and the true category probability sequence can be calculated, and the second distance can be determined as the second classification loss.

[0060] Optionally, the expression recognition method further includes: Calculate semantic correlation loss based on expression features and text features; Correspondingly, the total model loss is calculated based on the text feature, the first cross-modal reconstruction feature, the expression feature, and the second cross-modal reconstruction feature, including: The total model loss is calculated based on text features, first cross-modal reconstruction features, expression features, second cross-modal reconstruction features, first classification loss, second classification loss, and semantic correlation loss.

[0061] Among them, since the expression features in the image modality and the text features in the text modality represent the expression information of the same facial image, the semantic similarity between the expression features and the text features should be as high as possible. Therefore, this embodiment calculates the semantic correlation loss based on the expression features and the text features, and calculates the sum of the first reconstruction loss, the second reconstruction loss, the first classification loss, the second classification loss and the semantic correlation loss to obtain the total model loss, which further improves the accuracy of the trained image encoder and the trained text encoder.

[0062] Among them, see Figure 3 , is a schematic diagram of the model of this embodiment, where v is the expression sequence of the face image used for training, t is the corresponding text description information, CNN Enc is the image encoder, RNN Enc For text encoder, CNN Dnc For the image decoder, RNN Dnc is the text decoder, Class1 is the preset image classifier, and Class2 is the preset text classifier.

[0063] Using Image Encoder CNN Enc Encode the expression sequence v and get the expression feature f v , using a text encoder RNN Enc Perform feature encoding on the text description information t to obtain the text feature f t , using image decoder CNN Dnc For text feature f t Decode and obtain the first cross-modal reconstruction feature k t , using a text decoder RNN Dnc For facial features v Decode and obtain the second cross-modal reconstruction feature k v , use the preset image classifier Class1 to classify the expression feature f v Classify and obtain the first category probability sequence x1, and use the preset text classifier Class2 to classify the text feature f t Classify and obtain the second category probability sequence x2.

[0064] Then calculate the text feature f t and the first cross-modal reconstruction feature k t The third distance between them is determined as the first reconstruction loss l 11 , calculate the expression feature f v and the second cross-modal reconstruction feature k v The fourth distance between them is determined as the second reconstruction loss l 12; Get the true category probability sequence x0, calculate the first distance between the first category probability sequence x1 and the true category probability sequence x0, and determine the first distance as the first classification loss l 21 , calculate the second distance between the second category probability sequence x2 and the true category probability sequence x0, and determine the second distance as the second classification loss l 22 ; Expression features v and text features f t The semantic correlation loss l3 between them; finally calculate the first reconstruction loss l 11 , the second reconstruction loss l 12 , the first classification loss l 21 , the second classification loss l 22 The sum of the semantic correlation loss l3 is used to obtain the total model loss L, which is used to correct the parameters of the image encoder, text encoder, image decoder and text decoder to obtain the trained image encoder and the trained text encoder.

[0065] The embodiment of the present invention obtains a target expression sequence and corresponding target text description information of a target facial image to be identified, uses a trained image encoder to perform feature encoding on the target expression sequence to obtain a target expression feature, uses a trained text encoder to perform feature encoding on the target text description information to obtain a target text feature, uses a trained image decoder to decode the target text feature to obtain a first target reconstruction feature, uses a trained text decoder to decode the target expression feature to obtain a second target reconstruction feature, uses a preset classifier to classify the target expression feature, the target text feature, the first target reconstruction feature and the second target reconstruction feature to obtain the expression category of the target facial image, and respectively extracts the target expression feature of the target facial image in the image modality and the target text feature in the text modality, and performs cross-modal decoding on the target expression feature and the target text feature, thereby improving the interactivity of the features in the image modality and the text modality, thereby improving the representation capability of the expression information in the facial image, thereby improving the accuracy of expression recognition, and effectively improving the service efficiency and service quality of customer service in the financial technology field.

[0066] Corresponding to the expression recognition method in the above embodiment, Figure 4 A structural block diagram of an expression recognition device based on cross-modal reconstruction provided by a second embodiment of the present invention is given. For ease of explanation, only the parts related to the embodiment of the present invention are shown.

[0067] See also Figure 4 , the expression recognition device comprises: A data acquisition module 41 is used to acquire a target expression sequence and corresponding target text description information of a target face image to be recognized; A feature encoding module 42 is used to perform feature encoding on a target expression sequence using a trained image encoder to obtain a target expression feature, and to perform feature encoding on a target text description information using a trained text encoder to obtain a target text feature; A feature decoding module 43 is used to decode the target text feature using the trained image decoder to obtain a first target reconstruction feature, and to decode the target expression feature using the trained text decoder to obtain a second target reconstruction feature; The expression recognition module 44 is used to classify the target expression feature, the target text feature, the first target reconstruction feature and the second target reconstruction feature using a preset classifier to obtain the expression category of the target facial image.

[0068] Optionally, the preset classifier includes a preset image classifier and a preset text classifier, and the expression recognition module 44 includes: A first classification submodule is used to classify the target expression features and the second target reconstruction features using a preset image classifier to obtain a first target category probability sequence; A second classification submodule, used to classify the target text features and the first target reconstruction features using a preset text classifier to obtain a second target category probability sequence; The expression recognition submodule is used to calculate the sum of the first target category probability sequence and the second target category probability sequence to obtain the target category probability sequence, and determine the preset expression category corresponding to the maximum probability value in the target category probability sequence as the expression category of the target face image.

[0069] Optionally, the facial expression recognition device further includes: A training data acquisition module is used to obtain expression sequences and corresponding text description information of facial images used for training; A training feature extraction module is used to perform feature encoding on the expression sequence using an image encoder to obtain expression features, and to perform feature encoding on the text description information using a text encoder to obtain text features; A training feature decoding module is used to decode the text feature using an image decoder to obtain a first cross-modal reconstruction feature, and to decode the expression feature using a text decoder to obtain a second cross-modal reconstruction feature; A parameter correction module is used to calculate the total model loss based on text features, first cross-modal reconstruction features, expression features and second cross-modal reconstruction features, and to correct the parameters of the image encoder, text encoder, image decoder and text decoder based on the total model loss to obtain a trained image encoder, a trained text encoder, a trained image decoder and a trained text decoder.

[0070] Optionally, the parameter correction module includes: A first reconstruction loss calculation submodule, used to calculate a first reconstruction loss according to the text feature and the first cross-modal reconstruction feature; A second reconstruction loss calculation submodule, used to calculate a second reconstruction loss according to the expression feature and the second cross-modal reconstruction feature; The model total loss calculation submodule is used to calculate the sum of the first reconstruction loss and the second reconstruction loss to obtain the model total loss.

[0071] Optionally, the facial expression recognition device further includes: A real category acquisition module is used to obtain the real category corresponding to each face image; A feature classification module, used to classify expression features using a preset image classifier to obtain a first category probability sequence, and to classify text features using a preset text classifier to obtain a second category probability sequence; A classification loss calculation module, used to calculate a first classification loss according to a first category probability sequence and a true category, and to calculate a second classification loss according to a second category probability sequence and a true category; Correspondingly, the above parameter correction module includes: The total loss of the model is calculated based on text features, first cross-modal reconstruction features, expression features, second cross-modal reconstruction features, first classification loss, and second classification loss.

[0072] Optionally, the classification loss calculation module includes: A true category probability sequence determination submodule is used to determine the true category probability sequence of the face image according to the true category; A first classification loss calculation submodule, used to calculate a first distance between a first category probability sequence and a true category probability sequence, and determine the first distance as a first classification loss; The second classification loss calculation submodule is used to calculate a second distance between the second category probability sequence and the true category probability sequence, and determine the second distance as the second classification loss.

[0073] Optionally, the facial expression recognition device further includes: A semantic correlation loss calculation module is used to calculate the semantic correlation loss based on expression features and text features; Correspondingly, the above parameter correction module includes: The total model loss is calculated based on text features, first cross-modal reconstruction features, expression features, second cross-modal reconstruction features, first classification loss, second classification loss, and semantic relevance loss.

[0074] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0075] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned expression recognition method embodiments are implemented.

[0076] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 5 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0077] The processor may be a CPU, other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0078] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of an operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of a computer device, for example, a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on a computer device. Further, the memory may also include both an internal storage unit of a computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader, data, and other programs, such as program code of a computer program. The memory may also be used to temporarily store data that has been output or is to be output.

[0079] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0080] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.

[0081] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0082] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0083] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for facial expression recognition based on cross-modal reconstruction, characterized in that: The expression recognition method comprises: Obtain a target expression sequence and corresponding target text description information of a target face image to be recognized; Using a trained image encoder to perform feature encoding on the target expression sequence to obtain target expression features, and using a trained text encoder to perform feature encoding on the target text description information to obtain target text features; Decoding the target text feature using a trained image decoder to obtain a first target reconstruction feature, and decoding the target expression feature using a trained text decoder to obtain a second target reconstruction feature; The target expression feature, the target text feature, the first target reconstruction feature and the second target reconstruction feature are classified using a preset classifier to obtain the expression category of the target facial image.

2. The facial expression recognition method according to claim 1, characterized in that: The preset classifier includes a preset image classifier and a preset text classifier; The method of using a preset classifier to classify the target expression feature, the target text feature, the first target reconstruction feature, and the second target reconstruction feature to obtain the expression category of the target face image includes: Using the preset image classifier to classify the target expression feature and the second target reconstruction feature to obtain a first target category probability sequence; Using the preset text classifier to classify the target text feature and the first target reconstruction feature to obtain a second target category probability sequence; The sum of the first target category probability sequence and the second target category probability sequence is calculated to obtain a target category probability sequence, and a preset expression category corresponding to the maximum probability value in the target category probability sequence is determined as the expression category of the target facial image.

3. The facial expression recognition method according to claim 2, characterized in that: The training method of the trained image encoder, the trained text encoder, the trained image decoder and the trained text decoder comprises: Obtain expression sequences and corresponding text description information of facial images for training; Using an image encoder to perform feature encoding on the expression sequence to obtain expression features, and using a text encoder to perform feature encoding on the text description information to obtain text features; Using an image decoder to decode the text feature to obtain a first cross-modal reconstruction feature, and using a text decoder to decode the expression feature to obtain a second cross-modal reconstruction feature; The total model loss is calculated according to the text features, the first cross-modal reconstruction features, the expression features and the second cross-modal reconstruction features, and the parameters of the image encoder, the text encoder, the image decoder and the text decoder are modified according to the total model loss to obtain a trained image encoder, a trained text encoder, a trained image decoder and a trained text decoder.

4. The facial expression recognition method according to claim 3, characterized in that: The calculating the total model loss according to the text feature, the first cross-modal reconstruction feature, the expression feature and the second cross-modal reconstruction feature comprises: Calculating a first reconstruction loss according to the text feature and the first cross-modal reconstruction feature; Calculating a second reconstruction loss according to the expression feature and the second cross-modal reconstruction feature; The sum of the first reconstruction loss and the second reconstruction loss is calculated to obtain the total loss of the model.

5. The facial expression recognition method according to claim 4, characterized in that: The expression recognition method further comprises: Obtaining the true category corresponding to each of the face images; Using the preset image classifier to classify the expression features to obtain a first category probability sequence, and using the preset text classifier to classify the text features to obtain a second category probability sequence; Calculate a first classification loss according to the first category probability sequence and the true category, and calculate a second classification loss according to the second category probability sequence and the true category; Correspondingly, calculating the total model loss according to the text feature, the first cross-modal reconstruction feature, the expression feature, and the second cross-modal reconstruction feature includes: The total model loss is calculated according to the text feature, the first cross-modal reconstruction feature, the expression feature, the second cross-modal reconstruction feature, the first classification loss and the second classification loss.

6. The facial expression recognition method according to claim 5, characterized in that: The calculating of the first classification loss according to the first category probability sequence and the true category, and the calculating of the second classification loss according to the second category probability sequence and the true category include: Determine a true category probability sequence of the face image according to the true category; Calculating a first distance between the first category probability sequence and the true category probability sequence, and determining the first distance as a first classification loss; A second distance between the second category probability sequence and the true category probability sequence is calculated, and the second distance is determined as a second classification loss.

7. The facial expression recognition method according to claim 6, characterized in that: The expression recognition method further comprises: Calculating semantic correlation loss according to the expression feature and the text feature; Correspondingly, calculating the total model loss according to the text feature, the first cross-modal reconstruction feature, the expression feature, and the second cross-modal reconstruction feature includes: The total model loss is calculated according to the text feature, the first cross-modal reconstruction feature, the expression feature, the second cross-modal reconstruction feature, the first classification loss, the second classification loss and the semantic relevance loss.

8. An expression recognition device based on cross-modal reconstruction, characterized in that: The expression recognition device comprises: A data acquisition module, used to acquire a target expression sequence and corresponding target text description information of a target face image to be recognized; A feature encoding module is used to perform feature encoding on the target expression sequence using a trained image encoder to obtain target expression features, and to perform feature encoding on the target text description information using a trained text encoder to obtain target text features; A feature decoding module, used to decode the target text feature using a trained image decoder to obtain a first target reconstruction feature, and to decode the target expression feature using a trained text decoder to obtain a second target reconstruction feature; The expression recognition module is used to classify the target expression feature, the target text feature, the first target reconstruction feature and the second target reconstruction feature using a preset classifier to obtain the expression category of the target facial image.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the expression recognition method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the expression recognition method according to any one of claims 1 to 7 is implemented.

Images