Facial expression recognition method, device, electronic device and storage medium

Abstract

The present invention discloses a facial expression recognition method, device, electronic device and storage medium. The method comprises: obtaining multiple video frames containing targets to be detected; performing key point detection on each of the video frames containing the targets to be detected, and obtaining the facial key point coordinates corresponding to each of the video frames containing the targets to be detected; determining the key point coordinate offset corresponding to each of the facial key point coordinates; constructing a facial change spatiotemporal graph based on the key point coordinate offset corresponding to each of the facial key point coordinates; inputting the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result. The above technical solution performs facial expression recognition by using a facial change spatiotemporal graph constructed based on the key point coordinate offset, which reduces the amount of data calculation and improves the data processing speed compared to the full image feature analysis, thereby achieving efficient and real-time facial expression recognition.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a facial expression recognition method, device, electronic equipment and storage medium. Background Art

[0002] In intelligent vehicle systems, recognition of facial expressions of drivers and passengers can significantly improve driving safety and riding experience.

[0003] In the prior art, facial expression recognition solutions have the problems of large consumption of computing resources and insufficient real-time performance. Summary of the invention

[0004] The present invention provides a facial expression recognition method, device, electronic device and storage medium to reduce the amount of data calculation and improve the data processing speed, thereby realizing efficient and real-time facial expression recognition.

[0005] According to one aspect of the present invention, there is provided a facial expression recognition method, comprising:

[0006] Obtain multiple video frames containing the target to be detected;

[0007] Performing key point detection on each of the video frames containing the target to be detected, and obtaining the facial key point coordinates corresponding to each of the video frames containing the target to be detected;

[0008] Determine the key point coordinate offset corresponding to each of the facial key point coordinates;

[0009] Constructing a spatiotemporal graph of facial changes based on the key point coordinate offsets corresponding to the facial key point coordinates;

[0010] The facial change spatiotemporal graph is input into a pre-trained facial expression recognition model to obtain a facial expression recognition result.

[0011] According to another aspect of the present invention, there is provided a facial expression recognition device, comprising:

[0012] A video frame acquisition module is used to acquire multiple video frames containing the target to be detected;

[0013] A key point detection module, used to perform key point detection on each video frame containing the target to be detected, and obtain the facial key point coordinates corresponding to each video frame containing the target to be detected;

[0014] A key point coordinate offset determination module, used to determine the key point coordinate offset corresponding to each of the facial key point coordinates;

[0015] A facial change spatiotemporal graph construction module, used to construct a facial change spatiotemporal graph based on the key point coordinate offsets corresponding to the facial key point coordinates;

[0016] The facial expression recognition result prediction module is used to input the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] at least one processor;

[0019] and a memory communicatively coupled to the at least one processor;

[0020] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the facial expression recognition method described in any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the facial expression recognition method described in any embodiment of the present invention when executed.

[0022] The technical solution of the embodiment of the present invention obtains multiple video frames containing the target to be detected, performs key point detection on each video frame containing the target to be detected, obtains the facial key point coordinates corresponding to each video frame containing the target to be detected, and then determines the key point coordinate offset corresponding to each facial key point coordinate, and then constructs a facial change spatiotemporal graph based on the key point coordinate offset corresponding to each facial key point coordinate, and then inputs the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result. The above technical solution performs facial expression recognition by using the facial change spatiotemporal graph constructed based on the key point coordinate offset, which reduces the amount of data calculation and improves the data processing speed compared to the full image feature analysis, thereby achieving efficient and real-time facial expression recognition.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flowchart of a facial expression recognition method provided according to Embodiment 1 of the present invention;

[0026] Figure 2 is a flowchart of a facial expression recognition method provided according to Embodiment 2 of the present invention;

[0027] Figure 3 is a flowchart of a facial expression recognition method provided according to Embodiment 3 of the present invention;

[0028] Figure 4 is a structural schematic diagram of a facial expression recognition device provided according to a fourth embodiment of the present invention;

[0029] Figure 5 It is a structural schematic diagram of an electronic device for implementing the facial expression recognition method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0032] Embodiment 1

[0033] Figure 1This is a flowchart of a facial expression recognition method provided in the first embodiment of the present invention. This embodiment can be applied to the case of recognizing the facial expression of a driver or passenger in a vehicle-mounted system. The method can be executed by a facial expression recognition device, which can be implemented in the form of hardware and / or software, and can be configured in a vehicle-mounted terminal. Figure 1 As shown, the method includes:

[0034] S110, acquiring multiple video frames containing the target to be detected.

[0035] In this embodiment, the target to be detected refers to the driver and / or passengers located inside or outside the vehicle.

[0036] Specifically, real-time video capture of the driver and / or passengers can be performed through a camera installed in the vehicle, thereby obtaining multiple video frames containing the targets to be detected, wherein the camera can be installed at a location such as the dashboard and / or A-pillar of the vehicle, and no specific limitation is made here.

[0037] S120, performing key point detection on each video frame containing the target to be detected, and obtaining facial key point coordinates corresponding to each video frame containing the target to be detected.

[0038] Specifically, key point detection can be performed on each video frame containing the target to be detected through traditional machine learning, deep learning, or cascade regression methods, so as to obtain the facial key point coordinates corresponding to each video frame containing the target to be detected.

[0039] Optionally, key point detection is performed on each video frame containing the target to be detected to obtain the facial key point coordinates corresponding to each video frame containing the target to be detected, including: inputting multiple video frames containing the target to be detected into a pre-trained target detection model to obtain the facial key point coordinates corresponding to each video frame containing the target to be detected.

[0040] Exemplarily, the target detection model can be a Yolov8-based target detection network model, etc. Specifically, multiple video frames containing targets to be detected can be input into a pre-trained Yolov8-based target detection network model to obtain the facial key point coordinates corresponding to each video frame containing the target to be detected, wherein each video frame can correspond to 68 facial key point coordinates.

[0041] S130, determining the key point coordinate offset corresponding to each of the facial key point coordinates.

[0042] In this embodiment, the key point coordinate offset is used to characterize the dynamic process of facial expression changes.

[0043] Specifically, the coordinates of each facial key point corresponding to the current video frame can be subtracted from the coordinates of each facial key point corresponding to the previous video frame to obtain the key point coordinate offset; the coordinates of each facial key point can also be subtracted from the key point coordinates of the reference frame to obtain the key point coordinate offset, which is not specifically limited here.

[0044] S140, constructing a face change spatiotemporal graph based on the key point coordinate offsets corresponding to the facial key point coordinates.

[0045] Specifically, the key point coordinate offsets can be used to construct a spatiotemporal graph of facial changes, where nodes represent the offsets of key points and edges represent the spatial relationships between key points and the temporal relationships between key points in consecutive video frames.

[0046] S150, inputting the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result.

[0047] Specifically, the facial change spatiotemporal map can be used as the input data of the model, and then the facial change spatiotemporal map can be input into a pre-trained facial expression recognition model. The facial expression recognition model performs convolution operations on the facial change spatiotemporal map, which can simultaneously capture the spatial facial structure information and temporal dynamic change information, thereby achieving accurate recognition of facial expressions and outputting more accurate facial expression recognition results.

[0048] Among them, the facial expression recognition model can be a spatial-temporal graph convolutional network (STGCN), which can include a graph convolution layer, a spatial-temporal convolution layer and a fully connected layer.

[0049] The technical solution of the embodiment of the present invention obtains multiple video frames containing the target to be detected, performs key point detection on each video frame containing the target to be detected, obtains the facial key point coordinates corresponding to each video frame containing the target to be detected, and then determines the key point coordinate offset corresponding to each facial key point coordinate, and then constructs a facial change spatiotemporal graph based on the key point coordinate offset corresponding to each facial key point coordinate, and then inputs the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result. The above technical solution performs facial expression recognition by using the facial change spatiotemporal graph constructed based on the key point coordinate offset, which reduces the amount of calculation and improves the processing speed compared to the full image feature analysis, thereby achieving efficient and real-time facial expression recognition.

[0050] Embodiment 2

[0051] Figure 2A flowchart of a facial expression recognition method provided in Embodiment 2 of the present invention, the method of this embodiment can be combined with the various optional schemes in the facial expression recognition method provided in the above embodiments. The facial expression recognition method provided in this embodiment is further optimized. Optionally, the determination of the key point coordinate offset corresponding to each of the facial key point coordinates includes: obtaining the key point coordinates of the reference frame; determining the key point coordinate offset based on the key point coordinates of each of the facial key points and the key point coordinates of the reference frame.

[0052] like Figure 2 As shown, the method includes:

[0053] S210: Acquire multiple video frames containing the target to be detected.

[0054] S220, performing key point detection on each of the video frames containing the target to be detected, and obtaining the coordinates of the facial key points corresponding to each of the video frames containing the target to be detected.

[0055] S230, obtaining key point coordinates of the reference frame.

[0056] The reference frame may be a video frame corresponding to a calm facial expression, which may be obtained by manual selection, and then key points of the reference frame are extracted to obtain key point coordinates of the reference frame.

[0057] S240: Determine a key point coordinate offset based on the facial key point coordinates and the key point coordinates of the reference frame.

[0058] Specifically, the formula for determining the key point coordinate offset is:

[0059] ;

[0060] in, Indicates the key point coordinate offset, represents the coordinates of facial key points in any video frame, Represents the keypoint coordinates of the base frame.

[0061] It should be noted that the use of key point coordinate offsets can reduce the impact of external changes (such as head movement) on facial expression recognition results, making the facial expression recognition method more robust to different environments and conditions. Furthermore, the key point coordinate offsets as input features simplify the input of the facial expression recognition model, helping the facial expression recognition model to learn key information about expression changes more quickly.

[0062] S250, constructing a face change spatiotemporal graph based on the key point coordinate offsets corresponding to the facial key point coordinates.

[0063] S260, inputting the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result.

[0064] Optionally, after determining the key point coordinate offset corresponding to each facial key point coordinate, the method also includes: obtaining the distance between the left and right eyebrows of the target to be detected, wherein the distance between the left and right eyebrows is the distance between the center point of the left eyebrow and the center point of the right eyebrow; dividing the key point coordinate offset corresponding to each facial key point coordinate by the distance between the left and right eyebrows of the target to be detected to obtain a normalized key point coordinate offset.

[0065] It should be noted that through normalization, the coordinate offsets of key points of different scales and distances can be unified, making the model training more stable.

[0066] Optionally, a facial change spatiotemporal graph is constructed based on the key point coordinate offsets corresponding to the coordinates of each facial key point, including: using the key point coordinate offsets as nodes for constructing a spatial graph; using the spatial relationship of the target facial anatomical structure to be detected as edges for constructing the spatial graph; generating a facial spatial graph based on the nodes and edges of the spatial graph; generating a facial temporal graph based on the temporal relationship of the same key point between consecutive video frames; and merging the facial spatial graph and the facial temporal graph to obtain a facial change spatiotemporal graph.

[0067] Exemplarily, for each video frame, a facial space graph can be constructed using the normalized key point coordinate offsets. Specifically, in the facial space graph, each node represents a key point, and the node is characterized by the normalized key point coordinate offsets of the key point. The edges between nodes reflect the spatial relationship of the facial structure, that is, the connection between key points that can be predefined according to the facial anatomical structure. The connection between key points can be a proximity connection in physical distance or a functional connection, for example, the key points around the eyes are connected to each other to form a subgraph. For consecutive video frames, a facial time graph can be constructed to capture the changes of key points over time. In the facial time graph, the same key points establish connections between consecutive frames. For example, the i-th key point in the t-th frame is connected to the corresponding i-th key point in the t+1-th frame. Further, the facial space graph and the facial time graph can be merged to form a complete facial change spatiotemporal graph. Specifically, in the facial change spatiotemporal graph, the nodes contain both the spatial relationship between facial key points and the dynamic change relationship between key points over time.

[0068] The technical solution of the embodiment of the present invention can reduce the impact of external changes (such as head movement) on facial expression recognition results by using key point coordinate offsets, making the facial expression recognition method more robust to different environments and conditions. Furthermore, the key point coordinate offsets as input features simplify the input of the facial expression recognition model, helping the facial expression recognition model to learn key information about expression changes more quickly.

[0069] Embodiment 3

[0070] Figure 3 A flowchart of a facial expression recognition method provided in Embodiment 3 of the present invention, the method of this embodiment can be combined with the various optional schemes in the facial expression recognition method provided in the above embodiments. The facial expression recognition method provided in this embodiment is further optimized. Optionally, after inputting the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result, the method further includes: triggering a target function according to the facial expression recognition result, wherein the target function includes a music change function and / or an in-vehicle light adjustment function.

[0071] like Figure 3 As shown, the method includes:

[0072] S310: Acquire multiple video frames containing objects to be detected.

[0073] S320, performing key point detection on each of the video frames containing the target to be detected, and obtaining the coordinates of the facial key points corresponding to each of the video frames containing the target to be detected.

[0074] S330, determining the key point coordinate offset corresponding to each of the facial key point coordinates.

[0075] S340, constructing a face change spatiotemporal graph based on the key point coordinate offsets corresponding to the facial key point coordinates.

[0076] S350, inputting the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result.

[0077] S360: triggering a target function according to the facial expression recognition result, wherein the target function includes a music changing function and / or an in-vehicle light adjustment function.

[0078] For example, if the facial expression recognition result is a happy expression, the music is replaced with cheerful music or the lights in the car are adjusted to bright tones; if the facial expression recognition result is a sad expression, the music is replaced with sad music or the lights in the car are adjusted to dark tones.

[0079] The technical solution of the embodiment of the present invention triggers the target function according to the facial expression recognition result, thereby linking the facial expression with the function in the cabin, thereby improving the user's driving experience.

[0080] Embodiment 4

[0081] Figure 4 This is a schematic diagram of the structure of a facial expression recognition device provided by Embodiment 4 of the present invention. Figure 4 As shown, the device comprises:

[0082] The video frame acquisition module 410 is used to acquire multiple video frames containing the target to be detected;

[0083] A key point detection module 420 is used to perform key point detection on each video frame containing the target to be detected, and obtain the facial key point coordinates corresponding to each video frame containing the target to be detected;

[0084] A key point coordinate offset determination module 430, used to determine the key point coordinate offset corresponding to each of the facial key point coordinates;

[0085] A facial change spatiotemporal graph construction module 440 is used to construct a facial change spatiotemporal graph based on the key point coordinate offsets corresponding to the facial key point coordinates;

[0086] The facial expression recognition result prediction module 450 is used to input the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result.

[0087] The technical solution of the embodiment of the present invention obtains multiple video frames containing the target to be detected, performs key point detection on each video frame containing the target to be detected, obtains the facial key point coordinates corresponding to each video frame containing the target to be detected, and then determines the key point coordinate offset corresponding to each facial key point coordinate, and then constructs a facial change spatiotemporal graph based on the key point coordinate offset corresponding to each facial key point coordinate, and then inputs the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result. The above technical solution performs facial expression recognition by using the facial change spatiotemporal graph constructed based on the key point coordinate offset, which reduces the amount of calculation and improves the processing speed compared to the full image feature analysis, thereby achieving efficient and real-time facial expression recognition.

[0088] In some optional implementations, the key point detection module 420 may also be used to:

[0089] Multiple video frames containing the target to be detected are input into a pre-trained target detection model to obtain the coordinates of facial key points corresponding to each of the video frames containing the target to be detected.

[0090] In some optional implementations, the key point coordinate offset determination module 430 may also be used to:

[0091] Get the key point coordinates of the reference frame;

[0092] A key point coordinate offset is determined based on the facial key point coordinates and the key point coordinates of the reference frame.

[0093] In some optional implementations, the formula for determining the key point coordinate offset is:

[0094] ;

[0095] in, Indicates the key point coordinate offset, represents the coordinates of facial key points in any video frame, Represents the keypoint coordinates of the base frame.

[0096] In some optional implementations, the facial expression recognition device further includes:

[0097] The left and right eyebrow distance acquisition module is used to acquire the distance between the left and right eyebrows of the target to be detected, where the distance between the left and right eyebrows is the distance between the center point of the left eyebrow and the center point of the right eyebrow;

[0098] The normalization processing module is used to divide the key point coordinate offset corresponding to each of the facial key point coordinates by the distance between the left and right eyebrows of the target to be detected to obtain a normalized key point coordinate offset.

[0099] In some optional implementations, the facial change spatiotemporal graph construction module 440 may also be used to:

[0100] Using the key point coordinate offset as a node for constructing a spatial graph;

[0101] The spatial relationship of the facial anatomical structure of the target to be detected is used as the edge of the spatial graph;

[0102] generating a facial spatial graph based on the nodes of the spatial graph and the edges of the spatial graph;

[0103] Generate a facial temporal map based on the temporal relationship of the same key points between consecutive video frames;

[0104] The facial space map and the facial time map are combined to obtain a facial change space-time map.

[0105] In some optional implementations, the facial expression recognition device further includes:

[0106] A target function triggering module is used to trigger a target function according to the facial expression recognition result, wherein the target function includes a music changing function and / or an in-vehicle light adjustment function.

[0107] The facial expression recognition device provided in the embodiment of the present invention can execute the facial expression recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0108] Embodiment 5

[0109] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0110] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The I / O interface 15 is also connected to the bus 14.

[0111] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0112] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a facial expression recognition method, which includes:

[0113] Obtain multiple video frames containing the target to be detected;

[0114] Performing key point detection on each of the video frames containing the target to be detected, and obtaining the facial key point coordinates corresponding to each of the video frames containing the target to be detected;

[0115] Determine the key point coordinate offset corresponding to each of the facial key point coordinates;

[0116] Constructing a spatiotemporal graph of facial changes based on the key point coordinate offsets corresponding to the facial key point coordinates;

[0117] The facial change spatiotemporal graph is input into a pre-trained facial expression recognition model to obtain a facial expression recognition result.

[0118] In some embodiments, the facial expression recognition method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the facial expression recognition method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the facial expression recognition method in any other appropriate manner (e.g., by means of firmware).

[0119] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0124] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A facial expression recognition method, characterized in that: include: Obtain multiple video frames containing the target to be detected; Performing key point detection on each of the video frames containing the target to be detected, and obtaining the facial key point coordinates corresponding to each of the video frames containing the target to be detected; Determine the key point coordinate offset corresponding to each of the facial key point coordinates; Constructing a spatiotemporal graph of facial changes based on the key point coordinate offsets corresponding to the facial key point coordinates; Inputting the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result; The determining of the key point coordinate offset corresponding to each of the facial key point coordinates includes: Extracting key points from a reference frame to obtain key point coordinates of the reference frame, wherein the reference frame is a video frame corresponding to a calm facial expression obtained by manual selection; The key point coordinate offset is determined based on the coordinates of each of the facial key points and the key point coordinates of the reference frame. The formula for determining the key point coordinate offset is: ； in, Indicates the key point coordinate offset, represents the coordinates of facial key points in any video frame, Represents the keypoint coordinates of the base frame.

2. The method according to claim 1, characterized in that The step of performing key point detection on each video frame containing the target to be detected to obtain facial key point coordinates corresponding to each video frame containing the target to be detected includes: Multiple video frames containing the target to be detected are input into a pre-trained target detection model to obtain the coordinates of facial key points corresponding to each of the video frames containing the target to be detected.

3. The method according to claim 1, characterized in that After determining the key point coordinate offset corresponding to each of the facial key point coordinates, the method further includes: Obtaining the distance between the left and right eyebrows of the target to be detected, wherein the distance between the left and right eyebrows is the distance between the center point of the left eyebrow and the center point of the right eyebrow; The key point coordinate offset corresponding to each of the facial key point coordinates is divided by the distance between the left and right eyebrows of the target to be detected to obtain a normalized key point coordinate offset.

4. The method according to claim 1, characterized in that: The constructing of a face change spatiotemporal graph based on the key point coordinate offsets corresponding to the facial key point coordinates comprises: Using the key point coordinate offset as a node for constructing a spatial graph; The spatial relationship of the facial anatomical structure of the target to be detected is used as the edge of the spatial graph; generating a facial spatial graph based on the nodes of the spatial graph and the edges of the spatial graph; Generate a facial temporal map based on the temporal relationship of the same key points between consecutive video frames; The facial space map and the facial time map are combined to obtain a facial change space-time map.

5. The method according to any one of claims 1 to 4, characterized in that: After inputting the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result, the method further includes: A target function is triggered according to the facial expression recognition result, wherein the target function includes a music changing function and / or an in-vehicle light adjustment function.

6. A facial expression recognition device, characterized in that: include: A video frame acquisition module is used to acquire multiple video frames containing the target to be detected; A key point detection module, used to perform key point detection on each video frame containing the target to be detected, and obtain the facial key point coordinates corresponding to each video frame containing the target to be detected; A key point coordinate offset determination module, used to determine the key point coordinate offset corresponding to each of the facial key point coordinates; A facial change spatiotemporal graph construction module, used to construct a facial change spatiotemporal graph based on the key point coordinate offsets corresponding to the facial key point coordinates; A facial expression recognition result prediction module is used to input the facial change spatiotemporal graph into a pre-trained facial expression recognition model to obtain a facial expression recognition result; The key point coordinate offset determination module is specifically used for: Extracting key points from a reference frame to obtain key point coordinates of the reference frame, wherein the reference frame is a video frame corresponding to a calm facial expression obtained by manual selection; The key point coordinate offset is determined based on the coordinates of each of the facial key points and the key point coordinates of the reference frame. The formula for determining the key point coordinate offset is: ； in, Indicates the key point coordinate offset, represents the coordinates of facial key points in any video frame, Represents the keypoint coordinates of the base frame.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the facial expression recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the facial expression recognition method according to any one of claims 1 to 5 when executed.

Images