Landmark detection using deep neural networks with multi-frequency self-attention

Abstract

Systems and methods of landmark detection using deep neural networks with multi-frequency self-attention are provided. The system includes an encoder network that receives an image of an object of interest as an input and generates a multi-frequency feature map as an output. The system also includes an attention layer that receives the generated multi-frequency feature map and refines the generated multi-frequency feature map based on a correlation or association between the received multi-frequency feature maps. The system further includes a decoder network that receives the refined multi-frequency feature map from the attention layer as a second input and generates a landmark detection result based on the second input. The landmark detection result includes a heat map image of the object of interest, and the heat map image indicates locations of landmark points on the object of interest in the image.

Description

[0001] Cross-reference to related applications / incorporation by reference

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 211,127, filed June 16, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Various embodiments of this disclosure relate to neural networks and object landmark detection. More specifically, various embodiments of this disclosure relate to systems and methods for landmark detection using deep neural networks with multi-frequency self-attention. Background Technology

[0004] Advances in machine learning and artificial intelligence have led to the development of various types of neural networks (or models) that can be used for different applications, such as landmark detection (e.g., face landmark detection). Typically, landmark detection corresponds to the task of detecting multiple key points or landmarks in an image of an object (such as a face). For a face, such key points or landmarks can represent salient regions of the face, such as the mouth, eyes, nose, chin, or eyebrows. Face landmark detection has a variety of applications, such as face recognition, face deformation, head pose estimation, face alignment, motion detection, and 3D modeling. However, conventional techniques for landmark detection may not accurately detect landmarks on objects in an image, especially if the image quality of the object is poor (i.e., occluded or blurred). Furthermore, conventional techniques for landmark detection may not be robust enough to accurately detect landmarks in images of objects that differ from normal human faces and living objects (such as caricature or cartoon faces that often exhibit a lot of exaggerated facial features).

[0005] By comparing the described system with some aspects of this disclosure, the limitations and disadvantages of conventional and traditional methods will become apparent to those skilled in the art, as illustrated in the remainder of this application with reference to the accompanying drawings. Summary of the Invention

[0006] Essentially as shown in at least one of the figures and / or described with respect to at least one figure, a system and method for landmark detection using a deep neural network with multi-frequency self-attention are provided, as set forth more fully in the claims.

[0007] These and other features and advantages of this disclosure can be understood from the following detailed description of the disclosure, together with the accompanying drawings, in which similar reference numerals always refer to similar parts. Attached Figure Description

[0008] Figure 1 This is a diagram illustrating a network environment for landmark detection using a deep neural network with multi-frequency self-attention, according to an embodiment of the present disclosure.

[0009] Figure 2 This is an exemplary block diagram of a system for landmark detection using a deep neural network with multi-frequency self-attention, according to embodiments of the present disclosure.

[0010] Figure 3A and Figure 3B These are common illustrations of embodiments according to this disclosure. Figure 1 A diagram of an exemplary architecture of an encoder-decoder neural network.

[0011] Figure 3C This describes an embodiment according to the present disclosure. Figure 1 A diagram of an exemplary architecture of the attention layer of an encoder-decoder neural network.

[0012] Figure 4A This is a diagram illustrating an exemplary scenario of landmark detection using a deep neural network with multi-frequency self-attention according to an embodiment of the present disclosure.

[0013] Figure 4B This is a diagram illustrating an exemplary scenario of landmark detection using a deep neural network with multi-frequency self-attention according to an embodiment of the present disclosure.

[0014] Figure 5 This is a flowchart illustrating an exemplary method for landmark detection using a deep neural network with multi-frequency self-attention, according to an embodiment of the present disclosure. Detailed Implementation

[0015] The implementation described below can be found in the disclosed systems and methods for landmark detection using deep neural networks with multi-frequency self-attention. The system includes an encoder network that receives input including an image of an object of interest (such as a human face or cartoon face) and generates a multi-frequency feature map as the output of the final layer of the encoder network based on the received first input. The system also includes an attention layer that can be coupled to the final layer of the encoder network. The attention layer receives the generated multi-frequency feature map from the final layer of the encoder network and refines the multi-frequency feature map based on the correlation or association between the received multi-frequency feature maps. The system may also include a decoder network that can be coupled to the encoder network via the attention layer. The decoder network receives the refined multi-frequency feature map as input from the attention layer and generates a landmark detection result based on the input. The landmark detection result may include a heatmap image of the object of interest. The heatmap image may indicate the location of landmark points on the object of interest in the image.

[0016] Landmark detection corresponds to the task of locating predefined landmarks on an object of interest. For example, 68 landmarks can be predefined for a face. Landmark detection plays a crucial role in many face-related applications such as face frontalization, 3D face reconstruction, or face recognition. It may be required in a variety of tasks, including face super-resolution, emotion recognition, and other face reconstruction tasks involving facial quality enhancement or altering style and appearance through makeup and other techniques.

[0017] Several traditional networks have been developed specifically for landmark detection from images of objects of interest. However, most of these traditional networks may only detect landmarks associated with one type of object of interest. For example, if a network can detect facial landmarks on a real face image, the same network may face problems detecting landmarks on a cartoon face image that differs from a normal face and exhibits many exaggerated facial features. In some cases, these traditional networks often fail to deliver the desired results if the image of the object of interest is occluded or blurred, or if it has a different pose or lighting.

[0018] This disclosure provides a neural network architecture that addresses the challenges of conventional networks by enabling hierarchical multi-frequency spatial learning for efficiently locating landmarks in different variations of objects, such as cartoon faces, comical faces, or real human faces. This disclosure also provides a multi-frequency attention module that captures the correlations and complex interactions between one or more high-frequency feature maps and low-frequency feature maps, handling variability in size and shape of various facial features. Therefore, this disclosure provides a robust mechanism for landmark detection for each type of object of interest. Compared to conventional networks that struggle to detect landmarks when images are occluded or blurred, or when they have different poses or lighting conditions, the disclosed neural network is able to detect landmarks even in these situations.

[0019] Figure 1 This is a diagram illustrating a network environment for landmark detection using a deep neural network with multi-frequency self-attention, according to an embodiment of the present disclosure. (Refer to...) Figure 1 A diagram of network environment 100 is shown. Network environment 100 includes system 102. System 102 includes processor 104 and encoder-decoder neural network 106. Encoder-decoder neural network 106 may include encoder network 108, attention layer 110, decoder network 112, and one or more skip connections 114. (See reference...) Figure 1The diagram further illustrates a display device 116 and a communication network 118. The encoder network 108 may include multiple encoding layers. Such encoding layers may include an input layer 120A, a first set of convolutional layers 120B following the input layer 120A, and a final layer 120C. Similarly, the decoder network 112 may include multiple decoding layers. Such decoding layers may include an input layer 122A, a second set of convolutional layers 122B following the input layer 122A, and a final layer 122C. (See also...) Figure 1 The image also shows an image 124 of the object of interest 126, a heat map image 128 indicating the position of the landmark point 130, and a final image 132. The final image 132 can be displayed on a display device 116.

[0020] Processor 104 may include suitable logic, circuitry, and interfaces that can be configured to execute program instructions associated with different operations to be performed by system 102. For example, some of these operations may involve overlaying colored markers on image 124 to indicate the location of landmark point 130 on object of interest 126 to generate a final image 132, and controlling display device 116 to present an output including the generated final image 132. In some other embodiments, these operations may involve training encoder-decoder neural network 106. Processor 104 may be implemented based on many processor technologies known in the art. Examples of processor technologies may include, but are not limited to, central processing unit (CPU), x86-based processor, reduced instruction set computing (RISC) processor, application-specific integrated circuit (ASIC) processor, complex instruction set computing (CISC) processor, graphics processing unit (GPU), coprocessors (such as inference accelerators or artificial intelligence (AI) accelerators), and / or combinations thereof.

[0021] The encoder-decoder neural network 106 can be a computational network or system of artificial neurons arranged in multiple layers. The encoder-decoder neural network 106 can determine the location of landmark point 130 on object of interest 126 in image 124. For example, the encoder-decoder neural network 106 can be trained on a task of landmark detection on object of interest 126 (i.e., detecting the location of landmark point 130).

[0022] The encoder network 108 may be a computational network or system of artificial neurons arranged in multiple coding layers 120. The multiple coding layers 120 of the encoder network 108 may include an input layer 120A, a first set of convolutional layers 120B, and a final layer 120N. Each of the multiple coding layers 120 may include one or more nodes (or artificial neurons, e.g., represented by circles or nodes). The outputs of all nodes in the input layer 120A may be coupled to at least one node in one or more convolutional layers in the first set of convolutional layers 120B. Similarly, the inputs of each convolutional layer may be coupled to the outputs of at least one node in the other layers of the encoder network 108. The outputs of each convolutional layer may be coupled to the inputs of at least one node in the other layers of the encoder network 108. Nodes in one or more of the final layers 120N may receive inputs from at least one convolutional layer to output results. The number of layers and the number of nodes in each layer may be determined based on the hyperparameters of the encoder network 108. Such hyperparameters may be set before or after training the encoder network 108 on a training dataset.

[0023] Each node of the encoder network 108 may correspond to a mathematical function (e.g., a sigmoid function or a rectified linear unit) having a set of parameters that can be tuned during training of the encoder network 108. For example, this set of parameters may include weight parameters, regularization parameters, etc. Each node may use the mathematical function to compute an output based on one or more inputs from nodes in one or more other layers (e.g., previous layers). All or some nodes of the encoder network 108 may correspond to the same or different mathematical functions.

[0024] Encoder network 108 may include electronic data, which may be implemented as a software component of an application executable on system 102 (or on display device 116). Encoder network 108 may rely on libraries, external scripts, or other logic / instructions for execution by a processing device. Encoder network 108 may include code and routines configured to enable a computing device (such as processor 104) to execute one or more operations for generating multi-frequency feature maps. Additionally or alternatively, encoder network 108 may be implemented using hardware, including but not limited to processors, microprocessors (e.g., for performing one or more operations or controlling the execution of one or more operations), coprocessors, field-programmable gate arrays (FPGAs), or application-specific integrated circuits (ASICs). In some embodiments, encoder network 108 may be implemented using a combination of hardware and software.

[0025] Attention layer 110 may include appropriate logic, circuitry, and interfaces configured to receive generated multi-frequency feature maps from the final layer 120N of encoder network 108 and refine the multi-frequency feature maps based on the correlation or association between the received multi-frequency feature maps. Attention layer 110 may be coupled to the final layer 120N of encoder network 108. For example, in Figure 3C The document provides detailed information on the refinement process.

[0026] The decoder network 112 may be a computational network or system of artificial neurons arranged in multiple decoding layers 122. The multiple decoding layers 122 of the decoder network 112 may include an input layer 122A, a second set of convolutional layers 122B, and a final layer 122N. Each of the multiple decoding layers 122 may include one or more nodes or artificial neurons (e.g., represented by circles). The outputs of all nodes in the input layer 122A may be coupled to at least one second set of convolutional layers 122B. Similarly, the inputs of each layer in the second set of convolutional layers 122B may be coupled to the outputs of at least one node in the other layers of the decoder network 112. The outputs of each convolutional layer in the second set of convolutional layers 122B may be coupled to the inputs of at least one node in the other layers of the decoder network 112. Nodes(one or more) in the final layer 122N may receive input from at least one convolutional layer to output a result. The number of layers and the number of nodes in each layer may be determined based on the hyperparameters of the decoder network 112. Such hyperparameters may be set before or after training the decoder network 112 on a training dataset.

[0027] Each node of the decoder network 112 may correspond to a mathematical function having a set of parameters that can be adjusted during the training of the decoder network 112. For example, this set of parameters may include weight parameters, regularization parameters, etc. Each node may use the mathematical function to compute an output based on one or more inputs from nodes in one or more other layers (e.g., previous layers). All or some nodes of the decoder network 112 may correspond to the same or different mathematical functions.

[0028] Decoder network 112 may include electronic data, which may be implemented as a software component of an application executable on system 102 (or on display device 116). Decoder network 112 may rely on libraries, external scripts, or other logic / instructions for execution by processing devices. Decoder network 112 may include code and routines configured to enable computing devices (such as processor 104) to execute one or more operations for generating landmark detection results. Additionally or alternatively, decoder network 112 may be implemented using hardware, including processors, microprocessors (e.g., for performing one or more operations or controlling the execution of one or more operations), field-programmable gate arrays (FPGAs), coprocessors, or application-specific integrated circuits (ASICs). Alternatively, in some embodiments, decoder network 112 may be implemented using a combination of hardware and software.

[0029] Display device 116 may include suitable logic, circuitry, and interfaces configured to display a final image 132 that may indicate a detected landmark on object of interest 126. In an embodiment, display device 116 may be configured to display a thermal image 128. Display device 116 can be implemented using several known technologies, such as, but not limited to, at least one of: liquid crystal display (LCD), light-emitting diode (LED) display, plasma display, or organic LED (OLED) display technology, or other display devices. According to embodiments, display device 116 may refer to a display screen of a head-mounted device (HMD), a smart glass device, a see-through display, a projection-based display, an electrochromic display, or a transparent display.

[0030] In another embodiment, display device 116 may include suitable logic, circuitry, interfaces, and / or code that can implement encoder-decoder neural network 106 as part of a software program or service (such as an application programming interface (API)-based service) executable on display device 116. The encoder-decoder neural network 106 can be implemented on display device 116 after training of the encoder-decoder neural network 106 has been completed on system 102. Examples of display device 116 may include, but are not limited to, computing devices, mainframes, servers, computer workstations, smartphones, cellular phones, mobile phones, gaming devices, wearable displays, consumer electronics (CE) devices, and / or any other device with image processing capabilities.

[0031] Communication network 118 may include a communication medium through which system 102 and display device 116 can communicate with each other. Communication network 118 may include either a wired connection or a wireless connection. Examples of communication network 118 may include, but are not limited to, the Internet, cloud networks, cellular or wireless mobile networks (such as LTE and 5G New Radio), Wi-Fi networks, personal area networks (PANs), local area networks (LANs), or metropolitan area networks (MANs). Various devices in network environment 100 may be configured to connect to communication network 118 according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Li-Fi, 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocols, and Bluetooth (BT) communication protocol.

[0032] In operation, encoder network 108 may receive a first input, which may include an image, such as an image 124 of object of interest 126. Based on the first input, encoder network 108 may be configured to generate a multi-frequency feature map as the output of the final layer 120N in a plurality of coding layers 120 of encoder network 108.

[0033] The multi-frequency feature map may include a first frequency feature map and a second frequency feature map. The first frequency feature map may capture first spatial information associated with global shape features of the object of interest 126, and the second frequency feature map may capture second spatial information associated with local shape features of the object of interest 126. In embodiments, local shape features may be more refined and more numerous than global shape features on the object of interest 126. The first frequency feature map may be referred to as a low-frequency feature map, and the second frequency feature map may be referred to as a high-frequency feature map. By way of example and not limitation, if the object of interest 126 is a face or a cartoon of a person, the low-frequency feature map may cover the shape and size of the face or cartoon of a person, while the high-frequency feature map covers the shape and size of the lips, eyes, nose, etc. of the face or cartoon of a person.

[0034] Attention layer 110 can be coupled to the final layer 120N of encoder network 108. Attention layer 110 can receive generated multi-frequency feature maps from the final layer 120N of encoder network 108. Attention layer 110 can be configured to refine the multi-frequency feature maps based on the correlation or association between the received multi-frequency feature maps. For example, in… Figure 3CThe document provides detailed information about attention layer 110 and the multi-frequency feature map.

[0035] Decoder network 112 can be coupled to encoder network 108 via attention layer 110. Decoder network 112 can be configured to receive a refined multi-frequency feature map from attention layer 110 as a second input. Decoder network 112 can be configured to generate landmark detection results based on the second input. The landmark detection results may include a heatmap image 128 of object of interest 126. Heatmap image 128 may indicate the location of landmark point 130 on object of interest 126 in image 124. For example, in Figure 3B Details about the decoder network are provided in the document.

[0036] According to an embodiment, the encoder-decoder neural network 106 may include one or more skip connections 114 between intermediate coding layers of the encoder network 108 and intermediate decoding layers of the decoder network 112. The intermediate coding layers of the encoder network 108 may be configured to generate a first set of intermediate multi-frequency feature maps based on inputs from previous layers in the encoder network 108. Each of the one or more skip connections 114 may be configured to pass spatial information included in the first set of intermediate multi-frequency feature maps to the intermediate decoding layers of the decoder network 112. For example, in Figure 3A and Figure 3B Details about one or more jump connections 114 are provided in the document.

[0037] Processor 104 can be configured to extract landmark detection results from the final layer 122N of decoder network 112. Subsequently, processor 104 can overlay colored markers on image 124 to indicate the positions of landmark points 130 on object of interest 126, thereby generating final image 132. After generating final image 132, processor 104 can control display device 116 to present output including final image 132. For example, in Figure 4A and Figure 4B Details about colored markings are provided in the document.

[0038] Figure 2 This is an exemplary block diagram of a system for landmark detection using a deep neural network with multi-frequency self-attention, according to embodiments of the present disclosure. (Combined with...) Figure 1 To explain the elements Figure 2 . Reference Figure 2 , showed Figure 1 A block diagram 200 of system 102. The system includes a processor (such as processor 104), memory 202, input / output (I / O) devices 204, network interface 206, inference accelerator 208, and encoder-decoder neural network 106.

[0039] Memory 202 may include suitable logic, circuitry, and / or interfaces that can be configured to store program instructions executable by processor 104. Furthermore, memory 202 may store image 124 and thermal image 128. In at least one embodiment, memory 202 may also store encoder-decoder neural network 106, multi-frequency feature maps, and spatial information associated with image 124. Examples of implementations of memory 202 may include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), hard disk drive (HDD), solid-state drive (SSD), CPU cache, and / or secure digital card (SD card).

[0040] I / O device 204 may include suitable logic, circuitry, and / or interfaces that can be configured to receive one or more user inputs and / or present information generated by system 102. I / O device 204 may include various input and output devices that can be configured to communicate with different operating components of system 102. Examples of I / O device 204 may include, but are not limited to, touchscreens, keyboards, mice, joysticks, microphones, and display devices (such as display device 116).

[0041] Network interface 206 may include suitable logic, circuitry, interfaces, and / or code that can be configured to establish communication between system 102 and display device 116 via communication network 118. Network interface 206 may be configured to implement known technologies to support wired or wireless communication. Network interface 206 may include, but is not limited to, antennas, radio frequency (RF) transceivers, one or more amplifiers, tuners, one or more oscillators, digital signal processors, codec chipsets, subscriber identity module (SIM) cards, and / or local buffers.

[0042] Network interface 206 can be configured to communicate with networks (such as the Internet, intranets, and / or wireless networks, such as cellular telephone networks, wireless local area networks (WLANs), personal area networks, and / or metropolitan area networks (MANs)) via offline and online wireless communications. Wireless communications can use any of a variety of communication standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), LTE, 5G New Radio, Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (such as IEEE 802.11, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and / or any other IEEE 802.11 protocol), Voice over Internet Protocol (VoIP), Wi-MAX, Internet of Things (IoT) technologies, Machine Type Communication (MTC) technologies, protocols for email, instant messaging, and / or Short Message Service (SMS).

[0043] Inference accelerator 208 may include suitable logic, circuitry, interfaces, and / or code that can be configured as a coprocessor to accelerate computations associated with the operation of encoder-decoder neural network 106. For example, inference accelerator 208 can accelerate computations on system 102, enabling the generation of landmark detection results in less time than would typically be required without inference accelerator 208. Inference accelerator 208 can implement various acceleration techniques, such as parallelization of some or all operations of encoder network 108 and decoder network 112. Inference accelerator 208 can be implemented as software, hardware, or a combination thereof. Example implementations of inference accelerator 208 may include, but are not limited to, GPUs, Tensor Processing Units (TPUs), neuromorphic chips, Vision Processing Units (VPUs), Field-Programmable Gate Arrays (FGPAs), Reduced Instruction Set Computing (RISC) processors, Application-Specific Integrated Circuits (ASIC) processors, Complex Instruction Set Computing (CISC) processors, microcontrollers, and / or combinations thereof.

[0044] Figure 3A and Figure 3B The illustrations together depict embodiments according to this disclosure. Figure 1 An exemplary architecture of an encoder-decoder neural network. Combined with... Figure 1 and Figure 2 To explain the elements Figure 3A and Figure 3B . Reference Figure 3A and Figure 3B This shows that it can be Figure 1Figure 300 shows an exemplary variant of the encoder-decoder neural network 106, namely an encoder-decoder neural network 302. The encoder-decoder neural network 302 may include an encoder network 304, an attention layer 306, and a decoder network 308. The encoder network 304 and the decoder network 308 may be coupled via the attention layer 306.

[0045] The encoder network 304 and decoder network 308 can be deep neural networks (DNNs). The encoder network 304 can include multiple encoding layers 310. The multiple encoding layers 310 can include an input layer 310A, a first set of convolutional layers 312, and a final layer 310B. The first set of convolutional layers 312 can follow the input layer 310A and can include a first convolutional layer 312A, a second convolutional layer 312B, a third convolutional layer 312C, up to an Nth convolutional layer 312N. For example, the first set of convolutional layers 312 can include five convolutional layers. The first convolutional layer 312A can be coupled to the input layer 310A of the encoder network 304. Each of the remaining convolutional layers in the first set of convolutional layers 312 can be coupled to a previous convolutional layer in the multiple encoding layers 310. For example, a second convolutional layer 312B can be coupled to a first convolutional layer 312A, a third convolutional layer 312C can be coupled to a second convolutional layer 312B, and so on. Each convolutional layer in the first set of convolutional layers 312 may include two convolution operators. For example, these two convolution operators may correspond to an octave convolution operation. Similar to the encoder network 304, the decoder network 308 may include multiple decoding layers 314. The multiple decoding layers 314 may include an input layer 314A, a second set of convolutional layers 316, and a final layer 310B. The second set of convolutional layers 316 may follow the input layer 314A and may include a first convolutional layer 316A, a second convolutional layer 316B, a third convolutional layer 316C, and so on up to an Nth convolutional layer 316N. For example, the second set of convolutional layers 316 may include five or fewer convolutional layers. Specifically, the first convolutional layer 316A in the second set of convolutional layers 316 may be coupled to the input layer 314A of the decoder network 308. Each of the remaining convolutional layers in the second set of convolutional layers 316 may be coupled to a previous convolutional layer in the multiple decoding layers 314. For example, a second convolutional layer 316B can be coupled to a first convolutional layer 316A, a third convolutional layer 316C can be coupled to a second convolutional layer 316B, and so on. Each convolutional layer in the second set of convolutional layers 316 can include two convolution operators. For example, these two convolution operators can correspond to the transposed octave convolution operation (also known as the double transposed octave convolution operation).

[0046] In operation, the encoder network 304 of the encoder-decoder neural network 302 can be configured to receive a first input. Specifically, the input layer 310A of the plurality of coding layers 310 of the encoder network 304 can be configured to receive the first input. The first input may include an input image (such as image 124) of an object of interest (such as object of interest 126). Figure 3A As shown, for example, the input image can be a 3-channel image (such as a red, green, and blue (RGB) image) with an input size of 256x256 pixels. The object of interest 126 can correspond to a living or inanimate object. For example, the object of interest 126 can correspond to a cartoon character, fictional character, cartoon character, or the face of a real person.

[0047] To generate multi-frequency feature maps, encoder network 304 can be configured to divide an input image (such as image 124) included in the received first input into initial multi-frequency feature maps. The initial multi-frequency feature maps may include an initial first frequency feature map that captures first spatial information associated with global shape features of the object of interest 126 and an initial second frequency feature map that captures second spatial information associated with local shape features of the object of interest 126. Global shape features may be associated with the shape of the object of interest 126. Local shape features may be associated with the shape and / or expression of one or more parts of the object of interest 126. As an example, if the object of interest 126 is a face, global features may be associated with the structure of the face. Local features may be associated with the structure of facial parts (such as eyes, lips, nose, etc.) and facial expressions. The initial first frequency feature map and the initial second frequency feature map may be referred to as an initial low-frequency feature map and an initial high-frequency feature map, respectively. Each of the initial low-frequency feature map and the initial high-frequency feature map may have a size different from the input size of image 124. In this embodiment, the initial high-frequency feature map may have the same size as the input size of image 124. Furthermore, the initial low-frequency feature map may have a size one octave smaller than the high-frequency feature map. For example, the initial high-frequency feature map may have a size of 256x256, and the initial low-frequency feature map may have a size of 128x128.

[0048] The encoder network 304 can also be configured to pass the initial multi-frequency feature map through the first set of convolutional layers 312 to generate a multi-frequency feature map as the output of the final layer 310B of the encoder network 304. As discussed above, each convolutional layer in the first set of convolutional layers 312 may include two convolution operators that can be applied to each of the initial multi-frequency feature maps. Each of these two convolution operators may correspond to an octave convolution operation (also known as a double octave convolution operation).

[0049] Octave convolution can decompose a multi-frequency feature map into low-frequency and high-frequency feature maps. Specifically, octave convolution can capture spatial information associated with image 124 at multiple frequencies. For example, if I H and I L Let O represent the high-frequency feature map and low-frequency feature map of the input, respectively. Then, we can use the following equations (1) and (2) to obtain the high-frequency feature map and low-frequency feature map O of the output obtained from the octave convolution. H and O L :

[0050] O H =f H→H (I H )+f L→H (I L (1)

[0051] O L =f L→L (I L )+f H→L (I H (2)

[0052] in,

[0053] f H→H and f L→L Refers to intra-frequency update operations, and

[0054] f L→H and f H→L This indicates inter-frequency communication.

[0055] The encoder network 304 can also be configured to pass the initial multi-frequency feature map through a first set of convolutional layers 312 to generate a multi-frequency feature map as the output of the final layer 310B of the encoder network 304. Each convolutional layer between the input layer 310A and the final layer 310B of the encoder network 304 can generate a first set of intermediate multi-frequency feature maps. For example, the first convolutional layer 316A (i.e., "convolutional layer 1") can generate a first set of intermediate multi-frequency feature maps, the second convolutional layer 316B (i.e., "convolutional layer 2") can generate a second set of intermediate multi-frequency feature maps, and so on. The size of each intermediate feature map in the first set of intermediate multi-frequency feature maps can be smaller than the initial multi-frequency feature map and larger than the multi-frequency feature map (obtained from the final layer 310B).

[0056] In an embodiment, the downsampling operation can be applied to each of the first set of intermediate multi-frequency feature maps before the corresponding intermediate multi-frequency feature maps are passed to the next layer where the octave convolution operation can be applied. For example, a set of first intermediate multi-frequency feature maps generated by the first convolutional layer 316A can be downsampled before being passed to the second convolutional layer 316B.

[0057] By way of example, and not limitation, the initial high-frequency feature map and the initial low-frequency feature map can have sizes of 256x256 and 128x128, respectively. The first convolutional layer 312A can generate a set of first intermediate multi-frequency feature maps, which may include high-frequency feature maps and low-frequency feature maps with sizes of 128x128 and 64x64, respectively. Similarly, the second convolutional layer 312B can generate a set of second intermediate multi-frequency feature maps, which may include high-frequency feature maps and low-frequency feature maps with sizes of 64x64 and 32x32, respectively. Similarly, the Nth convolutional layer 312N can generate intermediate multi-frequency feature maps, which may include high-frequency feature maps and low-frequency feature maps with sizes of 16x16 and 8x8, respectively.

[0058] The final layer 310B of the encoder network 304 can generate a multi-frequency feature map as output. The multi-frequency feature map can include a first frequency feature map that captures first spatial information associated with global shape features of the object of interest 126. As shown, for example, the first frequency feature map can have a size of 16x16. The multi-frequency feature map can also include a second frequency feature map that captures second spatial information associated with local shape features of the object of interest 126, and can have a size of 8x8, one octave smaller than the first frequency feature map. As discussed, local shape features can be more refined and numerous compared to global shape features on the object of interest 126.

[0059] In an embodiment, each feature map generated by the first set of convolutional layers 312 and the final layer 310B can have multiple channels. The number of channels in each of the first set of intermediate multi-frequency feature maps can be based on a first constant (α) and a second constant (C). The first constant (α) can represent the ratio of channels allocated to the low-frequency feature map and the high-frequency feature map. The low-frequency feature map can be defined as being one octave lower than the high-frequency feature map, i.e., the spatial resolution of the low-frequency feature map can be half that of the high-frequency feature map. The value of the first constant (α) can be between 0 and 1 (inclusive). According to an embodiment, the value of the first constant (α) can be set to 0.25, and the value of the second constant (C) can be fixed at 128. The number of channels in the first frequency feature map of the generated multi-frequency feature map can be (1-α)^8C, and the number of channels in the second frequency feature map of the generated multi-frequency feature map can be α.C.

[0060] Attention layer 306 can be configured to receive generated multi-frequency feature maps from the final layer 310B of encoder network 304. Attention layer 306 can be coupled to the final layer 310B of encoder network 304. The final layer 310B of encoder network 304 can be referred to as a bottleneck layer. Attention layer 306 can be configured to refine the multi-frequency feature maps based on the correlation or association between the received multi-frequency feature maps. For example, in… Figure 3C The document provides detailed information about attention layer 306 and the multi-frequency feature map.

[0061] The decoder network 308 can be configured to receive a refined multi-frequency feature map as a second input from the attention layer 306. Specifically, the input layer 314A of the decoder network 308 can be configured to receive a refined multi-frequency feature map, which may include a first refined multi-frequency spectral feature map and a second refined feature map, as input to the decoder network 308. The decoder network 308 can be coupled to the encoder network 304 via the attention layer 306.

[0062] Decoder network 308 can be configured to generate landmark detection results based on the second input. To generate the landmark detection results, decoder network 308 can be configured to pass the refined multi-frequency feature map through a second set of convolutional layers 316. The landmark detection results can be generated as the output of the final layer 314B of decoder network 308. Similar to encoder network 304, each convolutional layer between the input layer 314A and the final layer 314B of decoder network 308 can generate a second set of intermediate multi-frequency feature maps. However, unlike the first set of intermediate multi-frequency feature maps, the size of each intermediate feature map in the second set of intermediate multi-frequency feature maps can be larger than the refined multi-frequency feature map.

[0063] In an embodiment, the decoder network 308 can be configured to upsample the refined multi-frequency feature map. By way of example, if the size of the first refined multi-frequency feature map received at input layer 314A is 16x16 and the size of the second multi-frequency feature map is 8x8, then the decoder network 308 can be configured to upsample the first refined multi-frequency feature map to 32x32 and the second multi-frequency feature map to 16x16, respectively. The decoder network 308 can then pass the upsampled multi-frequency feature map through the first convolutional layer 316A in the second set of convolutional layers 316 to generate first intermediate feature maps of sizes 32x32 and 16x16, respectively. As discussed, the first convolutional layer 316A may include two convolution operators that correspond to a transposed octave convolution operation. In some embodiments, these two convolution operators may correspond to a bilinear interpolation operation.

[0064] Using successive transposed octave convolution operations, spatial information that may have been captured by octave convolution operations at encoder network 304 can be retrieved. Decoder network 308 can be configured to apply transposed octave convolution operations to retrieve spatial information. The application of transposed octave convolution operations at the first convolutional layer 316A can result in upsampling the generated first intermediate feature maps of sizes 32x32 and 16x16 to intermediate feature maps of sizes 64x64 and 32x32, respectively. Similarly, the second convolutional layer 316B and subsequent convolutional layers of decoder network 308 can be configured to generate second intermediate feature maps, third intermediate feature maps, and so on. The final layer 314B of decoder network 308 can be configured to generate landmark detection results as the output of decoder network 308. According to an embodiment, decoder network 308 can be configured to apply a single transposed octave convolution to the intermediate output of the Nth convolutional layer 316N to generate landmark detection results as the output of the final layer 314B of decoder network 308.

[0065] The landmark detection result may include a heatmap image (or output image) of the object of interest 126. The heatmap image can indicate the positions of landmark points on the object of interest 126 in image 124. For example, if the object of interest 126 is a face, the count of landmark points on the face could be 68. In an embodiment, the final size of the heatmap image can be smaller than the input size of image 124. This can be done to reduce the computational overhead of processing intermediate feature maps to generate a heatmap image of the input size. Figure 3B As shown, the output image can be 64x64 in size and can have M channels. The number of channels (M) can correspond to the number of landmark points. For example, if the object of interest 126 is a face, the output image can be 64x64 in size and can have M = 68 channels (the same as the number of landmark points for a face).

[0066] In an embodiment, the encoder-decoder neural network 302 may further include one or more skip connections. These skip connections may include, but are not limited to, a first skip connection 318A and a second skip connection 318B. The first skip connection 318A may be located between a first intermediate encoding layer (such as convolutional layer 3) of the encoder network 304 and a first intermediate decoding layer (such as convolutional layer N-1) of the decoder network 308. Similarly, the second skip connection 318B may be located between a second intermediate encoding layer (such as convolutional layer N-1) of the encoder network 304 and a second intermediate decoding layer (such as convolutional layer 3) of the decoder network 308.

[0067] The first intermediate encoding layer of encoder network 304 can be configured to generate a first set of intermediate multi-frequency feature maps based on inputs from previous layers in encoder network 304, and the first skip connection 318A can be configured to transmit spatial information included in the first set of intermediate multi-frequency feature maps to the first intermediate decoding layer of decoder network 308. In other words, the first skip connection 318A can be incorporated into encoder-decoder neural network 302 to recover positional information in decoder network 308 and also reduce spatial loss incurred in encoder network 304 during the encoding phase. Similarly, the second intermediate encoding layer of encoder network 304 can be configured to generate a second set of intermediate multi-frequency feature maps based on inputs from previous layers in encoder network 304, and the second skip connection 318B can be configured to transmit spatial information included in the second set of intermediate multi-frequency feature maps to the second intermediate decoding layer of decoder network 308. In an embodiment, encoder-decoder neural network 106 may include more than two skip connections.

[0068] It should be noted that the attention layer 306 can be incorporated at the bottleneck layer (i.e., after the final layer 310B of the encoder network 304) because the final layer 310B contains the most important and abstract information that can be encoded at the encoder network 304. Furthermore, the spatial resolution of the high-frequency and low-frequency feature maps can be lower at the bottleneck layer. This reduces the computational cost of the operation of the attention layer 306 and can lead to faster execution of the encoder-decoder neural network 302.

[0069] The disclosed encoder-decoder neural network 302 is robust enough to detect landmarks of human faces, whether cartoon, fictional, or real-life, even though cartoon, fictional, or cartoon characters may include exaggerated facial features (e.g., complex jaw structures) compared to real-life faces. Furthermore, the disclosed system is capable of landmark detection even in challenging scenarios such as pose variations, lighting changes, blurring, and occlusion. This is possible because the disclosed encoder-decoder neural network 302 can capture multi-frequency spatial information at each stage of encoding and decoding. This allows the encoder-decoder neural network 302 to learn global and local structures using the multi-frequency, multi-scale information received at each stage. Although the spatial dimensions decrease at successive encoding layers, the encoder-decoder neural network 302 learns high-level abstract information as part of the process. Furthermore, to further improve the quality of the feature maps, an attention layer 306 added at the bottleneck layer can capture long-range dependencies. This attention layer 306 can further facilitate inter-frequency communication, thereby providing attention and ensuring refinement of high-level features. Furthermore, one or more skip connections can be incorporated into the encoder-decoder neural network 302 to recover positional information and also reduce spatial losses incurred during the encoding phase.

[0070] As part of the experiment, one or more evaluation metrics were computed on multiple datasets using the disclosed encoder-decoder neural network 302. These evaluation metrics included normalized mean error (NME) and failure rate. For example, the multiple datasets could include a cartoon dataset, a 300-image in-the-wild (300W) dataset, a Caltech in-the-wild occluded face (COFW) dataset, and a broader in-the-wild face landmark (WFLW) dataset. The NME error of the disclosed encoder-decoder neural network 302 is the lowest among all conventional methods known in the art for landmark detection.

[0071] Figure 3C The illustration is based on an embodiment of the present disclosure. Figure 3A and Figure 3B A diagram of an exemplary attention layer of the encoder-decoder neural network 302. Combined with data from... Figure 1 , Figure 2 , Figure 3A and Figure 3B To explain the elements Figure 3C . Reference Figure 3C An attention layer 306 can be incorporated between the encoder network 304 and the decoder network 308 of the encoder-decoder neural network 302.

[0072] In an embodiment, attention layer 306 may be coupled to the final layer 310B of encoder network 304. Attention layer 306 may be configured to receive generated multi-frequency feature maps from the final layer 310B of encoder network 304. In an embodiment, the final layer 310B of encoder network 304 may be referred to as a bottleneck layer. In other words, attention layer 306 may be implemented at the bottleneck layer. Attention layer 306 may be configured to refine the multi-frequency feature maps based on the correlation or association between the received multi-frequency feature maps.

[0073] As discussed above, the generated multi-frequency feature map may include the first frequency feature map (X). h )320A and Second Frequency Characteristic Map (X) l )320B. The first frequency characteristic diagram 320A may include C h Each channel can have a height "H" and a width "W". The second frequency characteristic diagram 320B can include C. l The second frequency feature map 320B has one channel, and since the second frequency feature map 320B is one octave lower than the first frequency feature map 320A, the second frequency feature map 320B can have a height of "H / 2" and a width of "W / 2". To refine the multi-frequency feature map, the attention layer 306 can be configured to obtain a set of embeddings from the received multi-frequency feature map, which may include a first embedding (φ) 322A, a second embedding (θ) 322B, and a third embedding (γ) 322C. The embeddings can correspond to a translation from a high-dimensional vector to a low-dimensional space. Specifically, each of the set of embeddings can be obtained by applying a 1x1 convolution operation 324 to the multi-frequency feature map. The first embedding 322A, the second embedding 322B, and the third embedding 322C are provided by way of example rather than limitation via the following equations (3), (4), and (5):

[0074] φ=W φ (X l (3)

[0075] θ=W θ (X h (4)

[0076] γ=W r (X h (5)

[0077] in,

[0078] X h The first frequency feature map 320A represents the multi-frequency feature map, and...

[0079] X lThe second frequency feature map 320B represents the multi-frequency feature map, and

[0080] H represents the height of the first frequency characteristic map 320A.

[0081] W represents the width of the first frequency characteristic map 320A.

[0082] C h This indicates the number of channels in the first frequency characteristic map 320A.

[0083] C l This indicates the number of channels in the second frequency characteristic map 320B.

[0084]

[0085] θ∈R CXHXW ,

[0086] Y∈R CXHXW ,and

[0087] C = 128 (fixed value).

[0088] Attention layer 306 can also be configured to flatten each of the set of embeddings to a specific size. For example, the first embedding can be flattened to a size of C x S. Each of the second and third embeddings can be flattened to a size of C x N, where S and N represent the counts of spatial locations, i.e., S = H / 2 x W / 2 and N = H * W.

[0089] Attention layer 306 can also be configured to determine one or more correlations between the first frequency feature map 320A and the second frequency feature map 320B. Specifically, attention layer 306 can be configured to compute a similarity matrix that captures long-range dependencies between low-frequency and high-frequency spatial information captured in the first frequency feature map 320A and the second frequency feature map 320B, respectively. In an embodiment, the similarity matrix can be computed using matrix multiplication as provided in equation (6):

[0090] Y = φ T X θ (6)

[0091] in,

[0092] Y represents the similarity matrix, and Y∈R SXN .

[0093] Attention layer 306 can also be configured to normalize the similarity matrix to obtain a unified similarity matrix. Specifically, the calculated similarity matrix can be normalized based on normalization function 326. In an embodiment, normalization function 326 can be a SoftMax function. The unified similarity matrix can be obtained using the following equation (7):

[0094]

[0095] in,

[0096] Y represents the similarity matrix, and Y∈R SXN ,

[0097] f represents the normalization function, and

[0098] This represents a uniform similarity matrix.

[0099] In the embodiment, the output of the attention layer 306(Z) for each location in Y (as described in Equation 7) can be provided by Equation (8) as given below:

[0100]

[0101] in,

[0102] Z represents attention layer 306, and Z∈R SxC ,and

[0103] γ Y This indicates the transpose of the third embedded 322C.

[0104] Attention layer 306 can also be configured to recover spatial locations to obtain H / 2 × W / 2 from S and Z respectively. Attention layer 306 can also pass S and Z through 1x1 convolutions to obtain C back from C. l The number of channels. The attention layer 306 can also be configured to enable the residual connections to obtain a first refined multi-frequency feature map 328. In an embodiment, the first refined multi-frequency feature map 328 can be provided by the following equation (9):

[0105] X′ l =W Z (Z)+X l (9)

[0106] Where X′ l It is the first refined multi-frequency feature map 328, and

[0107] The first refined multi-frequency feature map 328 (provided by equation (9)) may have captured the complex correlation between high-frequency spatial information and low-frequency spatial information.

[0108] In an embodiment, the second refined feature map may be X h Attention layer 306 can be configured to provide a first refined feature map (given by equation (9)) and a second refined feature map as input to decoder network 308. As discussed, decoder network 308 can receive the refined multi-frequency feature maps and can generate heatmap image 128 based on the refined multi-frequency feature maps as input.

[0109] Figure 4A This is a diagram illustrating an exemplary scenario of landmark detection using a deep neural network with multi-frequency self-attention, according to an embodiment of the present disclosure. Combined with... Figure 1 , Figure 2 , Figure 3A , Figure 3B and Figure 3C To explain the elements Figure 4A . Reference Figure 4A The following diagram illustrates scenario 400A. Scenario 400A shows a processor 404, ... Figure 3A and Figure 3B The encoder-decoder neural network 302 and Figure 1 The system 402 of the display device 116 also shows an image 406 of a cartoon face 408.

[0110] The encoder-decoder neural network 302 may include an encoder network 304, an attention layer 306, and a decoder network 308. The encoder network 304 may receive a first input, which may include an image 406 of an object of interest (i.e., a cartoon face 408). The encoder network 304 may also be configured to generate a multi-frequency feature map as the output of the final layer 310B of the encoder network 304. The multi-frequency feature map can be generated based on the received first input.

[0111] The encoder network 304 can also be configured to send multi-frequency feature maps to an attention layer 306 that can be coupled to the final layer 310B of the encoder network 304. The attention layer 306 can receive the generated multi-frequency feature maps from the final layer 310B of the encoder network 304 and can refine the multi-frequency feature maps based on the correlation or association between the received multi-frequency feature maps. The attention layer 306 can also send the refined multi-frequency feature maps to a decoder network 308 that can be coupled to the encoder network 304 via the attention layer 306.

[0112] The decoder network 308 can receive the refined multi-frequency feature map from the attention layer 306 as a second input. The decoder network 308 can also be configured to generate landmark detection results. In an embodiment, the processor 404 can be configured to extract the landmark detection results from the final layer 314B of the decoder network 308. The landmark detection results may include a heatmap image 410 of the cartoon face 408. The heatmap image 410 may indicate the location of landmark points 412 on the cartoon face 408 in image 406.

[0113] In an embodiment, processor 404 may be configured to overlay colored markers onto image 406 based on heatmap image 410 to indicate the positions of landmark points 412 on cartoon face 408. Based on the colored markers overlaid on image 406, a final image 414 may be generated. For example, a first color (such as white) or a combination of at least two colors (such as white and red) may be used to depict each position of landmark point 412. In an embodiment, a first color may be used to depict the positions of landmark points 412 associated with global shape features, and a second color may be used to depict the positions of landmark points 412 associated with local shape features. Processor 404 may also be configured to control display device 116 to render output, which may include the final image 414 with colored markers overlaid on image 406.

[0114] Figure 4B This is a diagram illustrating an exemplary scenario of landmark detection using a deep neural network with multi-frequency self-attention, according to an embodiment of the present disclosure. Combined with... Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 3C and Figure 4A To explain the elements Figure 4B . Reference Figure 4B Scene 400B is shown. In scene 400B, a processor 404 is shown. Figure 3A and Figure 3B The encoder-decoder neural network 302 and Figure 1 A diagram of system 402 of display device 116 is shown. An image 416 of a face 418 is also shown.

[0115] The encoder-decoder neural network 302 may include an encoder network 304, an attention layer 306, and a decoder network 308. The encoder network 304 may receive a first input, which may include an image 416 of an object of interest (i.e., a face 418). The encoder network 304 may also be configured to generate a multi-frequency feature map as the output of the final layer 310B of the encoder network 304. The multi-frequency feature map can be generated based on the received first input.

[0116] The encoder network 304 can also be configured to send multi-frequency feature maps to an attention layer 306 that can be coupled to the final layer 310B of the encoder network 304. The attention layer 306 can receive the generated multi-frequency feature maps from the final layer of the encoder network 304 and can refine the multi-frequency feature maps based on the correlation or association between the received multi-frequency feature maps. The attention layer 306 can also send the refined multi-frequency feature maps to a decoder network 308 that can be coupled to the encoder network 304 via the attention layer 306.

[0117] The decoder network 308 can receive the refined multi-frequency feature map from the attention layer 306 as a second input and can generate a landmark detection result including a heatmap image 420 of the face 418. The heatmap image 420 can indicate the position of the landmark point 422 on the face 418 in the image 416.

[0118] In an embodiment, processor 404 may be configured to extract landmark detection results from the final layer 310B of decoder network 308. Processor 404 may also be configured to overlay colored markers on image 416 to indicate the locations of landmark points 422 on face 418. Based on the colored markers overlaid on image 416, a final image 424 may be generated. For example, a first color (such as white) or a combination of at least two colors (such as white and red) may be used to depict each location of landmark point 422. In an embodiment, a first color may be used to depict the locations of landmark points 422 associated with global shape features, and a second color may be used to depict the locations of landmark points 422 associated with local shape features. Processor 404 may also be configured to control display device 116 to render output, which may include the final image 424 with colored markers overlaid on image 416.

[0119] Figure 5 This is a flowchart illustrating an exemplary method for landmark detection using a deep neural network with multi-frequency self-attention, according to an embodiment of the present disclosure. (Combined with...) Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 4A and Figure 4B To explain the elements Figure 5 . Reference Figure 5 The flowchart 500 is shown. The operation of the flowchart 500 can begin at 502 and can proceed to 504.

[0120] At point 504, a first input can be received as input. The first input may include an image 124 of the object of interest 126. In at least one embodiment, the encoder network 108 may be configured to receive the first input, which may include the image 124 of the object of interest 126.

[0121] At position 506, a multi-frequency feature map can be generated as the output of the final layer 120N of the encoder network 108. The output can be generated based on the received first input. In at least one embodiment, the encoder network 108 can be configured to generate a multi-frequency feature map as the output of the final layer 120N of the encoder network 108 based on the received first input.

[0122] At point 508, the generated multi-frequency feature map can be received from the final layer 120N of the encoder network 108. In at least one embodiment, the attention layer 110 can be configured to receive the generated multi-frequency feature map from the final layer 120N of the encoder network 108.

[0123] At point 510, the multi-frequency feature map can be refined. The multi-frequency feature map can be refined based on the correlation or association between the received multi-frequency feature maps. In at least one embodiment, the attention layer 110 can be configured to refine the multi-frequency feature map based on the correlation or association between the received multi-frequency feature maps.

[0124] At point 512, the refined multi-frequency feature map can be received from the attention layer 110 as a second input. In at least one embodiment, the decoder network 112 can be configured to receive the refined multi-frequency feature map from the attention layer 110 as a second input.

[0125] At point 514, a landmark detection result can be generated based on the second input. The landmark detection result may include a heatmap image 128 of the object of interest 126. The heatmap image 128 may indicate the location of landmark point 130 on the object of interest 126 in image 124. In at least one embodiment, the decoder network 112 may be configured to generate a landmark detection result including the heatmap image 128 of the object of interest 126 based on the second input, wherein the heatmap image 128 indicates the location of landmark point 130 on the object of interest 126 in image 124. Control may be passed to the end.

[0126] Various embodiments of this disclosure may provide a non-transitory computer-readable medium and / or storage medium having instructions thereon stored thereon, which are executable by a machine and / or computer to operate a system (e.g., system 102) for landmark detection using a deep neural network with multi-frequency self-attention. These instructions may cause the machine and / or computer to perform operations including: receiving as input an image (e.g., image 124) that may include an object of interest (e.g., object of interest 126); generating a multi-frequency feature map as output of a final layer (e.g., final layer 120N) of an encoder network (e.g., encoder network 108); receiving the generated multi-frequency feature map from the final layer 120N of encoder network 304; refining the multi-frequency feature map based on correlations or associations between the received multi-frequency feature maps; receiving the refined multi-frequency feature map as a second input from an attention layer (e.g., attention layer 110); and generating a landmark detection result that may include a heatmap image (e.g., heatmap image 128) of the object of interest. Heatmap images can indicate the location of landmarks (e.g., landmark 130) on objects of interest in the image.

[0127] Certain embodiments of this disclosure can be found in systems and methods for landmark detection using deep neural networks with multi-frequency self-attention. Various embodiments of this disclosure may provide a system 102 that may include an encoder network 108 configured to receive a first input including an image 124 of an object of interest 126. The object of interest 126 may correspond to a cartoon, fictional, or real-life human face. The encoder network 108 may also be configured to generate a multi-frequency feature map as the output of a final layer 120N of the encoder network 108 based on the received first input. The multi-frequency feature map may include a first frequency feature map capturing a first spatial information associated with global shape features of the object of interest 126 and a second frequency feature map capturing a second spatial information associated with local shape features, which are finer and more numerous than the global shape features on the object of interest 126.

[0128] According to an embodiment, the encoder network 108 includes a plurality of coding layers 120. The plurality of coding layers 120 may further include an input layer 120A and a first set of convolutional layers 120B following the input layer 120A. A first convolutional layer in the first set of convolutional layers 120B may be coupled to the input layer 120A of the encoder network 108, and each remaining convolutional layer in the first set of convolutional layers 120B is coupled to a previous convolutional layer in the plurality of coding layers 120. Each convolutional layer in the first set of convolutional layers 120B includes two convolution operators. Each of these two convolution operators corresponds to an octave convolution operation.

[0129] According to an embodiment, encoder network 108 can be configured to divide the image 124 included in the received first input into initial multi-frequency feature maps. Encoder network 108 can also be configured to pass the initial multi-frequency feature maps through a first set of convolutional layers 120B to generate multi-frequency feature maps as the output of the final layer 120N of encoder network 108. Each convolutional layer between the input layer 120A and the final layer 120N of encoder network 108 generates a first set of intermediate multi-frequency feature maps. Each intermediate feature map in the first set of intermediate multi-frequency feature maps has a size smaller than the initial multi-frequency feature map and larger than the initial multi-frequency feature map.

[0130] System 102 may further include an attention layer 110, which may be configured to receive generated multi-frequency feature maps from the final layer 120N of encoder network 108. Attention layer 110 may also be configured to refine the multi-frequency feature maps based on the correlation or association between the received multi-frequency feature maps.

[0131] According to an embodiment, system 102 may further include a decoder network 112, which may be configured to receive a refined multi-frequency feature map from attention layer 110 as a second input. Decoder network 112 may be coupled to encoder network 108 via attention layer 110. Decoder network 112 may include a plurality of decoding layers 122. The plurality of decoding layers 122 may include an input layer 122A and a second set of convolutional layers 122B following the input layer 122A of decoder network 112. A first convolutional layer in the second set of convolutional layers 122B may be coupled to the input layer 122A of decoder network 112, and each remaining convolutional layer in the second set of convolutional layers 122B may be coupled to a previous convolutional layer in the plurality of decoding layers 122. Each convolutional layer in the second set of convolutional layers 122B includes two convolution operators. Each of the two convolution operators corresponds to a transposed octave convolution operation.

[0132] According to an embodiment, the decoder network 112 can be configured to pass the refined multi-frequency feature map through a second set of convolutional layers 122B to generate a landmark detection result as the output of the final layer 122N of the decoder network 112. A second set of intermediate multi-frequency feature maps is generated in each convolutional layer between the input layer 122A and the final layer 122N of the decoder network 112. The size of each intermediate feature map in the second set of intermediate multi-frequency feature maps is larger than that of the refined multi-frequency feature map.

[0133] According to an embodiment, the landmark detection result may include a heatmap image 128 of the object of interest 126. The heatmap image 128 may indicate the position of the landmark point 130 on the object of interest 126 in the image 124.

[0134] According to an embodiment, system 102 may further include a skip connection between an intermediate coding layer of encoder network 108 and an intermediate decoding layer of decoder network 112. The intermediate coding layer of encoder network 108 may be configured to generate a first set of intermediate multi-frequency feature maps based on input from previous layers in encoder network 108. The skip connection may be configured to transmit spatial information included in the first set of intermediate multi-frequency feature maps to the intermediate decoding layer of decoder network 112.

[0135] According to an embodiment, system 102 may further include processor 104, which may be configured to extract landmark detection results from the final layer 122N of decoder network 112. Processor 104 may also be configured to overlay colored markers on image 124 to indicate the positions of landmark points on object of interest 126. Processor 104 may also be configured to control display device 116 to present output including the colored markers overlaid on image 124.

[0136] According to an embodiment, the encoder network 108 and the decoder network 112 may be deep neural networks (DNNs). The encoder network 108, the decoder network 112, and the attention layer 110 may together form an encoder-decoder neural network 106 that can be trained for the task of landmark detection.

[0137] This disclosure can be implemented in hardware or a combination of hardware and software. It can be implemented in a centralized manner on at least one computer system or in a distributed manner, wherein different elements can be distributed across several interconnected computer systems. Suitable computer systems or other apparatuses may be appropriate for performing the methods described herein. The combination of hardware and software can be a general-purpose computer system having a computer program that, when loaded and executed, can control the computer system to perform the methods described herein. This disclosure can be implemented in hardware including a portion of an integrated circuit that also performs other functions.

[0138] This disclosure can also be embedded in a computer program product that includes all features enabling the implementation of the methods described herein and, when loaded into a computer system, the execution of those methods. In this context, a computer program means any expression of a set of instructions in any language, code, or symbolic manner, which is intended to cause a system with information processing capabilities to directly perform a particular function, or to perform a particular function after any one or both of: a) being translated into another language, code, or symbolic form; or b) being copied in a different material form.

[0139] While this disclosure has been described with reference to specific embodiments, those skilled in the art will understand that various changes can be made and equivalents can be substituted without departing from the scope of this disclosure. Furthermore, many modifications can be made without departing from the scope of this disclosure to adapt particular situations or materials to the teachings of this disclosure. Therefore, this disclosure is not intended to be limited to the specific embodiments disclosed, but rather is intended to include all embodiments falling within the scope of the appended claims.

Claims

1. A system for boundary marker detection, the system comprising: An encoder network, the encoder network being configured as follows: Receive the first input, which includes an image of the object of interest; as well as A multi-frequency feature map is generated based on the received first input as the output of the final layer of the encoder network; An attention layer, coupled to the final layer of the encoder network, wherein the attention layer is configured to: The multi-frequency feature map is received from the final layer of the encoder network; Multiple embeddings are obtained from the multi-frequency feature map; Change the size of each of the plurality of embeddings to a specific size; Based on each of the plurality of embeddings having the specified size, a similarity matrix is ​​calculated that captures the dependencies between the multi-frequency feature maps; as well as The multi-frequency feature map is refined based on the calculated similarity matrix; as well as A decoder network, coupled to the encoder network via the attention layer, wherein the decoder network is configured to: The refined multi-frequency feature map is received from the attention layer as a second input; as well as Based on the second input, a landmark detection result including a heatmap image of the object of interest is generated. The heatmap image indicates the location of landmark points on the object of interest in the image.

2. The system of claim 1, wherein the object of interest corresponds to one of a human face, a cartoon character, a fictional character, a real-life human face.

3. The system according to claim 1, wherein The encoder network and the decoder network are deep neural networks (DNNs), and The encoder network, the decoder network, and the attention layer together form an encoder-decoder neural network trained for the landmark detection task.

4. The system according to claim 1, wherein the multi-frequency feature map comprises: A first frequency feature map capturing first spatial information associated with the global shape features of the object of interest, and A second frequency feature map that captures second spatial information associated with local shape features, which are more refined and numerous compared to the global shape features on the object of interest.

5. The system according to claim 1, wherein The encoder network includes multiple encoding layers. The plurality of coding layers include an input layer and a first set of convolutional layers following the input layer of the encoder network, and The first convolutional layer in the first group of convolutional layers is coupled to the input layer of the encoder network, and each remaining convolutional layer in the first group of convolutional layers is coupled to a previous convolutional layer in the first group of convolutional layers.

6. The system according to claim 5, wherein Each convolutional layer in the first group of convolutional layers includes two convolution operators, and Each of the two convolution operators corresponds to an octave convolution operation.

7. The system according to claim 5, wherein The encoder network is also configured to: The image included in the received first input is divided into an initial multi-frequency feature map; and The initial multi-frequency feature map is passed through the first set of convolutional layers to generate the multi-frequency feature map as the output of the final layer of the encoder network. Each convolutional layer in the first set of convolutional layers between the input layer and the final layer of the encoder network generates a first set of intermediate multi-frequency feature maps, and The size of each intermediate feature map in the first set of intermediate multi-frequency feature maps is smaller than the size of each initial multi-frequency feature map in the initial multi-frequency feature map and larger than the size of each multi-frequency feature map in the multi-frequency feature map.

8. The system according to claim 1, wherein The decoder network includes multiple decoding layers. The plurality of decoding layers include an input layer and a second set of convolutional layers following the input layer of the decoder network, and The first convolutional layer in the second group of convolutional layers is coupled to the input layer of the decoder network, and each remaining convolutional layer in the second group of convolutional layers is coupled to a previous convolutional layer in the second group of convolutional layers.

9. The system according to claim 8, wherein Each convolutional layer in the second group includes two convolution operators, and Each of the two convolution operators corresponds to the transpose octave convolution operation.

10. The system according to claim 8, wherein The decoder network is further configured to pass the refined multi-frequency feature map through the second set of convolutional layers to generate the landmark detection result as the output of the final layer of the decoder network. Each convolutional layer in the second set of convolutional layers between the input layer and the final layer of the decoder network generates a second set of intermediate multi-frequency feature maps, and The size of each intermediate feature map in the second set of intermediate multi-frequency feature maps is larger than the size of each refined multi-frequency feature map in the refined multi-frequency feature maps.

11. The system of claim 1, further comprising a skip connection between the intermediate encoding layer of the encoder network and the intermediate decoding layer of the decoder network.

12. The system according to claim 11, wherein The intermediate coding layers of the encoder network are configured to generate a first set of intermediate multi-frequency feature maps based on inputs from previous layers in the encoder network. The skip connection is configured to transmit spatial information, including in the first set of intermediate multi-frequency feature maps, to the intermediate decoding layer of the decoder network.

13. The system of claim 1, further comprising a processor, the processor being configured to: The landmark detection results are extracted from the final layer of the decoder network; Colored markers are overlaid on the image to indicate the positions of the landmark points on the object of interest; and The control display device presents an output including the colored markers overlaid on the image.

14. A method for boundary marker detection, the method comprising: The encoder network receives the first input, which includes an image of the object of interest. The encoder network generates a multi-frequency feature map based on the received first input, which is then used as the output of the final layer of the encoder network. The multi-frequency feature map is received from the final layer of the encoder network by an attention layer coupled to the final layer of the encoder network; The attention layer obtains multiple embeddings from the multi-frequency feature map; The attention layer changes the size of each of the plurality of embeddings to a specific size; The attention layer computes a similarity matrix that captures the dependencies between the multi-frequency feature maps based on each of the plurality of embeddings having the specific size; The attention layer refines the multi-frequency feature map based on the calculated similarity matrix; The refined multi-frequency feature map is received from the attention layer as a second input by the decoder network, which is coupled to the encoder network via the attention layer; as well as The decoder network generates landmark detection results, including a heatmap image of the object of interest, based on the second input. The heatmap image indicates the location of landmark points on the object of interest in the image.

15. The method of claim 14, wherein the object of interest corresponds to one of a human face, a cartoon character, a fictional character, a real-life human face.

16. The method of claim 14, wherein The encoder network and the decoder network are deep neural networks (DNNs), and The encoder network, the decoder network, and the attention layer together form an encoder-decoder neural network trained for the landmark detection task.

17. The method of claim 14, wherein The encoder network includes multiple encoding layers. The plurality of coding layers include an input layer and a first set of convolutional layers following the input layer of the encoder network, and The first convolutional layer in the first group of convolutional layers is coupled to the input layer of the encoder network, and each remaining convolutional layer in the first group of convolutional layers is coupled to a previous convolutional layer in the first group of convolutional layers.

18. The method of claim 17, wherein Each convolutional layer in the first group of convolutional layers includes two convolution operators, and Each of the two convolution operators corresponds to an octave convolution operation.

19. The method of claim 17, further comprising: The encoder network divides the image included in the received first input into an initial multi-frequency feature map; as well as The encoder network passes the initial multi-frequency feature map through the first set of convolutional layers to generate the multi-frequency feature map as the output of the final layer of the encoder network, wherein... Each convolutional layer in the first set of convolutional layers between the input layer and the final layer of the encoder network generates a first set of intermediate multi-frequency feature maps, and The size of each intermediate feature map in the first set of intermediate multi-frequency feature maps is smaller than the size of each initial multi-frequency feature map in the initial multi-frequency feature map and larger than the size of each multi-frequency feature map in the multi-frequency feature map.

20. The method of claim 14, wherein the decoder network comprises a plurality of decoding layers. The plurality of decoding layers include an input layer and a second set of convolutional layers following the input layer of the decoder network, and The first convolutional layer in the second group of convolutional layers is coupled to the input layer of the decoder network, and each remaining convolutional layer in the second group of convolutional layers is coupled to a previous convolutional layer in the second group of convolutional layers.

Images